network_stuff:infiniband
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| network_stuff:infiniband [2025/07/02 11:03] – jotasandoku | network_stuff:infiniband [2025/10/03 16:04] (current) – jotasandoku | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| __**INFINIBAND**__: | __**INFINIBAND**__: | ||
| + | \\ | ||
| + | {{ : | ||
| \\ | \\ | ||
| From [[https:// | From [[https:// | ||
| Line 10: | Line 12: | ||
| * RDMA provides access to the memory from one computer to the memory of another computer without involving either computer’s operating system. This technology enables high-throughput and low-latency networking with low CPU utilization. | * RDMA provides access to the memory from one computer to the memory of another computer without involving either computer’s operating system. This technology enables high-throughput and low-latency networking with low CPU utilization. | ||
| * Mellanox provides RDMA via the OFED package | * Mellanox provides RDMA via the OFED package | ||
| - | * lid : local indentifier (All devices in a subnet have a Local Identifier (LID)). Routing between different subnets is done on the basis of a Global Identifier (GID) | + | * **LID** |
| + | * GID: Is another identifier but is to route BETWEEN SUBNETS. Contains : Subnet Prefix and a GUID (Global Unique Identifier). | ||
| * NSD (Network Shared Disks): In our context, NSD is the server that connects to the storage via the Mellanox switch. The servers share the NSD's to the clients, creating some sort of distributed logical disk (a bit like the hyperflex technology). Particuartly in our setupm the servers dont share their local disks but they expose the DDN's disks. | * NSD (Network Shared Disks): In our context, NSD is the server that connects to the storage via the Mellanox switch. The servers share the NSD's to the clients, creating some sort of distributed logical disk (a bit like the hyperflex technology). Particuartly in our setupm the servers dont share their local disks but they expose the DDN's disks. | ||
| - | * SM (Subnet Manager): | + | |
| * SM master is the node truly acting as SM. The node with the highest priority [0-15] wins. | * SM master is the node truly acting as SM. The node with the highest priority [0-15] wins. | ||
| * In our setup, servers all have priority 14 while switch has priority 15. | * In our setup, servers all have priority 14 while switch has priority 15. | ||
| Line 32: | Line 35: | ||
| | | ||
| Useful server side ib commands | Useful server side ib commands | ||
| - | ibstatus | + | ibstatus |
| + | ibping -S -L 10 ---- ibping -L 20 -c 10 -n 3 # to ping, we need to run one in the server (LID 20) and one in the client (LID 10). This is because even ping makes RDMA calls | ||
| Line 43: | Line 47: | ||
| Tshoot commands: | Tshoot commands: | ||
| - | show ib smnodes | + | show ib smnodes |
| show ib smnode nyzsfsll51 sm-state | show ib smnode nyzsfsll51 sm-state | ||
| show guids # so we can identify the macs | show guids # so we can identify the macs | ||
| Line 128: | Line 132: | ||
| * On a working node, `sminfo` reports a different Subnet Manager than the one expected: | * On a working node, `sminfo` reports a different Subnet Manager than the one expected: | ||
| + | |||
| SM lid 1, lmc 0, smsl 0, priority 5, state: master | SM lid 1, lmc 0, smsl 0, priority 5, state: master | ||
| smguid 0x... " | smguid 0x... " | ||
| - | | ||
| **Fix: | **Fix: | ||
network_stuff/infiniband.1751454183.txt.gz · Last modified: by jotasandoku
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
