====== Rank and busbw for network troubleshooting ======

===== What a rank is, in network terms =====

A **rank** is one process of the training job. Normally there is one per GPU, so for you **a rank = one GPU and its NIC**: one physical endpoint on the fabric.

^ AI-team term ^ Your translation (H200, rail-optimised) ^
| Global rank | One GPU in the job, numbered 0 … N−1 across all nodes |
| Local rank | GPU index inside the node (0-7) |
| World size | Total GPUs in the job (nodes × 8) |
| Communicator | The set of GPUs taking part in a given collective, usually all of them |
| Rank → NIC | Local rank //i// → GPU //i// → NIC //i// → **rail //i//** → a port on rail //i//'s leaf |

So rank 13 on 8-GPU nodes is typically node 1, GPU 5, NIC 5, rail-5 leaf, port //x//. The "typically" matters, because two things can break the clean mapping:

  * **The launcher decides the rank order.** Rank → hostname comes from the job's hostfile or Slurm allocation, not from any fixed rule. Get it from the app team.
  * **NCCL's PXN feature** can move data over NVLink to another GPU's NIC so traffic stays on the destination's rail. A rank's traffic doesn't always leave via "its" NIC.

How to pin it down with facts rather than assumptions:

  - Get rank → hostname from the scheduler (Slurm job info or the launcher hostfile).
  - Get local rank → NIC from the NCCL debug output (''NCCL_DEBUG=INFO''), which logs which IB device each rank uses.
  - Get NIC → GUID/LID from ''ibstat'' on the host.
  - Get GUID → leaf switch and port from UFM, or ''iblinkinfo'' / ''ibnetdiscover''.

That chain turns "rank 13 is slow" into "node X, mlx5_5, leaf Y port Z", which is the language you troubleshoot in.

===== busbw, rewritten with that mapping =====

  * A multi-node all-reduce uses **every GPU, every NIC (all 8 rails) and every switch path between them** at the same time, plus NVLink inside each node.
  * NCCL splits the data into parallel channels across the NICs. The collective completes only when the **slowest** channel finishes.
  * So low busbw (vs. baseline) means **at least one element in this physical set is degraded**:
    * a GPU or NVLink/NVSwitch inside any node,
    * any of the N × 8 NICs (or its PCIe link),
    * any NIC-to-leaf cable or optic,
    * any leaf, uplink or spine on the paths between those leaves.
  * One bad element anywhere drags the **whole job** down. busbw tells you the set is unhealthy, not which element.
  * busbw is normalised for job size, so a 2-node test and a 64-node test are directly comparable. That's what makes bisection work.

===== Locating the element: shrink the set until the fault is isolated =====

^ Step ^ Test ^ What you've isolated ^ If low ^
| 1 | All-reduce inside each node (8 GPUs, no IB) | NVLink/GPU, per node | That node's internals. **Not the network** |
| 2 | All-reduce across all job nodes | Whole set | Continue. If normal, the **network is likely not the cause**: check compute stragglers, storage, data loading |
| 3 | 2-node tests (pairs on the same leaves) | Node pair | Outlier node found |
| 4 | Per-NIC on the outlier node (NCCL restricted to one NIC, or ''ib_write_bw'' per NIC pair) | One rail on one node | NIC, PCIe, optic, cable or leaf port of that rail |
| 5 | Same-leaf pair vs. cross-leaf vs. cross-SU | Switch tier | Uplinks, spine, adaptive routing, congestion |

Then confirm in telemetry for the specific leaf/port you've identified: error counters, BER, speed/width downgrade, congestion counters.

===== How to run it =====

==== Where from ====

  * Run from the cluster's **login/head node**, launching onto the suspect nodes through the scheduler (Slurm) or MPI.
  * The nodes must be **drained or freed** from the slow job first. The test needs all GPUs and NICs.
  * Use the **same software stack as the job** (driver, CUDA, NCCL version, container image if they use one). A different NCCL version can give a different busbw and invalidate the comparison with the job.
  * Build once on shared storage so every node sees the same binary.

==== Build nccl-tests (once) ====

<code bash>
git clone https://github.com/NVIDIA/nccl-tests.git
cd nccl-tests
make MPI=1 MPI_HOME=/path/to/mpi CUDA_HOME=/usr/local/cuda NCCL_HOME=/path/to/nccl
</code>

Binaries land in ''build/'', e.g. ''build/all_reduce_perf''.

==== Step 1: single node (NVLink only, no network) ====

One process driving all 8 GPUs:

<code bash>
./build/all_reduce_perf -b 8 -e 8G -f 2 -g 8
</code>

  * ''-b 8 -e 8G'': message sizes from 8 bytes to 8 GB
  * ''-f 2'': double the size each step
  * ''-g 8'': 8 GPUs in this process

==== Steps 2-3: multi-node (2 nodes = 16 ranks, 1 GPU per rank) ====

With Slurm:

<code bash>
srun -N 2 --ntasks-per-node=8 --gpus-per-node=8 \
     -w node01,node02 \
     ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1
</code>

With mpirun:

<code bash>
mpirun -np 16 -H node01:8,node02:8 \
       -x NCCL_DEBUG=INFO \
       ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1
</code>

''NCCL_DEBUG=INFO'' also prints which IB device each rank uses: this gives you the rank → NIC mapping.

==== Step 4: one rail only ====

Restrict NCCL to a single NIC (here rail 5):

<code bash>
mpirun -np 2 -H node01:1,node02:1 \
       -x NCCL_IB_HCA==mlx5_5 \
       -x CUDA_VISIBLE_DEVICES=5 \
       ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1
</code>

Or test the raw RDMA path without NCCL, NIC to NIC:

<code bash>
# on node02 (server)
ib_write_bw -d mlx5_5 --report_gbits
# on node01 (client)
ib_write_bw -d mlx5_5 --report_gbits node02
</code>

Repeat for each rail (mlx5_0 … mlx5_7).

==== Check the link you've isolated ====

<code bash>
ibstat mlx5_5          # state, rate (expect 400), physical state
iblinkinfo | grep -i <node-or-guid>   # which leaf and port it lands on
</code>

==== Reading the output ====

nccl-tests prints one line per message size, with ''algbw'' and ''busbw'' columns for out-of-place and in-place runs, then an ''Avg bus bandwidth'' summary line. Compare the **busbw at the largest sizes** (and the whole curve) with your baseline for the same node count and topology.

**Verify on your system:** Slurm may need ''--mpi=pmix'' (or similar) depending on how MPI was built. The ''NCCL_IB_HCA'' syntax (a leading ''='' for an exact device-name match) and the device names (''mlx5_N'') depend on your NCCL version and host. Check ''ibv_devices'' for the real names.

===== Reading the result =====

  * **Large messages, low plateau:** lost bandwidth. Look for a degraded or downgraded link, a NIC at the wrong PCIe width, or a link down shrinking the path set.
  * **Small messages slow:** latency. Look for extra hops, adaptive-routing problems, or SHARP not in use.
  * **Fine at 2 nodes, worse at scale:** spine, uplink or congestion problem.
  * **Normal busbw, slow job:** the fabric is likely fine. Ask the app team for per-rank step times. A slow rank points at a specific GPU/node, which you can map back with the chain above.

**Limit:** nccl-tests need the nodes, so the job must be drained or paused.