Rank and busbw for network troubleshooting

A rank is one process of the training job. Normally there is one per GPU, so for you a rank = one GPU and its NIC: one physical endpoint on the fabric.

So rank 13 on 8-GPU nodes is typically node 1, GPU 5, NIC 5, rail-5 leaf, port x. The “typically” matters, because two things can break the clean mapping:

• The launcher decides the rank order. Rank → hostname comes from the job's hostfile or Slurm allocation, not from any fixed rule. Get it from the app team.
• NCCL's PXN feature can move data over NVLink to another GPU's NIC so traffic stays on the destination's rail. A rank's traffic doesn't always leave via “its” NIC.

How to pin it down with facts rather than assumptions:

1. Get rank → hostname from the scheduler (Slurm job info or the launcher hostfile).
2. Get local rank → NIC from the NCCL debug output (NCCL_DEBUG=INFO), which logs which IB device each rank uses.
3. Get NIC → GUID/LID from ibstat on the host.
4. Get GUID → leaf switch and port from UFM, or iblinkinfo / ibnetdiscover.

That chain turns “rank 13 is slow” into “node X, mlx5_5, leaf Y port Z”, which is the language you troubleshoot in.

• A multi-node all-reduce uses every GPU, every NIC (all 8 rails) and every switch path between them at the same time, plus NVLink inside each node.
• NCCL splits the data into parallel channels across the NICs. The collective completes only when the slowest channel finishes.
• So low busbw (vs. baseline) means at least one element in this physical set is degraded:
  • a GPU or NVLink/NVSwitch inside any node,
  • any of the N × 8 NICs (or its PCIe link),
  • any NIC-to-leaf cable or optic,
  • any leaf, uplink or spine on the paths between those leaves.
• One bad element anywhere drags the whole job down. busbw tells you the set is unhealthy, not which element.
• busbw is normalised for job size, so a 2-node test and a 64-node test are directly comparable. That's what makes bisection work.

Then confirm in telemetry for the specific leaf/port you've identified: error counters, BER, speed/width downgrade, congestion counters.

• Run from the cluster's login/head node, launching onto the suspect nodes through the scheduler (Slurm) or MPI.
• The nodes must be drained or freed from the slow job first. The test needs all GPUs and NICs.
• Use the same software stack as the job (driver, CUDA, NCCL version, container image if they use one). A different NCCL version can give a different busbw and invalidate the comparison with the job.
• Build once on shared storage so every node sees the same binary.

git clone https://github.com/NVIDIA/nccl-tests.git
cd nccl-tests
make MPI=1 MPI_HOME=/path/to/mpi CUDA_HOME=/usr/local/cuda NCCL_HOME=/path/to/nccl

Binaries land in build/, e.g. build/all_reduce_perf.

One process driving all 8 GPUs:

./build/all_reduce_perf -b 8 -e 8G -f 2 -g 8

• -b 8 -e 8G: message sizes from 8 bytes to 8 GB
• -f 2: double the size each step
• -g 8: 8 GPUs in this process

With Slurm:

srun -N 2 --ntasks-per-node=8 --gpus-per-node=8 \
     -w node01,node02 \
     ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

With mpirun:

mpirun -np 16 -H node01:8,node02:8 \
       -x NCCL_DEBUG=INFO \
       ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

NCCL_DEBUG=INFO also prints which IB device each rank uses: this gives you the rank → NIC mapping.

Restrict NCCL to a single NIC (here rail 5):

mpirun -np 2 -H node01:1,node02:1 \
       -x NCCL_IB_HCA==mlx5_5 \
       -x CUDA_VISIBLE_DEVICES=5 \
       ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

Or test the raw RDMA path without NCCL, NIC to NIC:

# on node02 (server)
ib_write_bw -d mlx5_5 --report_gbits
# on node01 (client)
ib_write_bw -d mlx5_5 --report_gbits node02

Repeat for each rail (mlx5_0 … mlx5_7).

ibstat mlx5_5          # state, rate (expect 400), physical state
iblinkinfo | grep -i <node-or-guid>   # which leaf and port it lands on

nccl-tests prints one line per message size, with algbw and busbw columns for out-of-place and in-place runs, then an Avg bus bandwidth summary line. Compare the busbw at the largest sizes (and the whole curve) with your baseline for the same node count and topology.

Verify on your system: Slurm may need –mpi=pmix (or similar) depending on how MPI was built. The NCCL_IB_HCA syntax (a leading = for an exact device-name match) and the device names (mlx5_N) depend on your NCCL version and host. Check ibv_devices for the real names.

• Large messages, low plateau: lost bandwidth. Look for a degraded or downgraded link, a NIC at the wrong PCIe width, or a link down shrinking the path set.
• Small messages slow: latency. Look for extra hops, adaptive-routing problems, or SHARP not in use.
• Fine at 2 nodes, worse at scale: spine, uplink or congestion problem.
• Normal busbw, slow job: the fabric is likely fine. Ask the app team for per-rank step times. A slow rank points at a specific GPU/node, which you can map back with the chain above.

Limit: nccl-tests need the nodes, so the job must be drained or paused.