====== Slow JCT troubleshooting – InfiniBand (QM9700) example ======
//All values are invented but realistic. Outputs are abridged.//
===== Context =====
* **Fabric:** 2048 H200 GPUs = 256 nodes × 8 GPUs and 8 NICs (CX-7, 400G). Rail-optimised, 2-tier QM9700: 8 SUs × 32 nodes, 64 leaves, 32 spines.
* **Naming:** ''su3-n17'' = SU3 node 17. ''su3-l5'' = SU3's rail-5 leaf. Node //n// lands on port //n//. ''su3-s01'' = a spare node in SU3.
* **Job 48213:** 512 GPUs on ''su3-n[01-32]'', ''su4-n[01-32]''. Step time has gone from 1.10 s to 1.42 s; JCT from 18 h to 23 h.
^ Baseline (healthy) ^ Value ^
| Live all-reduce busbw, job's cross-node communicator | 360 GB/s |
| nccl-tests single node, 8 GB | 480 GB/s |
| nccl-tests 2 nodes, 8 GB | 370 GB/s |
| ''ib_write_bw'' per NIC (host or GPU memory) | 390 Gb/s |
===== Phase A: live, job keeps running, read-only =====
==== A1. Live busbw: is comms degraded, and since when? ====
# Prometheus (NCCL Inspector metrics; label names depend on your exporter)
avg by (coll) (nccl_bus_bandwidth_gbs{jobid="48213", coll="AllReduce"})
AllReduce 360 GB/s until Day-2 02:14 -> 185 GB/s since
AllGather / ReduceScatter: same step change
NVLink-only communicators (intra-node): unchanged
**Reading:** cross-node collectives halved at a precise time and NVLink is unaffected, so the fault is on the cross-node path, or a straggler.
==== A2. Straggler or transfer path? ====
Straggler panel (same telemetry): **no rank flagged**. Read-only GPU check across the job's nodes:
pdsh -w su3-n[01-32],su4-n[01-32] \
'nvidia-smi --query-gpu=index,clocks.sm,temperature.gpu,clocks_throttle_reasons.active --format=csv,noheader' \
| awk -F, '$2+0 < 1900'
(no output: all 512 GPUs at ~1980 MHz, no throttle reasons)
**Reading:** no slow GPU, so it's the transfer path.
==== A3. Link-rate sweep of all 512 NICs ====
pdsh -w su3-n[01-32],su4-n[01-32] \
'for d in mlx5_{0..7}; do echo "$d $(cat /sys/class/infiniband/$d/ports/1/rate)"; done' \
| grep -v "400 Gb"
su3-n17: mlx5_5 200 Gb/sec (4X HDR)
**Reading:** one NIC trained at 200G. Rail 5 of su3-n17.
==== A4. Which switch port? ====
iblinkinfo | grep -i "su3-n17 mlx5_5"
su3-l5 17[ ] ==( 4X 53.125 Gbps Active/ LinkUp)==> su3-n17 mlx5_5
**Reading:** ''su3-l5'' port 17, running at HDR lane speed instead of NDR.
==== A5. Counters and optics, both ends ====
Switch side:
perfquery -x 17
SymbolErrorCounter: 41230 (baseline 0)
LinkDownedCounter: 1 (matches Day-2 02:14)
PortXmitWait: normal
Host side, NIC optic:
mlxlink -d mlx5_5 -m -c
Rx Power Current [dBm] : -1.1, -1.3, -8.9, -1.2 <- lane 3 ~8 dB low
Effective Physical BER : 2E-08 (healthy ~1E-12 or better)
**Phase A conclusion, no drain so far:** a degraded optic or dirty connector on su3-n17 ''mlx5_5'' ↔ ''su3-l5'' port 17. Lane 3 is weak, the link renegotiated to 200G, and every collective's rail-5 channels now run at half speed.
===== Phase B: repair window agreed with the app team =====
==== B0. Take the node out ====
App team checkpoints, then:
scontrol update nodename=su3-n17 state=drain reason="rail5 link 200G, optic"
The job restarts on ''su3-s01'': live busbw back to 360 GB/s, step time 1.10 s. Now su3-n17 is free to test.
==== B1. Decouple GPU vs. host vs. network on su3-n17 ====
Each test exercises a different slice of the path:
^ Test ^ GPU / HBM ^ NVLink ^ GPU↔NIC PCIe ^ NIC, optic, cable, switch ^
| Single-node nccl-tests | ✔ | ✔ | — | — |
| ''ib_write_bw'' host memory | — | — | — | ✔ |
| ''ib_write_bw'' GPU memory | ✔ | — | ✔ | ✔ |
**GPU side:**
dcgmi diag -r 2
Overall: Pass
./build/all_reduce_perf -b 8 -e 8G -f 2 -g 8
# size ... algbw busbw
8589934592 ... 273.2 478.1
# Avg bus bandwidth : 478.1
**Network path without the GPU (host memory), against the spare:**
# on su3-s01 (server)
ib_write_bw -d mlx5_5 -F --report_gbits -s 1048576 -D 10
# on su3-n17 (client)
ib_write_bw -d mlx5_5 -F --report_gbits -s 1048576 -D 10 su3-s01
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec]
1048576 ... 197.3 196.1
Control on a healthy rail (''-d mlx5_4''): 389.6 Gb/s.
**Full path with GPU memory:**
ib_write_bw -d mlx5_5 -F --report_gbits -s 1048576 -D 10 --use_cuda=5 su3-s01
1048576 ... 196.4 195.2
**2-node NCCL, rail 5 only:**
mpirun -np 2 -H su3-n17:1,su3-s01:1 \
-x NCCL_IB_HCA==mlx5_5 -x CUDA_VISIBLE_DEVICES=5 \
./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1
# Avg bus bandwidth : 48.3 (single-rail baseline 96)
**Reading the matrix:**
^ GPU side ^ Host memory ^ GPU memory ^ Verdict ^
| Pass / 478 | **196** | **195** | Low even without the GPU → **network**, not GPU or PCIe |
==== B2. Locate the part: swap, one change at a time ====
^ Action ^ Result ^ Reading ^
| Move cable to spare port 33 on ''su3-l5'' | Still 200G, lane 3 low | Not the leaf port |
| Clean connectors, reseat | Still 200G | Not dirt |
| Replace host-side optic | **400G**, Rx lane 3 −1.2 dBm, BER < 1E-12 | **Root cause: host-side optic** |
==== B3. Verify and return ====
cat /sys/class/infiniband/mlx5_5/ports/1/rate -> 400 Gb/sec (4X NDR)
ib_write_bw ... (host memory) -> 390.2 Gb/s
2-node all_reduce_perf, all 8 rails -> Avg bus bandwidth 370.4
perfquery -x 33 (after 30 min) -> SymbolErrorCounter: 0
scontrol update nodename=su3-n17 state=resume
su3-n17 rejoins the spare pool. Live busbw on the job stays at 360 GB/s.
===== Why one half-speed link cost about 29% of JCT =====
NCCL splits each collective evenly across channels on all 8 NICs. The rail-5 channels through su3-n17 took about 2× longer, and every collective completes only when its slowest channel does. All 512 ranks waited, so exposed comm went from 0.25 s to 0.57 s per step.
===== Verify on your system =====
* Prometheus label names (exporter-dependent).
* ''clocks_throttle_reasons'' is renamed ''clocks_event_reasons'' in newer drivers.
* ''--use_cuda'' syntax; perftest must be built with CUDA.
* ''NCCL_IB_HCA'' exact-match ''='' syntax.
* ''mlxlink'' output layout varies by MFT version.