Slow JCT troubleshooting – InfiniBand (QM9700) example

All values are invented but realistic. Outputs are abridged.

• Fabric: 2048 H200 GPUs = 256 nodes × 8 GPUs and 8 NICs (CX-7, 400G). Rail-optimised, 2-tier QM9700: 8 SUs × 32 nodes, 64 leaves, 32 spines.
• Naming: su3-n17 = SU3 node 17. su3-l5 = SU3's rail-5 leaf. Node n lands on port n. su3-s01 = a spare node in SU3.
• Job 48213: 512 GPUs on su3-n[01-32], su4-n[01-32]. Step time has gone from 1.10 s to 1.42 s; JCT from 18 h to 23 h.

# Prometheus (NCCL Inspector metrics; label names depend on your exporter)
avg by (coll) (nccl_bus_bandwidth_gbs{jobid="48213", coll="AllReduce"})

AllReduce   360 GB/s  until Day-2 02:14   ->   185 GB/s  since
AllGather / ReduceScatter: same step change
NVLink-only communicators (intra-node): unchanged

Reading: cross-node collectives halved at a precise time and NVLink is unaffected, so the fault is on the cross-node path, or a straggler.

Straggler panel (same telemetry): no rank flagged. Read-only GPU check across the job's nodes:

pdsh -w su3-n[01-32],su4-n[01-32] \
  'nvidia-smi --query-gpu=index,clocks.sm,temperature.gpu,clocks_throttle_reasons.active --format=csv,noheader' \
  | awk -F, '$2+0 < 1900'

(no output: all 512 GPUs at ~1980 MHz, no throttle reasons)

Reading: no slow GPU, so it's the transfer path.

pdsh -w su3-n[01-32],su4-n[01-32] \
  'for d in mlx5_{0..7}; do echo "$d $(cat /sys/class/infiniband/$d/ports/1/rate)"; done' \
  | grep -v "400 Gb"

su3-n17: mlx5_5 200 Gb/sec (4X HDR)

Reading: one NIC trained at 200G. Rail 5 of su3-n17.

iblinkinfo | grep -i "su3-n17 mlx5_5"

su3-l5  17[  ] ==( 4X 53.125 Gbps Active/ LinkUp)==> su3-n17 mlx5_5

Reading: su3-l5 port 17, running at HDR lane speed instead of NDR.

Switch side:

perfquery -x <su3-l5 LID> 17

SymbolErrorCounter: 41230     (baseline 0)
LinkDownedCounter:  1         (matches Day-2 02:14)
PortXmitWait:       normal

Host side, NIC optic:

mlxlink -d mlx5_5 -m -c

Rx Power Current [dBm] : -1.1, -1.3, -8.9, -1.2     <- lane 3 ~8 dB low
Effective Physical BER : 2E-08                       (healthy ~1E-12 or better)

Phase A conclusion, no drain so far: a degraded optic or dirty connector on su3-n17 mlx5_5 ↔ su3-l5 port 17. Lane 3 is weak, the link renegotiated to 200G, and every collective's rail-5 channels now run at half speed.

App team checkpoints, then:

scontrol update nodename=su3-n17 state=drain reason="rail5 link 200G, optic"

The job restarts on su3-s01: live busbw back to 360 GB/s, step time 1.10 s. Now su3-n17 is free to test.

Each test exercises a different slice of the path:

GPU side:

dcgmi diag -r 2

Overall: Pass

./build/all_reduce_perf -b 8 -e 8G -f 2 -g 8

#       size    ...   algbw   busbw
  8589934592    ...   273.2   478.1
# Avg bus bandwidth : 478.1

Network path without the GPU (host memory), against the spare:

# on su3-s01 (server)
ib_write_bw -d mlx5_5 -F --report_gbits -s 1048576 -D 10
# on su3-n17 (client)
ib_write_bw -d mlx5_5 -F --report_gbits -s 1048576 -D 10 su3-s01

 #bytes   #iterations  BW peak[Gb/sec]  BW average[Gb/sec]
 1048576  ...          197.3            196.1

Control on a healthy rail (-d mlx5_4): 389.6 Gb/s.

Full path with GPU memory:

ib_write_bw -d mlx5_5 -F --report_gbits -s 1048576 -D 10 --use_cuda=5 su3-s01

 1048576  ...          196.4            195.2

2-node NCCL, rail 5 only:

mpirun -np 2 -H su3-n17:1,su3-s01:1 \
       -x NCCL_IB_HCA==mlx5_5 -x CUDA_VISIBLE_DEVICES=5 \
       ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

# Avg bus bandwidth : 48.3     (single-rail baseline 96)

Reading the matrix:

cat /sys/class/infiniband/mlx5_5/ports/1/rate     -> 400 Gb/sec (4X NDR)
ib_write_bw ... (host memory)                     -> 390.2 Gb/s
2-node all_reduce_perf, all 8 rails               -> Avg bus bandwidth 370.4
perfquery -x <su3-l5 LID> 33  (after 30 min)      -> SymbolErrorCounter: 0

scontrol update nodename=su3-n17 state=resume

su3-n17 rejoins the spare pool. Live busbw on the job stays at 360 GB/s.

NCCL splits each collective evenly across channels on all 8 NICs. The rail-5 channels through su3-n17 took about 2× longer, and every collective completes only when its slowest channel does. All 512 ranks waited, so exposed comm went from 0.25 s to 0.57 s per step.

• Prometheus label names (exporter-dependent).
• clocks_throttle_reasons is renamed clocks_event_reasons in newer drivers.
• –use_cuda syntax; perftest must be built with CUDA.
• NCCL_IB_HCA exact-match = syntax.
• mlxlink output layout varies by MFT version.