DokuWiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Slow JCT troubleshooting – InfiniBand (QM9700) example ======

//All values are invented but realistic. Outputs are abridged.//

===== Context =====

  * **Fabric:** 2048 H200 GPUs = 256 nodes × 8 GPUs and 8 NICs (CX-7, 400G). Rail-optimised, 2-tier QM9700: 8 SUs × 32 nodes, 64 leaves, 32 spines.
  * **Naming:** ''su3-n17'' = SU3 node 17. ''su3-l5'' = SU3's rail-5 leaf. Node //n// lands on port //n//. ''su3-s01'' = a spare node in SU3.
  * **Job 48213:** 512 GPUs on ''su3-n[01-32]'', ''su4-n[01-32]''. Step time has gone from 1.10 s to 1.42 s; JCT from 18 h to 23 h.

^ Baseline (healthy) ^ Value ^
| Live all-reduce busbw, job's cross-node communicator | 360 GB/s |
| nccl-tests single node, 8 GB | 480 GB/s |
| nccl-tests 2 nodes, 8 GB | 370 GB/s |
| ''ib_write_bw'' per NIC (host or GPU memory) | 390 Gb/s |

===== Phase A: live, job keeps running, read-only =====

==== A1. Live busbw: is comms degraded, and since when? ====

<code>
# Prometheus (NCCL Inspector metrics; label names depend on your exporter)
avg by (coll) (nccl_bus_bandwidth_gbs{jobid="48213", coll="AllReduce"})
</code>

<code>
AllReduce   360 GB/s  until Day-2 02:14   ->   185 GB/s  since
AllGather / ReduceScatter: same step change
NVLink-only communicators (intra-node): unchanged
</code>

**Reading:** cross-node collectives halved at a precise time and NVLink is unaffected, so the fault is on the cross-node path, or a straggler.

==== A2. Straggler or transfer path? ====

Straggler panel (same telemetry): **no rank flagged**. Read-only GPU check across the job's nodes:

<code bash>
pdsh -w su3-n[01-32],su4-n[01-32] \
  'nvidia-smi --query-gpu=index,clocks.sm,temperature.gpu,clocks_throttle_reasons.active --format=csv,noheader' \
  | awk -F, '$2+0 < 1900'
</code>

<code>
(no output: all 512 GPUs at ~1980 MHz, no throttle reasons)
</code>

**Reading:** no slow GPU, so it's the transfer path.

==== A3. Link-rate sweep of all 512 NICs ====

<code bash>
pdsh -w su3-n[01-32],su4-n[01-32] \
  'for d in mlx5_{0..7}; do echo "$d $(cat /sys/class/infiniband/$d/ports/1/rate)"; done' \
  | grep -v "400 Gb"
</code>

<code>
su3-n17: mlx5_5 200 Gb/sec (4X HDR)
</code>

**Reading:** one NIC trained at 200G. Rail 5 of su3-n17.

==== A4. Which switch port? ====

<code bash>
iblinkinfo | grep -i "su3-n17 mlx5_5"
</code>

<code>
su3-l5  17[  ] ==( 4X 53.125 Gbps Active/ LinkUp)==> su3-n17 mlx5_5
</code>

**Reading:** ''su3-l5'' port 17, running at HDR lane speed instead of NDR.

==== A5. Counters and optics, both ends ====

Switch side:

<code bash>
perfquery -x <su3-l5 LID> 17
</code>

<code>
SymbolErrorCounter: 41230     (baseline 0)
LinkDownedCounter:  1         (matches Day-2 02:14)
PortXmitWait:       normal
</code>

Host side, NIC optic:

<code bash>
mlxlink -d mlx5_5 -m -c
</code>

<code>
Rx Power Current [dBm] : -1.1, -1.3, -8.9, -1.2     <- lane 3 ~8 dB low
Effective Physical BER : 2E-08                       (healthy ~1E-12 or better)
</code>

**Phase A conclusion, no drain so far:** a degraded optic or dirty connector on su3-n17 ''mlx5_5'' ↔ ''su3-l5'' port 17. Lane 3 is weak, the link renegotiated to 200G, and every collective's rail-5 channels now run at half speed.

===== Phase B: repair window agreed with the app team =====

==== B0. Take the node out ====

App team checkpoints, then:

<code bash>
scontrol update nodename=su3-n17 state=drain reason="rail5 link 200G, optic"
</code>

The job restarts on ''su3-s01'': live busbw back to 360 GB/s, step time 1.10 s. Now su3-n17 is free to test.

==== B1. Decouple GPU vs. host vs. network on su3-n17 ====

Each test exercises a different slice of the path:

^ Test ^ GPU / HBM ^ NVLink ^ GPU↔NIC PCIe ^ NIC, optic, cable, switch ^
| Single-node nccl-tests | ✔ | ✔ | — | — |
| ''ib_write_bw'' host memory | — | — | — | ✔ |
| ''ib_write_bw'' GPU memory | ✔ | — | ✔ | ✔ |

**GPU side:**

<code bash>
dcgmi diag -r 2
</code>

<code>
Overall: Pass
</code>

<code bash>
./build/all_reduce_perf -b 8 -e 8G -f 2 -g 8
</code>

<code>
#       size    ...   algbw   busbw
  8589934592    ...   273.2   478.1
# Avg bus bandwidth : 478.1
</code>

**Network path without the GPU (host memory), against the spare:**

<code bash>
# on su3-s01 (server)
ib_write_bw -d mlx5_5 -F --report_gbits -s 1048576 -D 10
# on su3-n17 (client)
ib_write_bw -d mlx5_5 -F --report_gbits -s 1048576 -D 10 su3-s01
</code>

<code>
 #bytes   #iterations  BW peak[Gb/sec]  BW average[Gb/sec]
 1048576  ...          197.3            196.1
</code>

Control on a healthy rail (''-d mlx5_4''): 389.6 Gb/s.

**Full path with GPU memory:**

<code bash>
ib_write_bw -d mlx5_5 -F --report_gbits -s 1048576 -D 10 --use_cuda=5 su3-s01
</code>

<code>
 1048576  ...          196.4            195.2
</code>

**2-node NCCL, rail 5 only:**

<code bash>
mpirun -np 2 -H su3-n17:1,su3-s01:1 \
       -x NCCL_IB_HCA==mlx5_5 -x CUDA_VISIBLE_DEVICES=5 \
       ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1
</code>

<code>
# Avg bus bandwidth : 48.3     (single-rail baseline 96)
</code>

**Reading the matrix:**

^ GPU side ^ Host memory ^ GPU memory ^ Verdict ^
| Pass / 478 | **196** | **195** | Low even without the GPU → **network**, not GPU or PCIe |

==== B2. Locate the part: swap, one change at a time ====

^ Action ^ Result ^ Reading ^
| Move cable to spare port 33 on ''su3-l5'' | Still 200G, lane 3 low | Not the leaf port |
| Clean connectors, reseat | Still 200G | Not dirt |
| Replace host-side optic | **400G**, Rx lane 3 −1.2 dBm, BER < 1E-12 | **Root cause: host-side optic** |

==== B3. Verify and return ====

<code>
cat /sys/class/infiniband/mlx5_5/ports/1/rate     -> 400 Gb/sec (4X NDR)
ib_write_bw ... (host memory)                     -> 390.2 Gb/s
2-node all_reduce_perf, all 8 rails               -> Avg bus bandwidth 370.4
perfquery -x <su3-l5 LID> 33  (after 30 min)      -> SymbolErrorCounter: 0
</code>

<code bash>
scontrol update nodename=su3-n17 state=resume
</code>

su3-n17 rejoins the spare pool. Live busbw on the job stays at 360 GB/s.

===== Why one half-speed link cost about 29% of JCT =====

NCCL splits each collective evenly across channels on all 8 NICs. The rail-5 channels through su3-n17 took about 2× longer, and every collective completes only when its slowest channel does. All 512 ranks waited, so exposed comm went from 0.25 s to 0.57 s per step.

===== Verify on your system =====

  * Prometheus label names (exporter-dependent).
  * ''clocks_throttle_reasons'' is renamed ''clocks_event_reasons'' in newer drivers.
  * ''--use_cuda'' syntax; perftest must be built with CUDA.
  * ''NCCL_IB_HCA'' exact-match ''='' syntax.
  * ''mlxlink'' output layout varies by MFT version.