Hypothetical design and invented values. Outputs are abridged. Command syntax to verify is listed at the end.

• Fabric: 2048 H200 GPUs = 256 nodes × 8 GPUs and 8 BlueField-3 SuperNICs (400G). Rail-optimised, 2-tier Spectrum-X (SN5600, Cumulus/NVUE): 8 SUs × 32 nodes, 64 leaves (32 × 400G down via breakout, 16 × 800G up), 16 spines (64 × 800G).
• RoCE QoS design (the intended config): RoCE marked DSCP 26 → switch priority 3, lossless (PFC on priority 3) with ECN. CNPs on DSCP 48 / priority 6. Spectrum-X adaptive routing and congestion control on.
• Naming: su3-n17 = SU3 node 17. su3-l5 = SU3's rail-5 leaf. mlx5_5 / eth5 = rail-5 NIC (RDMA device / netdev). su3-s01, su3-s02 = spares.
• Job 51877: 512 GPUs on su3-n[01-32], su4-n[01-32]. The job was restarted on Day-1 15:30 after a node failure, with su3-n17 (freshly reimaged) as the replacement. Since then step time has gone from 1.10 s to 1.31 s, noisy (1.15-1.45 s), and JCT from 18 h to about 21.5 h.

# Prometheus (NCCL Inspector / Spectrum-X NetInspector; label names depend on your exporter)
avg by (coll) (nccl_bus_bandwidth_gbs{jobid="51877", coll="AllReduce"})

AllReduce   355 GB/s stable (previous runs)  ->  since Day-1 15:30: avg 240 GB/s, swinging 180-300
NVLink-only communicators: unchanged

Reading: a noisy drop rather than a clean step. This pattern suggests a congestion-dependent problem (loss or retransmits under load) rather than a hard link fault. Onset matches the job restart with the replacement node.

Straggler panel: no rank flagged. Same GPU sweep as the IB case:

pdsh -w su3-n[01-32],su4-n[01-32] \
  'nvidia-smi --query-gpu=index,clocks.sm,clocks_throttle_reasons.active --format=csv,noheader' \
  | awk -F, '$2+0 < 1900'

(no output)

Reading: transfer path.

pdsh -w su3-n[01-32],su4-n[01-32] \
  'for i in eth{0..7}; do echo "$i $(cat /sys/class/net/$i/speed)"; done' | grep -v 400000

(no output: all 512 NICs at 400G)

Switch-side errors on all job-facing leaf ports: clean. Unlike the IB case, there is no physical fault.

Per-priority counters on every NIC, sampled 10 s apart (delta):

pdsh -w su3-n[01-32],su4-n[01-32] \
  'for i in eth{0..7}; do a=$(ethtool -S $i | grep -E "tx_prio(0|3)_bytes"); sleep 10; \
   b=$(ethtool -S $i | grep -E "tx_prio(0|3)_bytes"); echo "$i $a $b"; done' | <delta script>

all NICs except one:   tx_prio3 ~ 38 GB/10s    tx_prio0 ~ 0
su3-n17 eth5:          tx_prio3 ~ 0            tx_prio0 ~ 31 GB/10s     <-- RDMA in priority 0

RoCE retransmit counters on the same NIC:

cat /sys/class/infiniband/mlx5_5/ports/1/hw_counters/{local_ack_timeout_err,roce_adp_retrans,rp_cnp_handled}

local_ack_timeout_err: 18342     (other NICs: 0)
roce_adp_retrans:      961204    (other NICs: ~0)
rp_cnp_handled:        0         (other NICs: steady non-zero)

Reading: su3-n17's rail-5 RDMA traffic is marked into the lossy default class. Under congestion it is dropped (no PFC protection), which forces retransmits. It also gets no ECN-based congestion control: rp_cnp_handled = 0.

lldpctl eth5          # on su3-n17

SysName: su3-l5    PortID: swp9s0

Switch side (Cumulus):

nv show interface swp9s0 qos counters

ingress traffic-class 0 : high rate   (should be ~0)
ingress traffic-class 3 : ~0          (should carry RoCE)
PFC rx/tx priority 3    : 0

What Just Happened (drop telemetry) on the spines and SU3/SU4 rail-5 leaves:

Drop reason: tail drop   src: su3-n17 rail5 IP   DSCP: 0   egress: spine downlinks / leaf downlinks under incast

# on su3-n17 vs a healthy node (su3-n16)
mlnx_qos -i eth5 | grep -i trust
cat /sys/class/infiniband/mlx5_5/tc/1/traffic_class

su3-n16:  Priority trust state: dscp     traffic_class: 106   (DSCP 26 + ECN-capable)
su3-n17:  Priority trust state: pcp      traffic_class: (empty)

The job doesn't set NCCL_IB_TC, so NCCL uses the node default. On the reimaged node, the host QoS role didn't run, and all 8 NICs on su3-n17 have the same drift (A4 only flagged eth5 at first because rail 5 carried the most congested traffic at the time).

Phase A conclusion, no drain so far: host-side config drift on su3-n17. Its RoCE traffic is not marked DSCP 26, lands in the lossy class, is dropped under congestion and retransmitted, and sees no ECN congestion control. The fabric hardware and switch config are fine.

The fix is applied on the host, but RDMA queue pairs pick up their traffic class when the job initialises. The job must restart either way, so this is the repair window.

scontrol update nodename=su3-n17 state=drain reason="RoCE QoS not applied after reimage"

The job restarts on su3-s01: live busbw 355 GB/s stable, step time 1.10 s.

GPU side:

dcgmi diag -r 2                               -> Overall: Pass
./build/all_reduce_perf -b 8 -e 8G -f 2 -g 8  -> Avg bus bandwidth : 477.6

Point-to-point, host memory and GPU memory (before the fix):

show_gids mlx5_5                     # find the RoCEv2 GID index, here 3
# on su3-s02 (server)
ib_write_bw -d mlx5_5 -x 3 -F --report_gbits -s 1048576 -D 10
# on su3-n17 (client)
ib_write_bw -d mlx5_5 -x 3 -F --report_gbits -s 1048576 -D 10 su3-s02
ib_write_bw -d mlx5_5 -x 3 -F --report_gbits -s 1048576 -D 10 --use_cuda=5 su3-s02

host memory: 384.7 Gb/s     GPU memory: 383.9 Gb/s     <- both NORMAL

Reading: a single uncongested flow never hits a full queue, so it's never dropped. Pair tests pass with a QoS fault present. On Ethernet, a clean pair test does not clear the network.

Reproduce under congestion (8-node all-to-all, includes incast):

mpirun -np 64 -H su3-n17:8,su3-s01:8,su3-s02:8,<5 more spares>:8 \
       ./build/alltoall_perf -b 1M -e 1G -f 2 -g 1

# Avg bus bandwidth : 24.6     (baseline 41)

Same test without su3-n17: 40.8. The fault shows only under congestion and only with su3-n17 in the set.

# on su3-n17, all 8 NICs (persist via config management, not by hand)
mlnx_qos -i eth5 --trust dscp
echo 106 > /sys/class/infiniband/mlx5_5/tc/1/traffic_class
# repeat for eth0-7 / mlx5_0-7; fix the reimage pipeline so the QoS role always runs

ethtool -S eth5 delta          -> tx_prio3 ~ 38 GB/10s, tx_prio0 ~ 0
nv show interface swp9s0 qos counters -> traffic in class 3, PFC/ECN active
8-node alltoall_perf with su3-n17     -> Avg bus bandwidth 40.9
WJH                                   -> no drops from su3-n17
hw_counters local_ack_timeout_err     -> 0 increase over 30 min

scontrol update nodename=su3-n17 state=resume

su3-n17 rejoins the spare pool. Also add a pre-flight check that compares mlnx_qos trust state and traffic_class with the golden config before a node can take jobs.

Every collective includes channels to and from su3-n17. Its packets are dropped whenever a queue on the path fills, and each drop stalls that channel until the retransmit. The collective finishes only when the stalled channel does, so all 512 ranks wait. Because drops depend on momentary congestion, the slowdown is noisy, not a clean step: that pattern alone points away from a hard link fault.

• NVUE command and counter names (nv show interface … qos counters and the WJH commands vary by Cumulus release).
• hw_counters names (roce_adp_retrans, rp_cnp_handled, local_ack_timeout_err) vary by driver/firmware.
• traffic_class 106 = DSCP 26 shifted into the ToS byte plus ECN-capable bits. Match it to your QoS design.
• The default Spectrum-X RoCE profile values (DSCP 26 / priority 3 / CNP DSCP 48) come from the standard RoCE profile. Confirm them with nv show qos roce.
• ib_write_bw -x (GID index) and –use_cuda syntax depend on the perftest version.