Hypothetical design and invented values. Values read from dashboards (Grafana/Prometheus, fabric telemetry, drop telemetry) are stated directly. CLI is shown only where it's the natural tool.

• Fabric: 2048 H200 GPUs = 256 nodes × 8 GPUs and 8 BlueField-3 SuperNICs (400G). Rail-optimised, 2-tier Spectrum-X (SN5600): 8 SUs × 32 nodes, 64 leaves, 16 spines.
• Intended RoCE QoS: RoCE marked DSCP 26 → priority 3, lossless (PFC) with ECN. CNPs on DSCP 48. Adaptive routing and congestion control on.
• Naming: su3-n17 = SU3 node 17. su3-l5 = SU3's rail-5 leaf. eth5 / mlx5_5 = rail-5 NIC. su3-s0x = spares.
• Job 51877: 512 GPUs on su3-n[01-32], su4-n[01-32]. Step time has gone from 1.10 s to 1.31 s and is noisy (1.15-1.45 s); JCT from 18 h to about 21.5 h.

• All-reduce busbw: 355 GB/s on previous runs. Since Day-1 15:30: average 240 GB/s, swinging 180-300.
• NVLink-only communicators: unchanged.

Reading: cross-node comms degraded. The pattern is noisy, not a clean step, which suggests a congestion-dependent problem (loss/retransmits under load) rather than a hard fault.

• Straggler panel: no rank flagged.
• GPU clocks/throttling: normal on all 512 GPUs.

Reading: the transfer path, not a slow GPU.

• All job-facing links up at 400G/800G, no errors, no flaps.

Reading: not physical. On Ethernet, A1-A3 together mean packets are being dropped, delayed or retransmitted under load. The next question is about the fabric: where is traffic dropped, and what traffic is it?

• No switch config changes.
• Day-1 15:30: job restarted after a node failure, with su3-n17 (freshly reimaged) as the replacement.

Reading: a suspect, not yet evidence.

• Tail drops on spine downlinks and SU3/SU4 leaf downlinks, peaking during collectives (incast).
• In a lossless RoCE design there should be none.

Reading: drops confirmed. Which traffic?

• Drops are all in class 0 (lossy default), but they are large RDMA-sized flows.
• Class 3 (RoCE): no drops; PFC and ECN behaviour normal.

Reading: RoCE traffic is arriving in the wrong class, so something is mis-marking it.

• Every dropped packet: DSCP 0, one source IP in the rail-5 subnet.

Reading: a single sender. Map the IP to a port.

• The IP is learned on su3-l5 port swp9s0. LLDP on that port shows su3-n17 eth5.
• That port's ingress: RDMA traffic in class 0; class 3 ~0.

Reading: matches the A4 suspect. Only now does the investigation touch the host.

Host telemetry for su3-n17 rail 5: RDMA bytes go out on priority 0, not 3. RoCE retransmits are high, and congestion notifications handled are 0 (it gets no ECN-based congestion control). Config diff against a healthy node:

mlnx_qos -i eth5 | grep -i trust
cat /sys/class/infiniband/mlx5_5/tc/1/traffic_class

The same drift is on all 8 NICs of su3-n17: the host QoS role didn't run on reimage. The job doesn't set NCCL_IB_TC, so NCCL used the node default.

Phase A conclusion, no drain so far: host-side QoS config drift on su3-n17. Its RoCE traffic is marked DSCP 0, lands in the lossy class, is dropped under congestion and retransmitted, and gets no ECN congestion control. Fabric hardware and switch config are fine.

RDMA queue pairs take their traffic class when the job initialises, so fixing the host requires a job restart anyway. That restart is the repair window.

scontrol update nodename=su3-n17 state=drain reason="RoCE QoS not applied after reimage"

The job restarts on a spare. Grafana: live busbw 355 GB/s stable, step time 1.10 s.

Key point: a single uncongested flow never meets a full queue, so it is never dropped. Pair tests pass even with a QoS fault present. On Ethernet, reproduce with a congested pattern (all-to-all or incast).

Apply the QoS role on su3-n17 via config management (trust DSCP, traffic class 106, all 8 NICs). Fix the reimage pipeline so the role always runs.

• Host telemetry: RDMA on priority 3, retransmits flat, CNPs handled again.
• Switch counters on su3-l5 swp9s0: traffic in class 3.
• WJH: no drops from su3-n17.
• 8-node all-to-all with su3-n17: 40.9 GB/s.

scontrol update nodename=su3-n17 state=resume

Add a pre-flight check: a node's trust state and traffic class must match the golden config before it can take jobs.

Every collective includes channels to and from su3-n17. Its packets are dropped whenever a queue on the path fills, and each drop stalls that channel until the retransmit. The collective finishes only when the stalled channel does, so all 512 ranks wait. Drops depend on momentary congestion, so the slowdown is noisy, not a clean step.

• Go fabric-wide first: drops, traffic classes and congestion on the switches you own.
• Let the fabric tell you which host. The legitimate early pointers to a host are a change correlation (A4) or a source address in drop telemetry (A7).
• Touch the host only to confirm and find the cause.

• Dashboard/metric names depend on your telemetry stack.
• traffic_class 106 = DSCP 26 plus ECN-capable bits. Match it to your QoS design.
• The Spectrum-X RoCE profile defaults (DSCP 26 / priority 3 / CNP DSCP 48): confirm them on your switches.