====== Slow JCT troubleshooting – Spectrum-X Ethernet (east-west fabric), hypothetical example ======

//Hypothetical design and invented values. Values read from dashboards (Grafana/Prometheus, fabric telemetry, drop telemetry) are stated directly. CLI is shown only where it's the natural tool.//

===== Context =====

  * **Fabric:** 2048 H200 GPUs = 256 nodes × 8 GPUs and 8 BlueField-3 SuperNICs (400G). Rail-optimised, 2-tier Spectrum-X (SN5600): 8 SUs × 32 nodes, 64 leaves, 16 spines.
  * **Intended RoCE QoS:** RoCE marked **DSCP 26** → priority 3, lossless (PFC) with ECN. CNPs on DSCP 48. Adaptive routing and congestion control on.
  * **Naming:** ''su3-n17'' = SU3 node 17. ''su3-l5'' = SU3's rail-5 leaf. ''eth5'' / ''mlx5_5'' = rail-5 NIC. ''su3-s0x'' = spares.
  * **Job 51877:** 512 GPUs on ''su3-n[01-32]'', ''su4-n[01-32]''. Step time has gone from 1.10 s to 1.31 s and is **noisy** (1.15-1.45 s); JCT from 18 h to about 21.5 h.

^ Baseline (healthy) ^ Value ^
| Live all-reduce busbw, cross-node communicator | 355 GB/s, stable |
| nccl-tests all-to-all, 8 nodes | 41 GB/s |
| ''ib_write_bw'' per NIC | 385 Gb/s |

===== Phase A: live, job keeps running =====

==== A1. Live busbw (Grafana, NCCL Inspector) ====

  * All-reduce busbw: 355 GB/s on previous runs. **Since Day-1 15:30: average 240 GB/s, swinging 180-300.**
  * NVLink-only communicators: unchanged.

**Reading:** cross-node comms degraded. The pattern is **noisy, not a clean step**, which suggests a congestion-dependent problem (loss/retransmits under load) rather than a hard fault.

==== A2. Straggler or transfer path? (straggler panel, GPU dashboard) ====

  * Straggler panel: **no rank flagged**.
  * GPU clocks/throttling: normal on all 512 GPUs.

**Reading:** the transfer path, not a slow GPU.

==== A3. Physical health (fabric telemetry dashboard) ====

  * All job-facing links up at 400G/800G, no errors, no flaps.

**Reading:** not physical. On Ethernet, A1-A3 together mean **packets are being dropped, delayed or retransmitted under load**. The next question is about the **fabric**: where is traffic dropped, and what traffic is it?

==== A4. What changed at onset? (change log, scheduler history) ====

  * **No switch config changes.**
  * Day-1 15:30: job restarted after a node failure, with **su3-n17 (freshly reimaged)** as the replacement.

**Reading:** a suspect, not yet evidence.

==== A5. Is the fabric dropping anything? (drop telemetry / WJH, switch counters) ====

  * **Tail drops** on spine downlinks and SU3/SU4 leaf downlinks, peaking during collectives (incast).
  * In a lossless RoCE design there should be **none**.

**Reading:** drops confirmed. Which traffic?

==== A6. Which traffic class? (per-class counters on those ports) ====

  * Drops are all in **class 0 (lossy default)**, but they are large RDMA-sized flows.
  * Class 3 (RoCE): no drops; PFC and ECN behaviour normal.

**Reading:** RoCE traffic is arriving in the wrong class, so **something is mis-marking it**.

==== A7. Who sends it? (WJH drop records) ====

  * Every dropped packet: **DSCP 0**, **one source IP** in the rail-5 subnet.

**Reading:** a single sender. Map the IP to a port.

==== A8. Where does it attach? (fabric telemetry: neighbor table + LLDP) ====

  * The IP is learned on **''su3-l5'' port ''swp9s0''**. LLDP on that port shows **su3-n17 ''eth5''**.
  * That port's ingress: RDMA traffic in class 0; class 3 ~0.

**Reading:** matches the A4 suspect. **Only now does the investigation touch the host.**

==== A9. Confirm on the host and find why ====

Host telemetry for su3-n17 rail 5: RDMA bytes go out on priority 0, not 3. RoCE retransmits are high, and congestion notifications handled are 0 (it gets no ECN-based congestion control). Config diff against a healthy node:

<code bash>
mlnx_qos -i eth5 | grep -i trust
cat /sys/class/infiniband/mlx5_5/tc/1/traffic_class
</code>

^ Node ^ Trust ^ traffic_class ^
| su3-n16 (healthy) | dscp | 106 (DSCP 26 + ECN-capable) |
| su3-n17 | **pcp** | **(empty)** |

The same drift is on **all 8 NICs** of su3-n17: the host QoS role didn't run on reimage. The job doesn't set ''NCCL_IB_TC'', so NCCL used the node default.

**Phase A conclusion, no drain so far:** **host-side QoS config drift** on su3-n17. Its RoCE traffic is marked DSCP 0, lands in the lossy class, is dropped under congestion and retransmitted, and gets no ECN congestion control. **Fabric hardware and switch config are fine.**

===== Phase B: repair window agreed with the app team =====

RDMA queue pairs take their traffic class when the job initialises, so fixing the host requires a job restart anyway. That restart is the repair window.

==== B0. Take the node out ====

<code bash>
scontrol update nodename=su3-n17 state=drain reason="RoCE QoS not applied after reimage"
</code>

The job restarts on a spare. Grafana: live busbw **355 GB/s stable**, step time 1.10 s.

==== B1. Confirm on su3-n17, and why pair tests mislead on Ethernet ====

^ Test ^ Result ^ Baseline ^ Reading ^
| GPU health (DCGM diag), single-node all-reduce | Pass / 478 GB/s | 480 | GPU side fine |
| ''ib_write_bw'' pair, host memory | **385 Gb/s** | 385 | Normal |
| ''ib_write_bw'' pair, GPU memory | **384 Gb/s** | 385 | Normal |
| 8-node all-to-all **with** su3-n17 | **24.6 GB/s** | 41 | Fault reproduces under congestion |
| Same, **without** su3-n17 | 40.8 GB/s | 41 | Fault follows the node |

**Key point:** a single uncongested flow never meets a full queue, so it is never dropped. **Pair tests pass even with a QoS fault present.** On Ethernet, reproduce with a congested pattern (all-to-all or incast).

==== B2. Fix ====

Apply the QoS role on su3-n17 via config management (trust DSCP, traffic class 106, all 8 NICs). Fix the reimage pipeline so the role always runs.

==== B3. Verify and return ====

  * Host telemetry: RDMA on priority 3, retransmits flat, CNPs handled again.
  * Switch counters on ''su3-l5 swp9s0'': traffic in class 3.
  * WJH: no drops from su3-n17.
  * 8-node all-to-all with su3-n17: **40.9 GB/s**.

<code bash>
scontrol update nodename=su3-n17 state=resume
</code>

Add a **pre-flight check**: a node's trust state and traffic class must match the golden config before it can take jobs.

===== Why a marking error on one node cost about 19% of JCT =====

Every collective includes channels to and from su3-n17. Its packets are dropped whenever a queue on the path fills, and each drop stalls that channel until the retransmit. The collective finishes only when the stalled channel does, so all 512 ranks wait. Drops depend on momentary congestion, so the slowdown is **noisy**, not a clean step.

===== Principle =====

  * Go **fabric-wide first**: drops, traffic classes and congestion on the switches you own.
  * **Let the fabric tell you which host.** The legitimate early pointers to a host are a **change correlation** (A4) or a **source address in drop telemetry** (A7).
  * Touch the host only to confirm and find the cause.

===== IB vs. Spectrum-X: what changes =====

^ Aspect ^ InfiniBand case ^ Spectrum-X case ^
| Typical fault signature | Clean step, link rate/BER visible | Noisy, congestion-dependent |
| Physical health check | Finds it | Clean |
| Key evidence | Link rate, BER, symbol errors | Drop telemetry, traffic class, DSCP, CNP/retransmits |
| Pair test | Shows the fault | **Can pass with a fault present**; needs a congested test |
| Locate the port | Fabric topology / ''iblinkinfo'' | Source IP → neighbor table → LLDP |

===== Verify on your system =====

  * Dashboard/metric names depend on your telemetry stack.
  * ''traffic_class'' 106 = DSCP 26 plus ECN-capable bits. Match it to your QoS design.
  * The Spectrum-X RoCE profile defaults (DSCP 26 / priority 3 / CNP DSCP 48): confirm them on your switches.