DokuWiki

User Tools

Site Tools

Trace:
network_stuff:machine_learning:networking:ai:tshoot-roce

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

network_stuff:machine_learning:networking:ai:tshoot-roce [2026/10/09 15:05] – created adminnetwork_stuff:machine_learning:networking:ai:tshoot-roce [2026/10/09 15:12] (current) – admin
Line 1: Line 1:
 ====== Slow JCT troubleshooting – Spectrum-X Ethernet (east-west fabric), hypothetical example ====== ====== Slow JCT troubleshooting – Spectrum-X Ethernet (east-west fabric), hypothetical example ======
  
-//Hypothetical design and invented values. Outputs are abridged. Command syntax to verify is listed at the end.//+//Hypothetical design and invented values. Values read from dashboards (Grafana/Prometheus, fabric telemetry, drop telemetry) are stated directly. CLI is shown only where it's the natural tool.//
  
 ===== Context ===== ===== Context =====
  
-  * **Fabric:** 2048 H200 GPUs = 256 nodes × 8 GPUs and 8 BlueField-3 SuperNICs (400G). Rail-optimised, 2-tier Spectrum-X (SN5600, Cumulus/NVUE): 8 SUs × 32 nodes, 64 leaves (32 × 400G down via breakout, 16 × 800G up), 16 spines (64 × 800G). +  * **Fabric:** 2048 H200 GPUs = 256 nodes × 8 GPUs and 8 BlueField-3 SuperNICs (400G). Rail-optimised, 2-tier Spectrum-X (SN5600): 8 SUs × 32 nodes, 64 leaves, 16 spines. 
-  * **RoCE QoS design (the intended config):** RoCE marked **DSCP 26** → switch priority 3, lossless (PFC on priority 3) with ECN. CNPs on DSCP 48 / priority 6. Spectrum-X adaptive routing and congestion control on. +  * **Intended RoCE QoS:** RoCE marked **DSCP 26** → priority 3, lossless (PFC) with ECN. CNPs on DSCP 48. Adaptive routing and congestion control on. 
-  * **Naming:** ''su3-n17'' = SU3 node 17. ''su3-l5'' = SU3's rail-5 leaf. ''mlx5_5'' / ''eth5'' = rail-5 NIC (RDMA device / netdev). ''su3-s01'', ''su3-s02'' = spares. +  * **Naming:** ''su3-n17'' = SU3 node 17. ''su3-l5'' = SU3's rail-5 leaf. ''eth5'' / ''mlx5_5'' = rail-5 NIC. ''su3-s0x'' = spares. 
-  * **Job 51877:** 512 GPUs on ''su3-n[01-32]'', ''su4-n[01-32]''. The job was restarted on **Day-1 15:30** after a node failure, with ''su3-n17'' (freshly reimaged) as the replacement. Since then step time has gone from 1.10 s to 1.31 s, **noisy** (1.15-1.45 s), and JCT from 18 h to about 21.5 h.+  * **Job 51877:** 512 GPUs on ''su3-n[01-32]'', ''su4-n[01-32]''. Step time has gone from 1.10 s to 1.31 s and is **noisy** (1.15-1.45 s); JCT from 18 h to about 21.5 h.
  
 ^ Baseline (healthy) ^ Value ^ ^ Baseline (healthy) ^ Value ^
 | Live all-reduce busbw, cross-node communicator | 355 GB/s, stable | | Live all-reduce busbw, cross-node communicator | 355 GB/s, stable |
-| nccl-tests 2 nodes, 8 GB | 365 GB/s | 
 | nccl-tests all-to-all, 8 nodes | 41 GB/s | | nccl-tests all-to-all, 8 nodes | 41 GB/s |
 | ''ib_write_bw'' per NIC | 385 Gb/s | | ''ib_write_bw'' per NIC | 385 Gb/s |
  
-===== Phase A: live, job keeps running, read-only =====+===== Phase A: live, job keeps running =====
  
-==== A1. Live busbw ====+==== A1. Live busbw (Grafana, NCCL Inspector) ====
  
-<code> +  * All-reduce busbw: 355 GB/s on previous runs. **Since Day-1 15:30: average 240 GB/s, swinging 180-300.** 
-# Prometheus (NCCL Inspector / Spectrum-X NetInspector; label names depend on your exporter) +  * NVLink-only communicators: unchanged.
-avg by (coll) (nccl_bus_bandwidth_gbs{jobid="51877", coll="AllReduce"}) +
-</code>+
  
-<code> +**Reading:** cross-node comms degraded. The pattern is **noisy, not a clean step**, which suggests a congestion-dependent problem (loss/retransmits under load) rather than a hard fault.
-AllReduce   355 GB/s stable (previous runs)  ->  since Day-1 15:30: avg 240 GB/s, swinging 180-300 +
-NVLink-only communicators: unchanged +
-</code>+
  
-**Reading:** a noisy drop rather than a clean step. This pattern suggests a **congestion-dependent** problem (loss or retransmits under load) rather than a hard link fault. Onset matches the job restart with the replacement node.+==== A2. Straggler or transfer path? (straggler panel, GPU dashboard) ====
  
-==== A2. Straggler or transfer path? ====+  * Straggler panel: **no rank flagged**. 
 +  * GPU clocks/throttling: normal on all 512 GPUs.
  
-Straggler panel: **no rank flagged**. Same GPU sweep as the IB case:+**Reading:** the transfer path, not a slow GPU.
  
-<code bash> +==== A3. Physical health (fabric telemetry dashboard) ====
-pdsh -w su3-n[01-32],su4-n[01-32] \ +
-  'nvidia-smi --query-gpu=index,clocks.sm,clocks_throttle_reasons.active --format=csv,noheader' \ +
-  | awk -F, '$2+0 < 1900' +
-</code>+
  
-<code> +  * All job-facing links up at 400G/800G, no errors, no flaps.
-(no output) +
-</code>+
  
-**Reading:** transfer path.+**Reading:** not physical. On Ethernet, A1-A3 together mean **packets are being dropped, delayed or retransmitted under load**. The next question is about the **fabric**: where is traffic dropped, and what traffic is it?
  
-==== A3. Link-rate sweep (comes back clean) ====+==== A4. What changed at onset? (change log, scheduler history) ====
  
-<code bash> +  * **No switch config changes.** 
-pdsh -w su3-n[01-32],su4-n[01-32] \ +  * Day-1 15:30: job restarted after a node failure, with **su3-n17 (freshly reimaged)** as the replacement.
-  'for i in eth{0..7}; do echo "$i $(cat /sys/class/net/$i/speed)"; done' | grep -v 400000 +
-</code>+
  
-<code> +**Reading:** a suspect, not yet evidence.
-(no output: all 512 NICs at 400G) +
-</code>+
  
-Switch-side errors on all job-facing leaf ports: clean. **Unlike the IB case, there is no physical fault.**+==== A5. Is the fabric dropping anything? (drop telemetry / WJH, switch counters) ====
  
-==== A4. RoCE traffic-class sweep: is RDMA traffic in the right queue? ====+  * **Tail drops** on spine downlinks and SU3/SU4 leaf downlinks, peaking during collectives (incast). 
 +  * In a lossless RoCE design there should be **none**.
  
-Per-priority counters on every NIC, sampled 10 s apart (delta):+**Reading:** drops confirmed. Which traffic?
  
-<code bash> +==== A6. Which traffic class? (per-class counters on those ports) ====
-pdsh -w su3-n[01-32],su4-n[01-32] \ +
-  'for i in eth{0..7}; do a=$(ethtool -S $i | grep -E "tx_prio(0|3)_bytes"); sleep 10; \ +
-   b=$(ethtool -S $i | grep -E "tx_prio(0|3)_bytes"); echo "$i $a $b"; done' | <delta script> +
-</code>+
  
-<code> +  * Drops are all in **class 0 (lossy default)**, but they are large RDMA-sized flows. 
-all NICs except one:   tx_prio3 ~ 38 GB/10s    tx_prio0 ~ 0 +  * Class 3 (RoCE): no drops; PFC and ECN behaviour normal.
-su3-n17 eth5:          tx_prio3 ~ 0            tx_prio0 ~ 31 GB/10s     <-- RDMA in priority 0 +
-</code>+
  
-RoCE retransmit counters on the same NIC:+**Reading:** RoCE traffic is arriving in the wrong class, so **something is mis-marking it**.
  
-<code bash> +==== A7. Who sends it? (WJH drop records) ====
-cat /sys/class/infiniband/mlx5_5/ports/1/hw_counters/{local_ack_timeout_err,roce_adp_retrans,rp_cnp_handled} +
-</code>+
  
-<code> +  * Every dropped packet: **DSCP 0**, **one source IP** in the rail-5 subnet.
-local_ack_timeout_err: 18342     (other NICs: 0) +
-roce_adp_retrans:      961204    (other NICs: ~0) +
-rp_cnp_handled:        0         (other NICs: steady non-zero) +
-</code>+
  
-**Reading:** su3-n17's rail-5 RDMA traffic is marked into the **lossy default class**. Under congestion it is dropped (no PFC protection), which forces retransmits. It also gets no ECN-based congestion control: ''rp_cnp_handled'' = 0.+**Reading:** a single sender. Map the IP to a port.
  
-==== A5. Locate the switch port and confirm drops in the fabric ====+==== A8. Where does it attach? (fabric telemetry: neighbor table + LLDP) ====
  
-<code bash> +  * The IP is learned on **''su3-l5'' port ''swp9s0''**. LLDP on that port shows **su3-n17 ''eth5''**. 
-lldpctl eth5          # on su3-n17 +  * That port's ingress: RDMA traffic in class 0; class 3 ~0.
-</code>+
  
-<code> +**Reading:** matches the A4 suspect. **Only now does the investigation touch the host.**
-SysName: su3-l5    PortID: swp9s0 +
-</code>+
  
-Switch side (Cumulus):+==== A9. Confirm on the host and find why ====
  
-<code bash> +Host telemetry for su3-n17 rail 5: RDMA bytes go out on priority 0, not 3. RoCE retransmits are high, and congestion notifications handled are 0 (it gets no ECN-based congestion control). Config diff against a healthy node:
-nv show interface swp9s0 qos counters +
-</code> +
- +
-<code> +
-ingress traffic-class 0 : high rate   (should be ~0) +
-ingress traffic-class 3 : ~0          (should carry RoCE) +
-PFC rx/tx priority 3    : 0 +
-</code> +
- +
-What Just Happened (drop telemetry) on the spines and SU3/SU4 rail-5 leaves: +
- +
-<code> +
-Drop reason: tail drop   src: su3-n17 rail5 IP   DSCP: 0   egress: spine downlinks / leaf downlinks under incast +
-</code> +
- +
-==== A6. Config diff: why is it marked DSCP 0? ====+
  
 <code bash> <code bash>
-# on su3-n17 vs a healthy node (su3-n16) 
 mlnx_qos -i eth5 | grep -i trust mlnx_qos -i eth5 | grep -i trust
 cat /sys/class/infiniband/mlx5_5/tc/1/traffic_class cat /sys/class/infiniband/mlx5_5/tc/1/traffic_class
 </code> </code>
  
-<code> +^ Node ^ Trust ^ traffic_class ^ 
-su3-n16:  Priority trust state: dscp     traffic_class: 106   (DSCP 26 + ECN-capable) +| su3-n16 (healthy) | dscp | 106 (DSCP 26 + ECN-capable) | 
-su3-n17:  Priority trust state: pcp      traffic_class: (empty) +| su3-n17 | **pcp** | **(empty)** |
-</code>+
  
-The job doesn't set ''NCCL_IB_TC'', so NCCL uses the node default. On the reimaged node, the host QoS role didn't run, and **all 8 NICs** on su3-n17 have the same drift (A4 only flagged eth5 at first because rail 5 carried the most congested traffic at the time).+The same drift is on **all 8 NICs** of su3-n17: the host QoS role didn't run on reimage. The job doesn't set ''NCCL_IB_TC'', so NCCL used the node default.
  
-**Phase A conclusion, no drain so far:** **host-side config drift** on su3-n17. Its RoCE traffic is not marked DSCP 26, lands in the lossy class, is dropped under congestion and retransmitted, and sees no ECN congestion control. **The fabric hardware and switch config are fine.**+**Phase A conclusion, no drain so far:** **host-side QoS config drift** on su3-n17. Its RoCE traffic is marked DSCP 0, lands in the lossy class, is dropped under congestion and retransmitted, and gets no ECN congestion control. **Fabric hardware and switch config are fine.**
  
 ===== Phase B: repair window agreed with the app team ===== ===== Phase B: repair window agreed with the app team =====
  
-The fix is applied on the host, but RDMA queue pairs pick up their traffic class when the job initialises. The job must restart either way, so this is the repair window.+RDMA queue pairs take their traffic class when the job initialises, so fixing the host requires a job restart anyway. That restart is the repair window.
  
 ==== B0. Take the node out ==== ==== B0. Take the node out ====
Line 145: Line 98:
 </code> </code>
  
-The job restarts on ''su3-s01'': live busbw 355 GB/s stable, step time 1.10 s.+The job restarts on a spare. Grafana: live busbw **355 GB/s stable**, step time 1.10 s.
  
-==== B1. Decoupling on su3-n17, and why pair tests mislead on Ethernet ====+==== B1. Confirm on su3-n17, and why pair tests mislead on Ethernet ====
  
-**GPU side:**+^ Test ^ Result ^ Baseline ^ Reading ^ 
 +| GPU health (DCGM diag), single-node all-reduce | Pass / 478 GB/s | 480 | GPU side fine | 
 +| ''ib_write_bw'' pair, host memory | **385 Gb/s** | 385 | Normal | 
 +| ''ib_write_bw'' pair, GPU memory | **384 Gb/s** | 385 | Normal | 
 +| 8-node all-to-all **with** su3-n17 | **24.6 GB/s** | 41 | Fault reproduces under congestion | 
 +| Same, **without** su3-n17 | 40.8 GB/s | 41 | Fault follows the node |
  
-<code bash> +**Key point:** a single uncongested flow never meets a full queue, so it is never dropped. **Pair tests pass even with a QoS fault present.** On Ethernet, reproduce with a congested pattern (all-to-all or incast).
-dcgmi diag -r 2                               -> Overall: Pass +
-./build/all_reduce_perf -b 8 -e 8G -f 2 -g 8  -> Avg bus bandwidth : 477.6 +
-</code> +
- +
-**Point-to-point, host memory and GPU memory (before the fix):** +
- +
-<code bash> +
-show_gids mlx5_5                     # find the RoCEv2 GID index, here 3 +
-# on su3-s02 (server) +
-ib_write_bw -d mlx5_5 -x 3 -F --report_gbits -s 1048576 -D 10 +
-# on su3-n17 (client) +
-ib_write_bw -d mlx5_5 -x 3 -F --report_gbits -s 1048576 -D 10 su3-s02 +
-ib_write_bw -d mlx5_5 -x 3 -F --report_gbits -s 1048576 -D 10 --use_cuda=5 su3-s02 +
-</code> +
- +
-<code> +
-host memory: 384.7 Gb/s     GPU memory: 383.9 Gb/s     <- both NORMAL +
-</code> +
- +
-**Reading:** a single uncongested flow never hits a full queue, so it's never dropped. **Pair tests pass with a QoS fault present.** On Ethernet, a clean pair test does not clear the network. +
- +
-**Reproduce under congestion (8-node all-to-all, includes incast):** +
- +
-<code bash> +
-mpirun -np 64 -H su3-n17:8,su3-s01:8,su3-s02:8,<5 more spares>:8 \ +
-       ./build/alltoall_perf -b 1M -e 1G -f 2 -g 1 +
-</code> +
- +
-<code> +
-# Avg bus bandwidth : 24.6     (baseline 41) +
-</code> +
- +
-Same test without su3-n17: 40.8. The fault shows only under congestion and only with su3-n17 in the set. +
- +
-^ GPU side ^ Pair, host mem ^ Pair, GPU mem ^ Congested all-to-all ^ Counters / config ^ Verdict ^ +
-| Pass / 478 | 385 (normal) | 384 (normal) | **24.6** (low) | DSCP 0, prio 0, no CNPs | **Host RoCE QoS config**, not hardware |+
  
 ==== B2. Fix ==== ==== B2. Fix ====
  
-<code bash> +Apply the QoS role on su3-n17 via config management (trust DSCP, traffic class 106, all 8 NICs). Fix the reimage pipeline so the role always runs.
-# on su3-n17, all 8 NICs (persist via config management, not by hand) +
-mlnx_qos -i eth5 --trust dscp +
-echo 106 > /sys/class/infiniband/mlx5_5/tc/1/traffic_class +
-# repeat for eth0-7 / mlx5_0-7; fix the reimage pipeline so the QoS role always runs +
-</code>+
  
 ==== B3. Verify and return ==== ==== B3. Verify and return ====
  
-<code> +  * Host telemetry: RDMA on priority 3, retransmits flat, CNPs handled again. 
-ethtool -S eth5 delta          -> tx_prio3 ~ 38 GB/10s, tx_prio0 ~ 0 +  * Switch counters on ''su3-l5 swp9s0'': traffic in class 3. 
-nv show interface swp9s0 qos counters -> traffic in class 3, PFC/ECN active +  * WJH: no drops from su3-n17. 
-8-node alltoall_perf with su3-n17     -> Avg bus bandwidth 40.9 +  * 8-node all-to-all with su3-n17: **40.9 GB/s**.
-WJH                                   -> no drops from su3-n17 +
-hw_counters local_ack_timeout_err     -> 0 increase over 30 min +
-</code>+
  
 <code bash> <code bash>
Line 212: Line 126:
 </code> </code>
  
-su3-n17 rejoins the spare pool. Also add a **pre-flight check** that compares ''mlnx_qos'' trust state and ''traffic_class'' with the golden config before a node can take jobs.+Add a **pre-flight check**: a node's trust state and traffic class must match the golden config before it can take jobs.
  
 ===== Why a marking error on one node cost about 19% of JCT ===== ===== Why a marking error on one node cost about 19% of JCT =====
  
-Every collective includes channels to and from su3-n17. Its packets are dropped whenever a queue on the path fills, and each drop stalls that channel until the retransmit. The collective finishes only when the stalled channel does, so all 512 ranks wait. Because drops depend on momentary congestion, the slowdown is **noisy**, not a clean step: that pattern alone points away from a hard link fault.+Every collective includes channels to and from su3-n17. Its packets are dropped whenever a queue on the path fills, and each drop stalls that channel until the retransmit. The collective finishes only when the stalled channel does, so all 512 ranks wait. Drops depend on momentary congestion, so the slowdown is **noisy**, not a clean step. 
 + 
 +===== Principle ===== 
 + 
 +  * Go **fabric-wide first**: drops, traffic classes and congestion on the switches you own. 
 +  * **Let the fabric tell you which host.** The legitimate early pointers to a host are a **change correlation** (A4) or a **source address in drop telemetry** (A7). 
 +  * Touch the host only to confirm and find the cause.
  
-===== IB vs. Spectrum-X: what changes in the method =====+===== IB vs. Spectrum-X: what changes =====
  
 ^ Aspect ^ InfiniBand case ^ Spectrum-X case ^ ^ Aspect ^ InfiniBand case ^ Spectrum-X case ^
 | Typical fault signature | Clean step, link rate/BER visible | Noisy, congestion-dependent | | Typical fault signature | Clean step, link rate/BER visible | Noisy, congestion-dependent |
-| Link-rate sweep | Finds it | Clean | +| Physical health check | Finds it | Clean | 
-| Key extra checks | ''perfquery'', ''mlxlink'' | Per-priority counters, DSCP/trust config, CNP/retransmit counters, WJH | +| Key evidence | Link rate, BER, symbol errors | Drop telemetry, traffic class, DSCP, CNP/retransmits | 
-| Pair test (''ib_write_bw'') | Shows the fault | **Can pass with a fault present**; needs a congested test (all-to-all, incast) | +| Pair test | Shows the fault | **Can pass with a fault present**; needs a congested test | 
-| Locate switch port | ''iblinkinfo'' | LLDP |+| Locate the port | Fabric topology / ''iblinkinfo'' | Source IP → neighbor table → LLDP |
  
 ===== Verify on your system ===== ===== Verify on your system =====
  
-  * NVUE command and counter names (''nv show interface ... qos counters'' and the WJH commands vary by Cumulus release). +  * Dashboard/metric names depend on your telemetry stack. 
-  * ''hw_counters'' names (''roce_adp_retrans'', ''rp_cnp_handled'', ''local_ack_timeout_err'') vary by driver/firmware. +  * ''traffic_class'' 106 = DSCP 26 plus ECN-capable bits. Match it to your QoS design. 
-  * ''traffic_class'' 106 = DSCP 26 shifted into the ToS byte plus ECN-capable bits. Match it to your QoS design. +  * The Spectrum-X RoCE profile defaults (DSCP 26 / priority 3 / CNP DSCP 48): confirm them on your switches.
-  * The default Spectrum-X RoCE profile values (DSCP 26 / priority 3 / CNP DSCP 48) come from the standard RoCE profile. Confirm them with ''nv show qos roce''. +
-  * ''ib_write_bw'' ''-x'' (GID index) and ''--use_cuda'' syntax depend on the perftest version.+
network_stuff/machine_learning/networking/ai/tshoot-roce.txt · Last modified: by admin