🛸 Hitchhiker's Guide — Phase 08: Distributed Systems & HPC Collectives

Read this if: you can train on one GPU but "all-reduce," "tensor parallel," and "why is MFU 50%" are fog. Field notes; derivations in the WARMUP.

0. The 30-second mental model

Many GPUs become one via collectives (all-reduce gradients/activations) over a bandwidth hierarchy (NVLink ≫ InfiniBand ≫ PCIe). Ring all-reduce moves ~2V/rank regardless of N (bandwidth-optimal). Tensor parallel must live on NVLink; data parallel can cross nodes. At scale, any one failure hangs everyone — so checkpoint, detect, recover. Low MFU = exposed communication, bubbles, or stragglers, not slow kernels.

1. Parallelism card

Frontier = compose them: TP in node, PP across few nodes, DP across rest.

2. Collective cost card

all-reduce = reduce-scatter + all-gather = 2·(N-1)/N·V ≈ 2V per rank   (N-independent!)
naive all-gather-then-sum = N·V per rank                                (N× worse)
ring: bandwidth-optimal, latency O(N)   |   tree: bandwidth worse, latency O(log N)
NCCL auto-picks by message size; hierarchical = reduce in-node then across-node

3. Topology stakes (the numbers)

HBM 3.35 TB/s  >  NVLink 900 GB/s  >  PCIe 64 GB/s  ≈  InfiniBand 50 GB/s

TP across nodes instead of NVLink ≈ 2–5× total slowdown (per-layer all-reduce ×20 slower × dozens of layers, unhideable). The #1 distributed misconfig. Schedulers must gang-place TP groups on one NVLink island (Phase 06).

4. NCCL survival kit

NCCL is fast when topology+config are right, silently 10× slow when not. Know what it's doing.

5. Diagnosing low MFU (the flowchart)

MFU < 70%? ->
  comm not overlapped?   bucketing on? all-reduce on critical path? (most common)
  topology wrong?        TP spanning nodes? PCIe instead of NVLink/IB? GPUDirect off?
  pipeline bubbles?      too few micro-batches for PP depth
  stragglers?            per-rank step-time variance -> one slow GPU (thermal/ECC)
  data/checkpoint stall? on the critical path?

Find what's not overlapped. Kernel micro-opt is rarely the answer at scale.

6. Failure reality

At 10k GPUs, ~1% of nodes have an issue daily. A synchronous collective hangs on any failure (no crash — a hang). Defense stack:

• Detect: collective timeouts + watchdogs (hang→crash+culprit), health monitoring (Xid/NIC), straggler detection.
• Recover: async sharded checkpoints (cadence = cost/risk lever), elastic training, hot spares.
• Minimize blast radius: placement so one failure stalls the smallest group.

Failures are the steady state; design for continuous partial failure. (And for sovereign/on-prem, no cloud auto-heal — you own recovery.)

7. War stories

• "8-GPU TP job at half speed" — scheduler spread it across 2 nodes; per- layer all-reduce hit IB not NVLink. Topology constraint → fixed. The classic.
• "Training hung at 3am, no error" — one GPU Xid'd; 1023 ranks blocked in NCCL forever. No NCCL_TIMEOUT set. Added it; now it crashes with the culprit rank in seconds.
• "Slow only on the new cluster" — NCCL_SOCKET_IFNAME picked the mgmt NIC, not IB. One env var, 10× collective speedup.
• "MFU 55% and kernels look fine" — gradient overlap was off (custom training loop bypassed DDP's bucketing). Re-enabled; MFU → 78%.
• "One slow GPU tanked a 256-GPU job" — thermal-throttled card did ECC retries; every all-reduce waited for it. Straggler detection drained the node.

8. Exit bar

You can derive ring all-reduce cost, contrast DP/TP/PP and place each on the topology, debug a slow collective via NCCL signals, diagnose low MFU, and design for failure at scale. Plus you've built ring all-reduce over real sockets and a topology cost model. Next: Phase 09 — the keystone, where all of this hardware knowledge becomes a hardware-independent platform (the JD's core mandate).