Phase 08 — Distributed Systems & HPC Collectives
Difficulty: ⭐⭐⭐⭐⭐ | Estimated Time: 2 weeks Roles supported: Head of Engineering [GPU], Distributed Systems Engineer, HPC/ML-Systems Engineer Hardware needed: none — the collectives lab runs over real OS sockets on one machine (multi-process); the topology lab is a simulator
Why This Phase Exists
The JD wants "distributed systems, HPC" and "AI infrastructure at production scale." A single GPU is never enough at the frontier: training a large model and serving it with tensor parallelism both turn many GPUs into one logical device, and the glue is collective communication — all-reduce, all-gather, reduce-scatter — running over NVLink and InfiniBand. When training "mysteriously" runs at half speed, or a node failure hangs an 1,024-GPU job, the cause is almost always in this layer.
You'll implement ring all-reduce over real sockets (the algorithm NCCL uses) and benchmark it against the theoretical bandwidth bound, then build a topology-aware collective simulator that shows why placement (Phase 06) and bandwidth (Phase 01 Ch. 9) decide distributed throughput — and where failures turn into hangs.
Concepts
- Why distribute: model/data don't fit one GPU; the parallelism taxonomy
- Data parallel (replicate model, split batch, all-reduce gradients)
- Tensor parallel (split each layer across GPUs, all-reduce per layer — NVLink-bound)
- Pipeline parallel (split layers into stages, micro-batch, bubble overhead)
- Expert/sequence/context parallel (overview) and how they compose (3D/4D parallelism)
- Collective operations: broadcast, reduce, all-reduce, all-gather, reduce-scatter, all-to-all — and their communication volumes
- Ring all-reduce: the algorithm and its
2(N-1)/N × databandwidth optimality; reduce-scatter + all-gather decomposition - Tree vs ring vs hierarchical collectives; latency vs bandwidth regimes; small vs large messages
- NCCL: what it does, how it picks algorithms/topology,
NCCL_DEBUG, the env knobs that matter - Topology: NVLink/NVSwitch islands, PCIe, InfiniBand/RoCE, rails, GPUDirect RDMA; why placement (Phase 06) sets collective bandwidth
- Overlap: computation/communication overlap, bucketing gradients, the critical path
- Failure domains at scale: a single GPU/NIC failure hangs a synchronous collective; detection, checkpointing, elastic/fault-tolerant training; the "1% of nodes fail daily at 10k-GPU scale" reality
- Serving-side distribution: tensor-parallel inference, KV-cache sharding, prefill/decode disaggregation traffic (Phase 07)
Labs
Lab 01 — Ring All-Reduce Over Real Sockets (Python)
| Field | Value |
|---|---|
| Goal | Implement ring all-reduce across N OS processes communicating over TCP sockets; verify correctness and benchmark against the 2(N-1)/N bandwidth bound. |
| Concepts | Reduce-scatter + all-gather, ring topology, bandwidth optimality, why all-reduce volume is ~2× the data regardless of N. |
| Steps | 1) python solution.py --world 4 — launches 4 worker processes, runs ring all-reduce on a vector, verifies the result equals the naive sum, prints bandwidth vs the bound. 2) Read the ring algorithm against WARMUP Ch. 3. 3) Compare against a naive all-gather-then-sum (more bytes). 4) Extensions: tree all-reduce, bucketing, a node-failure hang demo. |
| Stack | Python (stdlib socket, multiprocessing) |
| Output | A working multi-process ring all-reduce + a bandwidth report (achieved vs 2(N-1)/N bound). |
| How to Test | Built-in: all-reduced vector is bit-equal to the reference sum across all ranks; ring moves ≈2(N-1)/N × data per rank (verified by byte counting), beating the naive O(N) approach. |
| Talking Points | Why ring all-reduce is bandwidth-optimal and N-independent in per-link volume; the reduce-scatter/all-gather decomposition; when tree beats ring (latency-bound small messages); how this maps to gradient all-reduce in DDP. |
| Resume Bullet | "Implemented ring all-reduce over TCP sockets across N processes; verified bit-exact reduction and measured per-rank communication volume at 2(N-1)/N × data, matching the bandwidth-optimal bound NCCL targets." |
| Extensions | Tree all-reduce + compare latency on small messages; gradient bucketing/overlap; a failure-injection mode where one rank dies and the collective hangs — then add a timeout + detection. |
Lab 02 — Topology-Aware Collective Simulator (Python)
| Field | Value |
|---|---|
| Goal | Model a multi-node GPU cluster's interconnect (NVLink islands, InfiniBand between nodes) and compute collective time for different parallelism placements — proving topology decides throughput. |
| Concepts | Bandwidth hierarchy (NVLink/IB), tensor-parallel-within-node, data-parallel-across-node, rail topology, the cost of a badly-placed collective. |
| Steps | 1) python solution.py — models all-reduce time for TP/DP placements across topologies, prints a comparison. 2) Read the cost model against WARMUP Ch. 6. 3) Reproduce: TP within an NVLink island is N× faster than TP spread across nodes (the Phase 06 topology argument, quantified). 4) Extensions: pipeline-bubble model, hierarchical all-reduce, failure-domain blast radius. |
| Stack | Python (stdlib) |
| Output | A topology cost model + a placement-comparison table (collective time per strategy). |
| How to Test | Built-in asserts: TP-within-island beats TP-across-nodes by the NVLink/IB bandwidth ratio; hierarchical all-reduce beats flat across nodes; placement matters more than algorithm at scale. |
| Talking Points | Why tensor parallelism must stay within NVLink islands; how the scheduler (Phase 06) and the collective (this phase) jointly determine throughput; the failure-domain blast radius of different placements. |
| Resume Bullet | "Built a topology-aware collective cost model quantifying the NVLink-vs-InfiniBand placement gap; demonstrated tensor-parallel-within-island delivering an order-of-magnitude lower all-reduce time than cross-node placement." |
| Extensions | Pipeline-parallel bubble model; hierarchical (intra-then-inter-node) all-reduce; failure-domain analysis (how many GPUs one NIC/switch failure stalls). |
Deliverables Checklist
- Ring all-reduce: correct across N processes; bandwidth report matches the bound
- You can derive the 2(N-1)/N all-reduce volume and explain why it's N-independent
- Topology sim: placement-comparison table reproduced and explained
- You can explain TP/DP/PP and why TP must stay on NVLink
- You can describe how a single-GPU failure hangs a synchronous job and the mitigations
- One page: "our platform's multi-GPU/multi-node strategy" (feeds Phase 12)
Interview Relevance
- "Derive ring all-reduce's communication cost. Why is it bandwidth-optimal?"
- "Compare data, tensor, and pipeline parallelism. Which is NVLink-bound and why?"
- "Your 512-GPU training run is at 60% MFU. Where do you look?" (overlap, topology, stragglers)
- "One GPU fails in a 1024-GPU job. What happens, and how do you make it survivable?"
- "Why does tensor-parallel placement across nodes destroy throughput?"