Lab 02 — Topology-Aware Collective Simulator (Python)

Phase: 08 — Distributed & HPC | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–6 hours Language: Python (stdlib) | Hardware: none

Concept primer: ../WARMUP.md Ch. 4, 6.

Run

python solution.py

0. The mission

Model a cluster's interconnect hierarchy (NVLink within a node, InfiniBand between nodes) and compute collective time for different placements — proving the Phase-06-meets-Phase-08 thesis: where you put a parallel group decides throughput more than which algorithm you use.

1. What the output shows

== all-reduce time for an 8-way tensor-parallel group (256 MB activation, x80 layers/step) ==
placement                   links          per-step (ms)   vs best
TP within 1 NVLink node     NVLink                  44.0      1.0x
TP split across 2 nodes     InfiniBand             753.9     17.1x  <- catastrophic
TP split across 4 nodes     InfiniBand             753.9     17.1x

== data-parallel gradient all-reduce, 4 nodes x 8 GPUs (256 MB) ==
flat ring over all 32 GPUs (IB)          : 10.5 ms
hierarchical (NVLink in-node, IB across)  : 1.6 ms   (6.7x faster)

== failure blast radius (1 GPU dies, synchronous job) ==
TP-within-node placement : 1 GPU failure stalls 8 GPUs (one TP group)
TP-across-nodes placement: 1 GPU failure stalls 32 GPUs (spans all nodes)

(The TP cost is multiplied by 80 layers because tensor parallelism all-reduces inside every layer — which is exactly why crossing nodes is catastrophic: you pay the slow-link penalty dozens of times per step, unhideable.)

  • TP placement: tensor parallelism within one NVLink node is ~18× faster than spread across nodes — the WARMUP Ch. 6 result, quantified. Same algorithm, same data; only placement differs. This is why "my 8-GPU TP job is slow" is almost always "it got spread across nodes."
  • Hierarchical all-reduce: reduce within NVLink islands first, then across nodes over IB — does the big-volume reduction on fast links, only the small cross-node step on slow links (WARMUP Ch. 4). ~6× faster than a flat ring.
  • Blast radius: placement also decides how many GPUs one failure stalls (WARMUP Ch. 8).

2. Reading order (solution.py)

  1. Link bandwidths (NVLink/IB/PCIe) and the allreduce_time cost model (2(N-1)/N · V / bandwidth + latency).
  2. The placement scenarios — how the ranks map to nodes and which links the collective crosses.
  3. hierarchical_allreduce_time — the two-level decomposition.
  4. The blast-radius computation.

3. Extensions

  1. Pipeline bubble model: compute PP efficiency = microbatches / (microbatches + stages - 1); show how more micro-batches shrink the bubble.
  2. 3D parallelism placement: TP=8 (in node), PP=4 (across nodes), DP=rest; compute total step time and find the placement that minimizes it.
  3. Rail/GPUDirect model: add a PCIe-staging penalty when GPUDirect RDMA is misconfigured; show the 50× host-bounce cliff.
  4. Failure-domain optimization: given a job shape, find the placement that minimizes blast radius subject to the NVLink-island constraint.

4. Common pitfalls

  1. Using bandwidth without latency — for small messages latency dominates and ring loses to tree (WARMUP Ch. 4); the model includes a per-hop latency term.
  2. Forgetting that TP all-reduces per layer — the per-step TP cost is n_layers × per-layer-allreduce, which is what makes cross-node TP catastrophic (not a one-time cost).
  3. Treating all GPUs as equidistant — the entire lesson is that they aren't.

5. What this lab proves about you

You can quantify the interconnect's effect on distributed throughput and explain why placement (Phase 06) and collectives (Phase 08) must be co-designed. When a training job underperforms, you reason from topology — "is the TP group on one NVLink island?" — before touching kernels. That's the systems-level judgment the JD's "HPC at production scale" requires.