The Eight ML-Infrastructure Designs
The design round that a general SWE cannot fake. Eight systems worked end to end — nine sections each, then attacked by a hostile staff-level interviewer, then revised. Six critiques per design, each naming a real defect in the first draft.
These are Track C's designs with the physics changed. The primitives are the same — fencing, floors, leases, segments, level-triggered reconciliation — but the binding constraint is HBM bandwidth and KV bytes, not CPU or disk. That substitution is the whole round.
Table of Contents
- How to Use These
- The Eight
- The Numbers That Decide Each Design
- What Carries Over From Track C, and What Does Not
- Cross-Cutting Patterns
- What the Critiques Found
- The Order to Work Them
- References
How to Use These
Same loop as Track C's twelve, with one addition that is specific to this round:
- Read only the prompt. Stop.
- Do the arithmetic first, before any diagram — 5 minutes with
../gpu_math.py. In this round the numbers select the architecture, and every one of these eight designs is decided by a calculation that fits on an index card. - Write your own, 45 minutes, against the template.
- Score yourself against
RUBRIC.mdbefore reading further. - Read sections 1–10, then the critique — try to answer each one first — then the revision.
- Everything you missed goes into
../../../review/at 1 day.
The arithmetic step is not optional and it is the differentiator. Across all eight, the most common defect the critiques found was a claim that a two-line calculation disproves. If you write these designs without numbers you will produce a plausible architecture that a staff interviewer dismantles in ninety seconds.
The Eight
| # | Design | The two hard parts | Why it is in the set |
|---|---|---|---|
| m01 | Multi-tenant LLM API platform | fairness on a memory-bound resource · admission at the preemption cliff | The canonical AI-lab design round. Do this first |
| m02 | KV cache / prefix cache tier | which storage tiers beat recompute · cache-key correctness | The cleanest arithmetic in the set; one derivation deletes a component |
| m03 | GPU cluster scheduler | topology-aware placement · gang scheduling without deadlock | Training and inference on one fleet; where the money is |
| m04 | Pretraining data pipeline | global dedup at 15B docs · deterministic order and exact resumption | Correctness at petabyte scale, with almost no compute |
| m05 | Evaluation harness | reproducibility in a non-deterministic stack · when a delta is real | The statistics round hiding inside a systems round |
| m06 | Retrieval-augmented serving | where the latency budget actually goes · freshness vs index cost | Retrieval and serving; the tokens are the coupling |
| m07 | Multi-adapter (LoRA) serving | batching heterogeneous adapters · the cold-start long tail | A serving constraint on a research parameter |
| m08 | Training fault tolerance | the goodput equation · failures that do not announce themselves | 233 interruptions in 30 days, and what to do about them |
The Numbers That Decide Each Design
One calculation per design decides its architecture. If you can reproduce these eight, you can open any of these rounds with the sentence that reframes the question.
| Design | The calculation | What it decides |
|---|---|---|
| m01 | A 128k-context request holds 40 GiB of KV = 23% of a 4×H100 replica | Fairness must be in KV·seconds, not requests. A request limiter is off by 735× |
| m02 | Break-even BW = kv_bytes_per_token × FLOPS / 2N = 9.3 GB/s at TP4, independent of prefix length | DRAM and RDMA qualify; NVMe at 7 GB/s is slower than recomputing. No disk tier |
| m03 | At 50% cluster free, expected fully-free 8-GPU nodes ≈ 0.5 | Fragmentation, not capacity, is the constraint. Best-fit + shape segregation |
| m04 | All-pairs dedup = (15e9)²/2 = 1.1e20 comparisons = 3.5M core-years | Wrong algorithm, not slow one. MinHash + LSH, threshold (1/b)^(1/r) |
| m05 | 500-item benchmark: SE = 1.72 pp → resolves only differences > 6.7 pp | Most reported improvements are noise. Paired testing recovers 2.4× for free |
| m06 | Retrieval 85 ms; prefilling what it returned 207 ms | Optimize k and the reranker, not the index. Every chunk costs 17.8 ms + 0.15 GiB |
| m07 | Distinct adapters at batch 128: +6% (attn r=8) vs +151% (all-modules r=64) | A research hyperparameter moves serving cost 25×. Constrain it at registration |
| m08 | Restart term R/M = 5.4% of the run at 16,384 GPUs, invariant in the checkpoint interval | Everyone tunes the interval; the leverage is in restart time |
Say the number, then the consequence. That ordering — measurement before architecture — is what the rubric's highest-weighted line is actually measuring.
What Carries Over From Track C, and What Does Not
Carries over unchanged. These are the same problems in different clothes, and recognizing them is worth real credit:
| Track C primitive | Where it reappears |
|---|---|
| Reserved floors over strict priority | m01 (enterprise tier) · m03 (quota) · m07 (tail adapter slots) — and d05, d12 |
| Fencing tokens / epochs | m03 R5 (node boot epoch) · m08 (leader supervisor) |
| Leases with heartbeats | m01 R4 (in-flight KV accounting) · m03 (worker liveness) |
| Immutable segments + tombstones + compaction | m06 (vector index) — identical to d07, d09 |
| Session guarantees | m06 §8 (delete visibility as an ACL boundary) |
| Level-triggered reconciliation | m03 (allocations) — same as d12 |
| Estimate-then-reconcile | m01 §6 (token budgets) — same as d03 |
| Cache admission (TinyLFU) over pure LRU | m02 §8 · m07 §7 |
Does not carry over — and getting these wrong is the tell that you learned distributed systems and assumed it transferred:
| Instinct from Track C | Why it fails here |
|---|---|
| "Utilization is the load signal" | nvidia-smi reads ~100% during a batch-1 decode using 1/295th of the machine. Use KV occupancy |
| "Add a disk cache tier" | m02: below ~9 GB/s, recompute is faster. Storage hierarchies invert when the cached object is enormous |
| "Degradation is gradual near saturation" | KV exhaustion causes preemption + full prefill recompute, a positive feedback loop. It is a cliff, not a slope |
| "Scale out to fix latency" | Decode is bandwidth-bound; more replicas add throughput, not speed. And TP across nodes costs +52% |
| "Retry the failed request" | An in-flight generation's state is the KV cache. There is nothing to retry onto |
| "More data / more items is the fix" | m05: it is, but the arithmetic says how much — and paired testing is 2.4× cheaper |
The unifying sentence: in Track C the scarce resource is coordination; in Track D it is bandwidth. Every mechanism that assumed cheap memory movement has to be re-derived.
Cross-Cutting Patterns
Specific to this track — the ideas that recur across the eight and nowhere in Track C:
| Pattern | Where it appears |
|---|---|
| KV cache is the capacity unit | m01 (fairness) · m02 (the cache is the resource) · m06 (retrieval inflates it 7.2×) · m07 (adapters compete with it) |
| Prefill vs decode are different workloads | m01 (chunked prefill) · m06 (the 207 ms) · m07 (prefill reads adapters per chunk) |
| The break-even calculation | m02 (fetch vs recompute) · m03 (preempt vs wait) · m08 (checkpoint vs lose work) |
| A research parameter with a serving cost | m07 (rank, target modules) · m04 (vocab size doubles the corpus) · m02 (MLA changes every tier) |
| Cache made of the resource it caches for | m02 (KV cache vs in-flight KV) · m07 (adapters vs KV) |
| Affinity is required, not an optimization | m02 R2 · m06 (per-tenant index) · m07 R2 — a cache sized against the global working set is always wrong |
| Measured signal, not target signal | m01 R1 (measured TPOT) · m08 (measured MTBF drives the interval) · m05 (measured seed variance) |
| Silent wrongness over loud failure | m02 §7 (cache keys) · m05 §6 (batch composition) · m07 §8 (adapter/base mismatch) · m08 §7 (SDC) |
| The head and the tail want different architectures | m07 (merge the top 50, page the rest) · m06 (big tenants shard, small ones brute-force) · m05 (tiered evals) |
The last one is worth internalizing. Four of the eight designs conclude that an 80/20 traffic distribution should be served by two mechanisms, not one stretched across both. That is a generalizable move and it is rarely the first instinct.
What the Critiques Found
Forty-eight critiques across eight designs. The clusters, and what they say about how these designs fail:
| Defect class | Count | Example |
|---|---|---|
| Arithmetic never done | 9 | m07: host DRAM budget assumed 500 GB; m02's KV tier already claims 300 of it |
| A control loop with the wrong sign | 4 | m01: KV estimate uses target TPOT, so it under-estimates exactly when the fleet is loaded |
| A mechanism that needs the thing it provides | 5 | m06: defragmentation needs free capacity; it runs because there is none |
| A guarantee that is not the one sold | 4 | m01: enterprise bought latency; the floor delivered admission |
| An identity that can be reused | 3 | m03: a replacement node with the same hostname inherits allocations |
| A policy the caller must declare | 3 | m02: caller-supplied cache policy could not classify the highest-value case |
| A threshold on an unsized resource | 4 | m08: pinning guard at 50%, on a cache never sized with Little's law |
| A statistical claim without an interval | 3 | m05: seed variance conflated with engine variance |
| A rule with no path to yes | 2 | m07: rejecting rank-64 adapters that a customer genuinely needs |
| Alerting that will be muted | 2 | m08: paging on normal hardware failure at MTBF 3.1 h |
The two most instructive classes are new relative to Track C:
- Control loops with the wrong sign (m01 R1, m08 R2). A controller fed a target instead of a measurement behaves correctly in the healthy case and backwards in the loaded one — which is the only case that matters. Check every controller's behaviour at the point where target and measurement diverge.
- Thresholds on unsized resources (m08 R3, m07 R3). A threshold chosen before the resource was sized with Little's law either never fires or always fires. Size, then threshold.
And the perennial: arithmetic never done is still the largest single class, at 9 of 48 — down from 10 of 12 in Track C, which is what happens when the design opens with §2 instead of §5.
The Order to Work Them
Mandatory, in this order:
- m01 — the canonical round. Everything else references its numbers.
- m02 — the cleanest derivation in the program. Short, and it teaches the habit the whole track depends on.
- m08 — the training side, and the goodput equation is the most portable idea here.
Then by leverage:
- m03 — the fragmentation number is unforgettable and widely useful
- m06 — the most commonly asked in product-facing AI roles
- m05 — the one nobody prepares for; large differentiation per hour spent
- m07 — narrower, but the serving-constrains-research argument is unique
- m04 — deep, and only asked by labs that build corpora
One per week, 45 minutes to write plus 30 to critique — the same cadence as
Track C's twelve, and they
interleave: alternate a d and an m so the primitives reinforce across substrates.
References
../WARMUP.md— the roofline, KV budgets, batching and parallelism, from zero../README.md— Track D drills, the concept inventory, numbers to quote cold../gpu_math.py— every §2 in this directory, reproducible../../systems-design/designs/README.md— the twelve distributed designs these build on../../../CHEATSHEET.md#5-inference-infrastructure— the same material, dense, for the morning of a round../../../diagnostics/RUBRIC.md— how these are scored