Track D — Warmup: Inference Infrastructure, From Zero

Self-contained. Assumes you know what a matrix multiply is and nothing else about transformers. By the end you should be able to derive the memory budget and the decode latency floor on a whiteboard, explain every throughput technique and what it costs, and answer "design ChatGPT" at two altitudes.

This is the round most senior generalists lose. It is also the one where your search and ranking background transfers further than you would guess.

Table of Contents

• Chapter 0: Why Your Background Transfers
• Chapter 1: What Inference Actually Does
  • 1.1 A transformer, in the only detail that matters here
  • 1.2 Attention, and why it needs a cache
  • 1.3 Prefill and decode are different workloads
• Chapter 2: The Roofline — The One Derivation
  • 2.1 Arithmetic intensity
  • 2.2 Decode is memory-bandwidth-bound
  • 2.3 Prefill is compute-bound
  • 2.4 The H100 vs H200 proof
  • 2.5 Everything that follows from this
• Chapter 3: The Memory Budget
  • 3.1 Weights
  • 3.2 The KV cache, derived
  • 3.3 GQA and why long context is affordable
  • 3.4 Activations and overhead
  • 3.5 A complete worked budget
• Chapter 4: Batching
  • 4.1 Static batching and why it wastes everything
  • 4.2 Continuous batching
  • 4.3 PagedAttention
  • 4.4 Chunked prefill
  • 4.5 The scheduler is the product
• Chapter 5: The Rest of the Toolkit
  • 5.1 Prefix caching
  • 5.2 Speculative decoding
  • 5.3 Quantization
  • 5.4 Parallelism: TP, PP, EP
  • 5.5 Disaggregated prefill and decode
• Chapter 6: Autoscaling Non-Stationary Traffic
  • 6.1 Why request-count autoscaling fails
  • 6.2 The signals that work
  • 6.3 Predictive versus reactive
  • 6.4 Admission control and fairness
• Chapter 7: Worked Answer — Design ChatGPT
  • 7.1 The opening ninety seconds
  • 7.2 Altitude 1: the abstracted answer
  • 7.3 Altitude 2: "open up the serving layer"
  • 7.4 The follow-ups, with answers
• The Numbers Sheet
• The Thirty-Five Questions
• References

Chapter 0: Why Your Background Transfers

Ten years on multilingual search and recommendation means you have shipped: a serving tier with a hard latency budget, an index too large for one machine, ranking under a compute constraint, a cache hierarchy where hit rate is the economics, and traffic that is non-stationary by time zone.

LLM serving is the same problem class with substitutions:

The gap is vocabulary and the memory-bandwidth constraint, not concepts. That last row is the one genuinely new thing, and it is the source of almost every difference.

Chapter 1: What Inference Actually Does

1.1 A transformer, in the only detail that matters here

A decoder-only transformer is a stack of L identical layers. Text is converted to a sequence of tokens (sub-word units, ~4 characters of English each), each mapped to a vector of dimension d.

Each layer does two things:

1. Attention — every token looks at every previous token and pulls in information.
2. A feed-forward network — a per-token MLP, usually expanding to 4d and back.

After the last layer, a projection to vocabulary size produces a probability distribution over the next token. You sample one, append it, and run the whole stack again.

That last sentence is the entire performance story. Generating n tokens means running the full model n times, sequentially, because token k+1 depends on token k. There is no way to parallelize across the tokens of one response — the dependency is inherent.

1.2 Attention, and why it needs a cache

For each token, each layer computes three projections of its vector: Q (query), K (key), V (value). Attention for token i is:

\[ \text{out}i = \sum{j \le i} \text{softmax}\left(\frac{Q_i \cdot K_j}{\sqrt{d_h}}\right) V_j \]

Token i attends over the K and V of every token before it.

Here is the crucial observation: K and V for token j never change. They are a function of token j and the weights, both fixed. So when you generate token 501, you do not recompute K and V for tokens 1–500 — you cache them.

That is the KV cache, and it is the single most important object in LLM serving.

Without it, generating token n costs O(n) work for the whole prefix, so generating n tokens costs O(n²). With it, each new token costs O(1) new K/V plus an O(n) attention read. The cache converts quadratic to linear, and in exchange it consumes memory that grows with batch size × sequence length.

Everything hard about LLM serving is a consequence of the KV cache being large, growing, and unpredictable in final size.

1.3 Prefill and decode are different workloads

Prefill — processing the prompt. All prompt tokens are known up front, so all of them are processed in parallel, in one pass. It is a big matrix-multiply. Output: the KV cache for the prompt, plus the first generated token.

Decode — generating the rest, one token at a time, each depending on the last. Strictly sequential.

These are two different workloads sharing one accelerator, and that sentence is the compact form of most of this chapter. A scheduler that treats them identically lets one long prefill block everyone's decode — which is exactly the problem chunked prefill was invented to solve.

Chapter 2: The Roofline — The One Derivation

If you internalize one thing in this track, this is it. Derive it on a whiteboard in ninety seconds.

2.1 Arithmetic intensity

Any computation moves bytes from memory and does FLOPs on them. Define:

\[ I = \frac{\text{FLOPs performed}}{\text{bytes moved}} \]

A processor has peak compute \(P\) (FLOP/s) and peak bandwidth \(B\) (bytes/s). Their ratio, \(P/B\), is the machine balance — the arithmetic intensity at which the two are matched.

• \( I < P/B \) → memory-bound. Compute units idle waiting for data.
• \( I > P/B \) → compute-bound. Memory idle waiting for the ALUs.

For an H100: \( P = 989.5 \times 10^{12} \) FLOP/s (BF16, dense), \( B = 3.35 \times 10^{12} \) bytes/s. So:

\[ P/B \approx 295 \text{ FLOP per byte} \]

You must do ~295 floating-point operations on every byte you load to keep an H100 busy. That is a demanding bar, and it is the number that explains everything below.

⚠ The sparsity asterisk — a trap worth knowing

NVIDIA's H100 datasheet says 1,979 TFLOP/s BF16. That figure carries an asterisk: with sparsity. It assumes 2:4 structured sparsity — two of every four weights are zero and the Tensor Core skips them. Dense BF16 is exactly half: 989.5 TFLOP/s.

LLM inference weights are dense. Nothing in a standard decode step benefits from the sparsity path. So the honest machine balance for this workload is 295 FLOP/byte, not 590, and every ratio below is computed against the dense number.

Quoting 1,979 for a dense workload is one of the easiest ways to lose credibility in this round — it says you read a spec sheet rather than a benchmark. The safe phrasing:

"989 dense BF16, or 1,979 with 2:4 sparsity, which doesn't apply here. So call it a machine balance around 295."

It does not change any conclusion. Decode at \(I \approx 1\) is memory-bound against 295 by exactly as decisive a margin as against 590. But it changes the numbers, and the numbers are what get checked.

2.2 Decode is memory-bandwidth-bound

Take one decode step at batch size b, for a model with N parameters at 2 bytes each.

Bytes moved. Every weight must be read once. Batching does not change this — the same weights serve all sequences in the batch. Plus the KV cache for all b sequences.

\[ \text{bytes} = 2N + b \cdot s \cdot \text{kv_per_token} \]

FLOPs. Each of the b sequences does roughly \(2N\) FLOPs (one multiply-add per parameter).

\[ \text{FLOPs} = 2Nb \]

So arithmetic intensity, ignoring the KV term:

\[ I_{\text{decode}} \approx \frac{2Nb}{2N} = b \]

The arithmetic intensity of decode is approximately the batch size.

At batch 1, \(I = 1\) — against a machine balance of 295. You are using roughly 1/295th of the GPU's compute. At batch 64, \(I = 64\) — still 4.6× below balance.

Concrete numbers, 70B at FP16 on an H100:

weights            = 70e9 × 2 bytes = 140 GB
time to read them  = 140 GB / 3.35 TB/s = 42 ms
FLOPs at batch 1   = 2 × 70e9 = 140 GFLOP
time to compute    = 140e9 / 989.5e12 = 0.141 ms     (dense BF16)

ratio = 41.8 / 0.1415 = 295x    (exactly the machine balance, as the derivation predicts)

The GPU spends ~295× longer waiting on HBM than computing. It is idle almost all of the time. Note that the ratio comes out at exactly the machine balance — that is not a coincidence, it is what \( I = 1 \) against \( P/B = 295 \) means, and it is a good self-check that the arithmetic is right. (If you quote the sparsity number you get 590×, which is the same conclusion reached with the wrong constant.)

2.3 Prefill is compute-bound

Now prefill a prompt of s tokens. All s are processed together, so:

\[ \text{FLOPs} \approx 2Ns, \qquad \text{bytes} \approx 2N, \qquad I_{\text{prefill}} \approx s \]

A 2,000-token prompt gives \(I = 2000\), comfortably above the 295 balance — by nearly 7×. Prefill is compute-bound.

Same model and hardware:

FLOPs = 2 × 70e9 × 2000 = 280 TFLOP
time  = 280e12 / 989.5e12 = 283 ms  (compute, dense BF16)
bytes = 140 GB → 42 ms              (memory)
compute dominates by 6.7x

So one 2,000-token prefill costs ~283 ms of pure compute — during which, on a naive scheduler, nobody else's decode runs. That is the head-of-line blocking problem, and it is why TTFT for one user and TPOT for everyone else are in direct conflict.

2.4 The H100 vs H200 proof

The clean empirical confirmation, and the sentence to have ready:

The H200 has exactly the same compute and 43% more bandwidth, and it is materially faster at decode. If decode were compute-bound it would be exactly as fast.

That is a two-sentence, falsifiable, citable argument. Deploy it.

2.5 Everything that follows from this

Every technique in this track is a consequence:

1. Batching is nearly free on the compute axis. \(I \approx b\), so raising the batch raises intensity toward machine balance at almost no extra bandwidth cost — the weight read is amortized. This is why continuous batching is the single biggest throughput lever.
2. The KV cache, not the weights, limits your batch. Weights are fixed; KV grows with \(b \times s\). Memory management is therefore the hard engineering problem.
3. Prefill and decode want different scheduling, because one is compute-bound and the other is bandwidth-bound. Hence chunked prefill and disaggregation.
4. Quantization helps decode more than compute suggests, because halving the weight bytes halves the dominant term.
5. Speculative decoding trades spare compute for latency — and it only works because decode leaves ~99% of the compute idle. At high batch there is no spare compute and it stops paying.

Chapter 3: The Memory Budget

Three consumers of HBM: weights, KV cache, activations. Do this arithmetic out loud.

3.1 Weights

\[ \text{bytes} = N \times \text{bytes per parameter} \]

A 70B model at FP16 needs 140 GB and an H100 has 80 GB, so it does not fit on one GPU — you need at least two, and in practice four for KV headroom. That single fact drives the parallelism discussion (§5.4).

3.2 The KV cache, derived

For each token, each layer stores one K vector and one V vector per KV head:

\[ \text{KV bytes per token} = 2 \times L \times H_{kv} \times d_h \times \text{bytes} \]

• 2 — one K, one V
• \(L\) — layers
• \(H_{kv}\) — key/value heads (not query heads — see §3.3)
• \(d_h\) — head dimension
• bytes — 2 for FP16

Llama-70B: L = 80, \(H_{kv}\) = 8, \(d_h\) = 128, FP16.

\[ 2 \times 80 \times 8 \times 128 \times 2 = 327{,}680 \text{ bytes/token} = 320 \text{ KB/token} \]

Then:

A single 128k-context conversation needs 39 GB of KV cache — half an H100, for one user. That is the number that makes long context an infrastructure problem rather than a model feature, and it is worth saying out loud.

3.3 GQA and why long context is affordable

In original multi-head attention (MHA), every query head has its own K and V. So \(H_{kv} = H_q\).

For Llama-70B with 64 query heads, MHA would give:

\[ 2 \times 80 \times 64 \times 128 \times 2 = 2{,}621{,}440 \text{ bytes/token} = 2.5 \text{ MB/token} \]

8× larger. A 4,096-token sequence would need 10 GB of KV instead of 1.25 GB.

Grouped-Query Attention (GQA) shares one K/V head across a group of query heads — here, 8 query heads share each of 8 KV heads. Multi-Query Attention (MQA) is the extreme case with a single KV head.

The tradeoff: slightly lower quality, dramatically smaller KV cache — which means dramatically larger batch, which means dramatically lower cost per token. GQA is the architectural decision that makes long context economically possible, and it is one of the clearest examples of model architecture chosen for serving cost, which is a good thing to be able to point at.

3.4 Activations and overhead

Transient per-forward-pass memory: intermediate tensors, the attention workspace, CUDA context, framework overhead, and fragmentation.

Rule of thumb: reserve a few GB per GPU, more for large batch prefill. For napkin work, 4 GB is a reasonable placeholder — and say it is a placeholder, because a real number comes from profiling.

3.5 A complete worked budget

Llama-70B, FP16, on 4 × H100 (80 GB each), 4,096-token sequences.

Total memory       = 4 × 80 GB                        = 320 GB
Weights            = 70e9 × 2                         = 140 GB   (tensor-parallel: 35 GB/GPU)
Activations        = 4 GB × 4                         =  16 GB
                                                        --------
Available for KV                                      = 164 GB

KV per token       = 2 × 80 × 8 × 128 × 2             = 320 KB
KV per sequence    = 320 KB × 4096                    = 1.25 GB

Max batch          = 164 / 1.25                       ≈ 131 concurrent sequences

Now the decode floor at that batch:

bytes per step  = 140 GB (weights) + 131 × 1.25 GB (KV) = 304 GB
time            = 304 / 3.35 TB/s                        = 91 ms
throughput      = 131 tokens / 0.091 s                   ≈ 1,440 tok/s aggregate
per user        = 1 / 0.091                              ≈ 11 tok/s

Sanity check that against reality: ~11 tokens/second per user is roughly reading speed, which is about right for a chat product. If your arithmetic gives 200 tok/s per user or 0.5, you made an error — and knowing the plausible range is itself a useful check to state.

Two observations to volunteer:

• At batch 131 the KV cache (164 GB) exceeds the weights (140 GB). The cache is the dominant memory consumer, which is the opposite of most people's intuition.
• Doubling context to 8,192 halves the batch to ~65. Context length and concurrency trade directly against each other, one-for-one.

Run gpu_math.py to do this for any model and GPU.

Chapter 4: Batching

4.1 Static batching and why it wastes everything

The naive approach, and the one every non-specialist proposes: collect b requests, run them together, return all b, repeat.

It fails badly, for a reason specific to generation: sequences finish at different times.

batch of 4, output lengths 10, 200, 15, 180

step 10:  seq0 done. Its slot sits IDLE for 190 more steps.
step 15:  seq2 done. Idle for 185 more steps.
step 180: seq3 done.
step 200: seq1 done. Batch returns.

Two compounding wastes:

1. Idle slots. Utilization is mean(lengths) / max(lengths). With a realistic long-tailed output distribution, that is routinely 30–50%.
2. Head-of-line blocking. A request arriving at step 11 waits until step 200 to even start, even though three of four slots are empty.

4.2 Continuous batching

The fix (Orca, OSDI 2022; also called in-flight batching): schedule at iteration granularity, not batch granularity.

After every forward pass:

• Any sequence that finished is evicted and returned to its client immediately.
• Any waiting request is admitted into the freed slot.

The batch composition changes every step. There are no idle slots and no head-of-line blocking from long generations.

Reported gains are large — vendor benchmarks put continuous batching plus paged memory at several times naive throughput on identical hardware. Quote it as vendor-reported, or measure it yourself. This is exactly where the "numbers I measured vs numbers I read" distinction matters.

What it costs, which you should name: scheduler complexity, and the fact that batch composition now varies per step, so per-step latency is no longer uniform. That variance shows up as TPOT jitter, which users perceive as uneven streaming.

4.3 PagedAttention

Continuous batching creates a memory problem. If each sequence's KV cache is one contiguous allocation, you must reserve for the maximum possible length — you do not know how long the output will be.

Reserve 4,096 tokens for a sequence that generates 50, and 99% of that allocation is wasted. Reserve less and you must reallocate and copy mid-generation.

PagedAttention (vLLM, SOSP 2023) applies virtual-memory paging to the KV cache:

• The cache is divided into fixed-size blocks (e.g. 16 tokens).
• Each sequence has a block table mapping logical positions to physical blocks.
• Blocks are allocated on demand, as generation proceeds.
• Blocks need not be contiguous.

The analogy is exact — this is paging, with a page table, and it eliminates both internal fragmentation (waste inside an over-large allocation) and external fragmentation (free memory that is unusably scattered).

It also enables copy-on-write sharing: two sequences with the same prefix point at the same physical blocks until one diverges. That makes parallel sampling (n=4 completions of one prompt) nearly free in memory, and it is the mechanism underneath prefix caching.

What it costs: an indirection per attention operation, which needs a custom kernel. Small, and overwhelmingly worth it — but it is not free, and saying so is better than presenting it as a pure win.

4.4 Chunked prefill

Continuous batching still has a problem. Prefill is compute-bound and can take ~141 ms for a 2,000-token prompt (§2.3). During that step, every decoding sequence in the batch is stalled.

Result: one user's long prompt causes a visible stutter in everyone else's token stream.

Chunked prefill (Sarathi-Serve) splits a long prefill into chunks — say 512 tokens — and schedules each chunk alongside ongoing decodes:

step 1: [decode ×60] + [prefill chunk 1 of 4]
step 2: [decode ×60] + [prefill chunk 2 of 4]
...

Now every step contains decode work, so no sequence stalls for more than one chunk's duration.

The tradeoff, precisely: total prefill throughput drops slightly, because chunks are less efficient than one big matmul (less arithmetic intensity per chunk, plus the KV of earlier chunks must be re-read). In exchange, TTFT and TPOT tails improve substantially.

That is the shape of every decision in this chapter: you are moving along a throughput-versus-tail-latency curve, not getting a free win.

4.5 The scheduler is the product

Step back. Given continuous batching, paged memory, and chunked prefill, the scheduler decides, every iteration:

• Which waiting requests to admit (and which to reject)
• How many prefill chunks versus decode steps to include
• Which sequences to preempt when memory runs out — and preemption means either swapping KV to host memory (costly transfer) or recomputing it later (costly compute)
• How to honour priority classes

This is where the product's latency character actually lives. Two deployments of the same model on the same hardware with different scheduler policies have entirely different SLOs.

And this is the single most useful thing to say in the design round, because it reframes the question from "which techniques do you know" to "what policy would you choose, and for which traffic class". An interactive chat turn, a long agentic tool loop, and a batch API want different points on the curve — so the real question is whether you run separate pools or one priority-aware scheduler with preemption, and what each costs.

Chapter 5: The Rest of the Toolkit

5.1 Prefix caching

Many requests share a prefix: the same system prompt, the same tool definitions, the same few-shot examples, or — in a chat product — the entire conversation so far.

Prefill for that prefix is deterministic: same tokens plus same weights gives the same K and V. So cache the KV blocks and reuse them.

The gain is largest exactly where you most need it. In a multi-turn conversation, turn n's prompt is turn n−1's prompt plus two messages. Without prefix caching you re-prefill the whole history every turn, so a 20-turn conversation does O(n²) prefill work in total. With it, each turn prefills only the new tokens.

Implementation: hash the token prefix, look up cached blocks, reuse via copy-on-write. SGLang's RadixAttention organizes this as a radix tree so partial prefix matches are found efficiently.

Costs to name:

• Cache memory competes with KV cache for running sequences. It is a capacity allocation decision, not free.
• Eviction policy matters — LRU over prefixes, weighted by how much prefill each saves.
• Security: prefix cache keys must be scoped per tenant unless the prefix is genuinely public, or you have built a cross-tenant information leak. This is the kind of thing that reads very well when volunteered.

The routing consequence, and it is the important one for a design round: if replica A has a conversation's prefix cached, sending turn n+1 to replica B throws that away. So the load balancer must be prefix-aware / session-affine — which turns load balancing into a cache-affinity problem. That is a problem you have already solved in search, and it is the strongest single bridge from your background into this domain.

5.2 Speculative decoding

Decode is bandwidth-bound and leaves ~99% of the compute idle (§2.2). Speculative decoding spends that idle compute to reduce latency.

1. A small draft model (or a cheap heuristic, or the model's own earlier layers) proposes k tokens quickly.
2. The target model verifies all k in one forward pass — possible because the tokens are already known, so verification is a parallel prefill-shaped operation.
3. Accept the longest correct prefix; reject the rest and continue from there.

With a modified sampling rule, the output distribution is provably identical to sampling from the target model directly. It is not an approximation — that is what makes it acceptable in production.

Speedup ≈ mean accepted tokens per verification step. Typical reported acceptance gives 1.5–3× on latency.

When it loses, and this is the question:

• At high batch. The spare compute has been consumed by batching, so the verification pass is no longer nearly free. Speculative decoding is a low-batch, latency-oriented technique; it trades throughput for latency.
• Rejected tokens are wasted compute, so a poorly-matched draft model can make things worse.
• You now maintain and serve two models.

5.3 Quantization

Store weights (and optionally KV) in fewer bits.

Because decode is bandwidth-bound and weights dominate the bytes moved, halving weight precision nearly halves decode time. It also frees memory for a larger batch. So quantization helps twice.

What to say about quality, because this is where people overclaim: degradation is workload-specific. A model that looks fine on perplexity can degrade sharply on long-context retrieval, on code, or on the tail of a distribution that matters to you. So the answer is always "quantize, then evaluate on your own task-specific evals, and be prepared to keep the higher precision for the traffic that needs it."

KV-cache quantization is separately valuable: at 8-bit KV you halve the cache, doubling the batch. It is often the cheaper win because KV quantization tends to degrade quality less than weight quantization.

5.4 Parallelism: TP, PP, EP

When a model does not fit on one GPU, or one GPU is too slow, split it. Three axes.

Tensor parallelism (TP) — split each layer's matrices across GPUs; each computes a slice.

• Requires an all-reduce every layer, so it needs very fast interconnect (NVLink). Across PCIe or between nodes it collapses.
• Latency improves — the work per GPU shrinks.
• Practical limit: within one node (8 GPUs on NVLink).

Pipeline parallelism (PP) — split layers across GPUs; GPU 0 runs layers 1–20, GPU 1 runs 21–40, and so on.

• Only one activation transfer per boundary, so it tolerates slower interconnect and works across nodes.
• Introduces bubbles: GPU 1 idles until GPU 0 finishes. Micro-batching fills them, which helps throughput but not single-request latency.
• Latency does not improve — the request still traverses every stage.

Expert parallelism (EP) — for Mixture-of-Experts, place different experts on different GPUs.

• Only k of E experts activate per token, so total parameters can be huge while active parameters stay modest.
• Requires an all-to-all communication per MoE layer to route tokens to their experts, which becomes the bottleneck at scale.
• Load imbalance across experts is a real operational problem — a popular expert becomes a hot shard, which is exactly the hot-partition problem from Track C §8.4.

The decision rule to state: TP within a node for latency, PP across nodes for capacity, EP if the model is MoE. And the corollary — more parallelism is not free; every axis adds communication, and past a point you are paying for coordination rather than buying speed.

5.5 Disaggregated prefill and decode

The logical endpoint of "prefill and decode are different workloads": run them on different machines.

• A prefill pool — compute-optimized, sized by prompt-token rate.
• A decode pool — bandwidth-optimized (H200s, more memory), sized by concurrent sequences.
• The KV cache produced by prefill is transferred to a decode worker.

Benefits: each scales independently; neither interferes with the other; you can buy different hardware for each.

Costs: the KV transfer — potentially gigabytes over the network per request — plus a more complex control plane. This wins when the transfer cost is small relative to the interference you avoid, which depends on your prompt/output length distribution.

DistServe (OSDI 2024) is the reference. Mentioning it as "the direction this goes when prefill/decode interference is your binding constraint" is a good level of engagement.

Chapter 6: Autoscaling Non-Stationary Traffic

Named explicitly by the interviewer in the source report. This is where your search background both transfers and misleads.

6.1 Why request-count autoscaling fails

In a search tier, requests are roughly interchangeable — a query costs 8 ms ± a factor of a few. So QPS is an excellent proxy for load, and QPS-based HPA works.

In LLM serving:

A 10,000× spread. Request count is not merely a poor proxy — it is nearly uncorrelated with load. QPS-based autoscaling will scale up on a burst of trivial requests and fail to scale on a handful of expensive ones. It is measuring the wrong quantity.

GPU utilization is also misleading, and this catches people who know to avoid QPS. Decode is bandwidth-bound, so the SM utilization counter can read high while the GPU is stalled on memory and doing very little useful work. "GPU at 90%" does not mean "90% of achievable throughput".

6.2 The signals that work

KV cache occupancy is the one to lead with. It is the binding constraint (Chapter 3), and it rises before queueing begins — so it gives you the warning you need given that GPU scale-up takes minutes.

A practical composite: scale on max(kv_occupancy / 0.85, queue_wait_p95 / target) — take whichever is closer to its limit, so you respond to whichever constraint binds first.

6.3 Predictive versus reactive

GPU scale-up is minutes, not seconds: instance acquisition (30 s to several minutes, sometimes unavailable), container pull, model weight loading (140 GB from network storage is minutes unless cached locally), CUDA graph capture and warm-up.

Therefore purely reactive autoscaling is always late. By the time a signal crosses a threshold, you are five minutes from relief, and five minutes of overload is an outage.

What actually works:

1. Forecast from history. Traffic is strongly diurnal and weekly — it is non-stationary but not unpredictable. Scale ahead of the predicted curve.
2. Warm pools. Keep instances loaded and idle. You pay for idle GPU to buy latency, and the pool is sized by forecast error, not by average load. Say that; it is the non-obvious part.
3. Fast reactive as a safety net for the unforecastable — a product launch, a news event.
4. Admission control while scaling. Since you cannot scale instantly, you must be able to shed. Autoscaling and load shedding are the same control problem at two time scales.

The academic line here — SageServe, ENOVA — is exactly forecast-aware autoscaling for LLM serving, and naming it shows you have read past the blog posts.

6.4 Admission control and fairness

When you cannot scale further, choose what not to serve.

Admission control: reject at the edge, cheaply, before the request consumes a KV slot. Rejecting in 1 ms is vastly better than accepting and timing out at 60 s, because the timeout consumed capacity that could have served someone.

Queueing discipline matters more here than in most systems, because of the cost variance. A FIFO queue lets one 100k-token request block many small ones. Options:

• Separate queues by cost class, with a share of capacity each. Short requests are not stuck behind long ones.
• Shortest-job-first, if you can estimate cost — prompt length is known exactly, output length is not, but it can be predicted from history per endpoint.
• Per-tenant fair queueing on tokens, not requests. This is the crucial detail: a tenant sending ten 100k-token requests is using 1,000× the capacity of a tenant sending ten small ones. Request-based fairness is not fairness at all.

Preemption is the LLM-specific twist. Since a sequence's KV cache can be swapped or recomputed, you can evict a running low-priority sequence to admit a high-priority one. Recompute is usually cheaper than swapping over PCIe — a useful, specific detail.

Chapter 7: Worked Answer — Design ChatGPT

7.1 The opening ninety seconds

Clarify first. These questions change the design, and asking them is scored:

• "Interactive chat only, or also a batch/async API? They want different schedulers."
• "Roughly what scale — millions of DAU? And what's the p95 TTFT target? I'll assume 10M DAU and a 500 ms TTFT target."
• "Do we own the model and the serving stack, or is inference a service we call?"
• "Multi-tenant with per-tenant SLOs, or one tier?"
• "Is conversation history stored server-side, or does the client resend it?"

Then the scoping sentence, which is the most important thing you say in this round:

"I'll treat the inference engine as a service with three properties: it exposes capacity in tokens per second rather than requests per second, it has an admission interface I can apply backpressure to, and it streams. I'll spend my time on traffic, coordination and failure — tell me if you want me to open it up."

That sentence does three jobs at once: it proves you know the engine is special, it hands the interviewer the steering wheel, and it buys you time for the parts they asked about. It is inference I2 in ../../research/findings.md: the "abstract the serving layer" advice is a scoping test, not a hint about depth.

And be ready to open it in seconds when asked. Hesitating there undoes the credibility the abstraction bought.

7.2 Altitude 1: the abstracted answer

                          ┌──────────────┐
   client ───SSE stream───│   Edge / LB  │  TLS, DDoS, geo-routing
                          └──────┬───────┘
                                 ▼
                   ┌──────────────────────────┐
                   │       API gateway        │  authn, quota, per-tenant
                   │                          │  rate limit (TOKENS, not
                   └────┬──────────────┬──────┘  requests), request class
                        │              │
             ┌──────────▼───┐   ┌──────▼────────┐
             │ Conversation │   │  Safety /     │  input moderation
             │    store     │   │  moderation   │  (parallel where possible)
             └──────────┬───┘   └──────┬────────┘
                        │              │
                   ┌────▼──────────────▼─────┐
                   │   Context assembler     │  system prompt + history
                   │                         │  + retrieved docs + tools
                   └───────────┬─────────────┘
                               │  token budget enforced HERE
                   ┌───────────▼─────────────┐
                   │   Inference router      │  PREFIX-AWARE / session-affine
                   │   • admission control   │  ← the load-balancing decision
                   │   • priority classes    │
                   └───────────┬─────────────┘
                               ▼
        ┌──────────────────────────────────────────────┐
        │       Inference service  (abstracted)        │
        │  capacity in tokens/s · admission interface  │
        │  · streaming · KV occupancy exposed          │
        └───────────────────┬──────────────────────────┘
                            │ token stream
                   ┌────────▼─────────┐
                   │ Output moderation│  streaming, on a buffer
                   └────────┬─────────┘
                            ▼
                    back to the client

The five things to say at this altitude:

1. Streaming transport. SSE, not WebSocket. The token stream is one-directional after the request; SSE is plain HTTP, works through every proxy, and reconnects natively. WebSocket buys bidirectionality you do not need and costs you infrastructure compatibility. Abort handling matters: when the client disconnects, the generation must be cancelled promptly, or you keep paying for tokens nobody will read. That is real money at this scale.

2. Conversation storage and the token budget. Append-only per conversation, sharded by conversation ID, hot/cold tiered. The context window is a hard budget — the assembler must decide what to include (recent turns, a summary of older ones, retrieved documents) and enforce the limit. That is a product decision with a direct cost consequence: every token in the prompt costs prefill compute and KV memory.

3. The router is prefix-aware, and that is the key design choice. Sending turn n+1 of a conversation to a replica that does not have its prefix cached throws away the cache and re-prefills the entire history. So routing is session-affine with a fallback: prefer the replica holding the prefix; fall back to least-loaded if it is saturated or down. This is consistent hashing on conversation ID with load-aware overflow — the same structure as cache affinity in a search tier.

4. Admission control and priority classes. Interactive, batch and free tiers get different queues and different shares. Rate limits are on tokens per minute, not requests — because requests vary 10,000× in cost. When capacity is exhausted, reject at the edge with a Retry-After rather than queueing indefinitely.

5. Safety in the path. Input moderation can run in parallel with context assembly to hide its latency. Output moderation is the hard one — you are streaming, so you either buffer (adding latency, hurting the streaming experience) or scan incrementally and accept that you may retract text already shown. That is a genuine product tradeoff and worth naming as one.

7.3 Altitude 2: "open up the serving layer"

When asked, go straight to the constraint. Do not build up.

"The binding constraint is memory bandwidth, and the binding capacity is the KV cache."

Then, in order:

Memory budget. 70B at FP16 is 140 GB of weights, so 4 × H100 with tensor parallelism. KV is 2 × 80 layers × 8 KV heads × 128 dim × 2 bytes = 320 KB/token; at 4k context that is 1.25 GB per sequence; 320 GB total minus 140 weights minus ~16 activations leaves 164 GB, so ~130 concurrent sequences. The KV cache is larger than the weights — it is the capacity constraint, and doubling context halves concurrency.

Why decode is bandwidth-bound. One decode step must read all 140 GB of weights: 42 ms at 3.35 TB/s. The compute is 140 GFLOP: 0.14 ms. ~295× apart. The H100/H200 comparison is the proof — identical compute, 43% more bandwidth, materially faster decode.

Therefore batching. Arithmetic intensity of decode ≈ batch size, and machine balance is ~295 FLOP/byte dense, so at batch 130 we are still below balance and batching is nearly free. Continuous batching at iteration granularity, so no slot idles and no request waits behind a long generation. PagedAttention so the KV cache is allocated in blocks on demand rather than reserved for the worst case, which is where the batch size comes from.

Then the prefill/decode conflict. Prefill is compute-bound — a 2,000-token prompt is ~141 ms of solid compute during which every decode stalls. Chunked prefill interleaves prefill chunks with decodes, trading a little prefill throughput for much better TTFT and TPOT tails.

Then prefix caching, which is worth the most in a chat product specifically: turn n's prompt contains turn n−1's entirely, so without it a 20-turn conversation does O(n²) prefill. This is what makes the router's prefix-affinity so valuable.

Then the framing that ties it together:

"None of these are free wins stacked on top of each other. They're different points on a throughput-versus-tail-latency curve. And it's not one curve — an interactive turn, a long agentic tool loop, and a batch job want genuinely different scheduler policies. So the real question is whether we run separate pools per traffic class, or one priority-aware scheduler with preemption. Separate pools give hard isolation and waste capacity when the mix shifts; one scheduler is efficient and needs preemption to be correct, which for LLMs means either swapping KV over PCIe or recomputing it — and recompute is usually cheaper. I'd start with two pools, interactive and batch, and add preemption within interactive when the mix data justifies it."

That paragraph is the strong hire (staff) answer, because it does not recite techniques — it frames them as a policy decision and then makes one with a stated reason.

7.4 The follow-ups, with answers

Q: How do you autoscale this? Not on request count — requests vary 10,000× in cost, so QPS is nearly uncorrelated with load. Not on GPU utilization either, because decode is bandwidth-bound and the counter reads high while stalled on memory. I'd scale on KV cache occupancy as the leading indicator — it's the binding constraint and it rises before queueing starts — plus queue wait p95 and tokens/sec tracked separately for prefill and decode. And because scale-up is minutes (instance acquisition plus loading 140 GB of weights), reactive alone is always late: forecast from the diurnal curve, pre-warm ahead of it, size the warm pool from forecast error rather than average load, and keep admission control as the fast path for whatever the forecast missed.

Q: A user sends a 100k-token prompt. What happens to everyone else? On a naive scheduler, a stutter — that prefill is seconds of solid compute and every decode in the batch stalls behind it. Chunked prefill fixes the blocking. But it also consumes ~31 GB of KV cache, which is a fifth of my budget for one request, so it needs its own admission decision: either a separate long-context pool, or a per-request KV quota with cost-class queueing so it can't starve short requests. And I'd price it accordingly, because the cost genuinely is thousands of times higher.

Q: How do you do rolling model updates? Two versions live simultaneously; conversations pinned to one for their duration, because switching mid-conversation changes behaviour visibly. Shadow traffic to the new version first for quality metrics, then a canary by percentage, then ramp with automatic rollback on eval or latency regression. The operational constraint people miss: draining a replica takes as long as its longest in-flight generation — potentially minutes — so you cannot SIGTERM. You stop admitting, let decodes finish, then terminate.

Q: What happens when a GPU dies mid-generation? Every in-flight sequence on it loses its KV cache. You cannot recover it from another replica — it's ephemeral state. So those requests fail and must be retried, which means re-prefilling from the conversation history. That's why the conversation store is the durable record and the KV cache is explicitly a cache. The mitigation is fast detection and a client-side retry that lands on a healthy replica; the cost is a re-prefill, which prefix caching partly absorbs if another replica happens to hold the prefix.

Q: One tenant sends 10× everyone else. How do you keep it fair? Fair queueing on tokens, not requests — a tenant sending ten 100k-token requests is using 1,000× the capacity of one sending ten small ones, so request-based fairness isn't fairness. Per-tenant token-rate limits at the gateway, weighted fair queueing on admission, and per-tenant KV quotas so one tenant can't occupy the whole cache. If it's sustained rather than bursty, that's a capacity and pricing conversation, not a technical one.

Q: What's your cost per million tokens, and where does it go? Order of magnitude: 4 × H100 at ~$2.50/hr is $10/hr; at ~1,400 tokens/s aggregate and, say, 40% realized efficiency after scheduler overhead and ragged batches, that's ~560 tok/s, so ~$5 per million output tokens. I'd hold that loosely — it's a modelled number, not a measured one, and the 40% is a placeholder. The dominant lever is batch size, which is bounded by KV cache, which is why GQA and KV quantization matter more to unit economics than anything on the compute side.

Q: Would you use vLLM or build your own? vLLM, TensorRT-LLM or SGLang — building a serving engine is a multi-year effort and these implement continuous batching, paged attention, and chunked prefill well. What I'd build is the layer above: the router with prefix affinity, admission control, the priority scheduler across pools, and the autoscaling controller. That's where the product-specific policy lives, and it's the part no framework can make for you.

Q: How does this change for agentic workloads? Substantially, and it's the most interesting version of this question. One user action becomes tens of model calls, each with a growing context as tool results accumulate. So: traffic gets burstier and more correlated — a single user action produces a correlated burst, which breaks autoscaling signals tuned for smoothed request-response traffic. Prefix caching gets more valuable because each step shares the prefix with the last. Latency budgets compound — 50 sequential calls at 500 ms TTFT each is 25 seconds of user-visible latency, so TTFT matters far more than in a chat turn. And cancellation matters much more, because an abandoned agent loop can burn compute indefinitely if nobody stops it.

The Numbers Sheet

Memorize. Verify each yourself before quoting it, and attach a date to anything about price.

Hardware

Cloud pricing, 2026-reported, order of magnitude only: H100 ~$1.50–3.00/hr · H200 ~$3.80/hr · B200 ~$6.50/hr.

Formulas

Anchors for Llama-70B FP16

The Thirty-Five Questions

Fundamentals

1. What is the KV cache and why does it exist?
2. Why is generation sequential but prefill parallel?
3. Write the KV-bytes-per-token formula.
4. What does GQA change, and by how much for Llama-70B?
5. What is TTFT? TPOT? Which does prefill determine?

The roofline 6. Define arithmetic intensity. 7. What is an H100's machine balance, and what does that number mean? 8. Derive the arithmetic intensity of decode. 9. Derive it for prefill. 10. Give the H100/H200 argument in two sentences. 11. Why does batching raise intensity almost for free? 12. What is the decode-step floor for 70B FP16 on an H100?

Memory 13. Does 70B FP16 fit on one H100? How many do you need? 14. How much KV does a 128k-token conversation need? 15. At batch 130, is KV or weights the larger consumer? 16. What happens to batch size when you double context?

Batching 17. Why does static batching waste 30–50%? 18. What does continuous batching change, and at what granularity? 19. What problem does PagedAttention solve, and what is the analogy? 20. What does chunked prefill trade away? 21. Why is the scheduler "the product"?

Toolkit 22. Why is prefix caching worth more in chat than in single-turn? 23. What does prefix caching do to your load-balancing design? 24. Why does speculative decoding stop paying at high batch? 25. Why does quantization help decode more than the FLOP count suggests? 26. TP vs PP — which crosses nodes, and which improves latency? 27. What is the MoE all-to-all problem, and what does it resemble? 28. When does disaggregated prefill/decode win?

Serving 29. Why does QPS autoscaling fail here? Give the cost ratio. 30. Why is GPU utilization a misleading signal? 31. What is the best leading indicator, and why? 32. Why is reactive autoscaling always late? 33. How do you size a warm pool? 34. Why is fair queueing on tokens rather than requests? 35. What happens to a GPU's in-flight requests when it dies?

References

Papers — read at least the first four

• Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. https://arxiv.org/abs/2309.06180
• Yu et al. Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022 — continuous batching
• Agrawal et al. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. OSDI 2024. https://arxiv.org/abs/2403.02310 — chunked prefill
• Pope et al. Efficiently Scaling Transformer Inference. MLSys 2023. https://arxiv.org/abs/2211.05102 — the roofline analysis, done properly
• Leviathan et al. Fast Inference from Transformers via Speculative Decoding. ICML 2023. https://arxiv.org/abs/2211.17192
• Ainslie et al. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. https://arxiv.org/abs/2305.13245
• Zhong et al. DistServe: Disaggregating Prefill and Decoding. OSDI 2024. https://arxiv.org/abs/2401.09670
• Zheng et al. SGLang / RadixAttention. https://arxiv.org/abs/2312.07104
• Dettmers et al. LLM.int8(). NeurIPS 2022 · Frantar et al. GPTQ. ICLR 2023 · Lin et al. AWQ. MLSys 2024
• Williams, Waterman, Patterson. Roofline: An Insightful Visual Performance Model. CACM 2009 — the original model
• SageServe: Forecast Aware Auto-Scaling for LLM Serving. https://arxiv.org/pdf/2502.14617
• ENOVA: Autoscaling towards Cost-effective and Stable Serverless LLM Serving. https://arxiv.org/abs/2407.09486

Implementations to read

• vLLM — https://github.com/vllm-project/vllm · Inside vLLM: Anatomy of a High-Throughput LLM Inference System https://vllm.ai/blog/2025-09-05-anatomy-of-vllm
• NVIDIA TensorRT-LLM — https://github.com/NVIDIA/TensorRT-LLM
• SGLang — https://github.com/sgl-project/sglang
• NVIDIA Triton Inference Server — https://github.com/triton-inference-server/server

Operational grounding

• OpenAI. Scaling Kubernetes to 7,500 nodes. https://openai.com/index/scaling-kubernetes-to-7500-nodes/
• NVIDIA. H100 Tensor Core GPU. https://www.nvidia.com/en-us/data-center/h100/

In this repo

• gpu_math.py — every calculation in Chapter 3, runnable for any model/GPU
• README.md — Track D drills, failure modes, rubric
• ../systems-design/WARMUP.md — the distributed primitives this layer sits on
• ../../research/findings.md — what is confirmed vs reported about this round
• Related tracks: llm-inference-engineer · Senior AI Engineer · pretraining-lead