Warmup Guide — Inference Serving

Zero-to-expert primer for Phase 09 — the track's namesake phase: why LLM inference is shaped the way it is (prefill/decode, the KV cache you implement in the lab), and the serving stack built on top (batching, paging, quantization, speculation, SLOs).

Table of Contents

• Chapter 1: The Shape of the Problem
• Chapter 2: The KV Cache — Derivation and Cost
• Chapter 3: Prefill vs Decode — Two Workloads in One Request
• Chapter 4: Metrics and SLOs — TTFT, TPOT, Goodput
• Chapter 5: Batching — Where Throughput Comes From
• Chapter 6: Memory Management — PagedAttention and Prefix Caching
• Chapter 7: The Optimization Menu
• Chapter 8: Multi-GPU and the Serving Topology
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: The Shape of the Problem

Autoregressive generation (Phase 03 Ch. 1) produces one token per forward pass, and each pass at batch 1 must read every model weight. That makes decode memory-bandwidth-bound (the roofline argument, in full in the model-accuracy track's Phase 07 warmup):

$$\text{tokens/sec} \approx \frac{\text{memory bandwidth}}{\text{bytes read per token}}$$

A100 (~2 TB/s) serving a 7B FP16 model (14 GB): ceiling ≈ 140 tok/s at batch 1 — while the GPU's compute sits ~99% idle. The entire serving discipline is the consequence: every technique in this phase either reduces bytes-per-token (quantization, GQA, paging efficiency) or amortizes the bytes across more useful work (batching, speculation, prefix caching). Hold that sentence; the rest is elaboration.

Chapter 2: The KV Cache — Derivation and Cost

The lab's subject, derived from first principles:

Why it exists: attention at position $t$ needs $K_{1..t}, V_{1..t}$. Causality (Phase 04 Ch. 3) means past tokens' K/V never change — recomputing them per step makes generation $O(T^2)$ forward work for what should be $O(T)$. So: cache K and V per layer per head; each step computes only the new token's $q, k, v$, appends, and attends — a matvec against the cache instead of a recompute of history.

What it costs (the formula to know cold):

$$\text{bytes} = 2 \times n_{layers} \times n_{kv_heads} \times d_{head} \times \text{seq_len} \times \text{bytes/elem}$$

LLaMA-2-7B FP16 (32 layers, 32 KV heads, d=128): 512 KB per token — 2 GB at 4K context, per sequence. A 40 GB GPU holding 14 GB of weights fits ~13 such sequences — the KV cache, not compute, caps concurrency. This single calculation explains GQA (8× fewer KV heads → 8× more concurrent users — Phase 02 of the model-accuracy track), KV quantization, paging (Ch. 6), and most of vLLM's existence.

Correctness property (the lab's key test): cached and uncached generation must be bit-identical — the cache is pure memoization. Any divergence is an indexing/masking bug, typically at the position-embedding or mask boundary.

Chapter 3: Prefill vs Decode — Two Workloads in One Request

One request, two regimes (the most consequential fact in serving):

They want opposite optimizations (more FLOPs vs fewer bytes) and they interfere: a long prefill entering a batch stalls every decoding request for those iterations. Production answers: chunked prefill (slice prompts across iterations, blending smoothly with decodes) and prefill/decode disaggregation (separate pools, KV shipped between them — the frontier-scale architecture). Diagnosing which regime a complaint lives in ("slow" = TTFT or TPOT?) is always step one.

Chapter 4: Metrics and SLOs — TTFT, TPOT, Goodput

The vocabulary of serving quality:

• TTFT (time to first token): queue wait + prefill. Dominated by prompt length and batch interference. The "feels responsive" metric.
• TPOT / ITL (time per output token): decode speed. The "streams smoothly" metric; human reading speed (~10–15 tok/s) is the UX threshold worth knowing.
• E2E latency = TTFT + TPOT × output_len; throughput = total tokens/sec across the fleet — and throughput trades against latency via batching (Ch. 5).
• Goodput: throughput that meets SLO — the honest fleet metric, because raw tokens/sec is gameable by letting tails rot. Report p50/p99 of TTFT and TPOT, under a realistic arrival process (Poisson, not back-to-back), with the workload's prompt/output length distribution stated. A benchmark without these caveats is marketing.

Chapter 5: Batching — Where Throughput Comes From

Decode reads all weights per step regardless of batch size — so batching B requests amortizes the weight read B ways: throughput scales near-linearly until you leave the bandwidth-bound regime or exhaust KV memory. This is THE serving lever.

• Static batching fails for LLMs: requests finish at different lengths (head-of-line blocking, padding waste) and arrive continuously (batch-formation delay).
• Continuous batching (Orca's insight; every modern engine): scheduling at iteration granularity — after each decode step, retire finished sequences, admit new ones. Utilization stays high under length variance; admission is immediate.
• The induced problem: admitting a request commits to its KV growth → memory pressure, preemption policies, and the fragmentation problem Ch. 6 solves.
• The latency-throughput dial: deeper batches = better throughput, worse TPOT per user (more work per iteration). Goodput (Ch. 4) is how you tune the dial honestly.

Chapter 6: Memory Management — PagedAttention and Prefix Caching

(Shared ground with the model-accuracy track's Phase 08 warmup, Ch. 7–8 — recap + serving-side emphasis.)

• The fragmentation problem: contiguous per-request KV allocation at max-length wastes 60–80% of KV memory (vLLM's measurement) on reservation nobody uses.
• PagedAttention: fixed-size KV blocks (~16 tokens) + per-sequence block tables — the OS virtual-memory design transplanted. Near-zero waste → 2–4× more concurrent sequences → directly multiplies batch depth and therefore throughput (Ch. 5).
• Prefix caching, the feature serving engineers actually tune around: shared prefixes (system prompts, few-shot headers, conversation history re-sent each turn) hash to the same physical blocks — prefill for the shared part becomes a cache hit (TTFT drops dramatically for chat workloads). Design consequence for application engineers: put the static part of your prompt first; a timestamp at position 0 destroys prefix reuse for the whole fleet.
• Preemption under pressure: evict by swap-to-CPU or drop-and-recompute (recompute usually wins — it's a prefill, and prefill is fast); either way it lands in the p99.

Chapter 7: The Optimization Menu

The serving engineer's toolbox, each tagged with what it spends and saves (deep dives live in the model-accuracy track — Phases 03, 08, 10; here is the serving view):

The discipline: identify the binding constraint first (bandwidth? KV memory? launch overhead? interference?), then pick from the menu — the techniques are not additive freebies, and several (speculation + deep batching) actively conflict.

Chapter 8: Multi-GPU and the Serving Topology

When one GPU isn't enough — for memory or for latency:

• Tensor parallelism (TP): shard every matmul across GPUs (column/row splits with all-reduces per layer). Cuts both memory and latency per token; needs NVLink-class interconnect; the standard within-node answer (TP=2/4/8).
• Pipeline parallelism (PP): shard by layers; good for fitting giant models across nodes; adds bubble latency — for serving, used when TP runs out, with micro-batching to fill bubbles.
• Replication + routing: beyond one model instance, the problem becomes load balancing (session affinity for prefix-cache hits!), autoscaling on goodput, and the classic fleet questions — at which point you're running a distributed system and the PMC track's operational instincts apply.
• Rule of thumb for the lab's scale: a 7B fits one modern GPU (quantized: comfortably); TP enters at 70B-class or when TPOT SLOs demand splitting the bandwidth bill.

Lab Walkthrough Guidance

Lab 01 — KV Cache (implement it inside your Phase 04 mini-transformer):

1. Write the equivalence test first: full-recompute generation vs cached generation must match token-for-token (greedy) over many steps. This test is the lab.
2. Implement per-layer K/V append; mind the two classic bugs: position embeddings (the new token's position is cache_len, not 0) and mask shape (new query attends to all cached keys — no causal mask needed within a single new token's row beyond length).
3. Measure: tokens/sec with vs without cache as context grows (the O(T²)→O(T) curve, personally produced); peak memory vs context length (verify Chapter 2's formula against torch.cuda.max_memory_allocated to within bookkeeping).
4. Extensions in value order: batch the cache (per-sequence lengths — ragged batching is where continuous batching's bookkeeping becomes real); sliding-window eviction; INT8-quantize the cache and measure both memory and output drift.

Success Criteria

You are ready for Phase 10 when you can, from memory:

1. Derive the bandwidth ceiling formula and compute batch-1 tok/s for any (model, GPU) pair.
2. Write the KV-size formula and the 7B/4K worked example; state the equivalence property and the two classic implementation bugs.
3. Explain prefill vs decode bounds, their interference, and chunked prefill / disaggregation.
4. Define TTFT/TPOT/goodput and critique a "tokens/sec" benchmark's missing caveats.
5. Explain why batching is near-free for decode, what continuous batching changes, and what PagedAttention + prefix caching each fix.
6. Pick from the optimization menu for: batch-1 chatbot on one GPU; high-QPS summarization fleet; 32K-context analysis service — different answers, with reasons.

Interview Q&A

Q: Users say the bot "thinks forever then answers fast." What do you tune? That's high TTFT, fine TPOT: queueing + prefill. Check queue wait (admission/batching policy), prompt length (32K system prompts happen), prefill interference from other requests (→ chunked prefill), and prefix caching (a static system prompt should be a cache hit — if the app puts dynamic content first, fix the app). The instinct being tested: decompose "slow" into TTFT vs TPOT before touching anything.

Q: Doubling batch size doubled throughput at first, then stopped. Why? Three ceilings in order: (1) you left the bandwidth-bound regime — at large batch, decode matmuls become compute-bound and weight-read amortization stops paying; (2) KV memory — batch depth caps at cache capacity (paging mitigates waste but not physics); (3) per-iteration overhead (sampling, scheduling) growing with batch. Also check TPOT: throughput may have "scaled" while per-user latency blew the SLO — goodput is the honest scoreboard.

Q: When is speculative decoding the wrong answer? Large-batch high-QPS fleets (the regime is already compute-bound — verification no longer rides free bandwidth; deep batching is the better spend), tight memory (draft model + its KV eat capacity that batching wants), and high-temperature/creative workloads (acceptance rate collapses as draft and target diverge). It shines at low-batch, latency-sensitive, greedy-ish decoding — say where it wins and where it loses (full math in the model-accuracy Phase 08 warmup).

References

• Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention (vLLM) (SOSP 2023) — arXiv:2309.06180
• Yu et al., Orca: Iteration-Level Scheduling (OSDI 2022) — continuous batching's origin
• Pope et al., Efficiently Scaling Transformer Inference (2022) — arXiv:2211.05102 — the TP/latency analysis
• Agrawal et al., Sarathi: Chunked Prefill (2023) — arXiv:2308.16369
• Zhong et al., DistServe: Prefill/Decode Disaggregation (OSDI 2024) — arXiv:2401.09670
• Chen, Transformer Inference Arithmetic — kipp.ly/transformer-inference-arithmetic — the worked numbers
• vLLM docs — scheduler, prefix caching, and metrics pages
• Model-accuracy track companions: Phase 07 WARMUP (roofline) and Phase 08 WARMUP (FlashAttention/speculation/paging internals)