🛸 Hitchhiker's Guide — Phase 07: LLM Serving & KV-Cache Management
Read this if: you can run
vllm servebut couldn't yet derive its memory limits, explain PagedAttention as an allocator, or defend "goodput, not throughput" to a skeptical exec. Field notes; derivations in the WARMUP and the LLM Inference track Phase 09.
0. The 30-second mental model
Decode is bandwidth-bound ⇒ batching is the throughput lever. The KV cache grows per token and caps batch size ⇒ manage it like memory: page it (PagedAttention), share it (prefix caching), shrink it (GQA, FP8 KV), evict it under pressure. Schedule at iteration granularity (continuous batching). Optimize goodput (throughput meeting SLO), never raw throughput.
1. The one formula
KV bytes = 2 · layers · kv_heads · d_head · seq_len · bytes_per_elem
concurrent_seqs ≈ (HBM − weights) / (KV_bytes_per_token · context)
Llama-3-8B, 8K, BF16 ≈ 1 GB/seq ⇒ ~20 seqs on a 40 GB GPU. The cache, not compute, caps concurrency. Every optimization improves one term.
2. The KV-cache management ladder
| Technique | What it does | Win |
|---|---|---|
| GQA | fewer KV heads | 4–8× smaller cache (model-level) |
| FP8/INT8 KV | fewer bytes/elem | 2–4× smaller |
| PagedAttention | fixed blocks + block table | kills fragmentation → 2–4× more seqs |
| Prefix caching (COW) | share system-prompt KV | >90% prefill cut for chat |
| Eviction (recompute/swap) | free blocks under pressure | graceful degradation |
PagedAttention = OS virtual memory for the KV cache. Block size ≈16 tokens (table overhead vs internal fragmentation tradeoff).
3. The batching/scheduling ladder
- Static = collect, run to completion. Head-of-line blocking + formation delay. Don't.
- Continuous (Orca) = iteration-level admit/retire. 3–10× throughput. The standard.
- Chunked prefill = slice long prompts so they don't stall decodes.
- Disaggregation = separate prefill and decode GPU pools, ship KV between. Frontier-scale.
- The dial: deeper batch = more throughput, worse per-user TPOT. Tune for goodput, which peaks at an interior batch depth (Lab 02 proves it).
4. Metrics that resist gaming
| Metric | Means | Gotcha |
|---|---|---|
| TTFT | queue + prefill | prompt length, batch interference |
| TPOT/ITL | decode speed | UX floor ~10–15 tok/s |
| throughput | total tok/s | gameable by oversized batches |
| goodput | tok/s meeting SLO | the honest fleet metric |
Require goodput at p50/p99 SLOs, Poisson arrivals, real length distribution. Anything else is marketing (Phase 01 Ch. 10 energy).
5. Engine selection card
| Engine | Strength | Choose when |
|---|---|---|
| vLLM | throughput, model coverage, OSS velocity | default; multi-model fleets |
| TensorRT-LLM | peak latency/throughput (AOT kernels) | fixed model+GPU, last 20%, accept lock-in |
| SGLang | structured gen, RadixAttention prefix trees | agents, tool use, complex prompting |
| TGI | HF integration, simplicity | simpler deployments |
Platform answer: a serving gateway abstracting engines, route per model, select on goodput evidence. Not "one engine to rule them all."
6. Air-gapped/sovereign changes
No phone-home (offline weights, offline license — Phase 11), fixed on-prem capacity (admission/goodput matter more, no elastic burst), local-only observability (Phase 10), audit trails + data residency. Most stacks assume cloud; building serving that doesn't is the JD's sovereign-AI differentiator.
7. War stories
- "Throughput great in the benchmark, users furious" — benchmark ran giant batches back-to-back; real Poisson traffic blew p99 TTFT. Switched to goodput-at-SLO reporting; cut batch depth; users happy, "throughput" down 15%.
- "OOM crash takes down 80 in-flight requests" — no admission control, optimistic batching hit the KV wall. Added admission + preempt-with-recompute.
- "Chat TTFT randomly 2s" — no prefix caching; every request re-prefilled a 3K system prompt. Enabled it; TTFT → 80ms on cache hits.
- "New model 40% slower on TRT-LLM than vLLM" — TRT-LLM engine built for the old shape; nobody rebuilt. AOT engines need a rebuild per model/GPU (Phase 03). Routed it to vLLM until rebuilt.
8. Exit bar
You can derive KV memory and capacity cold, explain PagedAttention/prefix caching/continuous batching as the allocator-and-scheduler problems they are, defend goodput over throughput, select engines from requirements, and describe air-gapped serving. Plus you've built the paged allocator and the batching sim. Next: Phase 08 — when one GPU isn't enough, how collectives and topology turn many GPUs into one.