Warmup Guide — LLM Serving & KV-Cache Management

Zero-to-expert primer for Phase 07, from the platform/operations angle. Builds on Phase 01 (roofline), Phase 05 (allocators), Phase 06 (scheduling). For the model/algorithm depth (FlashAttention math, quantization formats, speculative decoding) see the LLM Inference track Phase 09; this guide cross-references rather than duplicates.

Table of Contents

Chapter 1: The Serving Problem in One Diagram

One LLM request is two workloads (Phase 01 Ch. 7's roofline, applied):

PREFILL  (process the whole prompt)      DECODE (generate token by token)
  all prompt tokens at once                one token per forward pass
  big matmuls -> COMPUTE-bound             reads all weights+KV -> BANDWIDTH-bound
  cost ~ prompt length                     cost ~ output length
  user feels it as: TTFT                   user feels it as: TPOT (smoothness)

Everything in serving follows from three facts: (1) decode is bandwidth-bound, so batching (reusing each weight read across many requests) is the master throughput lever; (2) the KV cache grows with every token and caps how many requests you can batch; (3) prefill and decode interfere (a long prefill stalls everyone's decode). The platform's job is to maximize goodput — useful throughput meeting latency SLOs — by managing the cache as a resource and scheduling at iteration granularity. Hold that; the rest elaborates.

Chapter 2: KV-Cache Memory — the Math That Caps Everything

The formula to know cold (and derive in interviews):

$$\text{KV bytes} = 2 \times n_{layers} \times n_{kv_heads} \times d_{head} \times \text{seq_len} \times \text{bytes/elem}$$

The 2 is K and V. Worked: Llama-3-8B (32 layers, 8 KV heads — GQA already applied — , d_head 128, BF16): per token = 2·32·8·128·2 = 131 KB/token; at 8K context that's ~1 GB per sequence, on top of the 16 GB of weights. A 40 GB GPU holds ~22 such sequences max — the KV cache, not compute, caps concurrency.

This single calculation explains the whole phase:

  • GQA (8 KV heads instead of 32) already cut this 4× — why every modern model uses it.
  • KV quantization (INT8/FP8 KV) halves/quarters it again.
  • Paging (Ch. 3) stops you from wasting most of it.
  • Batching limits (Ch. 5) are set by how many sequences' caches fit.

As a platform leader you carry this formula to capacity planning: "how many concurrent 8K-context users per H100?" is (HBM − weights) / (KV bytes/token × context), and every serving optimization is an attempt to improve one term.

Chapter 3: PagedAttention — KV Cache Is an Allocator Problem

Phase 05 Ch. 5 predicted this exactly: "fixed-size paging for the dominant consumer eliminates external fragmentation by design — that sentence is PagedAttention." Here it is, realized (Lab 01 builds it).

The naive way: allocate each sequence a contiguous KV buffer sized for max possible length. Problems: (1) you don't know the output length, so you reserve for the worst case → most of it is never used (internal fragmentation, vLLM measured 60–80% waste); (2) variable-length sequences leave un-fillable holes (external fragmentation, Phase 05's day-3 OOM).

PagedAttention: chop the KV cache into fixed-size blocks (e.g., 16 tokens). Each sequence gets a block table (logical block → physical block), exactly like OS virtual memory page tables. Allocate blocks on demand as the sequence grows; the attention kernel gathers K/V through the block table (one indirection).

Wins, each a direct allocator consequence:

  • No external fragmentation — all blocks identical size, any free block fits any sequence.
  • Near-zero internal fragmentation — waste is at most one partial block per sequence (vs reserving max length).
  • 2–4× more concurrent sequences in the same HBM → 2–4× throughput, because more sequences = bigger batches = better decode amortization (Ch. 5).
  • Sharing becomes trivial — blocks are relocatable units that multiple sequences can reference (Ch. 4).

The one tradeoff: block size. Bigger blocks = smaller block tables, more internal fragmentation; smaller blocks = more table overhead, finer packing. 16 tokens is the common sweet spot. Lab 01 lets you measure the curve.

Chapter 4: Prefix Sharing — Copy-on-Write for Prompts

The highest-ROI feature for real workloads, and a clean payoff of paging:

Chat and agent APIs send the same long system prompt on every request ("You are a helpful assistant... [2000 tokens of instructions]"). Naively, each request recomputes and stores that prompt's KV — wasted compute (prefill) and wasted memory (identical KV copies).

Prefix caching: hash the prompt prefix; if its KV blocks already exist, share them — multiple sequences' block tables point at the same physical blocks. New requests skip the shared prefill entirely (TTFT collapses) and add no memory for the shared part.

The mechanism is copy-on-write (Lab 01 implements it): shared blocks are read-only and ref-counted; when a sequence diverges (its tokens differ from the shared prefix, or it writes into a shared block), that block is copied first. This is fork() semantics for KV cache — and beam search / parallel sampling (N continuations of one prompt) use the same machinery.

ROI: for a chat assistant with a 2K-token system prompt at 1000 req/s, prefix caching can cut prefill compute by >90% and is often the single biggest serving win. As a platform decision: enable it, measure hit rate, and expose prompt structure guidance to customers (stable prefix first) — a product lever, not just a flag.

Chapter 5: Continuous Batching — Iteration-Level Scheduling

Batching is the master throughput lever (decode reads all weights per step regardless of batch size — Phase 01 Ch. 7 — so batch B amortizes that read B ways). But how you batch decides everything.

  • Static batching: collect B requests, run them together until all finish. Fails for LLMs: requests finish at wildly different lengths, so the batch runs at the speed of its longest member while finished slots sit idle (head-of-line blocking), and new arrivals wait for the whole batch (batch-formation latency). GPU utilization collapses under length variance.

  • Continuous batching (Orca, 2022; every modern engine): schedule at iteration granularity. After each decode step: retire finished sequences (free their KV blocks), admit waiting requests (run their prefill, add to the batch). The batch composition changes every step; utilization stays high under length variance; new requests start almost immediately. Typical win: 3–10× throughput over static. Lab 02 measures it.

The induced problems (which the rest of serving solves):

  • Admitting a request commits to its KV growth → memory pressure → preemption/swapping (Ch. 7).
  • A long prefill entering the batch stalls every decode that iteration → chunked prefill (slice the prompt across iterations) and prefill/decode disaggregation (separate GPU pools, ship KV between them).
  • The latency-throughput dial: deeper batches = more throughput, worse TPOT per user (more work per iteration). You don't maximize batch depth — you maximize goodput (Ch. 6).

Chapter 6: Metrics and SLOs — Why Goodput Is the Only Honest Number

The serving vocabulary, and the one metric that resists gaming:

  • TTFT (time to first token) = queue wait + prefill. The "feels responsive" metric; dominated by prompt length and batch interference.
  • TPOT / ITL (time per output token) = decode speed. The "streams smoothly" metric; human reading speed ~10–15 tok/s is the UX floor.
  • E2E latency = TTFT + TPOT × output_len.
  • Throughput = total tokens/sec across the fleet.
  • Goodput = throughput that meets SLO. The honest metric, because raw throughput is gameable: you can inflate tokens/sec by running enormous batches that blow every request's TPOT past the SLO — "high throughput," zero happy users. Goodput counts only tokens delivered within their latency target.

Why this is a leadership point: vendors and internal teams report throughput because it's bigger; you require goodput at stated p50/p99 SLOs under a realistic arrival process (Poisson, not back-to-back) with the real prompt/output length distribution. A benchmark missing those caveats is marketing (same discipline as Phase 01 Ch. 10's spec-sheet skepticism). Lab 02 shows goodput peaking at an interior batch depth — proof that "maximize batch" is wrong and "maximize goodput" is right.

Chapter 7: Memory Pressure — Preemption, Swapping, Recompute

Continuous batching admits requests optimistically; sometimes their combined KV growth exceeds HBM. The platform must degrade gracefully:

  • Preemption: evict a running sequence to free its blocks. Two recovery options when it's rescheduled:
    • Recompute: throw away its KV, re-run prefill later. Cheap memory, costs compute (re-prefill). Best when prefill is short.
    • Swap: copy its KV blocks to host RAM, copy back later. Costs the 50× PCIe bandwidth cliff (Phase 01 Ch. 9), but preserves the work. Best when the KV is large and prefill was expensive.
  • Admission control: the cleaner lever — don't admit a request you can't sustain; queue it. This protects p99 for admitted requests (the goodput argument) at the cost of TTFT for queued ones. SLO-aware admission (shed or queue load to protect tails) is the production posture.

The policy choice (recompute vs swap vs reject) is workload-dependent and is a real config surface you own. vLLM exposes both; knowing why you'd pick each is the Phase-07 depth interviewers probe.

Chapter 8: The Engine Landscape — A Platform Decision

The JD wants "multiple inference engines." The map, as a selection problem:

  • vLLM: PagedAttention's home; best-in-class throughput, broad model support, fast-moving OSS, great for heterogeneous/multi-model fleets and rapid adoption of new models. The default starting point.
  • TensorRT-LLM: NVIDIA's engine; compiles model-specific optimized kernels (ahead-of-time, per-GPU — Phase 03's AOT story) for peak latency/throughput on NVIDIA hardware — at the cost of build complexity and NVIDIA lock-in. Best when you've fixed your model and hardware and need the last 20%.
  • SGLang: strong on structured generation, complex prompting, and RadixAttention (advanced prefix sharing); excellent for agent/tool workloads.
  • TGI (HF): solid, integrated with the HF ecosystem; common for simpler deployments.

The platform reality is multi-engine: different models/SLAs/hardware want different engines, and you abstract them behind a common gateway (the capstone). The selection axes: throughput vs latency, model coverage vs peak per-model, build complexity vs portability (and the hardware-lock-in question that Phase 09's HAL exists to manage). "We standardized on one engine" is rarely the right answer at scale; "we have a serving abstraction and route models to the engine that serves them best" is.

Chapter 9: Sovereign and Air-Gapped Serving

The JD's regulated-industry nice-to-haves (defence, finance, healthcare, government) reshape serving:

  • No phone-home: no telemetry, no model-download-at-runtime, no license server callout. Everything — weights, engine, license validation (Phase 11) — must work offline. This changes your packaging, your update story, and your observability (local-only, Phase 10).
  • On-prem capacity is fixed: no elastic cloud burst, so admission control and goodput discipline (Ch. 6) matter more — you can't scale out of a traffic spike.
  • License enforcement without a server: signed, offline-verifiable licenses with hardware binding and grace periods (Phase 11 Lab 02) — because the customer's airtight network can't reach your licensing API.
  • Auditability: regulated customers need request/response audit trails, often with data residency guarantees — a serving-layer feature, not an afterthought.

This is where Phases 07, 09 (portability for on-prem hardware diversity), and 11 (licensing/protection) converge into the JD's "sovereign AI" pitch — and it's a genuine market differentiator, because most serving stacks assume the cloud.

Lab Walkthrough Guidance

Order: Lab 01 → Lab 02 (the batching sim's admission logic depends on the paged allocator's concepts).

Lab 01 (paged KV allocator):

  1. python solution.py; read the allocator against Ch. 3–4 — BlockAllocator, BlockTable, the COW logic.
  2. Reproduce the memory comparison: naive contiguous vs paged vs paged+prefix. Tie each number to Ch. 2–3 (internal/external fragmentation eliminated).
  3. The prefix-sharing demo: N sequences sharing a system prompt collapse to one physical copy; at divergence, COW kicks in. Trace the ref-counts.
  4. Extensions: eviction (recompute vs swap, Ch. 7), then a beam-search fork.

Lab 02 (continuous batching + goodput):

  1. python solution.py; read the scheduler against Ch. 5–6.
  2. Reproduce static-vs-continuous throughput; then sweep max_batch and find where goodput peaks (interior, not max) — the Ch. 6 thesis, measured.
  3. Watch a long prefill stall decodes; reason about chunked prefill (extension).
  4. Extension: integrate Lab 01's prefix cache (TTFT collapse on cache hits).

Success Criteria

  • You can derive KV-cache bytes and concurrent-sequence capacity for any model/context cold (Ch. 2)
  • You can explain PagedAttention as an allocator and the block-size tradeoff (Ch. 3)
  • You can explain prefix sharing + COW and when it's the biggest win (Ch. 4)
  • You can contrast static vs continuous batching and explain the latency-throughput dial (Ch. 5)
  • You can define goodput and argue why it beats throughput as a fleet metric (Ch. 6) — and your Lab 02 result proves it
  • You can choose among vLLM/TensorRT-LLM/SGLang from requirements (Ch. 8)
  • You can describe what air-gapped serving changes (Ch. 9)

Interview Q&A

Q1: Derive the KV cache size for Llama-3-8B at 8K context and tell me what it implies for serving. A: KV bytes = 2 × layers × kv_heads × d_head × seq × bytes = 2·32·8·128·8192·2 ≈ 1.07 GB per sequence (8 KV heads because of GQA; without it, 32 heads → 4× more). On a 40 GB GPU holding 16 GB of weights, ~22 GB remains → ~20 concurrent 8K sequences max, and that's the hard cap on batch size — not compute. So serving economics are dominated by fitting more sequences: GQA (already in the 8), KV quantization (FP8 → 2×), and paging to stop wasting the budget on max-length reservations. Capacity planning is (HBM − weights) ÷ (KV/token × context).

Q2: Explain PagedAttention, and why you'd call it an allocator rather than an attention algorithm. A: The attention math is unchanged; what changes is KV memory management. Naive serving reserves a contiguous max-length KV buffer per sequence, wasting 60–80% (you don't know the output length) and fragmenting under variable lengths. PagedAttention stores KV in fixed-size blocks (≈16 tokens) with a per-sequence block table mapping logical to physical blocks — OS virtual memory for the KV cache. That eliminates external fragmentation (uniform blocks) and caps internal fragmentation at one partial block per sequence, yielding 2–4× more concurrent sequences and therefore throughput. It's an allocator because the problem it solves is allocation — fixed-size paging for the dominant consumer, exactly the generic fragmentation fix; the attention kernel just gains one indirection through the block table.

Q3: Static vs continuous batching, and what is goodput? A: Static batching forms a fixed batch and runs it to completion — for LLMs that means the batch runs at its longest member's speed while finished sequences waste their slots (head-of-line blocking), and arrivals wait for batch formation; utilization collapses under length variance. Continuous batching schedules at iteration granularity: after every decode step it retires finished sequences and admits new ones, so the batch is always full of useful work — typically 3–10× throughput. Goodput is the throughput that meets the latency SLO; it matters because raw throughput is gameable by oversized batches that blow everyone's TPOT — high tokens/sec, zero satisfied users. You tune the batch depth to maximize goodput (an interior optimum), not throughput.

Q4: A chat product has a 2000-token system prompt on every request. What do you do? A: Prefix caching. Hash the shared prefix and share its KV blocks across all requests via copy-on-write: new requests skip re-prefilling the system prompt (TTFT collapses) and add zero memory for the shared part; blocks are ref-counted and copied only when a sequence diverges. For a 2K-token prompt at high QPS this can cut prefill compute >90% — usually the single biggest win. As a platform move I'd also advise the product to keep the stable instructions first (so the cacheable prefix is maximal) and monitor cache hit rate as a first-class metric. SGLang's RadixAttention generalizes this to tree-structured shared prefixes for agent workloads.

Q5: Under memory pressure mid-generation, what are your options? A: First, admission control — ideally don't admit a request you can't sustain; queue it (protects p99 for admitted requests, the goodput argument). For already- running sequences, preempt and recover one of two ways: recompute (drop the KV, re-prefill on reschedule — cheap memory, costs compute; good when prefill is short) or swap (copy KV to host RAM and back — preserves work but pays the 50× PCIe cliff; good when KV is large and prefill was expensive). The choice is workload-dependent and a config surface I'd own. The anti-pattern is admitting unboundedly and OOM-crashing the engine, which kills every in-flight request.

Q6 (leadership): How do you choose inference engines for a multi-model platform? A: I don't standardize on one. Different models, SLAs, and hardware want different engines: vLLM as the throughput-strong, broad-coverage default and for fast adoption of new models; TensorRT-LLM where a model and GPU are fixed and we need the last 20% of latency/throughput (accepting build complexity and NVIDIA lock-in — the lock-in our Phase 09 HAL exists to bound); SGLang for agent/structured-generation workloads via RadixAttention. The architecture is a common serving gateway abstracting the engines, routing each model to the engine that serves it best, with goodput-at-SLO (not vendor throughput numbers) as the selection evidence. For sovereign customers, add the constraint that the engine must run fully air-gapped with offline licensing — which narrows the field and shapes packaging. The platform value is the abstraction + routing + honest benchmarking, not any single engine.

References

  • Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention" (vLLM, SOSP 2023) — https://arxiv.org/abs/2309.06180
  • Yu et al., "Orca: A Distributed Serving System for Transformer-Based Generative Models" (OSDI 2022) — continuous batching
  • Zheng et al., "SGLang: Efficient Execution of Structured Language Model Programs" (RadixAttention) — https://arxiv.org/abs/2312.07104
  • Agrawal et al., "Sarathi-Serve: Taming Throughput-Latency Tradeoff" (chunked prefill) — https://arxiv.org/abs/2403.02310
  • vLLM docs + source (block_manager, scheduler) — https://docs.vllm.ai , https://github.com/vllm-project/vllm
  • NVIDIA TensorRT-LLM — https://github.com/NVIDIA/TensorRT-LLM
  • DistServe (prefill/decode disaggregation) — https://arxiv.org/abs/2401.09670
  • Cross-track: LLM Inference Engineer Phase 09 WARMUP (model/algorithm depth), Phase 05 WARMUP Ch. 5 (paging as allocator design)