Phase 07 — LLM Serving & KV-Cache Management
Difficulty: ⭐⭐⭐⭐⭐ | Estimated Time: 2.5 weeks Roles supported: Head of Engineering [GPU], LLM Inference Engineer, Serving Platform Lead Hardware needed: none — both labs are CPU models of the serving runtime (Python); optional GPU for the vLLM reading exercise
Why This Phase Exists
The JD names it explicitly: "GPU scheduling, KV-cache management, or LLM serving optimisation" and "deploying and optimising AI infrastructure at production scale, ideally with multiple inference engines." This is the single most commercially valuable phase for the role — a 30% throughput gain across a serving fleet is millions of dollars a year, and the lever is almost always KV-cache management and batching.
This is the consequence phase. Phase 01 proved decode is memory-bandwidth-bound; Phase 05 built allocators and stream scheduling; Phase 06 built request-vs-job scheduling. Here it all converges into the serving runtime: you'll build a paged KV-cache allocator with prefix sharing (PagedAttention as an allocator-design problem — exactly Phase 05 Ch. 5's prediction) and a continuous-batching simulator with TTFT/TPOT/goodput, then connect both to operating vLLM/TensorRT-LLM/SGLang at scale.
This phase deliberately overlaps the LLM Inference Engineer track's Phase 09. That track owns the model/algorithm depth (FlashAttention math, quantization formats, speculative decoding derivations); this phase owns the platform/operations view — the cache as a managed resource, the scheduler as a fleet control loop, multi-engine selection, and the leadership decisions. Read both; they're complementary, and the WARMUPs cross-reference rather than duplicate.
Concepts
- The prefill/decode split and why it dictates everything (recap from Phase 01 Ch. 7, serving-side)
- KV-cache memory math:
2 × n_layers × n_kv_heads × d_head × seq_len × dtype_bytes— and why it, not weights, caps concurrency - PagedAttention: fixed-size KV blocks + block tables = external fragmentation eliminated for the dominant consumer (Phase 05 Ch. 5, realized)
- Prefix caching / sharing: copy-on-write KV blocks for shared system prompts (chat APIs) — the highest-ROI serving feature for instruction-heavy workloads
- Continuous batching (Orca): iteration-level scheduling, admission/eviction, the latency-throughput dial
- Chunked prefill and prefill/decode disaggregation: keeping long prompts from stalling decodes
- Metrics & SLOs: TTFT, TPOT/ITL, E2E, throughput, and goodput (throughput meeting SLO — the only honest fleet metric)
- Admission control, preemption, and KV-cache swapping/recompute under memory pressure
- GQA, KV quantization (INT8/FP8 KV) — shrinking the cache (cross-ref to llm-inference track)
- The engine landscape as a platform decision: vLLM vs TensorRT-LLM vs SGLang vs TGI — what each optimizes, when to choose which, the multi-engine reality
- Sovereign/air-gapped serving: on-prem deployment, no telemetry phone-home, license enforcement (Phase 11) — the JD's regulated-industry nice-to-haves
Labs
Lab 01 — Paged KV-Cache Allocator + Prefix Sharing (Python)
| Field | Value |
|---|---|
| Goal | Implement a PagedAttention-style KV-cache block allocator: fixed-size blocks, per-sequence block tables, copy-on-write prefix sharing — and measure the memory win vs naive contiguous allocation. |
| Concepts | Block tables, fixed-size paging, internal vs external fragmentation, copy-on-write prefix sharing, KV-cache memory math. |
| Steps | 1) python solution.py — runs correctness tests + a memory-efficiency comparison + a prefix-sharing demo. 2) Read the allocator against WARMUP Ch. 2–4. 3) Reproduce: naive contiguous wastes 60–80% on max-length reservation; paged wastes < one block/seq; prefix sharing collapses N copies of a shared prompt to 1. 4) Extensions: eviction/preemption, swapping to host, block ref-count GC. |
| Stack | Python (stdlib) |
| Output | A block allocator + tests + a memory-comparison table (naive vs paged vs paged+prefix). |
| How to Test | Built-in asserts: paged supports ≥2× the concurrent sequences of naive at the same memory; prefix sharing yields exact block reuse with correct copy-on-write at divergence. |
| Talking Points | Why this is "just an allocator" (Phase 05); the block-size tradeoff (internal fragmentation vs table overhead); why prefix caching is the highest-ROI feature for chat workloads; how ref-counting handles fork/COW at the divergence token. |
| Resume Bullet | "Implemented a PagedAttention-style KV-cache allocator with copy-on-write prefix sharing; demonstrated 2.6× concurrent-sequence capacity over contiguous allocation and exact block reuse for shared system prompts." |
| Extensions | LRU block eviction with recompute-vs-swap policy; host-memory swapping (the 50× bandwidth cliff from Phase 01); ref-count GC with a fork/branch (beam search) workload. |
Lab 02 — Continuous Batching Simulator + Goodput (Python)
| Field | Value |
|---|---|
| Goal | Simulate an LLM server under a Poisson request load; implement static vs continuous batching with iteration-level admission/eviction; measure TTFT, TPOT, throughput, and goodput under an SLO. |
| Concepts | Static vs continuous batching, iteration scheduling, admission control, the latency-throughput dial, goodput, prefill interference + chunked prefill. |
| Steps | 1) python solution.py — runs a workload through static and continuous batching, prints the metrics table. 2) Read the scheduler against WARMUP Ch. 5–6. 3) Reproduce continuous batching's throughput win and observe the TTFT/TPOT tradeoff as you change max_batch. 4) Extensions: chunked prefill, prefix-cache hits (Lab 01), priority/SLO-aware admission. |
| Stack | Python (stdlib) |
| Output | A serving simulator + a metrics table (static vs continuous; goodput vs batch depth). |
| How to Test | Built-in asserts: continuous batching ≥2× static throughput at concurrency 16; goodput is maximized at an interior batch depth (not the largest), proving the latency-throughput dial. |
| Talking Points | Why static batching wastes GPU on long-tail requests; why goodput (not raw throughput) is the honest metric; how prefill interference motivates chunked prefill and disaggregation; the admission-control knobs you'd expose. |
| Resume Bullet | "Built a continuous-batching LLM-serving simulator with goodput accounting under SLO; demonstrated 3.1× throughput over static batching and identified the goodput-optimal batch depth, quantifying the latency-throughput tradeoff." |
| Extensions | Chunked prefill (slice long prompts across iterations); prefix-cache integration; SLO-aware admission that sheds load to protect p99; prefill/decode disaggregation model. |
Deliverables Checklist
- Paged allocator: tests pass; memory-comparison table reproduced and explained
- You can do the KV-cache memory math cold for any model/context/batch
- Continuous-batching sim: metrics table reproduced; goodput-optimal batch depth identified
- You can explain why goodput > throughput as a fleet metric
- One page: "our serving platform's KV-cache & batching strategy" (feeds Phase 12 capstone)
-
vLLM reading: skim
vllm/core/block_manager+ scheduler; map each to your Lab 01/02 code
Interview Relevance
This phase is the direct portfolio for the JD's serving requirement.
- "Walk me through KV-cache memory and why it caps concurrency."
- "Explain PagedAttention — and why it's really an allocator." (Phase 05 link)
- "Static vs continuous batching — and what's goodput?"
- "How does prefix caching work and when is it worth it?"
- "vLLM vs TensorRT-LLM vs SGLang — when each, and how do you run multiple engines?"
- "Design air-gapped LLM serving for a regulated customer." (→ Phase 11, system-design/)