Phase 09 — Inference Serving Internals (vLLM/TensorRT-class)

The phase where the cost math from Phase 00 becomes a running system. Phase 00 told you decode is memory-bandwidth-bound and the KV-cache is the wall. This phase builds the three mechanisms a real serving engine uses to fight exactly those facts: PagedAttention (so the KV-cache doesn't fragment), continuous batching (so the batch stays full and amortizes the bandwidth-bound weight read), and speculative decoding (so you do fewer sequential decode steps). Get these and you can read the vLLM scheduler, size a fleet, and answer the serving-design round of any senior interview.

Why this phase exists

A model that passes evals is worth nothing until it serves traffic at a price and a latency the business can live with. The gap between "I called model.generate()" and "I run a 100-QPS endpoint at p99 < 200 ms TTFT for $X/1M tokens" is the serving stack, and the serving stack is three ideas:

1. The KV-cache is paged. Naive serving reserves a contiguous max_len slab of KV per request and wastes 60–80% of it to fragmentation and over-allocation. PagedAttention stores KV in fixed blocks addressed through a per-sequence block table — exactly like OS virtual memory — so memory is near-fully utilized and sequences can share blocks (prefix caching).
2. Batching is continuous. Static batching runs a fixed batch to completion, so one long request holds the whole batch hostage (head-of-line blocking). Continuous (in-flight) batching admits and retires requests every decode step, keeping the batch full — the 2–4× throughput win.
3. Decoding can be speculative. Because decode is bandwidth-bound, you have spare compute. A cheap draft proposes several tokens; the target verifies them in one parallel pass and accepts with min(1, p_target/p_draft), resampling the residual on rejection — provably preserving the target distribution while doing fewer sequential target steps.

Everything else in serving — chunked prefill, KV quantization, the SLOs — hangs off these three.

Concept map

                  ┌──────────────────────────────────────────────┐
                  │  Phase 00 facts: decode is BW-bound;          │
                  │  the KV-cache is the memory wall.             │
                  └──────────────────────────────────────────────┘
                       │                  │                  │
        ┌──────────────┘        ┌─────────┘        ┌─────────┘
        ▼                       ▼                  ▼
   FRAGMENTATION           HEAD-OF-LINE       SEQUENTIAL DECODE
   (naive max_len KV)      (static batch)     (one token / step)
        │                       │                  │
        ▼                       ▼                  ▼
   PAGEDATTENTION          CONTINUOUS         SPECULATIVE DECODING
   blocks + block table    BATCHING           draft → verify → accept
   ~0 fragmentation        admit/retire/step  min(1, p/q) + residual
   prefix sharing / CoW    2–4× throughput    distribution-preserving
        │                       │                  │
        └───────────┬───────────┴──────────┬───────┘
                    ▼                       ▼
              SLOs: TTFT (prefill)   throughput vs latency
                    TPOT/ITL (decode)  ($/1M, p99, QPS)
                    │
                    ▼
        the stacks: vLLM · TensorRT-LLM · SGLang · DeepSpeed-Inference · ORT

The lab

The lab is a runnable, test-verified miniature — see the lab standard. Run it red (pytest test_lab.py), make it green, then read solution.py.

Integrated scenario ideas

• Size a vLLM deployment: given a 13B model, 80 GB GPU, 8k context — compute the number of KV blocks (gpu_memory_utilization × free / block_bytes), the max concurrent sequences, and the QPS. Show the KV blocks, not the weights, set the ceiling (ties to Phase 00).
• Defend continuous batching: take a skewed workload (one 2k-token answer + many 50-token ones) and show static batching's wasted slot-steps vs continuous — the throughput slide for a design review.
• Decide whether to add speculative decoding: estimate the speedup from an acceptance rate α and a draft cost, find the breakeven, and explain why it helps decode (bandwidth-bound, spare compute) but not prefill.
• Pick a stack: vLLM vs TensorRT-LLM vs SGLang for a given workload (open weights vs NVIDIA-only, throughput vs lowest latency, complex prefix-sharing vs simple) — and name the deciding constraint.

Deliverables checklist

• lab.py passes pytest test_lab.py -v (and LAB_MODULE=solution does too).
• You can explain PagedAttention from first principles: blocks, block table, why fragmentation goes to near zero, and how prefix sharing / copy-on-write works.
• You can state the append-allocates-a-new-block-only-when-full off-by-one and why it matters.
• You can explain head-of-line blocking and why continuous batching is 2–4× throughput.
• You can prove speculative decoding preserves the target distribution (min(1, p/q) + residual).
• You can define TTFT and TPOT/ITL and say which mechanism moves which.

Key takeaways

• PagedAttention is OS paging for the KV-cache. Fixed blocks + a per-sequence block table turn a fragmented, over-allocated slab into near-fully-utilized memory, and make prefix sharing free. This is the single change that let one GPU serve far more concurrent sequences.
• Continuous batching removes head-of-line blocking. Schedule at the iteration level — admit and retire requests every decode step — and the batch stays full, which (because decode is bandwidth-bound) is where the 2–4× throughput comes from.
• Speculative decoding buys latency with spare compute, for free quality. The acceptance rule is exactly engineered so the output distribution equals the target's — you trade idle FLOPs for fewer sequential steps without changing what the model would have said.
• Serving is a throughput-vs-latency contract. TTFT is prefill (compute-bound), TPOT/ITL is decode (bandwidth-bound); batching raises throughput but can hurt TTFT — the SLOs decide the knobs, and the stack (vLLM/TensorRT-LLM/SGLang) is chosen to fit them.

Next: Phase 10 — Distributed Training & Model Parallelism.