Hitchhiker's Guide — Inference Serving Internals

The compressed practitioner tour. If WARMUP.md is the professor, this is the senior who leans over and says "here's what you actually need to remember about vLLM."

The 30-second mental model

A serving engine is three tricks on top of a transformer. PagedAttention: store the KV-cache in fixed blocks like OS pages, addressed by a per-sequence block table — kills the 60–80% fragmentation of naive max_len allocation. Continuous batching: schedule per decode step (admit + advance + retire every iteration) so the batch stays full — no head-of-line blocking, 2–4× throughput. Speculative decoding: a cheap draft proposes tokens, the target verifies them in one parallel pass, accept with min(1, p/q) and resample the residual on reject — provably the target's distribution, just fewer sequential steps. Prefill is compute-bound (TTFT); decode is bandwidth-bound (TPOT). That's the whole phase.

The numbers to tattoo on your arm

Back-of-envelope one-liners

# "How many concurrent sequences fit?"  (KV blocks, not weights, set the ceiling)
gpu_blocks = (free_HBM * gpu_memory_utilization) / (block_size * 2 * n_layers * d_kv * bytes)
max_seqs   ≈ gpu_blocks / avg_blocks_per_seq        # Phase 00 math, now in blocks

# "blocks for a 1000-token prompt, block_size 16?"
ceil(1000 / 16) = 63 blocks  (last block holds 1000 % 16 = 8 of 16 slots)

# "spec-decode speedup if acceptance α and we draft k?"
expected accepted ≈ (1 - α^(k+1)) / (1 - α) tokens per target pass  → ~1/(1-α) for large k

# "is this workload spec-decode-friendly?"
verbatim repetition (code, RAG quotes) → n-gram drafter, huge α, near-free
surprising creative text → low α → maybe a net loss

The framework one-liners (where these live in real tools)

# vLLM — the knobs that ARE this phase
from vllm import LLM
llm = LLM(model="...",
          gpu_memory_utilization=0.90,   # size of the KV block pool
          max_num_seqs=256,              # continuous-batching slot cap
          max_num_batched_tokens=8192,   # per-iter budget → enables chunked prefill
          block_size=16,                 # PagedAttention block size
          enable_prefix_caching=True,    # prefix sharing / CoW
          speculative_config={...})      # draft model / ngram / EAGLE

# TensorRT-LLM: in-flight batching + compiled kernels; build an engine, set max_batch_size
# SGLang: RadixAttention (tree prefix sharing) is on by default; great for agents

War stories

• The "fits in memory" OOM at peak. Weights fit, load test at batch 8 passed; production hit batch 60 and OOM'd on the KV blocks, not the weights. The fix wasn't a bigger GPU — it was capping max_num_seqs, raising gpu_memory_utilization, and letting the scheduler queue. Capacity is KV blocks at peak concurrency, always (Phase 00, made real).
• The one long answer that tanked everyone's latency. A single 4k-token generation in a static batch held three slots hostage for the whole run; p99 TPOT exploded. Continuous batching reclaimed the slots the instant short requests finished. Same hardware, 3× throughput, smooth tails.
• The "speculative decoding made it worse" ticket. A 7B draft for a 13B target on creative writing: acceptance was ~30%, so we paid for draft + verify and barely won. Swapped to an n-gram drafter for the RAG-with-quotes workload where acceptance was ~80% — then it flew. Acceptance rate is the whole game.
• The prefix nobody was sharing. Every request prepended a 1500-token system prompt; we stored its KV 200 times. Turning on enable_prefix_caching cut KV pressure dramatically and dropped TTFT — the prefill ran once. Free win that sat unused for months.
• The chunked-prefill rescue. Long-document summarization spiked everyone else's inter-token latency during each prefill. Enabling chunked prefill (via max_num_batched_tokens) interleaved the prefill with decode and bounded TPOT.

Vocabulary (rapid-fire)

• PagedAttention — KV-cache in fixed blocks via a block table; OS paging for the cache.
• Block table — per-sequence map from logical token positions to physical block ids.
• Internal fragmentation — unused slots in a sequence's last block (≤ block_size−1).
• Copy-on-write (CoW) — shared KV blocks; copy a block only when a sharer writes/diverges.
• Continuous / in-flight / iteration-level batching — schedule each decode step; admit + retire.
• Head-of-line blocking — a long request holds a static batch's slots until it finishes.
• Slot-step — one batch slot for one decode step; the throughput proxy (used vs wasted).
• TTFT / TPOT (ITL) — time to first token (prefill) / time per output token (decode).
• Chunked prefill — split a long prefill across steps, interleaved with decode.
• Speculative decoding — draft proposes, target verifies; min(1,p/q) accept + residual resample.
• Acceptance rate α — fraction of draft tokens the target accepts; drives the speedup.
• RadixAttention — SGLang's tree-structured prefix sharing for branching prompts.

Beginner mistakes

• Sizing capacity by weight memory and forgetting the KV blocks at peak concurrency.
• Getting the append off-by-one wrong: a new block iff the last is exactly full, not "nearly."
• Treating static and continuous batching as the same thing ("we batch, so we're fine").
• Thinking speculative decoding lowers quality — it's distribution-preserving; it only changes speed.
• Adding speculation to a low-acceptance workload and paying for draft + verify with no gain.
• Tuning for one latency number and wrecking another (throughput batch size vs TTFT).
• Leaving enable_prefix_caching off on a fat-fixed-system-prompt workload.

The one thing to take away

Serving is keep the bandwidth-bound decode batch full of useful work, and do fewer sequential steps — that's PagedAttention (fit more sequences) + continuous batching (no idle slots) + speculation (fewer steps), served against a TTFT/TPOT/$/1M contract. If you can explain those three and prove the speculative rule preserves the target distribution, you can hold the serving-design round.