Warmup Guide — Inference Serving Internals (vLLM/TensorRT-class)

Zero-to-senior primer for Phase 09. Phase 00 gave you the physics: decode is memory-bandwidth-bound, and the KV-cache — not the weights — is the inference memory wall. This phase turns that physics into a running serving engine. We start from "what actually happens between a request arriving and a token leaving" and build, from first principles, the three mechanisms every production stack shares: PagedAttention (the KV-cache as virtual memory), continuous batching (iteration-level scheduling that keeps the batch full), and speculative decoding (draft + verify, with an acceptance rule that provably preserves the target distribution). Along the way: prefill vs decode, the SLOs (TTFT, TPOT), chunked prefill, and how to choose between vLLM, TensorRT-LLM, SGLang, DeepSpeed-Inference, and ONNX Runtime.

Table of Contents

• Chapter 1: What Serving Actually Is — Prefill, Decode, and the SLOs
• Chapter 2: The KV-Cache Memory Wall and Why Naive Allocation Fails
• Chapter 3: PagedAttention — the KV-Cache as Virtual Memory
• Chapter 4: Prefix Sharing and Copy-on-Write
• Chapter 5: Static vs Dynamic vs Continuous Batching
• Chapter 6: The Throughput–Latency Tradeoff and Chunked Prefill
• Chapter 7: Speculative Decoding — Draft, Verify, and the Acceptance Rule
• Chapter 8: The Serving Stacks — vLLM, TensorRT-LLM, SGLang, and Friends
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: What Serving Actually Is — Prefill, Decode, and the SLOs

From zero. A serving engine takes a stream of requests — each a prompt plus a request for some number of generated tokens — and turns them into tokens flowing back, while sharing one (or a few) GPUs across many concurrent users. Every request goes through two phases with completely different physics, and confusing them is the most common junior mistake in a serving design.

Prefill. When a prompt arrives, the model processes all its tokens in one parallel forward pass to (a) produce the first output token and (b) populate the KV-cache for every prompt position. A 1000-token prompt is one big, dense matmul over 1000 positions. Recall the Phase 00 roofline: a big dense matmul has high arithmetic intensity, so prefill is compute-bound — it benefits from more FLOP/s and tensor cores.

Decode. After prefill, the model generates the rest one token at a time, each step a skinny matrix-times-vector that reads every weight from HBM to produce a single token. Arithmetic intensity ≈ 1 FLOP/byte, far left of any GPU's ridge — so decode is memory-bandwidth-bound. This single fact (from Phase 00) is why this whole phase exists: paging, batching, and speculation are all strategies to win at a bandwidth-bound, one-token-at-a-time workload.

request ─┐
         ▼
   ┌───────────┐   first token + KV for all prompt positions
   │  PREFILL  │ ───────────────────────────────►  (compute-bound, one parallel pass)
   └───────────┘
         │
         ▼
   ┌───────────┐  token  ┌───────────┐  token  ┌───────────┐
   │  DECODE 1 │ ──────► │  DECODE 2 │ ──────► │  DECODE 3 │ ─►  …  (memory-bound, sequential)
   └───────────┘         └───────────┘         └───────────┘
   reads ALL weights     reads ALL weights     reads ALL weights   (per step!)

The SLOs you serve against. Because the two phases differ, so do the latency metrics:

The senior's habit. When someone says "make it faster," the senior asks faster on which metric? TTFT is prefill + queue depth (cut it with chunked prefill, fewer queued tokens). TPOT is decode (cut it with speculation, smaller/quantized model). Throughput is batch fullness (continuous batching). These pull against each other — Chapter 6.

Common misconception. "One latency number describes the system." A config tuned for throughput (big batches) can wreck TTFT, and vice versa. You serve a contract — a TTFT target and a TPOT target and a $/1M budget — and the knobs trade between them.

Chapter 2: The KV-Cache Memory Wall and Why Naive Allocation Fails

Recall the cache. From Phase 00: to avoid recomputing attention over the whole prefix every step, we store each past token's key and value vectors — the KV-cache — and its size is

$$ \text{bytes}{\text{KV}} = 2 \times n{\text{layers}} \times T \times d_{\text{kv}} \times \text{bytes_per_elem} \times \text{batch}. $$

It grows linearly with context T and with the number of concurrent sequences, and at scale it is bigger than the weights — the thing that actually OOMs a server.

The naive allocation scheme. The obvious way to store a sequence's KV-cache is one contiguous buffer sized to the maximum possible length, max_len, reserved when the request starts. This is how pre-vLLM systems worked, and it bleeds memory three ways:

• Internal fragmentation. A request that will only generate 100 tokens but is allocated max_len = 2048 wastes 2048 - 100 slots — reserved, never used, can't be lent out.
• Reservation / over-allocation. You must reserve for the worst case up front because the buffer is contiguous and can't grow into someone else's space, so you provision for the longest possible output even though most requests are short.
• External fragmentation. As requests of different sizes come and go, the free memory shatters into holes too small to fit the next contiguous max_len request — even though the total free memory is plenty.

The vLLM paper measured this: naive contiguous allocation wastes 60–80% of KV memory. You're buying GPUs to store nothing.

NAIVE (contiguous, reserve max_len = 8):
  seq A (len 2): [A A . . . . . .]   ← 6 slots reserved, wasted
  seq B (len 5): [B B B B B . . .]   ← 3 slots wasted
  free holes:    scattered, none big enough for a new max_len request

The analogy that unlocks the fix. This is exactly the problem operating systems solved decades ago. Early systems gave each process a contiguous chunk of physical RAM and suffered the same fragmentation. The fix was virtual memory: break memory into fixed-size pages, let each process see a contiguous virtual address space, and map it through a page table to scattered physical pages. PagedAttention is that idea applied to the KV-cache (Chapter 3).

Common misconception. "Just set max_len smaller." That caps the feature (no long outputs) and still over-reserves for the requests that are long. The real fix isn't a smaller worst case — it's not reserving the worst case at all.

Chapter 3: PagedAttention — the KV-Cache as Virtual Memory

The core idea. Stop storing each sequence's KV-cache contiguously. Instead, carve the cache into fixed-size blocks (vLLM default block_size = 16 tokens), keep a global pool of free blocks, and give each sequence a block table — an ordered list of physical block ids — that maps its logical token positions to wherever the blocks physically live. The blocks need not be adjacent.

PAGED:
  block pool (block_size = 4):   [b0][b1][b2][b3][b4][b5] ...

  seq A block table → [b0, b2]        logical tokens 0..5 live in b0 (0-3), b2 (4-5, partial)
  seq B block table → [b1, b5, b3]    physically scattered; logically contiguous to A's view

  attention reads K/V by gathering through the block table — order is logical, not physical.

Why fragmentation nearly vanishes. Because allocation is per-block on demand:

• No external fragmentation — any free block fits any sequence; there's no "contiguous hole" requirement.
• Internal fragmentation is bounded — the only waste is the unused slots in each sequence's last (partial) block, at most block_size - 1 slots per sequence. With block_size = 16 that's ≤ 15 slots wasted per sequence, regardless of length — versus max_len - len before.
• No over-allocation — you allocate the prompt's blocks, then grow one block at a time as decode produces tokens. You never reserve for a future you might not reach.

The off-by-one that is the soul of the lab. Growing a sequence by one token (append) needs a new block if and only if the current last block is exactly full — i.e. len % block_size == 0. If you allocate too eagerly you leak blocks (waste memory); if too late you write past a block (corrupt the cache). The clean test: blocks_needed(8, 4) == 2, and the 9th token (after filling two 4-slot blocks) is the one that takes a third block.

def blocks_needed(num_tokens, block_size):
    return (num_tokens + block_size - 1) // block_size   # ceil division

# append: a NEW block is needed iff the last block is exactly full
if length % block_size == 0:        # 4 % 4 == 0  → take a new block before writing token 5
    take_one_free_block()
length += 1

The attention kernel side (so you can speak to it). The reason this needs a custom kernel is that attention must now gather K/V through the block table instead of reading a contiguous buffer. The PagedAttention CUDA kernel does exactly that: for each query, walk the sequence's block table and attend over the K/V in those (scattered) physical blocks. Our miniature models the allocator (blocks as counts), which is where the engineering judgment lives; the kernel is the [extension].

Production significance. PagedAttention is the change that made vLLM. By recovering the wasted 60–80% of KV memory, one GPU holds far more concurrent sequences' caches — and more concurrency is exactly what continuous batching needs (Chapter 5) to keep the bandwidth-bound decode batch full. Paging and batching are two halves of one win.

Common misconception. "Paging adds overhead, so it must be slower." The block-table indirection is cheap; the win — fitting many more sequences in memory so the batch can stay full — dwarfs it. Paging enables the throughput, it doesn't tax it.

Chapter 4: Prefix Sharing and Copy-on-Write

The opportunity. Many requests share a prefix — the same long system prompt, the same few-shot examples, the same RAG document prepended to every query. With contiguous allocation each request stores its own copy of that prefix's KV-cache: pure duplication. With blocks, the identical prefix is identical blocks, and identical blocks can be shared.

The mechanism (ref-counted blocks + copy-on-write). Borrowed wholesale from OS shared pages:

• Hash the prefix; if its blocks already exist, point the new sequence's block table at the same physical blocks and bump a reference count.
• All sharers read those blocks freely — no copy, no extra memory.
• When a sequence needs to write into a shared block (it diverges — generates a different token, or appends into a partially-shared last block), copy that one block first, decrement the original's ref count, and write into the private copy. This is copy-on-write (CoW).

system prompt blocks: [P0][P1]   (ref-count 3 — three requests share them)
  req A → [P0, P1, A2]    private A2 (its own continuation)
  req B → [P0, P1, B2]    private B2
  req C → [P0, P1, C2]    private C2
  → the prefix's KV is stored ONCE, not three times.

Production significance. This is automatic prefix caching (vLLM enable_prefix_caching) and the heart of SGLang's RadixAttention, which organizes shared prefixes in a radix tree so even branching prompts (an agent exploring several continuations) share their common stem. For agent and RAG workloads with a fat fixed prefix, this is a large, free memory and TTFT win — the prefix's prefill is computed once.

Common misconception. "Sharing risks one request seeing another's tokens." Copy-on-write is exactly the guarantee against that: the instant a sequence would change a shared block, it gets its own copy. Reads are shared; writes are private.

Chapter 5: Static vs Dynamic vs Continuous Batching

Why batch at all. Decode is memory-bandwidth-bound: each step reads every weight from HBM to make one token. If you read those weights and apply them to one sequence, the FLOPs are wasted; apply the same read to a batch of sequences and throughput scales ~linearly with batch (Phase 00). So keeping the batch full is the entire throughput game — and the three batching strategies differ only in how full they keep it.

Static batching. Collect B requests, run them as a fixed batch to completion, then take the next B. Simple, and how HF generate works on a batch. The flaw: requests in a batch finish at different times (one wants 20 tokens, another 500), but the batch slot stays occupied until the longest request in the batch finishes. The finished sequences' slots sit idle, doing nothing, until the wave ends. This is head-of-line blocking, and it wastes a huge fraction of slot-steps on skewed workloads (which real traffic always is).

STATIC (batch of 4, lengths 20 / 2 / 2 / 2):
  step:  1 2 3 ............................ 20
  slot0: ████████████████████████████████████   (20 useful)
  slot1: ██ . . . . . . . . . . . . . . . . . .   (2 useful, 18 WASTED — held hostage)
  slot2: ██ . . . . . . . . . . . . . . . . . .
  slot3: ██ . . . . . . . . . . . . . . . . . .
  → the whole batch runs 20 steps; 3 slots wasted 18 each.

Dynamic batching. A half-step: a server-side queue that forms a batch from whatever arrived in a short window, then runs it (often still to completion). It improves utilization at admission but still suffers head-of-line blocking within the batch.

Continuous (in-flight / iteration-level) batching. The Orca insight, productionized by vLLM: schedule at the iteration (single-decode-step) level, not the request level. Each step:

1. Admit waiting requests into any free batch slots (if KV blocks are available).
2. Advance every running sequence by one token.
3. Retire any sequence that just finished — immediately freeing its slot and KV blocks so a new request can take them next step.

Because a finished sequence's slot is reused the instant it retires, there is no head-of-line blocking: the batch stays full of useful work.

CONTINUOUS (max_batch 4): slots refill the moment a request finishes
  step:  1 2 3 4 5 ...
  slot0: A A A A A ...   (long request keeps running)
  slot1: B B C C D ...   (B finishes at step 2 → C admitted at step 3 → ...)
  slot2: ...             (idle only when the queue is truly empty)
  → far fewer wasted slot-steps; throughput up 2–4× on skewed traffic.

Why 2–4×. On realistic length-skewed traffic, static batching's wasted slot-steps are a large fraction of the total; continuous batching reclaims them. Since decode throughput ∝ batch fullness (bandwidth-bound), reclaiming idle slots directly buys throughput. The lab proves this: same useful work, continuous wastes ~0 slot-steps vs static's many.

Common misconception. "Bigger batch is always better." Bigger batches raise throughput but (a) cost KV memory — you can run out of blocks (the scheduler must then queue or preempt), and (b) can hurt TTFT, because a fat decode batch competes with incoming prefills. Continuous batching is about keeping the batch full of useful work, not maximally large.

Chapter 6: The Throughput–Latency Tradeoff and Chunked Prefill

The tension. Throughput wants big, full batches (amortize the weight read). Latency — especially TTFT — wants prefills to run now, not wait behind a fat decode batch. These pull opposite directions, and tuning a server is choosing where on the curve to sit for your SLOs.

The prefill–decode interference problem. A long prompt's prefill is one big compute-bound matmul that can monopolize the GPU for several steps. If it runs as one chunk, every decode step for every other request in the batch stalls behind it — their TPOT spikes, and the new request's own TTFT is fine but everyone else's smoothness dies. Conversely, if you always prioritize decode, a long prompt waits forever for prefill and its TTFT blows up.

Chunked prefill. Split a long prompt's prefill into chunks and interleave those chunks with ongoing decode steps in the same batched iteration. Now a 4000-token prefill becomes, say, eight 500-token chunks spread across eight steps, each step also advancing the decoders. The result: bounded TPOT for running requests and steady progress on the new prefill — you trade a slightly higher TTFT on the long prompt for not wrecking everyone else.

WITHOUT chunked prefill:        WITH chunked prefill:
  step: [ big 4k prefill ]        step1: [prefill chunk][decode×N]
        [decode][decode]...       step2: [prefill chunk][decode×N]
   → decoders stall for the       step3: [prefill chunk][decode×N]
     whole prefill                 → decoders keep flowing; prefill drips in

The knobs (so you can speak to a real config). In vLLM these are max_num_seqs (batch slot cap), max_num_batched_tokens (the per-iteration token budget that enables chunked prefill — prefill chunks and decode tokens share it), gpu_memory_utilization (how much HBM to give the KV block pool), and the scheduling policy. Tuning them is picking your throughput/TTFT/TPOT point.

Common misconception. "Latency and throughput are the same optimization." They're a Pareto frontier. You can almost always buy more throughput by accepting worse tail latency (bigger batches, longer queue windows) — the engineering is choosing the point that meets all three SLO numbers, then proving it with a load test, not picking "fast."

Chapter 7: Speculative Decoding — Draft, Verify, and the Acceptance Rule

The opportunity. Decode is memory-bandwidth-bound, so the GPU's math units are idle most of the time — you're waiting on HBM. Speculative decoding spends that idle compute to cut the number of sequential target-model steps.

The scheme. Use a cheap draft to propose k tokens fast (a small model, or Medusa/EAGLE heads, or even an n-gram lookup). Then run the target model once over all k proposed positions in parallel — one forward pass gives you the target's distribution at every proposed position (the same parallelism that makes prefill efficient). Now verify: accept a prefix of the draft, and on the first disagreement, correct it. If you accept m tokens from one target pass, you produced m+1 tokens for the cost of one sequential target step instead of m+1 of them.

The acceptance rule (and why it's exactly this). Let the target give probability p = p_target(t) and the draft give q = p_draft(t) for the proposed token t. Accept it with probability

$$ a = \min!\left(1, \frac{p}{q}\right). $$

If accepted, keep t and move on. If rejected (the first rejection in the sequence), discard t and the rest of the draft, and resample a corrected token from the normalized residual:

$$ p'(x) = \frac{\max(0,\ p_{\text{target}}(x) - p_{\text{draft}}(x))}{\sum_{y}\max(0,\ p_{\text{target}}(y) - p_{\text{draft}}(y))}. $$

Why this provably preserves the target distribution. Consider any token x for a single position. It can be emitted two ways: (1) the draft proposed x and we accepted it, or (2) the draft proposed something else, we rejected, and the residual resample produced x. Summing those two paths:

$$ \Pr[\text{emit } x] = q(x)\cdot\min!\left(1, \tfrac{p(x)}{q(x)}\right) + (\text{reject mass})\cdot p'(x). $$

The first term is min(q(x), p(x)). The reject mass is 1 - Σ_y min(q(y), p(y)), and the residual p'(x) is max(0, p(x)-q(x)) normalized by that exact same reject mass — so the second term is max(0, p(x) - q(x)). Adding them: min(q, p) + max(0, p - q) = p(x). The emitted token is distributed exactly as the target. No quality is traded; only latency is bought. This is the part interviewers love, because the algebra is the guarantee.

draft proposes:  t1  t2  t3
target verifies (1 parallel pass) → p1, p2, p3 (+ a bonus row p4)
  accept t1 with min(1, p1/q1)?  yes → keep
  accept t2 with min(1, p2/q2)?  yes → keep
  accept t3 with min(1, p3/q3)?  NO  → resample from residual(p3, q3); STOP
  emitted: [t1, t2, corrected]  (3 tokens, 1 sequential target step)
  if all 3 accepted → emit [t1, t2, t3, sample(p4)]  (4 tokens, 1 step!)

The speedup. It depends on the acceptance rate α (how often the draft agrees with the target) and the draft's cost. A good draft on easy text accepts long runs (big speedup); on hard, surprising tokens it accepts little (and you've paid for the draft + verify with little gain). The acceptance rate, not raw draft speed, is the number that matters — and it's why draft quality and the draft/target agreement are what you tune.

The drafter zoo (know when to use which):

Production significance. Speculative decoding is the main lever for TPOT (the inter-token latency) without changing the model's outputs. It shines when decode is bandwidth-bound with spare compute (the common case) and the workload is predictable enough for high acceptance. It does not help prefill (already compute-bound and parallel).

Common misconception. "Speculative decoding approximates the big model, so quality drops." No — the acceptance rule makes the output exactly the target's distribution (the proof above). The draft only affects speed (acceptance rate), never what gets said. A bad draft makes it slow, not wrong.

Chapter 8: The Serving Stacks — vLLM, TensorRT-LLM, SGLang, and Friends

You won't write these kernels at work; you'll choose and tune one of these engines. Know what each is and the deciding constraint.

The decision framing (ties to Phase 00's tradeoff resolver). It's the same exercise: score the candidates on throughput, latency, hardware fit, feature need (prefix sharing? speculation? structured output?), and operational complexity, weight them for your workload, and name the deciding constraint. "All-NVIDIA, latency-critical, simple prompts → TensorRT-LLM." "Open weights, agentic with branching prefixes → SGLang." "General high-throughput open-weights API → vLLM." There is no best engine, only best for these constraints.

Common misconception. "vLLM is always the answer." vLLM is the best default, but a latency-bound NVIDIA-only workload may want TensorRT-LLM, and a heavily prefix-sharing agent workload may want SGLang's RadixAttention. Pick on the deciding constraint, not the popularity.

Lab Walkthrough Guidance

The lab (lab-01-paged-attention-scheduler) turns these chapters into code. Suggested order (matches the file):

1. blocks_needed — Chapter 3. Pure ceil division; the test hammers the exact-multiple boundary (blocks_needed(8, 4) == 2). Get this and the rest of the allocator follows.
2. kv_cache_blocks_fragmentation — Chapter 2. Paged waste = Σ rounded-up blocks − used; naive waste = max_len × n − used. The test proves paged << naive on a skewed batch.
3. BlockManager — Chapter 3. allocate (ceil blocks, OOM → None), free (return all blocks), and append — the soul: a new block iff len % block_size == 0. The test_append_allocates_new_block_only_when_last_is_full test is the off-by-one.
4. Request / static_batching_steps — Chapter 5. The baseline: each wave runs to its longest request; wasted = wave_len − max_new_tokens per slot.
5. ContinuousBatchingScheduler — Chapter 5. _admit (slot + blocks), step (admit → advance via append → retire+free), run. The scheduler soul test shows it beats static on wasted slot-steps while never exceeding max_batch or over-allocating blocks.
6. speculative_accept — Chapter 7. min(1, p/q) accept; on reject resample the normalized residual. The spec-decode soul test is statistical: over many seeded trials the emitted distribution matches the target, within tolerance.

Run red → green: pytest test_lab.py -v, then LAB_MODULE=solution pytest test_lab.py -v, then python solution.py for the worked numbers.

Success Criteria

• All tests pass against your lab.py and against LAB_MODULE=solution.
• You can explain PagedAttention as OS virtual memory: blocks, the per-sequence block table, why external fragmentation vanishes and internal fragmentation is bounded by block_size - 1.
• You can state and defend the append off-by-one: a new block iff len % block_size == 0, and what breaks if you get it wrong (leak vs corruption).
• You can describe prefix sharing + copy-on-write and name the production feature (vLLM prefix caching, SGLang RadixAttention).
• You can explain head-of-line blocking and why iteration-level (continuous) batching is 2–4×, using slot-steps as the proxy.
• You can derive min(q, p) + max(0, p - q) = p and conclude speculative decoding preserves the target distribution.
• You can define TTFT vs TPOT, say which mechanism moves which, and pick a serving stack by the deciding constraint.

Interview Q&A

• "Explain PagedAttention." KV-cache stored in fixed blocks (like OS pages) addressed through a per-sequence block table, so the cache is non-contiguous → no external fragmentation, internal waste ≤ block_size−1 per sequence, no max_len over-reservation. Recovers the 60–80% naive schemes waste; enables prefix sharing.
• "Why does the KV-cache fragment under naive allocation?" Contiguous max_len reservation per sequence → internal waste (short request in a big reservation), over-allocation (reserve worst case), and external fragmentation (free memory shatters into too-small holes). Paging fixes all three.
• "Static vs continuous batching?" Static runs a fixed batch to completion → head-of-line blocking (finished slots idle until the longest finishes). Continuous schedules per-iteration: admit + advance
  • retire every step → batch stays full of useful work → 2–4× throughput on skewed traffic.
• "Prove speculative decoding doesn't change the output distribution." Emit x two ways: draft-and- accept = min(q,p), or reject-and-residual = max(0,p−q). Sum = p(x). So the emitted token ~ target. The draft only affects speed (acceptance rate), never what is said.
• "When does speculative decoding help and when not?" Helps decode (bandwidth-bound, spare compute) when acceptance is high (predictable text, good draft, n-grams for verbatim repetition). Doesn't help prefill. Low acceptance can be a net loss (you pay draft + verify).
• "TTFT vs TPOT — what moves each?" TTFT = prefill + queueing (chunked prefill, less queued work, prefix caching). TPOT = decode speed (speculation, smaller/quantized model, full batch). They trade against throughput.
• "What's chunked prefill and why?" Split a long prefill into chunks interleaved with decode steps so a big prompt doesn't stall everyone's TPOT; trades a little TTFT on that prompt for system-wide smoothness.
• "Pick a serving stack for an all-NVIDIA, latency-critical chatbot." TensorRT-LLM (compiled kernels, lowest NVIDIA latency, FP8), accepting build complexity — vs vLLM if you need flexibility/open-weights breadth, or SGLang if the workload is prefix-heavy/agentic.

References

• Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention (vLLM, SOSP 2023) — the paging idea, the 60–80% fragmentation measurement, prefix sharing / CoW.
• Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022) — iteration-level (continuous / in-flight) scheduling.
• Leviathan, Kalman, Matias, Fast Inference from Transformers via Speculative Decoding (ICML 2023) — the acceptance rule and the distribution-preservation proof.
• Chen et al., Accelerating Large Language Model Decoding with Speculative Sampling (DeepMind, 2023) — the concurrent speculative-sampling formulation.
• Cai et al., Medusa: Simple LLM Inference Acceleration with Multiple Decoding Heads (2024).
• Li et al., EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (2024).
• Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs (RadixAttention, 2024).
• Agrawal et al., SARATHI / chunked prefill and the vLLM docs on max_num_batched_tokens.
• vLLM, TensorRT-LLM, and DeepSpeed-Inference (FastGen) documentation — the real knobs: gpu_memory_utilization, max_num_seqs, block_size, enable_prefix_caching, speculative_config.