« Phase 05 · Warmup · Track Overview

Core Contributor Notes — How the Real Serving Stacks Work

Table of Contents

• 1. vLLM's scheduler
• 2. PagedAttention in detail
• 3. Chunked prefill
• 4. TGI and Triton, briefly
• 5. Provisioned capacity as the providers implement it
• 6. Sharp edges
• 7. What the miniature simplifies
• 8. References

1. vLLM's scheduler

Our ContinuousBatcher is a faithful miniature of the shape of vLLM's scheduler and a simplification of its policy. The real loop, per step:

1. Schedule. Decide which sequences run: continuing decodes first, then admit waiting sequences if blocks are available, then possibly preempt running sequences if memory is tight.
2. Execute. One forward pass over the whole batch.
3. Process outputs. Append tokens, detect stop conditions, free blocks for finished sequences.

Three policy elements the lab omits:

Preemption. When memory runs out mid-flight, vLLM can evict a sequence — either by recomputing its KV later (cheap for short sequences) or by swapping its blocks to CPU memory (cheaper for long ones). This is what makes over-commitment safe, and it is the mechanism our conservative admission control substitutes for. Preemption converts an OOM into a latency penalty, which is a strictly better failure mode.

Priority and fairness. Real schedulers support priorities and, in multi-tenant deployments, need some fairness policy or one tenant's long generations starve everyone. Our FIFO queue is the simplest possible policy and is wrong for a shared bank platform — which is a good extension and a good design-review question.

Prefix caching (enable_prefix_caching). Blocks are hashed by content; two sequences sharing a prompt prefix share the underlying blocks. This is the serving-side mechanism behind the provider-side prompt caching discount discussed in Phase 04, and on a self-hosted deployment it is a configuration flag with a large effect on an agent workload where every request repeats the same system prompt and tool schemas.

2. PagedAttention in detail

The core idea: KV is stored in fixed-size blocks (commonly 16 tokens' worth), and a per- sequence block table maps logical positions to physical blocks. The attention kernel is modified to gather from non-contiguous blocks.

What this buys, precisely:

The measured effect in the original paper is a large increase in achievable batch size at the same memory, which translates directly into throughput. The mechanism is worth understanding because it explains a practical observation: a real system serves more concurrent sequences than the naive budget ÷ max_length arithmetic predicts, so our lab's numbers are a conservative floor rather than an estimate.

Copy-on-write matters too: when two sequences share blocks and one diverges (parallel sampling, beam search), only the diverging block is copied. That is what makes n>1 sampling cheap.

3. Chunked prefill

The problem: prefill for a 30 000-token prompt is a single large compute chunk. While it runs, no decode steps happen, so every other request in the batch stalls. TTFT for the long request is fine; ITL for everyone else spikes.

The fix (Sarathi-Serve and now standard in vLLM): split the prefill into chunks and interleave them with decode steps. Each step processes a slice of the long prompt plus a decode token for everyone else.

The consequences are worth knowing because they change how you configure a deployment:

• Long-prompt TTFT gets slightly worse (its prefill is spread over more steps).
• Everyone else's ITL gets dramatically better.
• The scheduler now has a token budget per step rather than a request budget, which is a different tuning knob.

For an agent platform with a mix of short turns and document analysis, this is usually the single most impactful serving configuration after continuous batching itself. Our lab has no notion of it — every request's prefill is one tick — which is exactly why the PRINCIPAL DEEP-DIVE flags TTFT tail behaviour as something the model does not capture.

4. TGI and Triton, briefly

Hugging Face TGI — continuous batching, tensor parallelism, quantization support, and a production-shaped HTTP/gRPC server. Similar ideas to vLLM with a different operational surface; the choice between them is usually about ecosystem fit and which model architectures are supported today, not about the scheduling model.

NVIDIA Triton Inference Server — a general inference server (any framework, any model type) with an LLM backend (TensorRT-LLM) that provides in-flight batching. Triton's value is when you serve more than LLMs: embeddings, rerankers, classical models, all with one deployment and monitoring story. Its cost is complexity, and TensorRT-LLM requires an engine build step per model per GPU per configuration, which is a real operational burden.

The decision heuristic: vLLM or TGI if you serve LLMs; Triton if you serve a zoo. In a bank that has classical models, rerankers and LLMs, "a zoo" is the honest description more often than people expect.

5. Provisioned capacity as the providers implement it

Azure OpenAI PTUs. Capacity is purchased in units, with minimums per model and per deployment type, and availability varies by region. Throughput per unit depends on your prompt/generation shape, which is why Microsoft publishes a calculator rather than a constant — and why the lab makes tokens_per_unit_month an input rather than a derived value. Reservations (monthly/yearly) discount the hourly rate in exchange for commitment. Spillover to a standard deployment is a supported pattern and is exactly the floor-plus-spill shape.

AWS Bedrock Provisioned Throughput. Model units with commitment terms (no-commitment, 1-month, 6-month), where a model unit provides a specified throughput. Same structural decision, different vocabulary.

Three practical notes that apply to both:

• Capacity is regional and finite. "We'll buy PTUs" assumes they are available in your region for your model. In a sovereignty-constrained deployment, check this first — it can eliminate options before any arithmetic.
• Commitment terms interact with model lifecycles. A one-year commitment on a model family outlives most model generations. Read the terms on what happens when the model you committed to is retired.
• Measure throughput yourself, on your traffic. The single most common capacity error is planning with a published figure and discovering your input:output mix gives you materially less.

6. Sharp edges

gpu_memory_utilization is not a fraction of free memory. In vLLM it is the fraction of total GPU memory the engine may use, weights included. Setting it to 0.9 on a card that also hosts something else is how you OOM at startup — and it is why our usable_kv_bytes subtracts weights explicitly rather than applying a fraction to the total.

Max model length silently caps concurrency. Setting max_model_len to the model's full context makes the engine reserve for that worst case in some configurations. If your real p99 prompt is 4 000 tokens, advertising a 128 000-token context can cost you most of your batch size.

Quantization changes more than memory. Weight-only quantization halves or quarters the weight bytes (and therefore speeds up decode, which is bandwidth-bound) but leaves the KV cache untouched. KV-cache quantization is a separate feature and is the one that increases concurrency. Confusing the two produces a plan that does not deliver.

Tensor parallelism must divide the heads. TP degree has to divide the number of attention heads (and KV heads, for GQA). A model with 8 KV heads cannot be split 16 ways in the naive scheme — which constrains your options in a way capacity arithmetic alone will not reveal.

Engine startup is minutes. Weight loading, and for TensorRT-LLM an engine build. Autoscaling that assumes seconds will not work; keep warm capacity.

Speculative decoding's benefit is workload-dependent. A draft model proposing k tokens with acceptance rate a helps when a is high, which depends on the task. It also costs memory (two models resident). Measure on your traffic; the published speedups are for benchmarks, not for agent scratchpads.

7. What the miniature simplifies

Everything the real stacks add makes the numbers better than the miniature predicts (paging, prefix caching, chunked prefill) or worse in ways the miniature flags (TP communication, scheduling gaps). That is why it is usable as a conservative capacity floor, which is the right posture for a procurement conversation.

8. References

• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023 — the vLLM paper.
• Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models, OSDI 2022 — iteration-level scheduling, the origin of continuous batching.
• Agrawal et al., Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve, OSDI 2024 — chunked prefill.
• Shazeer, Fast Transformer Decoding (MQA), 2019; Ainslie et al., GQA, 2023.
• Dao et al., FlashAttention and FlashAttention-2 — why attention is memory-bound.
• Leviathan et al., Fast Inference from Transformers via Speculative Decoding, 2023.
• vLLM documentation — scheduler configuration, gpu_memory_utilization, max_model_len, prefix caching, quantization, tensor parallelism.
• Hugging Face TGI and NVIDIA Triton / TensorRT-LLM documentation.
• Azure OpenAI provisioned throughput — units, the capacity calculator, reservations, spillover.
• AWS Bedrock Provisioned Throughput — model units and commitment terms.
• Williams, Waterman & Patterson, Roofline, CACM 2009.