« Track Overview · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 05 — Serving Patterns, Capacity & the Economics of Inference

Answers these JD lines: "Design model serving patterns for managed APIs, dedicated capacity (PTUs / provisioned throughput), and self-hosted open-weight models on GPU infrastructure (vLLM, TGI, Triton, or equivalent), with appropriate trade-offs across cost, latency, sovereignty, and compliance" · "provisioned throughput / PTUs, self-hosted open-weight models, GPU scheduling, vLLM / TGI / Triton, KV-cache management, batching strategies, and the cost-latency-quality trade-offs that drive model selection."

Why this phase exists

Phase 04 built the gateway that routes to a deployment. This phase is about what a deployment actually is, and it is the phase where the JD stops asking for architecture and starts asking for arithmetic.

Three conversations you will have repeatedly, all of which end badly without numbers:

• "Should we buy provisioned capacity?" — answerable only as a break-even utilization, and that break-even moves a long way with your input:output mix.
• "Why is our self-hosted model slower than the managed one?" — because a single-stream decode wastes almost all of a GPU's FLOPs, and nobody configured batching.
• "Can we run the restricted model on-shore?" — a memory question first (does it even fit?), a cost question second, and a sovereignty question third.

The unifying insight is that decode is memory-bandwidth-bound and prefill is compute-bound, and almost every serving decision follows from that asymmetry.

Five ideas carry the phase:

1. Concurrency is a memory question. The KV cache grows linearly with sequence length × batch size, and it — not FLOPs — is what caps how many requests a GPU can hold.
2. Batching works because the weight read amortizes and the KV read does not. One expression explains why continuous batching is the biggest throughput win in modern serving.
3. Continuous batching wins most when output lengths are uneven — which is always, in an agent platform where some turns are a sentence and some are a report.
4. The PTU break-even is a function of your traffic mix, and quoting a single number without stating the mix is the most common error in this conversation.
5. Self-hosting is a utilization bet with an engineering line. An idle GPU costs the same as a busy one, and the team that runs it is not free.

Concept map

• Model shape: params, layers, KV heads, head dim, precision → weight bytes and KV bytes per token (2·L·H·d·b).
• GQA/MQA: the query:KV head ratio as the dominant lever on KV size, and therefore on concurrency.
• Memory budget: total memory − weights − working-space reserve = KV budget → max concurrent sequences.
• Prefill vs decode: parallel and compute-bound versus sequential and bandwidth-bound; arithmetic intensity and the roofline ridge point; TTFT versus inter-token latency.
• Batching: static (everyone waits for the longest) versus continuous / in-flight (retire and admit every step); admission control on the final projected length.
• PagedAttention: KV in fixed blocks rather than contiguous reservations — why real systems can over-commit and ours cannot.
• Capacity types: managed PAYG (elastic, congested, per-token) · provisioned / PTU (predictable latency, a monthly commitment) · self-hosted (sovereignty and unit cost, at the price of operating GPUs).
• Spillover: fill the dedicated floor, spill the peak to PAYG — degrade in cost, not availability.
• The trade-off space the JD names: cost, latency, sovereignty, compliance.

The lab

Integrated scenario (how this shows up at work)

The CTTO's office asks for a three-year AI infrastructure plan. Procurement wants to know how many PTUs to commit to. Group Risk wants the restricted-data workload on-shore. The platform team has been told the self-hosted deployment is "too slow."

The answers are all in this lab. The self-hosted deployment is slow because it is running batch-1 decode at an arithmetic intensity of ~1 FLOP/byte against a GPU whose ridge point is ~300 — it is using under half a percent of its compute, and the fix is continuous batching, not more GPUs. The PTU commitment should be sized to p50 demand with spillover to PAYG, because sizing to peak buys capacity that sits idle and sizing to p10 causes 429s. And the restricted workload fits on-shore only if the model fits in the available memory at the required context length — which for a 70B model at fp16 means tensor parallelism across a whole node before anything else is even discussable.

Deliverables checklist

• Lab 01 green under LAB_MODULE=solution pytest and under your own lab.py.
• You can write the KV-bytes-per-token formula from memory and compute concurrency with it.
• You can explain why decode is memory-bound and prefill is compute-bound.
• You can explain, in one sentence, why batching helps decode.
• You can describe continuous batching and say when its advantage over static is largest.
• You can compute a PTU break-even and state why the output fraction changes it.
• You can lay out the floor-plus-spillover pattern and say what it optimizes.
• You can list what belongs in a self-hosting business case beyond GPU hours.

Key takeaways

• KV bytes per token is 2·L·H·d·b. Memorize it; every concurrency question reduces to it.
• Concurrency is memory, not compute. The GPU that "should" serve hundreds of streams serves forty because of context length.
• Weights amortize across a batch; KV does not. That is why batching works and why it stops working at long context.
• Continuous batching's win scales with output-length variance, which in an agent platform is large.
• Admit on the final length, or the batch OOMs and kills half-served requests.
• Never quote a PTU break-even without the traffic mix.
• Spillover degrades cost, not availability — which is almost always the right trade.
• Self-hosting is a utilization bet, and the engineering line is the one that decides it.