Lab 01 — Capacity Planning & the Serving Simulator

The problem

Finance asks whether to commit to a year of provisioned capacity. Wholesale asks why the self-hosted model is slower than the managed one. Security asks whether the restricted-data model can run on-shore. Every one of those is the same question — what does serving actually cost, and what determines it — and every one is answerable with arithmetic you can do on a whiteboard.

This lab builds that arithmetic, plus a simulator that shows why continuous batching replaced static batching.

What you build

#	Component	What it does
1	`ModelShape`, `GPU`	the shapes that determine serving cost, including a tensor-parallel group as one aggregate device
2	`kv_bytes_per_token`, `usable_kv_bytes`, `max_concurrent_sequences`	the KV-cache arithmetic that turns concurrency into a memory question
3	`prefill_ms`, `decode_ms_per_token`, `arithmetic_intensity`, `ridge_point`	why prefill is compute-bound and decode is memory-bound, with the numbers
4	`ContinuousBatcher`	in-flight batching: retire, admit, step — with KV-budget admission on the final length
5	`StaticBatcher`	the baseline it replaced, so the win is measured rather than asserted
6	`PaygPricing`, `ProvisionedPricing`, `break_even`	the PTU-vs-PAYG break-even, as a function of the output mix
7	`plan_with_spillover`	the floor-plus-spill shape every mature deployment converges on
8	`SelfHostedPricing`, `cheapest_option`	the utilization bet, including the engineering line everyone forgets

Key concepts

Concept	Where	Why it matters
KV bytes = `2·L·H·d·b`	`kv_bytes_per_token`	the one formula to memorize; every concurrency question reduces to it
GQA is the biggest lever	the `kv_heads` tests	64 → 8 KV heads is 8× the concurrency, same model
Working-space reserve	`usable_kv_bytes`	assuming 100% of free memory is KV over-admits and OOMs the batch
Weights amortize, KV does not	`decode_ms_per_token`	the entire argument for batching, in one expression
Ridge point	`arithmetic_intensity`	decode at batch 1 wastes almost all of a GPU's FLOPs
Admit on the final length	`_projected`	admitting on the prompt OOMs mid-flight and kills half-served requests
Retire before admit	`ContinuousBatcher.run`	reusing freed capacity in the same step is continuous batching
Never-fits ⇒ reject	same	a request larger than the whole budget must not queue forever
Break-even moves with the mix	`break_even`	output costs several times input; never quote one number
Spillover	`plan_with_spillover`	degrade in cost, not in availability
Utilization bet	`self_hosted_cost_per_1k_tokens`	an idle GPU costs the same as a busy one

Files

File	Role
lab.py	your implementation
solution.py	reference; `python solution.py` runs six worked sections
test_lab.py	55 tests
requirements.txt	`pytest`

Run

pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v
python solution.py

Success criteria

All 55 tests green against your lab.py.
A 70B fp16 model on one 80 GiB card yields a KV budget of 0 — and a positive one on an 8-way group.
With zero context, doubling the batch exactly halves per-token decode time; with a very long context, it does not.
Batch-1 decode sits far below the GPU's ridge point.
Continuous batching finishes an uneven workload in fewer ticks than static, and the short request finishes first.
A request whose prompt fits but whose prompt+output does not is rejected.
peak_kv_bytes never exceeds the budget.
A heavier output mix reaches the PTU break-even at a lower volume.
Ties in cheapest_option go to PAYG.

How this maps to the real stack

This lab	The real thing	What we simplified
`ContinuousBatcher`	vLLM's scheduler, TGI's continuous batching, TensorRT-LLM's in-flight batching	real schedulers also do chunked prefill, preemption/swapping and priority; ours never preempts
KV budget	vLLM's PagedAttention block allocator	ours reserves the final length contiguously; paging allocates in blocks on demand, which is why real systems can over-commit safely and we cannot
`max_concurrent_sequences`	`gpu_memory_utilization` and the profiling step vLLM runs at startup	ours uses a flat overhead fraction; real systems measure
`decode_ms_per_token`	a roofline estimate	ignores kernel efficiency, TP communication, and scheduling gaps
`ProvisionedPricing`	Azure OpenAI PTUs, AWS Bedrock Provisioned Throughput	real capacity has minimums, regional availability and reservation terms
`plan_with_spillover`	Azure's PTU + standard "spillover" deployments; a gateway routing rule	ours is monthly arithmetic; production spills per-request
`SelfHostedPricing`	GPU node pools on AKS/EKS, plus the platform team that runs them	the engineering line is a guess; make it your own

Honest limits. No chunked prefill (so a long prompt blocks a step), no preemption, no paging (so fragmentation is invisible), no speculative decoding, no quantization modelling, and a decode model that ignores tensor-parallel communication. Each of those changes the numbers; none changes the reasoning.

Extensions

Paged KV. Allocate in fixed blocks on demand rather than reserving the final length. Then over-commit and add preemption when the budget is exceeded — and watch the goodput/latency trade-off appear.
Chunked prefill. Split a long prompt across steps so it does not stall decode. Measure TTFT for short requests arriving behind a long one, before and after.
Quantization. Add bytes_per_param=1 and a KV-cache quantization option, and recompute the concurrency table. The result usually surprises people.
Speculative decoding. Model a draft model that proposes k tokens with acceptance rate a, and find where it stops paying.
A real workload trace. Replace the synthetic arrivals with a sampled distribution of prompt and output lengths and compute p50/p95 TTFT and end-to-end latency.
Reserved-capacity risk. Add a one-year commitment with a discount, then price the option of the model being deprecated mid-term. That is the conversation finance actually needs.

Interview / resume bullets

"Built the platform's capacity model: KV-cache arithmetic that made maximum concurrency a memory calculation rather than a guess, and showed that moving from MHA to 8-way GQA multiplied servable concurrency by eight on the same hardware."
"Simulated continuous versus static batching on our real output-length distribution and quantified the win — 2.8× shorter wall-clock on an uneven workload — which turned a serving-stack choice into an evidenced decision."
"Made the PTU-versus-pay-as-you-go decision a function of measured traffic mix rather than a vendor datasheet, and adopted a floor-plus-spillover topology so peak demand degrades in cost rather than in availability."
"Included GPU-hours and platform-engineering cost in the self-hosting business case, and showed the crossover point as a function of monthly token volume."

Enterprise AI & Agentic Platform — Engineering & Architecture