« Phase 05 · Warmup · Track Overview
Hitchhiker's Guide — Serving, Capacity & Inference Economics
The 30-second mental model
Prefill is parallel and compute-bound. Decode is sequential and memory-bandwidth-bound. Everything else follows.
Concurrency is a memory question: the KV cache grows linearly with context × batch, and it is what caps how many requests a GPU holds. Batching works because the weight read amortizes and the KV read does not. And the capacity decision — PAYG vs PTU vs self-hosted — is arithmetic whose answer moves with your input:output mix.
The formulas
| Thing | Formula |
|---|---|
| KV bytes per token | \( 2 \cdot L \cdot H_{kv} \cdot d_{head} \cdot b \) |
| KV budget | total_memory − weights − ~10% working space |
| Max concurrency | KV budget ÷ (KV/token × context) |
| Prefill FLOPs | \( 2 \cdot \text{params} \cdot N \) |
| Decode bytes/step | weights + B × KV(context) |
| Arithmetic intensity | FLOPs ÷ bytes — compare to peak FLOPs/s ÷ bandwidth |
| Blended PAYG price | \( c_{in}(1-f) + c_{out} f \), f = output fraction |
| Break-even tokens | monthly_commitment ÷ blended × 1000 |
| Self-hosted unit cost | fixed monthly cost ÷ monthly tokens |
The numbers to carry
| Thing | Value |
|---|---|
| 70B, L=80, H_kv=8, d=128, fp16 | 320 KiB/token → 2.5 GiB per 8k sequence |
| 70B fp16 weights | ~130 GiB — does not fit one 80 GiB card |
| 8×80 GiB node, 8k context | 183 concurrent sequences |
| Same model, MHA (64 KV heads) | 22 · MQA (1 head) → 1 467 |
| Typical ridge point | ~300 FLOPs/byte |
| Batch-1 decode intensity | ~1 → ~0.3% of the GPU's compute |
| Continuous vs static, uneven outputs | 2.8× shorter wall-clock, 5× lower mean latency |
| PTU break-even, 10% → 50% output mix | 77% → 40% utilization |
One-liners
- Concurrency is memory, not FLOPs.
- GQA is the biggest lever — 64 → 8 KV heads is 8× the concurrency, same model.
- A 70B model at fp16 needs tensor parallelism to exist, not to be fast.
- Batch-1 decode wastes 99.7% of a GPU. The fix is batching, not more GPUs.
- Retire before admit — that is what makes batching continuous.
- Admit on the final length, or the batch OOMs and kills half-served requests.
- A request bigger than the whole budget is rejected, not queued.
- PagedAttention = KV in blocks, on demand → no fragmentation, safe over-commit, prefix sharing.
- Never quote a PTU break-even without the mix.
- Size the floor to p50 and spill — degrade in cost, not availability.
- Self-hosting is a utilization bet, and the engineering line decides it.
- Smallest TP degree that fits, not the largest the node allows.
Vocabulary
KV cache · stored keys/values so decode is linear not quadratic. GQA / MQA · fewer KV heads. Prefill / decode · parallel prompt processing vs sequential generation. TTFT · time to first token (prefill-dominated). ITL / TPOT · inter-token latency (decode-dominated). Arithmetic intensity · FLOPs per byte. Ridge point · where memory-bound becomes compute-bound. Continuous / in-flight batching · retire and admit every step. PagedAttention · block-allocated KV. Preemption · swapping a sequence out under memory pressure. Chunked prefill · splitting a long prompt so it does not stall decode. PTU · a unit of dedicated Azure OpenAI capacity. Spillover · overflow from dedicated to PAYG. TP degree · how many devices a model is split across.
War stories
"The GPUs are too slow." A self-hosted deployment serving one request at a time. Arithmetic intensity ~1 against a ridge point of ~300. The team's proposal was to double the fleet; the fix was a serving-stack config change.
The batch that OOMed at 3 a.m. Admission control budgeted the prompt length. Sequences grew, memory ran out, and the OOM killed the entire in-flight batch — including twenty requests that were 90% done.
The PTU commitment sized to peak. A year of capacity that ran at 22% utilization. The break-even was 57%; nobody had computed it, and nobody had asked what the output mix was.
The break-even that moved. A team computed 40% utilization using a 50% output mix. Then their agents started stuffing retrieved context into prompts, the mix fell to 10%, and the real break-even was 77%. The commitment had been signed.
The self-hosting case that won and then lost. GPU hours only. Eighteen months later the platform team had two engineers permanently on inference operations, and the true unit cost was triple the business case.
The shared deployment. Interactive chat and overnight batch on one endpoint. Whichever batch size was configured, one tier was wrong — and it was always the interactive one that complained.
Beginner mistakes
- Sizing concurrency from FLOPs.
- Using attention heads instead of KV heads (8× error on a GQA model).
- Forgetting the working-space reserve, then OOMing.
- Assuming a 70B fp16 model fits on one card.
- Benchmarking batch-1 decode and blaming the hardware.
- Static batching with variable output lengths.
- Admitting on prompt length.
- Queueing a request that can never fit.
- Quoting a PTU break-even with no mix.
- Treating "tokens per PTU" as a datasheet constant.
- Sizing dedicated capacity to peak instead of p50.
- A self-hosting case with only GPU hours.
- Maximizing TP degree.
- Sharing one deployment between interactive and batch tiers.
What "good" sounds like
"Concurrency is memory: KV per token is 2·L·H_kv·d·b, so 320 KiB/token for that model, 2.5 GiB per 8k conversation, and after weights and a 10% working reserve an 8-way node holds about 183 of them. The self-hosted endpoint is slow because batch-1 decode runs at ~1 FLOP/byte against a ridge point near 300 — continuous batching, not more GPUs. On capacity I'd size the dedicated floor to p50 and spill the peak to pay-as-you-go so we degrade in cost rather than availability, and I won't quote a break-even without the output mix, because it moves from 40% to 77% utilization between a 50% and a 10% mix. Self-hosting crosses over somewhere around 4–5B tokens a month on our numbers — but for restricted data it isn't a cost decision at all, it's the only admissible option, and then the utilization arithmetic is an obligation rather than a choice."