« Phase 05 · Track Overview

Warmup — Serving, Capacity & Inference Economics, From Zero

Assumes arithmetic and Phase 00's budget thinking. Assumes nothing about GPUs, transformers' memory behaviour, batching, or provisioned capacity.

Table of Contents

• 1. What happens when you call a model
• 2. The KV cache
  • 2.1 Why it exists
  • 2.2 Its size, derived
  • 2.3 GQA and MQA — the biggest lever
  • 2.4 The memory budget
• 3. Prefill and decode
  • 3.1 Two phases, two bottlenecks
  • 3.2 Arithmetic intensity and the roofline
  • 3.3 TTFT and inter-token latency
• 4. Batching
  • 4.1 Why batching works
  • 4.2 Static batching and its waste
  • 4.3 Continuous batching
  • 4.4 Admission control
  • 4.5 PagedAttention
• 5. Tensor parallelism
• 6. Capacity types
  • 6.1 Managed pay-as-you-go
  • 6.2 Provisioned throughput / PTUs
  • 6.3 The break-even, derived
  • 6.4 Spillover
  • 6.5 Self-hosting
• 7. Sovereignty and compliance as serving constraints
• 8. Lab walkthrough
• 9. Success criteria
• 10. Common mistakes
• 11. Interview Q&A
• 12. References

1. What happens when you call a model

You send a prompt; you get tokens back. Underneath, two very different things happen:

1. Prefill — the model processes your entire prompt in parallel, producing one output token and a cache of intermediate state.
2. Decode — the model produces the remaining output tokens one at a time, each one attending to everything before it.

That "one at a time" is not a software limitation. Token n+1 depends on token n, so decode is inherently sequential. Almost every fact in this phase follows from prefill being parallel and decode being sequential.

2. The KV cache

2.1 Why it exists

At each decode step the model must attend to every previous token. Recomputing the attention keys and values for the whole history at every step would make generation quadratic in output length — generating 1 000 tokens would cost ~500 000 token-equivalents of work.

So the model caches the keys and values it has already computed. Each new token computes its own K and V, appends them to the cache, and attends against the whole cache. Decode becomes linear instead of quadratic.

The cost of that speedup is memory, and that memory is the binding constraint on how many requests a GPU can serve at once.

2.2 Its size, derived

For one token, one layer, one KV head: you store a key vector and a value vector, each of head_dim elements. So:

$$\text{bytes per token} = \underbrace{2}{K \text{ and } V} \times L \times H{kv} \times d_{head} \times b$$

where L is layers, H_kv is KV heads, d_head is head dimension, and b is bytes per element (2 for fp16/bf16).

Worked, for a 70B-class model with L=80, H_kv=8, d_head=128, b=2:

$$2 \times 80 \times 8 \times 128 \times 2 = 327,680 \text{ bytes/token} \approx 320\text{ KiB}$$

An 8 192-token sequence therefore holds 2.5 GiB of KV cache. Read that again: one conversation, 2.5 GiB. That is why concurrency is a memory question.

Total KV = bytes/token × sequence length × batch size. Linear in both, which is the whole story.

2.3 GQA and MQA — the biggest lever

Notice the formula uses KV heads, not attention heads. Classic multi-head attention (MHA) has one KV head per query head. Two variants shrink that:

From the lab, on the same hardware at 8 192 tokens: MHA gives 22 concurrent sequences, GQA-8 gives 183, MQA gives 1 467. Same model size, same GPU, an 8× and then a 67× difference in how many customers you can serve.

This is why every modern serving-oriented model uses GQA, and it is the single most useful thing to know when someone asks why one model serves more cheaply than another of the same parameter count.

2.4 The memory budget

GPU memory
  − model weights              (params × bytes_per_param)
  − working space              (activations, context, fragmentation — reserve ~10%)
  = KV budget
  ÷ KV bytes per sequence      (bytes/token × sequence length)
  = max concurrent sequences

Worked, 70B at fp16 on one 80 GiB card: weights are 140 GB ≈ 130 GiB. That is more than the card holds. The model does not fit at all, and tensor parallelism (§5) is not an optimization here — it is a precondition.

On an 8-way group: 640 GiB − 130 GiB = 510 GiB, reserve 10% → 459 GiB for KV → 183 concurrent 8k sequences.

The working-space reserve is not optional. Activations, the CUDA context, allocator fragmentation and the framework itself all take memory. A planner that assumes 100% of the remainder is usable will over-admit and OOM under load — and an OOM kills the in-flight batch, not just the new request, which is the worst failure mode available.

3. Prefill and decode

3.1 Two phases, two bottlenecks

Prefill processes N tokens in parallel. Work ≈ \( 2 \times \text{params} \times N \) FLOPs (one multiply and one add per parameter per token). This is a big matrix multiply: the GPU's compute units are the constraint. Compute-bound.

Decode produces one token. Work ≈ \( 2 \times \text{params} \) FLOPs — tiny. But to do it, the GPU must read every weight from memory: 140 GB of traffic to produce one token. At ~3.35 TB/s that is ~42 ms of pure memory movement for ~0.00003 ms worth of arithmetic. Memory-bandwidth- bound, overwhelmingly.

3.2 Arithmetic intensity and the roofline

Arithmetic intensity = FLOPs performed ÷ bytes moved. Compare it to the GPU's ridge point = peak FLOPs/s ÷ memory bandwidth. Below the ridge you are memory-bound; above it, compute-bound.

For the lab's accelerator: \( 989 \times 10^{12} / 3.35 \times 10^{12} \approx 295 \) FLOPs per byte.

Decode at batch 1 has an intensity of ~1. You are running at roughly 0.3% of the machine's compute capability. That single number explains why a naive self-hosted deployment feels slow and expensive, and why the answer is not "buy more GPUs."

Batching raises it:

Still below the ridge at 128 — decode is very hard to make compute-bound — but per-token cost has fallen 75×.

3.3 TTFT and inter-token latency

• TTFT (time to first token) is dominated by prefill, so it scales with prompt length. A 20 000-token document produces a slow first token no matter how fast your GPU generates.
• ITL (inter-token latency) is dominated by decode, so it scales with batch size and context length, not prompt length.

Consequences for an agent platform:

• Long prompts hurt TTFT, which is what a user feels. This is a direct argument for prompt compaction (Phase 01) and for prefix caching (Phase 04).
• Large batches improve throughput and worsen individual ITL. That trade-off is a serving configuration decision, and it is exactly the knob to turn when an interactive tier and a batch tier share a deployment. Usually: don't share.

4. Batching

4.1 Why batching works

At each decode step the GPU reads:

$$\text{bytes} = \underbrace{\text{weights}}{\text{read once for the whole batch}} + \underbrace{B \times \text{KV}}{\text{per sequence}}$$

The weight read is amortized across the batch; the KV read is not. So per-token time is:

$$t = \frac{\text{weights} + B \cdot \text{KV}}{\text{bandwidth} \times B}$$

At small context, weights dominates and doubling B halves per-token time. At long context, B · KV dominates and batching stops helping — which is exactly the regime a long-context agent platform lives in, and a good reason to care about context length as a cost variable.

4.2 Static batching and its waste

The obvious implementation: collect requests until you have a batch, run it to completion, start the next.

The problem is output-length variance. A batch of 16 where one generation is 400 tokens and fifteen are 20 tokens runs for 400 steps, and for 380 of them fifteen slots produce nothing. You have paid for a batch of 16 and received the throughput of a batch of 1.

In an agent platform this is the normal case, not an edge case: some turns are "yes, released", some are a three-page investigation summary.

4.3 Continuous batching

Also called in-flight batching. Every decode step:

1. retire sequences that finished;
2. admit queued requests into the freed slots;
3. run one decode step for whatever is now in the batch.

A request no longer waits for the batch it arrived with. The short generation retires at step 20 and its slot immediately serves someone else.

The lab measures it: the same uneven workload takes 1 220 ticks static and 440 ticks continuous — 2.8× faster, with mean latency falling from 337 to 68 ticks. Same hardware, same model, same work. This is the single biggest throughput win in modern LLM serving and it is why vLLM and TGI exist.

Note the ordering in step 1–2: retire before admit. Reusing freed capacity in the same step is what makes it continuous; a scheduler that admits only at the start of a step wastes a step of capacity every time something finishes.

4.4 Admission control

You cannot admit a request whose KV will not fit. Two rules:

Budget the final length, not the prompt. A sequence that fits at 4 000 prompt tokens will need room for 4 000 + its output. Admitting on the prompt over-admits, and the batch OOMs mid-flight — killing requests that were already half-served. The lab tests exactly this.

A request larger than the whole budget must be rejected, not queued. It will never fit at any batch size, and queueing it forever is a memory leak with a customer attached. Reject it early with a clear error so the caller can chunk, summarize, or route to a larger deployment.

(Real schedulers soften the first rule via paging — see §4.5 — but the reasoning is unchanged: something must bound admission, or you OOM.)

4.5 PagedAttention

vLLM's contribution, and the reason it displaced everything else. Instead of reserving a contiguous KV region for each sequence's maximum length, it stores KV in fixed-size blocks, allocated on demand — exactly like OS virtual-memory pages.

Three consequences:

1. Fragmentation nearly vanishes. Contiguous reservation wastes everything between actual and maximum length; blocks waste at most one partial block per sequence.
2. Over-commitment becomes safe, because you allocate as sequences grow rather than up front — with preemption (swap a sequence out, recompute or restore later) as the escape valve when memory runs out.
3. Prefix sharing becomes possible: two sequences with the same system prompt can point at the same blocks, which is the mechanism behind provider-side prompt caching.

Our lab reserves the final length contiguously, which is simpler and strictly more conservative. Knowing the difference is the point: it is why a real system can run at higher occupancy than the naive arithmetic suggests.

5. Tensor parallelism

A 70B model at fp16 needs ~130 GiB of weights. No single accelerator holds that, so the model is split across devices: each GPU holds a slice of every layer's weights, computes its part, and the results are combined with an all-reduce at each layer.

Modelled simply: memory, bandwidth and FLOPs all scale with the group size. The lab does exactly that.

What the simple model ignores — and what you should mention when using it — is the communication cost. Every layer needs an all-reduce, so TP degree is limited by interconnect bandwidth, and scaling is sub-linear. The practical rule: choose the smallest TP degree that fits the model plus a useful KV budget, not the largest the node allows. Going wider costs latency to buy memory you may not need.

(Pipeline parallelism splits layers across devices instead, and data parallelism replicates the whole model. In serving, TP within a node and DP across nodes is the common shape.)

6. Capacity types

6.1 Managed pay-as-you-go

Per-token billing on the provider's shared pool.

Wins: zero commitment, instant elasticity, no operations, the newest models first. Costs: variable latency (you queue behind everyone else), 429s when the pool is busy, and a unit price that is the highest of the three options.

The right default at low or unpredictable volume, and the right spill target at any volume.

6.2 Provisioned throughput / PTUs

You buy dedicated capacity — Azure OpenAI calls the unit a PTU, AWS Bedrock calls it provisioned throughput with model units — for a fixed monthly cost.

Wins: predictable latency (no shared-pool congestion), no 429s within capacity, and a lower effective unit price at high utilization. Costs: a commitment, capacity that is idle when you are not using it, and a bet that the model you committed to will still be the one you want.

The most common mistake: treating "tokens per unit" as a datasheet constant. It depends on your input:output mix, because prefill and decode consume capacity differently. Measure it with your own traffic shape before sizing.

6.3 The break-even, derived

Let \( C_p \) be the monthly cost of the dedicated capacity and \( c_b \) the blended PAYG price per 1 000 tokens:

$$c_b = c_{in}(1-f) + c_{out},f$$

where \( f \) is the fraction of tokens that are output. Then:

$$\text{break-even tokens} = \frac{C_p}{c_b} \times 1000 \qquad \text{break-even utilization} = \frac{\text{break-even tokens}}{\text{capacity}}$$

Worked, from the lab: \( C_p \) = $12 000/month, capacity 4B tokens, input $3/1M, output $12/1M:

The break-even utilization nearly halves between a 10% and a 50% output mix. Never quote a single break-even number without stating the mix — and note that agents have unusually high output fractions when they generate long structured plans, and unusually low ones when they stuff retrieved context into a prompt to produce one word.

Two things the arithmetic does not capture, and you should say so:

• Latency is often the real reason for dedicated capacity, not cost. Removing shared-pool congestion and 429s can justify it well below break-even.
• A commitment is a bet on the model. Price the exit: what happens if the model is deprecated or superseded mid-term?

6.4 Spillover

Almost every mature deployment converges on the same shape: size the dedicated floor to p50 demand, spill everything above it to PAYG.

• Sizing to peak buys capacity that idles most of the month.
• Sizing to p10 means you are on the shared pool most of the time and get 429s.
• Sizing to p50 keeps the floor well utilized and converts peak demand into cost rather than unavailability.

That last phrase is the principle: degrade in cost, not in availability. It is the same instinct as Phase 00's degradable dependency, applied to capacity.

6.5 Self-hosting

Your own GPUs running an open-weight model on vLLM / TGI / Triton.

Wins: the lowest unit cost at high utilization; full control over residency and sovereignty; no third-party processing of your data; no deprecation risk. Costs: you now operate GPUs. Scheduling, node pools, driver and CUDA versions, model updates, evaluation, capacity planning, and an on-call rotation for an inference service.

The arithmetic is a utilization bet, because the cost is fixed and the volume is not:

$$\text{unit cost} = \frac{\text{monthly fixed cost}}{\text{monthly tokens}}$$

From the lab, 8 GPUs at $3/hour plus $25 000/month of engineering = $17 545/month:

The engineering line decides it. A business case for self-hosting that counts only GPU hours is the standard way this decision is won and then regretted. In a bank the honest number includes the platform engineers, the security review, the model-evaluation work, and the on-call cost — and even then, sovereignty is often the real driver rather than price.

7. Sovereignty and compliance as serving constraints

The JD lists "cost, latency, sovereignty, and compliance" together, and the last two frequently override the first two.

The design consequence: your capacity strategy is downstream of your data classification. Restricted workloads may have exactly one admissible option, and if that option is self-hosting, the utilization arithmetic in §6.5 becomes an obligation rather than a choice. This is why routing in Phase 04 treats residency and classification as gates that a cost-based rule cannot override.

8. Lab walkthrough

Work Lab 01 in this order.

1. ModelShape, GPU (§2, §5). kv_bytes_per_token is the formula; the aggregate properties are the TP model.
2. usable_kv_bytes, max_concurrent_sequences (§2.4). Floor at 0 when the model does not fit — the test asserts a 70B fp16 model yields 0 on one card.
3. prefill_ms, decode_ms_per_token, arithmetic_intensity, ridge_point (§3). The two decode tests are the lesson: with no context, batching is exactly proportional; with a long context, it is not.
4. ContinuousBatcher._projected and run (§4.3–4.4). Order inside the loop: arrive, retire, admit, terminate-check, step. Reject the never-fits case rather than queueing.
5. StaticBatcher.run (§4.2). Everyone in a batch finishes at start + max(output_tokens).
6. Pricing types and break_even (§6.3). Blended price first, then the division.
7. plan_with_spillover, SelfHostedPricing, cheapest_option (§6.4–6.5). Ties go to PAYG.

Then python solution.py and read the six sections against §§2–6.

9. Success criteria

Without the guide open:

• Write 2·L·H_kv·d·b and compute KV bytes for a given model and context.
• Explain why concurrency is a memory question, and do the budget subtraction.
• Explain what GQA changes and quantify it.
• Explain why prefill is compute-bound and decode is memory-bound.
• Give the arithmetic intensity of batch-1 decode and compare it to a ridge point.
• Explain why batching works, in one expression.
• Explain when batching stops working.
• Describe continuous batching and say why its win scales with output-length variance.
• State both admission rules and the failure each prevents.
• Explain what PagedAttention changes.
• Derive a PTU break-even and explain why the output fraction moves it.
• Lay out floor-plus-spillover and say what it optimizes.
• List what belongs in a self-hosting business case beyond GPU hours.

10. Common mistakes

Sizing concurrency from FLOPs. It is memory. Every time.

Forgetting the working-space reserve. You over-admit and OOM the batch, killing in-flight requests.

Using attention heads instead of KV heads. Your KV estimate is 8× too large on a GQA model.

Assuming a 70B model fits on one card. At fp16 it needs ~130 GiB.

Running batch-1 decode and concluding the GPU is slow. It is running at ~0.3% of its compute.

Static batching with variable output lengths. You paid for a batch and got the throughput of one request.

Admitting on prompt length. The batch OOMs mid-flight.

Queueing a request that can never fit. A leak with a customer attached.

Quoting a PTU break-even without the traffic mix. It moves by nearly 2× between a 10% and 50% output fraction.

Treating "tokens per PTU" as a constant. Measure it with your own mix.

Sizing dedicated capacity to peak. You bought idle capacity; size to p50 and spill.

A self-hosting case with only GPU hours. The engineering line is the one that decides it.

Maximizing TP degree. Communication cost is real; use the smallest degree that fits.

Sharing one deployment between interactive and batch tiers. Their batch-size preferences are opposite.

11. Interview Q&A

Q: How many concurrent users can one GPU serve?

A: "It's a memory question, not a compute one, and the formula is KV bytes per token equals 2 × layers × KV heads × head dim × bytes per element. For a 70B-class model with 80 layers, 8 KV heads and head dim 128 at fp16 that's 320 KiB per token — so an 8 000-token conversation holds 2.5 GiB of KV cache. Then it's arithmetic: total memory, minus 130 GiB of weights, minus about 10% for activations and fragmentation — and note that at fp16 a 70B model doesn't fit on one 80 GiB card at all, so tensor parallelism is a precondition rather than an optimization. On an 8-way group you get about 459 GiB of KV budget, which is 183 concurrent 8k sequences. The lever nobody mentions is the attention layout: the same model with MHA instead of 8-way GQA gives 22 sequences instead of 183, on identical hardware."

Q: Why is our self-hosted deployment slower than the managed API?

A: "Almost certainly because it's running batch-1 decode. Decode is memory-bandwidth-bound — to produce one token the GPU reads every weight from memory, so 140 GB of traffic for about 140 GFLOPs of arithmetic. That's an arithmetic intensity of roughly 1 FLOP per byte against a ridge point around 295, so you're using about 0.3% of the machine's compute. The fix is continuous batching, not more GPUs: the weight read amortizes across the batch while the KV read doesn't, so per-token time falls roughly proportionally until the KV term takes over. On our simulator, moving an uneven workload from static to continuous batching cut wall-clock 2.8× and mean latency 5×. The second thing I'd check is whether prefill is being blocked by long prompts — that's a TTFT problem with a different fix, chunked prefill."

Q: Should we buy provisioned capacity?

A: "Only as a break-even utilization, and I'd refuse to give a single number without the traffic mix. The blended pay-as-you-go price is input price times one-minus-the-output-fraction plus output price times the output fraction, and output is typically four times input — so at a 10% output mix the break-even is around 77% of the committed capacity, and at 50% it's 40%. Nearly a 2× swing on mix alone. I'd also push back on 'tokens per unit' as a datasheet number: it depends on your input:output shape, so measure it with your own traffic before sizing. Then the shape I'd actually propose is a floor plus spillover — size the dedicated capacity to p50 demand so it stays utilized, and spill peaks to pay-as-you-go, because that converts peak demand into cost rather than into 429s. Degrade in cost, not in availability. And two things the arithmetic misses: latency is often the real reason to buy dedicated capacity rather than price, and a commitment is a bet on the model still being the one you want — so price the exit."

Q: When would you self-host?

A: "Three reasons, in decreasing order of how often they're the real one. First sovereignty — if restricted data can't be processed by a third party or can't leave the jurisdiction and no provider offers an in-region deployment, self-hosting isn't a cost decision, it's the only admissible option. Second, deprecation and reproducibility: you control the weights, so an auditor's 'reproduce this decision from six months ago' has an answer. Third, unit cost at high utilization — and that's a bet, because a fixed monthly cost divided by a variable volume means an idle GPU costs the same as a busy one. On our numbers, 8 GPUs plus a realistic engineering line is about $17.5k a month, which is $17.50 per million tokens at 1B tokens and $0.88 at 20B. So it crosses over somewhere around 4–5B tokens a month. The line that decides it is the engineering cost, and a business case that counts only GPU-hours is how this decision is won and then regretted — you're taking on scheduling, driver versions, model updates, evaluation, and an on-call rotation for an inference service."

Q: What does PagedAttention actually change?

A: "It stores the KV cache in fixed-size blocks allocated on demand rather than reserving a contiguous region for each sequence's maximum length — the same idea as OS virtual memory pages. Three consequences. Fragmentation almost disappears, because you waste at most a partial block per sequence instead of everything between actual and maximum length. Over-commitment becomes safe, because you allocate as sequences grow, with preemption as the escape valve. And prefix sharing becomes possible, since two sequences with the same system prompt can point at the same blocks — which is the mechanism behind provider-side prompt caching. Practically it means a real system runs at higher occupancy than the naive contiguous arithmetic suggests, so if I'm doing capacity planning with the simple model I state that it's conservative."

Q: You have one deployment serving an interactive chat tier and an overnight batch tier. Thoughts?

A: "Separate them. They want opposite batch sizes: the batch tier wants the largest batch it can get, because throughput per GPU-hour is all that matters and inter-token latency is irrelevant; the interactive tier wants a small batch, because a large one increases everyone's inter-token latency even as it improves aggregate throughput. Sharing means one of them is always configured wrong. If they must share hardware, then at minimum separate the queues with priority and cap the batch size the interactive tier can be pulled into — and I'd want the degradation ladder to say explicitly that batch work is what gets shed first when the interactive tier is under pressure."

12. References

Serving systems

• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023 — the vLLM paper. Read it; it is short and it is the reference for §4.5.
• Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models, OSDI 2022 — the origin of iteration-level (continuous) scheduling.
• Agrawal et al., Sarathi-Serve / chunked prefill — how prefill and decode are interleaved without stalling.
• vLLM, Hugging Face TGI, and NVIDIA Triton / TensorRT-LLM documentation — the configuration surface these ideas appear as in practice.

Attention and memory

• Shazeer, Fast Transformer Decoding: One Write-Head is All You Need, 2019 — MQA.
• Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models, 2023.
• Dao et al., FlashAttention — why attention is memory-bound and what tiling does about it.

Performance modelling

• Williams, Waterman & Patterson, Roofline: An Insightful Visual Performance Model, CACM 2009 — arithmetic intensity and the ridge point.
• Chip Huyen and others have written good practitioner summaries of LLM inference economics; treat any specific price or throughput figure as perishable and re-derive with current numbers.

Capacity

• Azure OpenAI provisioned throughput / PTU documentation, including the capacity calculator and spillover deployments.
• AWS Bedrock Provisioned Throughput documentation, including model units and commitment terms.
• Your own provider's current price list — every number in this guide is illustrative and will be wrong by the time you read it. The method is what transfers.