Lab 03 — Autoscaling, Cold Starts & Deployment Economics

Phase: 20 — Custom Models in Production Difficulty: ⭐⭐⭐☆☆ (the arithmetic is small; the conclusions are what get argued about in design reviews) Time: 3–4 hours

This is the lab that decides whether your beautifully fine-tuned model ever ships. Three questions arrive in every deployment review — which GPU, how many, and is this cheaper than just calling the API? — and all three are arithmetic that people guess at. You build the calculator: KV-cache and VRAM sizing that picks a SKU (and shows that quantizing an 8B model moves you from an A100 to an L4, a 3× price cut that has nothing to do with quality), a discrete-time autoscaler with cold starts, a scale-down delay, and scale-to-zero — where the invariant is that no request is ever lost, only delayed — and the self-host vs. API model whose most valuable output is the uncomfortable one: against a cheap hosted small model, no volume ever makes self-hosting cheaper on token price, because a second GPU adds cost and capacity in the same proportion.

What you build

• kv_cache_gb(layers, kv_heads, head_dim, tokens, bytes) — 2·L·H_kv·d·b·T. It is n_kv_heads, not n_heads: GQA is why long context fits on one card.
• model_vram_gb(params_b, bytes_per_param, kv_gb, overhead_gb) — weights + KV pool + the 1–3 GB of runtime overhead everyone forgets right before they OOM.
• pick_gpu(required_gb) — cheapest SKU that fits, ties broken toward more VRAM; raises when nothing fits, which is the signal to quantize (P06) or shard (P10).
• Autoscaler — ticks of tick_s, replicas that boot for cold_start_s before serving, a queue, desired = clamp(ceil(backlog/target), min, max), immediate scale-up, delayed scale-down, and scale-to-zero when min=0.
• summarize(records, $/hr) — replica-seconds, cost, peak queue, ticks at zero replicas, and the conservation check (arrivals == served + queued).
• usd_per_1m_tokens, break_even_analysis, cost_compare — the money model, with utilization as a first-class term and integer GPUs making the self-hosted curve a step function.

Key concepts

Files

Run

pip install -r requirements.txt
pytest test_lab.py -v                       # against your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # against the reference
python solution.py                          # the worked example

Success criteria

• All 58 tests pass against your implementation (and LAB_MODULE=solution).
• You can explain why test_no_request_is_ever_lost and test_queue_conservation_holds_tick_by_tick are the soul tests of the scaler.
• You can explain test_scale_down_waits_for_the_delay and describe the flapping failure mode it prevents (scale down → burst returns → cold start → scale up → repeat, with every cycle costing a full cold start of queued latency).
• You can explain test_break_even_is_infeasible_when_self_hosting_costs_more_per_token in one sentence to a finance partner.
• You can explain test_halving_utilization_doubles_the_token_cost and name three levers that raise utilization (batching, multi-tenancy via Lab 02, off-peak batch work).
• Given "Llama-3-8B, fp16, 100k cached tokens", you can name the VRAM (~29 GB), the cheapest SKU (A100), and what int4 changes (~18 GB → L4).
• You can state the trade in test_warm_pool_trades_cost_for_latency as a number: what min_replicas=2 costs per month, and what it buys in peak queue depth.

How this maps to the real stack

Limits of the miniature (be honest in the interview): this is a fluid queue model — every request costs one unit and finishes inside a tick, so there is no token-level service time, no prefill/decode split, no per-request latency distribution, and no notion of p50/p99 beyond queue depth. Real autoscalers act on smoothed, delayed metrics (a 30–60 s window), so they react later and overshoot more than this one does. GPU prices vary by provider, region, commitment, and spot availability by 2–5×, and tokens_per_s depends on batch size, sequence length, and quantization — measure it, never assume it. Finally, cost_compare prices only GPU-hours: it ignores engineering time, on-call, storage, egress, and the eval/observability stack, which is often the real reason the API wins at small scale.

Extensions (build these for real)

• Replace the fluid queue with a discrete-event simulator: per-request prefill and decode times drawn from a fixed table, and report real p50/p95/p99.
• Add metric smoothing and a decision delay (a 30 s trailing window) and watch overshoot and flapping appear — then tune the window against cold-start time.
• Add spot/preemptible replicas with a preemption schedule and a fallback to on-demand; measure the cost saving against the reliability cost.
• Feed the real tokens_per_s you measure from a vllm serve benchmark (vllm bench serve or a small load script) at several batch sizes into usd_per_1m_tokens and plot cost vs. batch size.
• Combine with Lab 02: give each replica an adapter cache and route with tenant affinity, then show utilization (and therefore $/1M) improving.
• Add a request deadline: drop or 429 a request that has waited longer than T, and watch conserved become a conscious trade instead of an invariant.

Interview / resume

• Talking points: "How do you size the GPU for a 70B model at 32k context?" "Why does your autoscaler scale up faster than it scales down?" "What does scale-to-zero actually cost your users?" "At what volume should we stop calling the API?" — and the senior follow-up: "and when is that the wrong question?" "Your $/1M doubled with no code change — what happened?" (utilization).
• Resume bullet: Built the deployment-economics model for a custom-LLM platform — KV-cache/VRAM sizing with automatic GPU-SKU selection (showing int4 quantization moving an 8B replica from A100 to L4, −69% GPU cost), a discrete-time autoscaler with cold starts, anti-flapping scale-down delay and scale-to-zero under a strict no-request-lost invariant, and a self-host-vs-API break-even analysis that treats utilization and integer GPU capacity as first-class terms; 58-test deterministic suite.