Lab 05 — Size a Deployment

Goal: given a workload and an SLO, produce a one-page sizing memo — model choice, quantization, parallelism, #accelerators, $/token, and TCO — that you could defend to a customer's CFO. No hardware required; this is pure analysis and the single most interview-relevant lab (K08 §11).

Knowledge prereq: K08 — Capacity Sizing & Solutions Architecture, K05 — Distributed Inference, K01 — Inference Internals.

Stack: a spreadsheet or sizing_calculator.py (pure Python, no deps).

Steps

  1. Pick a scenario (or invent one): e.g., "Serve Llama-3-70B, 1k-in/400-out, 200 req/s peak, p95 TTFT < 1s, p95 TPOT < 40 ms, on H100-80GB, budget-sensitive."
  2. Compute the memory budget (K01 §3,11): weights (after quantization) + KV cache at target batch/context + overhead. Run sizing_calculator.py to get KV/token and the per-GPU concurrency ceiling.
  3. Choose parallelism (K05 §9): does it fit one chip? If not, TP up to the NVLink domain, then PP across nodes, then DP to scale throughput. Quantize before you shard.
  4. Estimate per-instance goodput at the SLO (cite Lab 03 numbers or a defensible assumption), then #instances = ⌈peak throughput / goodput⌉ × headroom (1.3–2×) → total accelerators → whole nodes.
  5. Compute $/1M tokens and monthly TCO (K08 §3): include power/perf-watt, utilization, and the assumptions. The calculator does the arithmetic.
  6. Build the trade-off frontier (K08 §5): at least 3 options (e.g., FP16 / INT8 / rightsized-smaller-model), each with #GPUs, $/token, and accuracy implication. Recommend one and justify.
  7. List the assumptions and risks explicitly (K08 §2,4): the I/O lengths, the goodput estimate, peak-vs-average, autoscaling lag, accuracy-validation plan.
  8. Write the memo in the K08 §11 format. One page. Defensible. That's the deliverable.

What you must be able to explain

  • Why you quantize before sharding (fewer chips) and size to the SLO, not the average.
  • Why the binding constraint might be KV memory or power, not FLOPs (K01 §3, K08 §3).
  • Why you'd recommend a smaller model if it clears the floor (K08 §5).

Result / résumé bullet

"Authored defensible inference-sizing memos translating SLOs into accelerator counts, parallelism strategy, and $/token across a 3-option trade-off frontier — including a model-rightsizing recommendation that cut projected GPU spend ~60% while clearing the accuracy floor."