Warmup Guide — Capstone: the End-to-End Multimodal Agentic Serving Platform

Zero-to-senior primer for the finale. The earlier warmups each taught one mechanism from first principles. This one teaches the method that composes them: how to take a workload, pull each subsystem from the phase that built it, budget memory/latency/cost, check the SLO, name the tradeoff, and try to break it. We walk the whole design method, build a worked reference design (a multimodal agentic assistant) with the real KV/SLA/cost math, analyze its failure modes, and ship it with an eval-gated rollout. The lab turns every chapter into code. Nothing here is new physics — it is Phase 00's arithmetic, applied to the whole stack.

Table of Contents

• Chapter 1: What the Capstone Tests — Composition Under Constraint
• Chapter 2: The Design Method (Every Time)
• Chapter 3: Workload Facts First
• Chapter 4: The Model Strategy — Pulling P00–P07 Into the Spec
• Chapter 5: Memory — Weights, the KV-Cache, and the Fit (P00/P02/P09)
• Chapter 6: Latency Composes Additively (P09/P11/P12)
• Chapter 7: Throughput, Speculative Decoding & Cost (P00/P09)
• Chapter 8: Quality, Safety & the SLO Gate (P11/P12/P14/P16)
• Chapter 9: The Decision — Filter, Rank, Name the Deciding Dial (P00 Grown Up)
• Chapter 10: A Worked Reference Design
• Chapter 11: Fail It — The Failure-Mode Analysis
• Chapter 12: The Production Rollout (P16/P17)
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: What the Capstone Tests — Composition Under Constraint

From zero. You have spent eighteen phases building mechanisms: a BPE tokenizer, attention with a KV-cache, an autograd tape, a CLIP-style VLM, a LoRA delta, a quantizer, the DPO loss, a constrained sampler, a PagedAttention block manager, a ring all-reduce, an HNSW index, a ReAct loop, a multi-agent bus, a neurosymbolic verifier, a VLA decoder, an eval harness, an MLOps gate, and a roofline model. Each is a part. The capstone asks the question a real job asks: can you assemble the parts into a system that meets a contract, and prove it?

Why this is a distinct skill. Knowing how attention works does not tell you how many GPUs the platform needs. Knowing how RAG works does not tell you whether retrieval blows the TTFT budget. The senior skill — the one a system-design interview probes for an hour — is composition under constraint: choosing which mechanisms to include, budgeting each one's share of memory/latency/cost, and showing the whole clears the SLO. The capstone certifies that you can do this and drop down to debug any layer when it breaks.

The senior's habit. When asked to "design a multimodal agentic assistant," the senior does not start drawing boxes. They first write the numbers: QPS, prompt/response token sizes, TTFT/TPOT SLO, the quality bar, the safety/compliance line, the budget. Then they pick a model strategy, lay out the serving and offline paths, and quantify — KV-cache at peak, tokens/sec, $/1k, p99 latency — before they commit to anything. The boxes come last, and they come labelled with budgets.

Common misconception. "The capstone is where I show off the fanciest architecture." No — the capstone is where you show restraint. The best answer is usually the smallest composition that clears the bar: the model you can quantize onto one GPU, RAG instead of a bigger model, an agent loop only where the task is actually actionable. Multi-agent, multi-GPU, multi-region are costs you justify with a number, not defaults you reach for.

Chapter 2: The Design Method (Every Time)

There is one method, and every reference design in system-design/ follows it. Memorize the five steps; they are the spine of every answer.

1. Workload facts first — task + modality, QPS / peak concurrency, prompt and response token sizes, latency SLO (TTFT and TPOT separately), quality bar + how you eval it, safety/compliance, budget. (Chapter 3.)
2. Two paths, always —
   • Serving (online): tokenize (P01) → perceive (P04) → retrieve (P11) → model with quant/sampler/KV/batching/speculative (P02/P06/P08/P09) → agent loop (P12) → verify + guardrails (P14/P16). (Chapters 4–8.)
   • Offline: data → train (P03) → PEFT (P05) → align (P07) → quantize (P06) → eval gate (P16) → registry → canary (P17); plus the RAG indexing pipeline (P11). (Chapter 12.)
3. Quantify — tokens/sec per GPU, KV-cache bytes at peak concurrency, $/1k (or $/1M), p99 TTFT/TPOT, recall@k, eval score with a confidence interval. (Chapters 5–7.)
4. Tradeoff — quality vs latency vs cost vs memory vs safety; name what you trade and write the one-sentence decision record. (Chapter 9.)
5. Fail it — hallucination, prompt injection, tool misuse, KV-cache OOM, cost blowup, model regression, drift — and the rollback. (Chapter 11.)

Why the order matters. Each step constrains the next. The workload facts decide the model strategy; the strategy decides the memory; the memory decides the GPU count; the GPU count and throughput decide the cost; the SLO decides which candidates are even eligible. Skip step 1 and every later number is unanchored — the single most common way candidates fail the interview.

Chapter 3: Workload Facts First

What it is. Before any architecture, you nail down the request: who calls it, how often, how big the input and output are, how fast it must respond, how good the answer must be, and what it must never do. These become the inputs to every formula.

The facts and where they go in the model:

The TTFT/TPOT split is non-negotiable. "Latency < 1s" is meaningless. A user perceives two different latencies: TTFT (time to first token — the spinner) and TPOT (time per output token — the streaming smoothness). They are governed by different physics: TTFT is prefill (compute-bound) plus retrieval; TPOT is decode (bandwidth-bound). Quoting one number without saying which it is, and whether it's prefill or decode, batched or single-stream, is the fastest way to sound junior.

Common misconception. "I'll figure out the numbers once I have the architecture." Backwards. The numbers are the architecture. The KV-cache at peak concurrency decides the GPU count; the TTFT budget decides whether you can afford a reranker; the cost ceiling decides the quantization. Architecture is the consequence of the workload facts, not the other way round.

Chapter 4: The Model Strategy — Pulling P00–P07 Into the Spec

What it is. The model strategy is the set of offline decisions baked into the weights you serve: how big (P00 scaling), trained how (P03), adapted how (P05 PEFT), aligned how (P07), and served at what precision (P06). In the capstone these collapse into a few PlatformDesign knobs.

• Size (n_params) — the smallest model that clears the quality bar, per Phase 00. Bigger raises quality with diminishing returns and raises every inference cost forever. The default senior move is small + RAG + a good prompt before reaching for a bigger model.
• Precision (quant_bytes_per_param) — fp16 (2), int8 (1), int4 (0.5). This is the lever between one GPU and two (P06). It dings quality (_quant_penalty in the lab), so you quantize after measuring the eval impact, not before.
• Adaptation (P05) — the domain LoRA/QLoRA that raised base_quality cheaply, frozen into the served weights. In the model it shows up as a higher base_quality, not a runtime cost.
• Alignment (P07) — the DPO/RLHF pass that raised quality and safety to a usable level. Again baked in: higher base_quality and a better-behaved model.
• Geometry (P02) — n_layers and d_model are not just quality knobs; they size the KV-cache. A deep, wide model has a bigger cache, which is the next chapter.

The reflex. "Deploy model X" → in two minutes: weights = N·bytes (P00), pick the quant that fits with KV headroom, note the quality ding, and ask "does RAG let me use a smaller X?" The strategy is chosen against the workload facts, not in a vacuum.

Common misconception. "Fine-tune it." Often RAG (P11) or a better prompt solves the task more cheaply and more controllably — and fine-tuning is right only when the knowledge is reasoning or style, not retrievable facts. The panel listens for this exact distinction.

Chapter 5: Memory — Weights, the KV-Cache, and the Fit (P00/P02/P09)

The two terms. Resident GPU memory for serving is, to first order, weights + KV-cache (we ignore activations, as Phase 00 does). The weights are the number everyone quotes:

$$ \text{weights} = N \times \text{bytes_per_param}. $$

The KV-cache is the number everyone forgets, and the one that actually OOMs the server:

$$ \text{KV} = 2 \times n_{\text{layers}} \times \text{seq_len} \times \text{batch} \times d_{\text{model}} \times \text{bytes_per_elem}. $$

The KV-cache is the swing factor. Weights are fixed once you pick the model and precision. The KV-cache grows with batch × context — so the same design that fits comfortably at batch=1 can OOM at batch=64. This is why fits_on_gpu in the lab flips on the KV-cache, not the weights, and why the boundary test raises only the batch. In production you provision GPUs for the KV-cache at peak concurrency, then check the weights fit too — the opposite of what juniors do.

A worked number. A 70B int8 model: weights = 70e9 × 1 = 70 GB. At 40 layers... no — 80 layers, d_model 8192, 2048 context, batch 8, int8: KV = 2 × 80 × 2048 × 8 × 8192 × 1 ≈ 21.5 GB. Total ≈ 91.5 GB — won't fit one 80 GB card, needs two (or tensor-parallel, P10). Cut the context to 1024 or the batch to 4 and it fits one. That is the capacity conversation, and it is arithmetic, not opinion.

The fixes you already built. GQA (P02) shrinks the cache by the KV-head ratio; PagedAttention (P09) eliminates fragmentation so the budget is usable; KV quantization stores the cache in int8/int4; shorter context caps seq_len. Each is a knob you turn when the fit check fails.

Common misconception. "It fit in the load test." The load test ran at batch 8; production peaks at batch 60, and the KV-cache is 7× bigger. Capacity = weights + KV at peak concurrency × context + activations. Always multiply by the real peak.

Chapter 6: Latency Composes Additively (P09/P11/P12)

The core idea — and the first soul of the lab. End-to-end latency is a sum of independent stage costs. Each subsystem you add contributes its own milliseconds, and the total is the sum:

$$ \text{e2e} = \underbrace{\text{RAG_retrieve} + \text{prefill}}{\text{TTFT}} + \underbrace{\text{out} \cdot \text{TPOT}}{\text{decode}} + \underbrace{(\text{steps}-1)(\text{step_ms} + \text{out}\cdot\text{TPOT})}_{\text{agent loop}}. $$

Stage by stage:

• RAG retrieval (P11) — embed → ANN top-k → rerank. Runs before the model sees the prompt, so it lands in TTFT, not in per-token decode. Adding RAG adds exactly its retrieval ms — the lab's test_e2e_adding_rag_increases_by_exactly_retrieval_ms asserts precisely this.
• Prefill (P09) — the compute-bound pass over the prompt; FLOPs ≈ 2N·seq_len at peak FLOP/s. Also part of TTFT.
• Decode (P09) — out_tokens × TPOT, where TPOT = 1000 / decode_tokens_per_sec, bandwidth-bound.
• Agent loop (P12) — each extra ReAct step pays its orchestration overhead (agent_step_ms, the tool call + observation) and another decode pass over its reasoning tokens. The lab's test_e2e_agent_loop_adds_steps_worth asserts each step adds exactly that.

Why "additive" is the whole point. If the stages compose additively, you can budget each one independently: "retrieval gets 150 ms, prefill gets 300 ms, decode gets 256 × 4 ms, the loop gets 2 × (80 + decode)" — and if the total breaks the SLO, you know exactly which stage to cut. A latency model where stages interact unpredictably is one you cannot budget, and budgeting is the job. (Real systems add queueing and jitter on top — Chapter 11 — but the mean budget is additive, and that is what you design against.)

Common misconception. "RAG adds latency to every token." No — retrieval happens once, up front, so it inflates TTFT only. Conversely the agent loop inflates the tail, because every extra step is another full decode. Knowing where each stage lands in the latency curve is what lets you fix the right number.

Chapter 7: Throughput, Speculative Decoding & Cost (P00/P09)

Decode throughput. From Phase 00: single-stream decode re-reads every weight per token, so it is bandwidth-bound at bandwidth / weight_bytes tokens/sec. Batching reads the weights once for batch sequences, so throughput scales ~linearly with batch (until you become compute-bound or run out of KV room):

$$ \text{tok/s} \approx \text{batch} \times \frac{\text{bandwidth}}{\text{weight_bytes}}. $$

Speculative decoding (P09). A small draft model proposes several tokens; the big model verifies them in one forward pass and accepts the prefix that matches. If a fraction a of draft tokens are accepted per verify step, you get more tokens per weight read — a throughput multiplier of

$$ \frac{1}{1 - a}. $$

At a = 0.5 you roughly double tokens/sec; at a = 0.6, ~2.5×. Crucially this does not change the FLOPs — it changes the bytes-moved efficiency, which is exactly the bandwidth-bound lever Phase 00 told you to pull. The lab's test_speculative_raises_tps_by_accept_factor asserts the multiplier.

Cost. Serving cost is produced tokens ÷ throughput × GPU price:

$$ \frac{$}{1\text{k req}} = \frac{\text{out_tokens} \times \text{agent_steps}}{\text{tok/s}} \times \frac{\text{gpu_$/h}}{3600} \times 1000. $$

Two facts fall out: batching lowers $/request (throughput rises, price fixed — the lab's test_cost_decreases_with_batch shows $ ∝ 1/batch), and the agent loop multiplies cost (each step re-generates its reasoning). The agent loop is the most common cost blowup, and the fix is a hard step cap.

Common misconception. "Speculative decoding is free quality risk." It is lossless — the big model verifies, so accepted tokens are exactly what it would have produced. The only cost is the draft model's compute and the wasted work on rejected tokens; the win is real and risk-free on quality.

Chapter 8: Quality, Safety & the SLO Gate (P11/P12/P14/P16)

Quality is composed (P16 over P11/P12/P06). The platform's quality is the model's intrinsic base_quality adjusted by the composition: RAG bumps grounding (fewer hallucinations on retrievable facts), the agent loop bumps capability (it can act, not just answer), and quantization dings it (P06). In the lab:

$$ \text{quality} = \text{clamp}\big(\text{base} + \text{rag_bump} + \text{agent_bump} - \text{quant_penalty}\big). $$

These directions are the point: RAG and agents add quality, quant subtracts it — and the tests assert each moves the right way.

Safety compounds — defence in depth (P16 + P14). Guardrails (input/output filters) and verification (a neurosymbolic or critic check, P14) are complementary layers. The residual unsafe fraction is the product of what each lets through, so

$$ \text{safety} = 1 - (1 - \text{guardrails})(1 - \text{verification}). $$

Either layer alone helps; together they compound. Guardrails 0.8 + verification 0.6 → safety 1 − 0.2·0.4 = 0.92. This is why production safety is layered, never a single filter.

The SLO is the contract. meets_slo checks the composed design against the workload's SLO and returns the exact ordered list of violations — ["ttft", "cost", "memory", "safety"] — so you know precisely which subsystem to fix. An SLO with a violation list is a to-do list; an SLO that just says "fail" is useless.

Common misconception. "One guardrail is enough." A single filter has a single failure mode; a determined prompt injection or an unusual output slips it. Layering an independent verifier turns the leak into a product of two small numbers — the difference between 0.2 and 0.08.

Chapter 9: The Decision — Filter, Rank, Name the Deciding Dial (P00 Grown Up)

The second soul of the lab. Phase 00 taught the tradeoff resolver over a single model. The capstone grows it into choose() over full platforms, with one critical addition: constraints before objective.

1. Filter to SLO-meeting designs. Drop every candidate that violates the SLO and record why (its violation list). A faster, cheaper design that misses the safety floor is not a candidate — it never enters the ranking. This is the production-correct order: a design that breaks the contract cannot win on points.
2. Rank the survivors by the weight-normalized dial score, exactly as Phase 00 does:

$$ \text{score}(d) = \sum_{\text{dial}} \text{score}d[\text{dial}] \cdot \frac{w{\text{dial}}}{\sum_j w_j}. $$

(Quality/safety are absolute 0..1; latency/cost/memory are "lower is better," normalized against the survivor pool so they are comparable.) 3. Name the deciding dial — the dial whose weighted (winner − runner-up) gap is largest. That single sentence is the ADR: "We chose the int4-13B+RAG because the workload weights cost 4× and it wins decisively on cost, accepting a measured 0.18 quality gap that clears the eval bar."

The winner flips. The same two candidates — a cheap int4-13B+RAG and a high-quality int8-70B +agent — pick opposite winners under a quality-weighted vs a cost-weighted workload. The lab's test_choose_winner_flips_with_priorities is the soul test, the same lesson as Phase 00's test_weights_flip_the_winner, now over a composed platform. "Best platform" is a category error; "best for these weights, among the SLO-feasible designs" is engineering.

Common misconception. "Just pick the highest score." Not if it violates the SLO — and not without the deciding dial. A score of 0.71 vs 0.68 tells a reviewer nothing; "we traded 0.18 of quality to win 4× on cost, here's the eval showing the gap is acceptable" is an argument that survives the six-months-later "why didn't we use the bigger model?" question.

Chapter 10: A Worked Reference Design

The brief. A multimodal agentic customer-support assistant: users upload a screenshot + a question; the assistant retrieves from a product KB, reasons in a short tool-using loop (lookup, check-order), and answers with citations. Workload: 16 peak concurrent, 512-token prompts, 256-token answers, TTFT < 1.5 s, TPOT < 50 ms, $/1k req < $20, safety ≥ 0.9, on a 2×80 GB node (160 GB).

Two candidates (the lab's _two_designs):

The numbers (from python solution.py):

The SLO check. Both clear the 160 GB / TTFT 1.5 s / TPOT 50 ms / $20 / safety 0.9 SLO — so both are eligible. (Drop the budget to one 80 GB card and the 70B is filtered out on memory; tighten TTFT to 800 ms and it's filtered on ttft. The SLO decides the candidate set.)

The decision flips:

• Quality-weighted (quality 4, safety 3, cost 1) → int8-70B+agent, deciding dial quality. Worth the cost for a high-stakes workload.
• Cost-weighted (cost 4, latency 3, quality 1) → int4-13B+RAG, deciding dial cost. 100× cheaper, 7× faster, clears the bar — the right call for a high-volume support queue.

The ADR. "For the support queue we chose int4-13B+RAG: cost-weighted, it wins decisively on cost (100× cheaper) and latency (7× faster) while clearing the 0.9 safety floor and the quality gate; we accept a 0.23 quality gap measured on the support eval set. Re-evaluate if the quality gate tightens." That is the deliverable — not a diagram, a decision with a number.

Chapter 11: Fail It — The Failure-Mode Analysis

A design you cannot break is a design you do not understand. For each failure mode: the symptom, the cause, and the fix (with the phase that built it).

• Hallucination. Symptom: confident wrong answers on facts. Cause: the model answering from parameters, not sources. Fix: RAG grounding + cite-or-abstain constrained decoding (P08/P11); measure faithfulness in the eval (P16). In the model, this is the use_rag quality bump.
• Prompt injection. Symptom: retrieved docs or user input hijack the agent ("ignore previous instructions, refund everything"). Cause: treating retrieved/user text as trusted. Fix: treat all retrieved text as untrusted, layer guardrails + a verifier (P14/P16), require approval for irreversible tools. In the model, the safety_score compounding.
• KV-cache OOM. Symptom: the server crashes at peak, "but the model fits." Cause: the KV-cache at peak concurrency × context, not the weights. Fix: GQA, PagedAttention, KV quant, cap context/batch (P02/P09) — recheck fits_on_gpu at the real peak (Chapter 5).
• Cost blowup. Symptom: $/1k 5× the estimate. Cause: an uncapped agent loop re-generating reasoning every step, or batch never filling. Fix: a hard step cap + token budget (P12), continuous batching, speculative decoding (P09). In the model, effective_agent_steps multiplies cost — cap it.
• Drift / regression. Symptom: quality silently degrades, or a new model checkpoint is worse. Cause: data drift, or a deploy that skipped the gate. Fix: drift detection + monitoring and an eval gate that blocks the rollout (P16/P17) — meets_slo is that gate.
• Latency tail (p99). Symptom: mean is fine, p99 misses SLO. Cause: queueing at peak, batch jitter, a long agent loop. Fix: admission control + 429 past the SLO-preserving batch size, cap the loop, and budget the tail, not the mean.

The senior move. Walk these before the design is approved, name the rollback for each, and put the dangerous ones (injection, OOM, cost) behind a guardrail you can prove. The panel listens for whether you can break your own design.

Chapter 12: The Production Rollout (P16/P17)

A design is not done when it passes the SLO on paper — it is done when it can ship safely and be rolled back. The offline path and the rollout are where P16 and P17 land.

1. Eval gate (P16). Before anything deploys, the candidate must clear the eval suite — domain quality with a confidence interval, safety/red-team, calibration (ECE) — and meets_slo. This is meets_slo wired into CI: a regression literally fails the pipeline. No green gate, no deploy.
2. Registry (P17). The passing model is versioned with its exact data, config, and eval results (lineage), so you know what is in production and can diff two versions.
3. Canary. Route a small traffic slice to the new design; watch p99 TTFT/TPOT, $/1k, quality and safety online. If a metric regresses past threshold, roll back automatically to the previous registry version.
4. Monitor + drift (P17). Steady-state: trace every request (retrieve → prefill → decode → tools, each a span), alert on SLO breach and drift, feed failures back into the eval set.

Why this closes the loop. The capstone's choose() picks a design; the rollout is what makes that choice reversible and observable. The eval gate is the same meets_slo you built — the design method and the production pipeline are the same SLO, checked offline and online.

Common misconception. "We'll add monitoring later." The trace IDs, the SLO gate, and the rollback are part of the design, not an afterthought — because the first incident is when you need them, and that is the worst time to build them.

Lab Walkthrough Guidance

The lab (lab-01-platform-capstone) turns the method into code. Suggested order (matches the file):

1. PlatformDesign.__post_init__ — Chapter 3/4. Validate every range; the dataclass must load for the tests to import it.
2. Memory (weights_bytes, kv_cache_bytes, total_gpu_bytes, fits_on_gpu) — Chapter 5. The boundary test is the lesson: the KV-cache, not the weights, swings the fit.
3. Latency (prefill_latency_ms, decode_tokens_per_sec, tpot_ms, ttft_ms, e2e_latency_ms) — Chapters 6–7. Compose additively; the test_e2e_* tests are the first soul. Speculative multiplies tokens/sec by 1/(1−accept).
4. Cost (cost_per_1k_requests) — Chapter 7. Falls with batch; multiplied by agent steps.
5. Quality / safety (quality_score, safety_score) — Chapter 8. Directions matter: RAG/agent up, quant down; safety compounds.
6. SLO (SLO, meets_slo) — Chapter 8. Return the exact ordered violation list.
7. The decision (_validate_weights, _dial_scores, score_design, choose) — Chapter 9. Filter to SLO first, then rank, then the deciding dial. test_choose_winner_flips_with_ priorities is the second soul.

Run red → green: pytest test_lab.py -v, then LAB_MODULE=solution pytest test_lab.py -v, then python solution.py for the worked reference design.

Success Criteria

• All tests pass against your lab.py and against LAB_MODULE=solution.
• You can name, for each subsystem in the architecture, the phase that built it and the PlatformDesign knob that represents it.
• You can derive e2e_latency_ms as a sum of stage budgets and explain why "additive" lets you size each stage independently (the first soul).
• You can show fits_on_gpu flips on the KV-cache (batch × context), not the weights, and compute the 70B-int8 fit number in your head (~91.5 GB → needs two cards).
• You can explain speculative decoding's 1/(1−accept) multiplier and why it changes bytes-moved, not FLOPs.
• You can run choose(), show the winner flip with the weights, and read off the deciding dial for the ADR (the second soul).
• Given a "design a multimodal agentic assistant under an SLO" prompt, you can walk the five-step method, quantify it, name the tradeoff, and list the failure modes with rollbacks.

Interview Q&A

• "Design a multimodal agentic assistant under a TTFT/TPOT/cost SLO — walk me through it." — Workload facts first (concurrency, token sizes, TTFT/TPOT split, quality bar, safety, budget); two paths (serving: tokenize→perceive→retrieve→model→agent→verify; offline: train→PEFT→align→ quantize→eval-gate→registry→canary); quantify KV at peak, tok/s, $/1k, p99; name the tradeoff; fail it. Lead with numbers, end with the deciding dial.
• "What sets your GPU count — the weights or the KV-cache?" — The KV-cache, at peak concurrency × context. Weights are fixed; the cache grows with batch and OOMs you. Show the 2·L·T·batch·d·bytes number; that is the capacity plan.
• "How does end-to-end latency decompose, and where does RAG land?" — Additively: TTFT (RAG retrieval + prefill) + decode (out·TPOT) + agent loop ((steps−1)(overhead + decode)). RAG lands in TTFT (it runs once, up front), not per-token. Budget each stage to the SLO.
• "How does speculative decoding help, and does it risk quality?" — It multiplies throughput by 1/(1−accept_rate) by verifying draft tokens per weight read — a bandwidth-bound win. It's lossless: the big model verifies, so accepted tokens are exactly what it would have produced.
• "Why does batching lower cost, and what's the limit?" — One weight read serves batch sequences, so tok/s ∝ batch and $/request ∝ 1/batch. The limit is becoming compute-bound or running out of KV-cache room — cap the batch at the largest size that still meets p99.
• "RAG or fine-tune?" — RAG when the gap is retrievable facts (cheaper, controllable, citable, fresh); fine-tune when it's reasoning or style. In the model, RAG bumps grounding quality without a training run. Don't fine-tune what a better prompt or retrieval solves.
• "How do you make the agent safe and bounded?" — Hard step cap + token budget (cost), typed idempotent tools (no double-acts), errors returned as observations (recovery), human approval for irreversible actions, and a verifier/guardrail layer (safety compounds: 1−(1−g)(1−v)).
• "Two candidate platforms — defend your choice." — Filter both to the SLO first (a design that breaks the contract isn't a candidate), then rank by the weighted dials, then name the deciding dial and the number you traded. Show the winner flips under a different weighting — that proves you understand it's "best for these priorities," not "best."
• "How does the eval gate block a bad rollout?" — meets_slo + the eval suite (quality CI, safety, calibration) is a CI gate: no green, no deploy. Then canary with auto-rollback on a metric regression. The offline gate and the online SLO are the same contract.
• "What's the difference between knowing the parts and being senior?" — Composition under constraint. Anyone can describe attention or RAG; the senior budgets eighteen mechanisms into one SLO-meeting design, quantifies it, names the tradeoff, and can break it — and debug any layer at 2 a.m.
• "Your design OOMs in production though it passed the load test — what happened?" — The load test ran at a lower batch; production peaked higher and the KV-cache (∝ batch × context) blew the budget. Fix: recheck fits_on_gpu at real peak, then GQA / PagedAttention / KV quant / cap context.
• "Where would you spend the next dollar of latency budget?" — Decompose the e2e into stages, find the largest one against its budget. Often it's the agent loop (cap steps) or retrieval (smaller k / cached embeddings); rarely is it raw decode if you've batched and speculated.

References

• The track's per-phase warmups are the primary sources for each subsystem — pull each one as you budget its stage:
  • Phase 00 — FLOPs/KV/roofline/cost/tradeoff arithmetic (the spine of this capstone).
  • Phases 02/06/09 — KV-cache geometry, quantization, PagedAttention + continuous batching + speculative decoding.
  • Phases 11/12/14/16 — RAG retrieval, the agent loop, neurosymbolic verification, eval + guardrails.
  • Phase 17 — experiment tracking, registry, drift, and the CI/CD eval gate + canary.
• system-design/README.md — the worked reference architectures and the design method; this capstone is design #0 of that catalogue.
• interview-prep/README.md — drills to turn these answers into spoken, defensible ones.
• Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention (vLLM, 2023) — the KV-cache and continuous-batching mechanism.
• Leviathan et al., Fast Inference from Transformers via Speculative Decoding (2023) — the 1/(1−accept) throughput win, lossless.
• Lewis et al., Retrieval-Augmented Generation (2020) — grounding to fight hallucination.
• Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models (2022) — the agent loop.
• The system-design canon (DDIA-style SLO/capacity thinking) for the TTFT/TPOT, queueing, and rollback framing applied to LLM serving.