Hitchhiker's Guide — Capstone: Composing the Whole Stack

The compressed practitioner tour. If WARMUP.md is the professor walking the design method, this is the senior who leans over and says "here's how you actually run a system-design round."

The 30-second mental model

A platform is the eighteen mechanisms budgeted into one design that meets an SLO. Workload facts first → model strategy → two paths (serving + offline) → quantify → tradeoff → fail it. Three sentences size any platform: latency composes additively (RAG + prefill + decode + agent loop sum), memory is dominated by the KV-cache (batch × context, not the weights), and cost amortizes with batch (and blows up with the agent loop). The decision is: filter to SLO-feasible designs, rank by weighted dials, name the deciding dial — and the winner flips with the weights. Senior is composition under constraint, not a fancier box diagram.

The numbers to tattoo on your arm

Back-of-envelope one-liners

# "70B int8, 2k ctx, batch 8 — fit on one 80GB card?"
weights 70e9*1 = 70 GB;  KV 2*80*2048*8*8192*1 ≈ 21.5 GB;  total 91.5 GB → no, need 2 (or TP)

# "Does adding RAG break my 1.5s TTFT?"
TTFT = retrieve(120ms) + prefill;  if prefill is 300ms → 420ms, fine. RAG only hits TTFT, once.

# "Speculative at accept 0.6 — how much faster?"
1/(1-0.6) = 2.5x tokens/sec, same FLOPs, lossless

# "3-step agent loop — what's the cost vs 1 step?"
cost ∝ out_tokens * steps → 3x the decode bill. Cap the steps.

# "Two candidates, cost-weighted — which wins?"
filter both to SLO; rank by {cost:4, latency:3}; small int4+RAG wins on cost; that's the ADR

The framework one-liners (where these live in real tools)

# Capacity / fit (the KV-cache budget)
#   vLLM: gpu_memory_utilization, max_model_len (T), max_num_seqs (batch)
# Throughput + speculative
#   vLLM: --speculative-model, continuous batching; TGI / TensorRT-LLM throughput logs
# Latency split
#   load test reports p50/p99 TTFT and TPOT separately — never one "latency" number
# The trace (additive stages, made visible)
#   OpenTelemetry / Langfuse: spans for retrieve → prefill → decode → each tool call
# The eval/SLO gate in CI
#   your pipeline's gate step runs the eval suite + meets_slo before promote (MLflow/W&B registry)

War stories

• The capstone that was all boxes, no numbers. A candidate drew a beautiful diagram — perception, RAG, agents, multi-agent critique, multi-region — and could not say the GPU count or the $/1k. The panel asked "what's your KV-cache at peak?" and it was over. Boxes without budgets is a junior answer wearing a senior costume.
• The agent loop that ate the budget. Demo cost was fine at 1 step; production averaged 5 steps per ticket because the loop had no cap, and $/ticket was 5×. The fix was one line — a hard step cap — but it should have been in the design, because cost ∝ steps.
• The RAG that blew TTFT. Someone added a cross-encoder reranker over top-50 and TTFT doubled, because retrieval is in the TTFT budget, not free. Cut k to 20 and reranked the top-10; back under SLO. Know where each stage lands in the latency curve.
• The "best model" that lost the review. Team picked the 70B because it scored 2 points higher. Cost-weighted, the int4-13B+RAG cleared the same eval bar at 1/100th the cost and 7× the speed. Same candidates, opposite winners — they optimized the benchmark, not the workload.
• The OOM that "couldn't happen." "The model fits, we checked" — at load-test batch 8. Production peaked at batch 60, KV-cache 7×, OOM at 3pm. Capacity is weights + KV at real peak, always.

Vocabulary (rapid-fire)

• Composition under constraint — the senior skill: budget the parts into one SLO-meeting whole.
• The two paths — serving (online) and offline (train→align→quantize→gate→deploy).
• TTFT / TPOT — time to first token (prefill + retrieve) / time per output token (decode).
• Additive latency — stages sum; budget each independently.
• KV swing factor — the cache (∝ batch × context) flips the fit, not the weights.
• Speculative multiplier — 1/(1−accept), lossless throughput.
• Defence in depth — layered safety: 1−(1−g)(1−v).
• Filter then rank — constraints (SLO) before objective (weighted score).
• Deciding dial — the one sentence in the ADR; the dial with the biggest weighted gap.
• The flip — same candidates, opposite winners under different weights.

Beginner mistakes

• Drawing the architecture before writing the workload numbers (the numbers are the architecture).
• Quoting one "latency" number instead of the TTFT/TPOT split, prefill vs decode, batched vs single.
• Sizing GPUs by the weights and forgetting the KV-cache at peak concurrency.
• An uncapped agent loop (cost blowup) or no idempotency / human gate on irreversible tools.
• Ranking by score before filtering out SLO-violators — a design that breaks the contract can't win.
• Reaching for multi-agent / multi-GPU / multi-region with no number justifying the complexity.
• Treating retrieved/user text as trusted (prompt injection) or shipping one guardrail (no depth).
• Saying "best model" instead of "best for these weights, among the SLO-feasible designs."

The one thing to take away

Before you draw a single box, write the workload facts; before you commit, quantify memory (KV first), latency (additive stages), and cost (per batch); before you decide, filter to the SLO then name the deciding dial. If you can compose the eighteen mechanisms into one design, size it, defend the tradeoff, and break it — you are the Senior AI Engineer.