Phase 19 — Capstone: Production Multimodal Agentic Serving Platform

The finale. Every earlier phase built one mechanism in isolation; this one composes the whole track into a single design you can size, SLO-check, and defend. You take the workload facts, pull each subsystem from the phase that built it, quantify the result with the Phase 00 arithmetic, name the tradeoff, and try to break it. If Phase 00 made you able to answer "how much compute, how much memory, how fast, what cost, what am I trading" for one model, this phase makes you able to answer it for the whole platform — the question a system-design interview and a real design review actually ask.

Why this phase exists

A Senior AI Engineer is not "the person who knows attention" or "the person who knows vLLM." They are the person who can stand at a whiteboard and assemble a tokenizer, a transformer, a fine-tune, a quantizer, a sampler, a KV-cache scheduler, a retriever, an agent loop, an evaluator, and an MLOps gate into one system that meets a latency/cost/quality/safety SLO — and drop down to debug any layer of it. The capstone is where the eighteen mechanisms stop being separate labs and become one architecture. The skill it certifies is composition under constraint: not inventing new tricks, but choosing and budgeting the right ones, in the right order, and proving the whole meets the contract.

How the capstone integrates Phases 00–18

The platform you model is a multimodal agentic assistant: it perceives images and text, retrieves grounding, reasons in a tool-using loop, and serves the result under an SLO. Each phase contributes exactly one slot:

Phase	Contributes to the platform
P00 Foundations	the arithmetic everything plugs into — `6ND`/`2N` FLOPs, weight + KV-cache memory, the roofline, `$/1M`, and the tradeoff resolver the capstone's `choose()` grows up from
P01 Tokenization	the input boundary: prompt + chat template + image placeholders → token IDs (sets `in_tokens`)
P02 Transformer/KV	the model's geometry — `n_layers`, `d_model` — that sizes the KV-cache, the memory wall
P03 Autograd/training	the offline training that produced the base weights (`6ND`, optimizer memory)
P04 VLMs	the perception stage — ViT encode + projector — that turns images into prepended tokens (extra context, extra KV)
P05 PEFT (LoRA/QLoRA)	how the base was cheaply adapted to the domain before serving
P06 Quantization	the serving precision (`quant_bytes_per_param`) — the lever between 1 GPU and 2, and the quality ding
P07 Alignment (RLHF/DPO)	the alignment that raised `base_quality` and `safety` before the model was frozen
P08 Sampling/constraints	the decode policy — temperature/top-p for chat, constrained JSON for tool calls
P09 Serving internals	the serving path — PagedAttention (KV fit), continuous batching (`batch` → throughput), speculative decoding (`accept_rate`)
P10 Distributed	the multi-GPU branch when weights + KV exceed one card (tensor/pipeline parallel)
P11 RAG	the retrieval stage — ANN top-k + rerank — whose latency lands in TTFT and whose grounding bumps quality
P12 Agents	the reasoning loop — ReAct steps + tool calls — whose overhead lands in e2e latency and whose capability bumps quality
P13 Multi-agent	the planner/executor/critic decomposition when one loop is not enough
P14 Neurosymbolic	the verification layer that compounds with guardrails into the safety score
P15 Embodied/VLA	the action contract — when the agent moves a motor, latency becomes a safety property
P16 Eval/guardrails	the quality and safety scores and the SLO gate the design must clear
P17 MLOps	the rollout — eval gate → canary → monitor → rollback; `meets_slo` is the CI gate
P18 CUDA/roofline	the hardware ceiling — `bandwidth`, `peak_flops` — that bounds prefill and decode

Architecture — the multimodal agentic serving platform

                         ┌──────────────────────────────────────────────┐
   user: image + text →  │              ONLINE / SERVING PATH            │
                         └──────────────────────────────────────────────┘
        │
        ▼  P01 tokenize + chat template          P04 ViT encode → projector → image tokens
   ┌─────────────┐     ┌──────────────┐
   │  TOKENIZER  │────►│  PERCEPTION  │──── image tokens ──┐
   └─────────────┘     └──────────────┘                   │
        │                                                  ▼
        │  P11 RAG: embed → ANN top-k → rerank      ┌────────────────────┐
        └────────────► [ RETRIEVE ] ── grounding ──►│   PROMPT ASSEMBLY  │  (→ TTFT)
                                                     └────────────────────┘
                                                              │
                              ┌───────────────────────────────┘
                              ▼  P12 ReAct loop ↺ (think → act → observe), step-capped
                     ┌──────────────────────────────────────────────────────┐
                     │                  MODEL  +  SERVER                      │
                     │  P02 transformer/KV   P06 quant   P08 sampler/JSON     │
                     │  P09 PagedAttention · continuous batch · speculative   │
                     │  P10 tensor/pipeline parallel (if it won't fit)        │
                     └──────────────────────────────────────────────────────┘
                              │ tool calls          │ tokens stream out (→ TPOT)
                              ▼                      ▼
                     ┌──────────────┐      ┌────────────────────┐
                     │ TYPED TOOLS  │      │ P14 VERIFY + P16    │
                     │ (idempotent) │      │ GUARDRAILS (safety) │
                     └──────────────┘      └────────────────────┘
                                                    │
                                                    ▼  composed answer + citations
   ┌──────────────────────────────────────────────────────────────────────────────┐
   │  OFFLINE PATH:  P03 train → P05 PEFT → P07 align → P06 quantize →              │
   │                 P16 EVAL GATE → P17 registry → canary → monitor/rollback       │
   │  observability spans every stage: trace IDs, p99 TTFT/TPOT, $/1k, drift        │
   └──────────────────────────────────────────────────────────────────────────────┘

   latency = RAG_retrieve + prefill + Σ(decode + agent_step)   ← composes ADDITIVELY
   memory  = weights + KV-cache(batch × context)               ← KV is the swing factor
   cost    = produced_tokens / throughput × $/h                ← batching amortizes it
   decision = filter to SLO ▸ rank by weighted dials ▸ deciding dial   ← the ADR

The lab

Lab	You build	Difficulty	Time
lab-01 — Platform Capstone	a `PlatformDesign` and the functions that compose it: weight + KV-cache memory and `fits_on_gpu`; additively-composed TTFT/TPOT/agent-loop latency; speculative throughput; `$/1k` cost; quality/safety scores; `meets_slo` with the exact violation list; and `choose()` that filters to SLO-meeting designs, ranks them, and names the deciding dial	⭐⭐☆☆☆ math / ⭐⭐⭐⭐⭐ integration	4–6 h

The lab is a runnable, test-verified miniature — see the lab standard. Run it red (pytest test_lab.py), make it green, then read solution.py. The two soul tests — test_e2e_* (latency composes additively) and test_choose_winner_flips_with_priorities (the tradeoff flips) — are the whole point; if those pass for the right reasons, you have the phase.

Integrated scenario ideas

Size the assistant end-to-end: a 13B int4 with RAG, 16 concurrent users, 4k context on one 80 GB GPU — compute weights, KV-cache at peak, TTFT (with retrieval), TPOT, $/1k, and show whether it clears a TTFT<1.5s / TPOT<50ms SLO. Show that the KV-cache sets the GPU count.
Defend "small + RAG over big": two candidates (frontier 70B vs int4-13B+RAG); prove with choose() that the cost-weighted workload picks the small one and the quality-weighted workload picks the big one — same candidates, opposite winners.
Trace the latency budget: take a 3-step agent request and decompose its e2e into retrieve + prefill + decode + loop, then find which stage to cut to meet a tighter SLO.
Fail it on purpose: raise batch until fits_on_gpu flips to OOM; raise agent_steps until the cost SLO breaks; lower precision until quality drops below the gate — and name the fix.
Wire the gate: make meets_slo the CI/CD eval gate (P17) so a regression literally fails the rollout, then canary the chosen design.

Deliverables checklist

lab.py passes pytest test_lab.py -v (and LAB_MODULE=solution does too).
You can name, for each subsystem, which phase built it and which PlatformDesign knob represents it.
You can explain why e2e_latency_ms is a sum of independent stage costs and size each stage's budget separately.
You can show that fits_on_gpu flips on the KV-cache (batch × context), not the weights.
You can run choose() and demonstrate the winner flips when the weights change, and read off the deciding dial for the ADR.
You can take a "design a multimodal agentic assistant" prompt and produce, on a whiteboard: the two paths, the memory/latency/cost numbers, the SLO check, the tradeoff, and the failure modes with rollbacks.

Key takeaways

Senior is composition under constraint. The capstone invents nothing — it budgets the eighteen mechanisms into one design that meets the SLO. Knowing each piece is the floor; making the whole defensible is the job.
Latency composes additively; memory is dominated by the KV-cache; cost amortizes with batch. These three sentences let you size any serving platform on paper, and they are exactly the Phase 00 arithmetic, now over a full stack.
Constraints before objective. choose() filters to SLO-feasible designs first, then optimizes — because a faster, cheaper design that misses the safety floor is not a candidate.
"Best platform" is still a category error. The winner flips with the weights; the deliverable is the deciding dial and the number that makes the trade acceptable — the one sentence in the ADR.

Where to go next

This is the end of the build track — and the start of the interview. Use the two running references to turn the eighteen mechanisms and this capstone into spoken answers:

interview-prep/ — the coding, ML-theory, and behavioral drills; rehearse explaining each subsystem from first principles and defending the tradeoffs.
system-design/ — the worked reference architectures (the high-throughput serving platform, production RAG, the VLM assistant, the tool-using agent, the train/align pipeline, edge deployment). The capstone is design #0 of that catalogue; read the others and practice walking the design method (workload facts → model strategy → serving + offline paths → quantify → tradeoff → fail it) until it is reflex.

You started Phase 00 able to size one model. You finish Phase 19 able to architect, quantify, SLO-gate, and defend a multimodal agentic platform — and debug it at the token, KV-block, and tool-call level. That is the Senior AI Engineer.

Senior AI Engineer — LLMs, VLMs & Agentic Systems