Lab 01 — Platform Capstone: the End-to-End Multimodal Agentic Serving Model

Phase: 19 — Capstone: Production Multimodal Agentic Serving Platform Difficulty: ⭐⭐☆☆☆ (the arithmetic reuses Phase 00; the integration judgment is ⭐⭐⭐⭐⭐) Time: 4–6 hours

The finale. Every earlier lab built one mechanism in isolation — the attention block, the LoRA delta, the PagedAttention block table, the ReAct loop, the eval harness. This lab composes them into one design you can score, SLO-check, and defend. You build a PlatformDesign whose knobs are the whole track (model size + quant from P00/P06, KV-cache from P00/P02, RAG retrieval from P11, the agent loop from P12, speculative decoding from P09, guardrails + verification from P16/P14), then the functions that turn those knobs into memory (weights + KV-cache + fit), latency that composes additively across stages, cost per 1k requests, quality/safety scores, an SLO check that returns the exact violations, and the decision: choose() filters to SLO-meeting designs, ranks them by weighted score, and names the deciding dial — and the winner flips when the workload's priorities change. That flip, and the additive composition, are the two souls of the lab.

What you build

• PlatformDesign — one dataclass holding every knob the track exposes: n_params, quant_bytes_per_param, n_layers, d_model, seq_len, batch, bandwidth, peak_flops, gpu_hourly_usd, and the stage flags use_rag / use_agent (agent_steps) / use_speculative (accept_rate), plus the quality/safety component scores. Range-validated.
• Memory — weights_bytes, kv_cache_bytes (the Phase 00 formulas, unchanged), total_gpu_bytes, and fits_on_gpu — where the KV-cache is the swing factor between a design that just fits and one that just OOMs.
• Latency — prefill_latency_ms (compute-bound), decode_tokens_per_sec (bandwidth-bound; speculative multiplies by the accept-rate factor), ttft_ms (prefill + RAG retrieval), tpot_ms, and e2e_latency_ms = TTFT + output_tokens·TPOT + agent-loop overhead — latency composes additively across stages.
• Cost — cost_per_1k_requests from throughput and GPU price; falls with batch because batching amortizes the weight read.
• Quality / safety / SLO — quality_score (RAG bumps grounding, agent bumps capability, quant dings it), safety_score (guardrails + verification compound, defence in depth), meets_slo returning the exact ordered violation list.
• The decision — DIALS, score_design, and choose: filter to SLO-meeting designs, rank by the weighted dial score, report the deciding dial — and watch the winner flip.

Key concepts

Files

Run

pip install -r requirements.txt
pytest test_lab.py -v                       # against your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # against the reference
python solution.py                          # the worked example

Success criteria

• All tests pass against your implementation (and LAB_MODULE=solution).
• You can explain why test_fits_on_gpu_boundary_kv_is_the_swing flips on the KV-cache and not the weights — and why that is the #1 capacity mistake in a serving design.
• You can explain why test_e2e_adding_rag_increases_by_exactly_retrieval_ms and test_e2e_agent_loop_adds_steps_worth are the composition soul: a real platform's latency is a sum of stage budgets, and you size each stage independently.
• You can explain why speculative decoding multiplies throughput by 1/(1−accept_rate) and why that does not change the FLOPs, only the bytes-moved efficiency.
• You can read meets_slo's violation list and say which subsystem to fix for each tag.
• You can explain why test_choose_winner_flips_with_priorities is the tradeoff soul — the same Phase 00 lesson, now over a full composed platform.
• Given a "design a multimodal agentic assistant" prompt, you can populate a PlatformDesign, compute its memory/latency/cost, check it against an SLO, and name the deciding dial — on paper.

How this maps to the real stack

Limits of the miniature (be honest in the interview): latency here is a clean additive model — real systems have queueing delay, prefix-cache hits, batch-formation jitter, and p99 tails that a mean stage budget hides; the cost model charges by produced tokens at the decode ceiling and ignores prefill GEMM time and idle GPUs at low traffic; quality_score / safety_score are stand-ins for real eval suites and red-team results; and the dial normalization in choose is relative to the candidate pool, so adding a candidate can shift scores. The point is the structure of the decision, not five-decimal precision.

Extensions (build these on real hardware)

• Add queueing: model TTFT as prefill + queue_wait(batch, arrival_rate) and find the arrival rate where p99 TTFT breaks the SLO (the real autoscaling trigger).
• Add a prefix-cache hit rate that discounts prefill for shared system prompts (P09), and show how it changes the cost/latency of a high-traffic chat workload.
• Add a multi-GPU branch: tensor-parallel splits weights and KV across tp GPUs; recompute fits_on_gpu and the $/1k with the extra GPUs (P10).
• Replace quality_score/safety_score with the outputs of the real Phase 16 eval + guardrail labs, and wire meets_slo into the Phase 17 CI/CD gate so a bad design literally fails the pipeline.
• Run the chosen design on a real GPU with vLLM and check decode_tokens_per_sec and $/1k against measured throughput — then write the ADR with the deciding dial.

Interview / resume

• Talking points: "Design a multimodal agentic assistant under a TTFT/TPOT/cost SLO — walk the stages and where each phase's mechanism lands." "Why is the KV-cache, not the weights, your capacity number?" "How does speculative decoding change throughput without changing FLOPs?" "Two candidate platforms — defend the choice with the deciding dial." "How does the eval/SLO gate block a bad rollout?"
• Resume bullet: Built an end-to-end serving-platform model that composes weight + KV-cache memory, additively-composed TTFT/TPOT/agent-loop latency, speculative-decoding throughput, $/1k-request cost, and composed quality/safety into an SLO-gated, priority-weighted architecture decision with an explainable deciding dial — the design-review artifact for a multimodal agentic LLM platform.