Phase 09 — Model Serving Platform
Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 2 weeks (40–50 hours) Roles Supported: the JD's "model-serving infrastructure … performance, reliability, latency, and cost efficiency in live environments"
Why This Phase Exists
Serving is where ML meets SLOs. The model is a function; the system around it decides whether p99 is 40 ms or 4 s, whether a GPU runs at 8% or 80% utilization, and whether a bad model version reaches 1% or 100% of users. The three levers that dominate every serving conversation — batching (throughput vs latency), caching (work avoidance), and progressive delivery (shadow/canary) — are exactly what the flagship lab builds, against a simulated clock so the tradeoffs are measurable and the tests deterministic.
The WARMUP covers the surrounding architecture: serving patterns, latency budgets and percentile thinking, autoscaling and cost, and the LLM-serving specifics this track inherits from the llm-inference track.
Concepts
- Serving patterns: online request/response, batch scoring, streaming, async/queue
- Latency as a distribution: p50/p95/p99, tail amplification under fan-out, budgets
- Dynamic batching: amortizing per-call overhead; max-batch-size and max-wait knobs; the latency-throughput frontier
- Caching tiers: exact-match request cache, feature cache (Phase 03's online store), embedding cache; TTLs and staleness contracts
- Progressive delivery: shadow (mirror traffic, compare offline), canary (small live fraction + guardrails), blue/green; automated rollback
- Cost: GPU utilization economics, right-sizing, autoscaling signals (QPS vs queue depth vs latency), cold starts
Labs
Lab 01 — Batching Server Simulator (flagship, implemented)
| Field | Value |
|---|---|
| Goal | Build a discrete-event serving simulator: a dynamic micro-batcher (max-size + max-wait flush policy), a latency recorder with honest percentiles, and a canary router with guardrail-based promote/rollback decisions |
| Concepts | Flush policies, queueing, percentile math, batching's latency-throughput tradeoff, canary guardrails |
| How to Test | pytest test_lab.py — 13 tests: flush-policy invariants, latency accounting, the throughput-win and latency-cost of batching (measured!), percentile correctness, canary decision logic |
| Talking Points | Why does batching help even on CPU? What sets max-wait? Why compare canary on guardrails rather than just the success metric? |
| Resume Bullet | Built a serving simulator quantifying dynamic-batching tradeoffs (5× throughput at +18 ms p99) and canary promotion logic with statistical guardrails |
→ Lab folder: lab-01-batching-server/
Extension Project A — Real Async Server (spec)
Wrap an actual model (Phase 05's ranker) in FastAPI with an asyncio dynamic batcher;
load-test with locust/hey; reproduce the simulator's frontier on real hardware;
find the knee point.
Extension Project B — Cost Model (spec)
Build the QPS × latency-budget × instance-type cost calculator: given a traffic profile and SLO, output the cheapest fleet (CPU vs GPU, batch sizes, replica counts, with/without cache at assumed hit rates). Defend three scenarios in a one-pager.
Guides in This Phase
- WARMUP.md — the full zero-to-expert guide
Deliverables Checklist
- Lab 01 implemented; all 13 tests pass
- Extension A latency-throughput frontier measured on real hardware
- Extension B cost memo