Phase 7 — Production Serving
How to turn a running model into a dependable service for many users: the request path, serving engines (vLLM and friends), the batching/paging/caching that make it economical, streaming, routing + fallbacks, observability, cost controls, and the runbook that keeps it alive.
Why this phase matters
Phase 6 got a model running for one user; Phase 7 is the leap to many users, reliably, affordably, observably — the gap between a notebook demo and a 99.9%-SLO product. The economic core is simple: a GPU is expensive and decode is bandwidth-bound, so you survive by batching many users' tokens per weight-read (Phase 6.01) and wrapping that token factory in a serve layer (streaming, routing, limits, observability) and an ops discipline (the runbook). Everything reduces to throughput + p95/p99 + cost, gated by the KV-cache ceiling.
Documents
| # | Document | What you'll be able to do |
|---|---|---|
| 00 | Serving Architecture | Map the request path; decide managed vs self-host (vs hybrid) |
| 01 | vLLM | Serve at scale; tune the KV pool; read /metrics |
| 02 | TGI, SGLang, TensorRT-LLM | Pick the right engine for the workload |
| 03 | Continuous Batching | Get 3–5× throughput; find the latency knee |
| 04 | PagedAttention | Understand the KV ceiling and raise concurrency |
| 05 | Prefix and Prompt Caching | Cut TTFT + cost via stable-prefix-first ordering |
| 06 | Streaming | Build a correct SSE proxy (forward, cancel, capture usage) |
| 07 | Routing and Fallbacks | Get reliability + economics; survive outages |
| 08 | Observability | Dashboard p95/p99, KV, cost, quality; set SLOs |
| 09 | Cost Controls | Bound spend; manage cost as COGS |
| 10 | Production Runbook | Launch, change, and recover predictably |
How to work through it
Read 00 (the map) first, then 01 (vLLM) and the mechanism trio 03–05 (batching → paging → caching) that explain why serving is cheap and where the KV ceiling is. 02 is the engine-choice survey. 06–09 are the serve-layer concerns (streaming, routing, observability, cost). 10 is the operational capstone — keep it open while you run the labs. Every doc opens with a from-zero plain-English primer and ends with a measured, artifact-producing lab.
Phase 7 artifacts
- A labeled request-path diagram + managed-vs-self-host decision (00).
- A vLLM endpoint with a throughput-vs-concurrency table and identified KV ceiling (01, 03, 04).
- An engine comparison on your workload (02).
- A prefix-cache TTFT/cost comparison (05) and a correct streaming proxy (06).
- A router + fallback + circuit breaker with a blended-cost result (07).
- A Grafana dashboard + SLO/alerts (08) and a cost model + budget guardrail with gross margin (09).
- A production runbook: executed checklist, tested rollback, an incident playbook + post-mortem (10).