Production Serving — Architecture and the Request Path

Phase 7 · Document 00 · Production Serving Prev: Phase 6 — Local Inference · Up: Phase 7 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Phase 6 got a model running on one machine for one user. Production serving is the leap to many users, reliably, affordably, observably — the gap between "it works in my notebook" and "it handles 1,000 concurrent users with a 99.9% SLO." That gap is where most LLM products live or die: an endpoint that's fast at batch-1 can fall over at batch-50; a model that's cheap per token can bankrupt you without max_tokens and caching; an outage with no fallback is a customer-facing incident. This phase teaches the architecture (the request path, the serving engines, batching/paging/caching, streaming, routing, observability, cost control, and the runbook) that turns a model into a dependable service. This overview is the map; the dedicated docs go deep on each layer.

2. Core Concept

Plain-English primer: serving = keep an expensive GPU busy, safely, for many users

A served LLM endpoint has one hard economic fact underneath it: the GPU is expensive and decode is memory-bandwidth-bound, so the only way to make it pay is to batch many users' tokens into each weight-read (Phase 6.01, what-happens §1.D–1.E). Everything in production serving is built around that:

• The inference engine (vLLM/TGI/SGLang/TensorRT-LLM) keeps the GPU maximally busy with continuous batching (03), packs the KV cache efficiently with PagedAttention (04), and reuses shared prefixes with prefix caching (05). This is the raw token factory.
• The serving layer around it makes that factory usable and safe: it streams tokens back (06), routes requests across models/providers with fallbacks (07), enforces cost controls (09), and emits observability so you can see TTFT/TPOT/errors/cost (08).
• The operations wrap it in a runbook: pre-launch checks, incident response, rollout/rollback (10).

The full request path

Every production LLM request flows through layers — know them and you can debug and design anything:

Client
  → TLS / Load balancer
  → API Gateway:  auth → rate limit → request validation → prompt normalization → policy/safety
  → Model Router:  pick model+provider · check budget/quota · choose instance (region/health)   [07]
  → Inference: Provider Adapter (cloud API)  OR  Self-hosted engine (vLLM/TGI/…)                 [01,02]
       └ prefill → decode (continuous batching [03], PagedAttention [04], prefix cache [05])
  → Response Normalizer:  unify shape · extract usage/cost · stream (SSE) [06]
  → Observability:  logs · metrics (TTFT/TPOT/$) · traces · billing events [08]
  → Client (streamed or complete)

The one architectural decision: managed API vs self-hosted

The fork that shapes the whole stack (deepened in Phase 5.02):

• Managed API (OpenAI/Anthropic/etc.): you skip the engine, GPUs, and batching — you build the gateway layers (routing, limits, observability, cost) around someone else's factory (Phase 8).
• Self-hosted (vLLM on your GPUs): you own the whole stack, including the engine and capacity planning — more control and better economics at scale, more operational burden.

Most real systems are hybrid: a gateway routes across managed APIs and self-hosted models with fallback.

3. Mental Model

        ┌──────────────────────── THE SERVING STACK ───────────────────────────┐
 OPS    │ runbook: pre-launch · incident response · rollout/rollback      [10]  │
 ───────┼───────────────────────────────────────────────────────────────────────┤
 SERVE  │ gateway: auth·limits·routing+fallback [07] · streaming [06] ·          │
 LAYER  │          cost controls [09] · observability [08]                       │
 ───────┼───────────────────────────────────────────────────────────────────────┤
 ENGINE │ vLLM / TGI / SGLang / TensorRT-LLM  [01,02]                            │
 (token │   continuous batching [03] · PagedAttention KV [04] · prefix cache [05]│
 factory)│   ← keeps the bandwidth-bound GPU BUSY for many users (Phase 6)        │
        └───────────────────────────────────────────────────────────────────────┘
   the economic law underneath: batch many users' tokens per weight-read = throughput = margin
   managed API  → you build only the SERVE+OPS layers     self-host → you own ENGINE too

Mnemonic: engine makes tokens cheap (batching/paging/caching); the serve layer makes them safe (routing/streaming/limits/observability); ops keeps it alive.

4. Hitchhiker's Guide

What to look for first: your workload shape (interactive vs batch, concurrency, context length) and the managed-vs-self-host decision — they determine the whole architecture and your cost model.

What to ignore at first: exotic engines, multi-node tensor/pipeline parallelism, and hand-tuned kernels. Start with one engine (vLLM) or one managed API behind a thin gateway; add complexity when metrics demand it.

What misleads beginners:

• Benchmarking at batch 1. Single-stream latency tells you nothing about throughput under load — the metric that sets cost (03).
• Forgetting the KV cache caps concurrency. You run out of KV memory long before compute (Phase 6.02, 04).
• No max_tokens / no budgets. Runaway generation and traffic spikes silently blow the bill (09).
• No fallback. One provider blip = a full outage (07).
• Latency means p50. Production lives and dies on p95/p99 (08).

How experts reason: they think in throughput and percentiles, not averages; size KV/concurrency before launch; enable continuous batching + prefix caching by default; put routing + fallback + budgets + observability in front of any model; and load-test at expected concurrency before shipping.

What matters in production: p95/p99 TTFT & TPOT, KV-cache utilization (the capacity ceiling), error rate + fallback success, cost per request and per resolved task, and a tested rollback.

How to debug/verify: read engine metrics (/metrics: cache usage, running/waiting queue, tok/s), trace the request path, and bisect by layer (gateway? router? engine? provider?).

Questions to ask vendors/providers: served context cap, rate limits (RPM/TPM), batch-discount endpoint, streaming support, SLA/SLO, region/data-residency, and whether the served model is quantized (Phase 5.10).

What silently gets expensive/unreliable: idle GPUs at low utilization, uncapped context/tokens, no caching on repeated prompts, and missing p99/queue alerts.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Serving economics = batching: decode is bandwidth-bound, so throughput (and margin) come from packing many users' tokens per weight-read (what-happens §1.D).
• The KV cache, not compute, usually caps concurrency (04, Phase 6.02).
• Continuous batching gives ~3–5× throughput over static batching (03).
• Prefix caching cuts TTFT and cost for repeated system prompts (05).
• Production is measured at p95/p99, not averages (08).
• Always set max_tokens and budgets — the two cheapest guards against cost blowups (09).
• Always have a fallback before launch (07).
• Managed vs self-host decides whether you own the engine or just the serve layer (Phase 5.02).

9. Observations from Real Systems

• vLLM is the de-facto open serving engine (PagedAttention + continuous batching + OpenAI-compatible API) — the backbone of most self-hosted deployments (01).
• OpenRouter / LiteLLM are the serve-layer (gateway) for managed APIs — routing, fallback, usage metering (Phase 8).
• Cloud model endpoints (Bedrock, Vertex, Azure OpenAI) are managed factories you wrap with your own gateway/observability.
• Real platforms are hybrid: premium managed models for hard requests, self-hosted open-weight for bulk/private traffic, unified behind one gateway (07).
• Outages are usually downstream: the discipline that saves you is fallback + observability + a runbook (07, 10).

10. Common Misconceptions

11. Engineering Decision Framework

DESIGN a serving stack:
 1. CLASSIFY workload: interactive vs batch · concurrency · context length · privacy.
 2. MANAGED vs SELF-HOST?
       data must stay in-house / high steady volume / need control → SELF-HOST (vLLM) [01]
       speed-to-market / spiky / want frontier quality            → MANAGED API + gateway
       usually → HYBRID behind one gateway                        [07, Phase 8]
 3. ENGINE (if self-host): vLLM default; TGI/SGLang/TensorRT-LLM for specific needs [02].
       enable continuous batching [03] + PagedAttention [04] + prefix caching [05].
 4. SERVE LAYER (always): streaming [06] · routing+fallback [07] · budgets/max_tokens [09]
                          · observability (p95/p99, KV, $, errors) [08].
 5. SIZE: KV/concurrency from Phase 6.02; load-test at expected concurrency.
 6. OPERATE: runbook, alerts, rollback, canary [10].

12. Hands-On Lab

Goal

Map the request path for your own setup and measure the metrics that define production: p50/p95 TTFT, throughput under concurrency, and the KV-cache ceiling.

Prerequisites

• A served endpoint: vLLM on a GPU (01) or a managed API behind a thin client. pip install httpx.

Setup

# Self-host example:
vllm serve Qwen/Qwen2.5-1.5B-Instruct --max-model-len 4096 \
  --gpu-memory-utilization 0.85 --enable-prefix-caching

Steps

1. Draw the path: for your setup, label each layer in §2 (what does auth/routing/streaming/observability for you?). Note which you own vs which the provider owns.
2. Measure single-stream: one request, record TTFT and total. (This is the floor, not production.)
3. Measure under concurrency: fire 1, 8, 32 concurrent requests; record aggregate throughput (tok/s) and p50/p95 TTFT. Watch throughput rise and per-request latency degrade.
4. Find the KV ceiling: raise concurrency/context until the engine queues (vllm:num_waiting_requests > 0) or memory caps; that's your capacity (04).
5. Prefix cache on/off: repeat a long shared system prompt with prefix caching on vs off; compare TTFT (05).

Expected output

A table: concurrency → throughput, p50/p95 TTFT, queue depth — plus the concurrency at which you hit the KV ceiling, and the prefix-cache TTFT delta.

Debugging tips

• Throughput flat as concurrency rises → batching off or KV-capped; check engine flags/metrics.
• p95 ≫ p50 → queuing under load; you've found the capacity limit.

Extension task

Add a fallback: point a tiny gateway at two endpoints and kill one mid-test; confirm requests reroute (07).

Production extension

Wire /metrics into Prometheus + a Grafana dashboard with p95/p99 TTFT, KV usage, and cost/request (08).

What to measure

p50/p95 TTFT, aggregate throughput, queue depth, KV ceiling concurrency, prefix-cache delta.

Deliverables

• A labeled request-path diagram for your setup.
• A throughput-vs-concurrency table with the KV ceiling.
• A prefix-cache TTFT comparison.

13. Verification Questions

Basic

1. Why is batching the economic engine of LLM serving?
2. List the layers of the production request path.
3. What usually caps serving concurrency — compute or KV cache?

Applied 4. For a private, high-volume internal copilot, would you go managed or self-host? Justify with the framework. 5. Why measure p95/p99 instead of average latency?

Debugging 6. Throughput doesn't rise as you add concurrency. Two causes. 7. p95 TTFT spikes under load while p50 is fine. What's happening?

System design 8. Design a hybrid stack (managed + self-hosted) with routing, fallback, budgets, and observability for a mixed workload.

Startup / product 9. Which serving choices most affect gross margin, and how do batching/caching/routing improve it?

14. Takeaways

1. Serving = keep a bandwidth-bound GPU busy for many users, safely — batching is the economic engine.
2. Know the request path; the engine makes tokens cheap, the serve layer makes them safe, ops keeps it alive.
3. KV cache caps concurrency; measure throughput and p95/p99, not batch-1 averages.
4. Always ship max_tokens + budgets + fallback + observability.
5. Managed vs self-host (often hybrid) is the architectural fork — usually a gateway over both.

15. Artifact Checklist

• A labeled request-path diagram for your stack.
• A throughput-vs-concurrency table + the KV-ceiling concurrency.
• A managed-vs-self-host (or hybrid) decision memo.
• A prefix-cache TTFT comparison.
• A short SLO draft (p95 TTFT, availability) to refine in 08.

Up: Phase 7 Index · Next: 01 — vLLM