System Design 02 — LLM Inference Gateway at Scale

"Design a 100k-QPS LLM inference platform." Phases: 07 (serving/KV), 06 (scheduling), 08 (multi-GPU for big models), 10 (observability/release).

1. Drive the requirements

  • QPS & latency SLOs: 100k QPS of what — short or long prompts? p99 TTFT/TPOT targets? (P07 — goodput is the real metric.)
  • Models: how many, what sizes? A 70B model needs TP across an NVLink island (P08); many small models = multi-tenant packing (P06).
  • Workload shape: chat (shared system prompts → prefix caching) vs varied?
  • Hardware/sovereignty: heterogeneous (HAL, P09)? on-prem tier (P11)?

Assume: a mix of 8B and 70B models, chat-heavy (long shared prompts), strict p99 TTFT, cloud + an on-prem tier.

2. Architecture

clients -> [LB] -> [Gateway: routing, auth, rate-limit, licensing(P11)]
                      |
              +-------------------- per-model serving pools --------------------+
              | 8B pool: vLLM, continuous batching, paged KV, prefix cache (P07)|
              | 70B pool: TP across NVLink island (P08), KV sharded (P07/P08)   |
              +----------------------------------------------------------------+
                      |
              [Observability: TTFT/TPOT/goodput, KV pressure (P10)]

3. The hard decisions (and tradeoffs)

  • Serving engine per pool (P07 Ch. 8): vLLM default for throughput+coverage; TensorRT-LLM for a fixed hot model needing the last 20%; SGLang for agent workloads. Tradeoff: peak-per-model vs coverage vs build complexity. Route on goodput evidence, not vendor numbers.
  • Batching & admission (P07): continuous batching tuned for goodput (interior batch depth, not max); admission control sheds/queues under overload to protect p99. Tradeoff: latency vs throughput dial.
  • KV-cache management (P07/P05): paged KV (no fragmentation), prefix caching for the shared chat prompts (huge TTFT win), KV quantization to fit more.
  • Big-model placement (P08): 70B is TP-sharded on one NVLink island — gang-place it (P06); never spread across nodes.
  • Prefill/decode interference (P07): chunked prefill or disaggregation so long prompts don't stall decodes.

4. Scaling to 100k QPS

  • Horizontal: many replicas per model behind the LB; autoscale on goodput + queue depth.
  • The KV wall (P07): concurrency is capped by KV memory, not compute — capacity planning is (HBM − weights) / (KV/token × context); quantize and use GQA models to raise it.
  • Prefix-cache hit rate as a first-class scaling metric for chat (P07).

5. Failure modes & operations (P10)

  • GoodputBelowSLO: throttling (P03), bad deploy (auto-rollback, P10), or overload (admission).
  • KV pressure / fragmentation (P05/P07): largest-free-block alert; admission already shedding.
  • A replica/backend down: LB health-check + failover; the gateway routes around it.
  • Release: compatibility gate (engine/model/driver, P03/P04/P10) → canary on goodput → progressive → auto-rollback.

6. Commercial framing (P12)

Goodput per GPU is gross margin — every serving optimization (batching, prefix cache, quantization) is margin. The on-prem tier (P11) and heterogeneous hardware (P09) open markets. Honest SLO reporting (goodput, not throughput) is a trust asset with customers.

The senior signal

You separated pools by model size (small = packing, 70B = TP-on-NVLink, connecting P07↔P08), tuned for goodput not throughput (P07's thesis), identified the KV wall as the real scaling limit, routed engines on evidence (P07 Ch. 8), and closed with margin + operations — the inference-platform leadership altitude.