Phase 8 — LLM Gateways
How to build the control plane for an organization's LLM usage: one OpenAI-compatible front door that fronts every provider — with adapters, a model registry, a routing engine, usage metering, a streaming proxy, an admin dashboard, and an enterprise policy engine. The two canonical shapes (aggregator and self-hosted proxy) plus the components you'd build to ship one.
Why this phase matters
Phase 7 taught you to serve models; Phase 8 packages that serve-layer as a product that sits in front of all your models and providers (cloud APIs and your self-hosted vLLM). A gateway is why apps stay simple (one API, swap models by string), why you survive provider outages (routing + fallback), why you can see and cap spend (metering + budgets), and why enterprises can adopt LLMs at all (policy + audit + residency). This is the architecture behind OpenRouter, LiteLLM, Portkey, every enterprise AI platform — and a top startup category (Phase 15).
The mechanism is largely Phase 7 (routing/fallback, streaming, observability, cost); Phase 8 makes it multi-provider and multi-tenant.
Documents
| # | Document | What you'll be able to do |
|---|---|---|
| 00 | What Is an LLM Gateway? | Frame the control plane; gateway vs proxy vs aggregator; build vs buy |
| 01 | OpenRouter-Style Architecture | Use/rebuild the aggregator: namespace, provider routing, transforms, BYOK |
| 02 | LiteLLM-Style Proxy | Self-host a unifying proxy with pools, fallbacks, virtual keys |
| 03 | Provider Adapters | Normalize request/response/usage/stream/errors across providers |
| 04 | Model Registry | Build the (model × provider) source of truth and keep it fresh |
| 05 | Routing Engine | Filter→select→fallback; aliases; fail-closed data-policy routing |
| 06 | Usage Metering | Record priced, attributed events; budgets, rate limits, chargeback |
| 07 | Streaming Proxy | Normalize many upstream streams; cancel-on-disconnect; capture usage |
| 08 | Admin Dashboard | Read (cost/health) + write (keys/routes/budgets) surfaces, secured |
| 09 | Enterprise Policy Engine | RBAC, PII/guardrails, residency, audit, tenant isolation → compliance |
How to work through it
Read 00 (the control-plane framing) first, then the two shapes — 01 (aggregator/OpenRouter) and 02 (self-hosted/LiteLLM) — to decide build vs buy. The component docs 03–07 are what you'd assemble to build one: adapters → registry → routing → metering → streaming. 08 (admin) and 09 (policy) make it operable and enterprise-ready. Many docs build directly on Phase 7 (routing/fallback [7.07], streaming [7.06], observability [7.08], cost [7.09]) — cross-links are explicit. Every doc opens with a from-zero plain-English primer and ends with a buildable lab; together they form the capstone gateway project (Phase 15).
Phase 8 artifacts
- A running gateway (LiteLLM) fronting ≥2 backends + a build-vs-buy note (00, 02).
- An aggregator provider-variance comparison + model fallback + transform demo (01).
- A provider adapter layer + conformance suite (03).
- A model registry (model × provider) seeded from a real catalog, with freshness/reconciliation (04).
- A filter→select→fallback routing engine with fail-closed data-policy routing (05).
- A metering layer (priced events, budgets, rate limits, chargeback, invoice reconciliation) (06).
- A normalizing streaming proxy (cancel-on-disconnect, per-provider usage capture) (07).
- An admin surface (Grafana reads + admin API/app writes, RBAC + audit) (08).
- A policy engine (RBAC, PII, fail-closed residency, audit, isolation) + red-team report + compliance map (09).