Cheatsheet 13 — Model Gateway Architecture
The components of an LLM gateway/proxy. Full: Phase 8. Lab: labs/lab-08-llm-gateway.
Why a gateway
One internal API in front of many providers/models → routing, fallback, cost control, metering, governance, observability, no client-side keys, no vendor lock-in.
Components (the building blocks)
client → [ AUTH ] → [ RATE LIMIT / QUOTA ] → [ ROUTER ] → [ PROVIDER ADAPTER ] → provider/model
│ │
[ MODEL REGISTRY ] [ FALLBACK / RETRY ]
│
[ USAGE METERING ] ← [ STREAMING PROXY ] ← [ CACHE ] ← [ GUARDRAILS / POLICY ]
│
[ OBSERVABILITY / AUDIT ]
| Component | Job | Phase |
|---|---|---|
| Provider adapters | Normalize each provider's API to one shape | 8.03 |
| Model registry | Catalog of (model × provider) + metadata/price/limits | 8.04 |
| Routing engine | Pick model/provider by cost/latency/quality/availability | 8.05 |
| Fallback/retry | Failover on error/timeout; circuit breakers | 7.07 |
| Streaming proxy | Pass SSE tokens through transparently | 8.07 |
| Usage metering | Per-key/team/tenant token + cost accounting | 8.06 |
| Caching | Prompt/prefix/response cache | 7.05 |
| Guardrails/policy | Input/output checks, PEP/PDP, fail-closed | 14.05 |
| Auth + key custody | Real provider keys server-side; issue scoped internal keys | 14.03 |
| Observability/audit | Traces, p95/p99, cost, audit logs | 7.08 / 14.06 |
Routing strategies
| Strategy | When |
|---|---|
| Cost-optimized (cheapest that meets quality) | Bulk/easy requests |
| Latency-optimized | User-facing/interactive |
| Quality/difficulty-based (small↔premium) | Mixed workloads (Phase 11.06) |
| Availability/failover | Provider outage resilience |
| Tenant/policy-based | Residency/compliance routing (fail-closed, 14.07) |
Build vs buy
- Buy/OSS: OpenRouter (aggregator API), LiteLLM (self-hosted proxy) — start here.
- Build: when you need custom routing/policy/metering or it's your product (Phase 8.01–8.02).
Interview line
"A gateway gives me provider abstraction, routing/fallback for resilience, central cost metering, and key custody — so clients never hold provider keys, I can fail over on outages, and I route easy requests to cheap models to protect margin."
Next: 14 — RAG Design · Full: Phase 8