What Is an LLM Gateway?

Phase 8 · Document 00 · LLM Gateways Prev: Phase 7 — Production Serving · Up: Phase 8 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

An LLM gateway is the control plane for an organization's LLM usage — one endpoint that fronts every model and provider, with centralized routing, fallback, cost tracking, budgets, policy, and observability. Phase 7 taught you to serve models; Phase 8 is the product/platform that sits in front of all of them (your self-hosted vLLM and OpenAI/Anthropic/Google) so applications get a single, provider-agnostic, governed interface. Without a gateway, every team hardcodes keys and provider quirks, no one can see or cap spend, one provider outage breaks everything, and there's no audit trail. With one, you swap providers without touching app code, enforce budgets and policy in one place, and survive outages automatically. This is the architecture behind OpenRouter, LiteLLM, Portkey, Cloudflare AI Gateway, and every serious enterprise AI platform — and a top startup category in its own right (Phase 15).

2. Core Concept

Plain-English primer: one front door for all LLMs

A gateway is an HTTP proxy that speaks the OpenAI-compatible API and sits between your apps and every LLM backend. Apps call one endpoint with one API shape; the gateway does auth, routing, format translation, metering, and policy, then forwards to the right backend and normalizes the response.

App → POST /v1/chat/completions {"model":"auto/coding", ...}  (OpenAI shape)
   → GATEWAY:  auth → budget/quota → policy → ROUTE (pick model+provider)
             → adapter (translate to provider format) → forward → normalize response
             → meter (tokens/cost) → log/trace
   → Backend: OpenAI / Anthropic / Google / your self-hosted vLLM   [Phase 7]
App ← unified OpenAI-compatible response (+ usage)

The reason everything speaks OpenAI-compatible is interoperability: it's the de-facto lingua franca, so any SDK, IDE BYOK, or tool that talks to OpenAI also talks to your gateway — and you can swap the backend model by changing a string, not code (Phase 6.04, Phase 7.01).

Gateway vs proxy vs aggregator vs router (the words people conflate)

They overlap: a gateway contains a router and is a proxy; an aggregator is a hosted gateway across third-party providers. This phase builds the gateway and studies the two canonical shapes: the aggregator (OpenRouter, 01) and the self-hosted unifying proxy (LiteLLM, 02).

The components (the map of this phase)

A gateway decomposes into parts you'll build one per doc:

• Provider adapters (03) — translate the unified format ↔ each provider's API (and stream shape).
• Model registry (04) — the source-of-truth DB of models, capabilities, and pricing.
• Routing engine (05) — filter to eligible models, then pick by cost/latency/quality/policy; aliases like auto/coding.
• Usage metering (06) — record tokens/cost per request; enforce budgets/quotas; chargeback.
• Streaming proxy (07) — forward SSE across providers, normalize events, capture usage, cancel on disconnect.
• Admin dashboard (08) — manage models/routes/keys; see usage/cost/errors/health.
• Policy engine (09) — auth/RBAC, PII/guardrails, data residency, audit, tenant isolation.

Much of the mechanism came from Phase 7 (routing/fallback [7.07], streaming [7.06], observability [7.08], cost [7.09]); Phase 8 packages it as a multi-provider, multi-tenant product.

Build vs buy

You don't always build one: buy/host OpenRouter (hosted aggregator) or LiteLLM/Portkey (self-host) when you want speed-to-value; build when you need deep custom policy/routing, full data control, or it is your product. Most teams start with LiteLLM/OpenRouter and build only what's differentiated.

3. Mental Model

        ┌──────────────────────── LLM GATEWAY (control plane) ────────────────────────┐
  app → │  AUTH → BUDGET/QUOTA → POLICY → ROUTING ENGINE → ADAPTER → (forward) → NORMALIZE │ → app
        │            [06]          [09]       [05]          [03]                    [07]    │
        │  backed by: MODEL REGISTRY [04]   ·   USAGE METER [06]   ·   ADMIN UI [08]        │
        └──────────────────────────────────┬───────────────────────────────────────────────┘
                                            ▼  forwards to ANY backend (OpenAI shape everywhere)
              OpenAI · Anthropic · Google · your self-hosted vLLM/TGI  [Phase 7]

  TWO canonical shapes:  AGGREGATOR (OpenRouter, hosted, many providers) [01]
                         SELF-HOSTED PROXY (LiteLLM, your infra)         [02]
  one OpenAI-compatible front door → swap models by string, govern centrally

Mnemonic: a gateway is one OpenAI-compatible front door = proxy + router + registry + meter + policy + admin, fronting every provider.

4. Hitchhiker's Guide

What to look for first: the unified OpenAI-compatible API and the routing + fallback + metering trio. Those deliver the headline value (provider-agnostic, reliable, cost-visible) before any fancy policy.

What to ignore at first: building your own aggregator from scratch, exotic routing ML, and a heavy admin UI. Start with LiteLLM (config + fallbacks + budgets); add custom components only where you're differentiated.

What misleads beginners:

• "A gateway is just a proxy." The value is the control plane — registry, metering, policy, observability — not just forwarding.
• Inventing a new API shape. You lose the OpenAI-ecosystem interop that makes a gateway useful.
• Adding latency you don't budget for. A gateway adds a network hop + processing; keep its overhead small and stream through it (07).
• Centralizing without reliability. A gateway is now a single point of failure for all LLM traffic — it needs its own HA, fallbacks, and circuit breakers (Phase 7.07).

How experts reason: they treat the gateway as the place to centralize cross-cutting concerns (auth, cost, policy, routing, observability) so apps stay simple; they keep the data plane thin/fast (stream-through, low overhead) and the control plane rich; and they make the gateway itself highly available because everything depends on it.

What matters in production: added latency (p95 overhead of the hop), HA of the gateway, correct usage/cost attribution (06), policy enforcement (esp. data residency/PII, 09), and fallback success (Phase 7.07).

How to debug/verify: trace a request through the gateway's stages; confirm routing picked the expected backend, usage was metered, policy applied, and streaming forwarded without buffering. Load-test the gateway itself.

Questions to ask (build vs buy): Do I need custom policy/routing, or will LiteLLM/OpenRouter do? Where must data stay (self-host vs hosted aggregator)? What's the gateway's own HA/SLA? Per-tenant budgets + audit?

What silently gets expensive/unreliable: the gateway as an un-redundant SPOF, latency creep from the extra hop, broken cost attribution (billing drift), and policy gaps (sensitive data to a disallowed provider).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• A gateway is one OpenAI-compatible front door fronting every provider — apps swap models by string, not code.
• It's a control plane, not just a proxy: registry + routing + metering + policy + observability + admin.
• Two canonical shapes: hosted aggregator (OpenRouter, 01) and self-hosted unifying proxy (LiteLLM, 02).
• Mechanism is mostly Phase 7 (routing/fallback, streaming, observability, cost) — the gateway packages it multi-provider, multi-tenant.
• It becomes a single point of failure for all LLM traffic — it needs its own HA + fallbacks.
• Keep the data plane thin/fast (stream-through, low overhead); put richness in the control plane.
• Correct usage/cost attribution and policy enforcement are the gateway's reason to exist for enterprises.
• Build vs buy: start with LiteLLM/OpenRouter; build only differentiated parts.

9. Observations from Real Systems

• OpenRouter = hosted aggregator: one key → hundreds of models across providers, with routing, fallbacks, and unified billing (01).
• LiteLLM = the self-hosted proxy of choice: a config.yaml of models, fallbacks, budgets, and an OpenAI-compatible endpoint (02).
• Portkey, Kong AI Gateway, Cloudflare AI Gateway, Helicone productize gateway features (caching, guardrails, analytics) — the category is crowded and growing.
• Enterprises build internal gateways to centralize keys, budgets, audit, and data-policy routing across many teams (09, Phase 14).
• IDE BYOK and coding tools point at gateways precisely because of the OpenAI-compatible standard (Phase 11).

10. Common Misconceptions

11. Engineering Decision Framework

DO I NEED A GATEWAY?  (multiple models/providers, multiple teams, cost/policy/reliability needs → yes)
 1. BUILD vs BUY:
      speed-to-value, standard needs        → BUY/HOST: OpenRouter (aggregator) or LiteLLM (self-host)
      data must stay in-house / deep custom  → SELF-HOST LiteLLM or BUILD
      it IS your product                     → BUILD (this phase)                       [Phase 15]
 2. COMPONENTS to stand up (in order of value):
      unified OpenAI API + adapters [03] → routing + fallback [05,7.07] → usage metering + budgets [06]
      → streaming proxy [07] → admin [08] → policy/RBAC/residency/audit [09]
 3. BACK with a MODEL REGISTRY [04] (source of truth for caps+pricing+routes).
 4. MAKE THE GATEWAY HA (it's now a SPOF): redundancy, health checks, circuit breakers.
 5. KEEP DATA PLANE THIN (stream-through, low overhead); measure the gateway's own p95.   [7.08]

12. Hands-On Lab

Goal

Stand up a working gateway (buy-then-extend): run LiteLLM in front of two backends, then trace a request through routing → metering → response, proving the provider-agnostic + cost-visible value.

Prerequisites

• A managed API key (OpenAI/Anthropic) and a local model (Phase 6.04); pip install 'litellm[proxy]'.

Setup

# config.yaml
model_list:
  - {model_name: smart,  litellm_params: {model: anthropic/claude-..., api_key: os.environ/ANTHROPIC_API_KEY}}
  - {model_name: cheap,  litellm_params: {model: openai/gpt-...-mini, api_key: os.environ/OPENAI_API_KEY}}
  - {model_name: local,  litellm_params: {model: openai/llama3.2:3b, api_base: http://localhost:11434/v1, api_key: none}}
litellm_settings: {fallbacks: [{"smart": ["cheap","local"]}], request_timeout: 30}
general_settings: {master_key: sk-master}

litellm --config config.yaml --port 4000

Steps

1. Unified API: call http://localhost:4000/v1/chat/completions with the OpenAI SDK for smart, cheap, and local — same client, three backends. This is the gateway value.
2. Fallback: break the smart backend (bad key) and confirm requests reroute to cheap/local and still succeed (Phase 7.07).
3. Metering: enable LiteLLM's DB/spend tracking; send 10 requests; query the recorded usage (tokens, cost, model) — confirm cost attribution (06).
4. Budget: set a low per-key budget; exceed it; confirm the gateway rejects with 429 (06).
5. Trace the stages: log/observe auth → route → backend → usage for one request; map each to the §2 diagram.

Expected output

One OpenAI-compatible endpoint serving three backends with working fallback, recorded per-request usage/cost, and a budget rejection — the core gateway value demonstrated.

Debugging tips

• 401 → master key/headers; 404 model → name not in model_list.
• Fallback didn't trigger → check fallbacks mapping and that the error is a failover type (Phase 7.07).

Extension task

Add auto/coding as an alias routing to the best available coding model, and a data-policy rule forcing "sensitive" requests to local (05, 09).

Production extension

Front it with your own auth/RBAC and an admin view of usage/cost (08, 09); make the gateway itself HA.

What to measure

Gateway overhead (added p95), fallback success, usage/cost attribution accuracy, budget enforcement.

Deliverables

• A running gateway (LiteLLM) fronting ≥2 backends via one OpenAI-compatible API.
• A fallback demonstration + recorded usage/cost + a budget rejection.
• A request-stage trace mapped to the component diagram.

13. Verification Questions

Basic

1. What is an LLM gateway, and what does it centralize?
2. Distinguish gateway, proxy, aggregator, and router.
3. Why do gateways standardize on the OpenAI-compatible API?

Applied 4. List the gateway components and what each does. 5. Build vs buy: when do you self-host LiteLLM vs use OpenRouter vs build your own?

Debugging 6. All LLM traffic is down though providers are up. What likely failed, and what was missing? 7. Latency rose after introducing the gateway. What do you check?

System design 8. Design a multi-tenant gateway: unified API, routing+fallback, metering+budgets, policy, admin, HA.

Startup / product 9. Why are gateways a strong startup category, and what would differentiate yours?

14. Takeaways

1. A gateway is one OpenAI-compatible front door fronting every provider — provider-agnostic apps, centralized control.
2. It's a control plane = proxy + router + registry + metering + policy + admin, not just forwarding.
3. Two shapes: hosted aggregator (OpenRouter) and self-hosted proxy (LiteLLM); mechanism is mostly Phase 7, packaged multi-provider/tenant.
4. It's a SPOF — make it HA; keep the data plane thin, the control plane rich.
5. Build vs buy: start with LiteLLM/OpenRouter; build the differentiated parts.

15. Artifact Checklist

• A running gateway fronting ≥2 backends via one OpenAI-compatible API.
• A fallback demo + recorded usage/cost attribution.
• A budget enforcement test (429 over limit).
• A request-stage trace mapped to the component map.
• A build-vs-buy decision note for your context.

Up: Phase 8 Index · Next: 01 — OpenRouter-Style Architecture