Lab 08 — Build an LLM Gateway

Core path works offline with a mock provider; add real providers with API keys. Build the gateway that fronts multiple providers. Concepts: Phase 8, cheatsheet 13.

Goal

A minimal OpenAI-compatible gateway: provider adapters + model registry + routing + fallback + usage metering + streaming passthrough.

Run (offline, zero deps)

python3 gateway.py    # runnable demo: cost routing, capability routing, fallback on outage, per-tenant metering

gateway.py is a complete, dependency-free implementation (adapters + registry + router + meter) using an offline MockAdapter. Swap in a real OpenAIAdapter (same interface) to go live. The "suggested files" below are how you'd split it for a production service.

Suggested files

lab-08-llm-gateway/
  README.md
  adapters.py     # base Adapter + OpenAIAdapter + MockAdapter (offline)
  registry.py     # model × provider × price × limits × capabilities
  router.py       # cost/difficulty routing + fallback + retries
  server.py       # FastAPI /v1/chat/completions (OpenAI-compatible), streaming
  metering.py     # per-key/tenant token + cost accounting

Key snippet (adapter + router shape)

# adapters.py
class Adapter:
    def complete(self, model, messages, **kw): raise NotImplementedError

class MockAdapter(Adapter):           # offline: deterministic, free
    def complete(self, model, messages, **kw):
        return {"content": f"[mock:{model}] " + messages[-1]["content"][:50],
                "in_tokens": 20, "out_tokens": 10}

# router.py
def route(req, registry):
    candidates = registry.for_capability(req.needs)          # filter
    candidates.sort(key=lambda m: m.price_out)               # cheapest first
    for m in candidates:                                     # fallback chain
        try:
            return m.adapter.complete(m.name, req.messages)
        except Exception:
            continue
    raise RuntimeError("all providers failed")

Steps

  1. Implement MockAdapter (offline) + a registry of 2–3 fake models with prices/capabilities.
  2. Implement cost-based routing + a fallback chain (kill a provider → it fails over).
  3. Add per-tenant usage metering (tokens + cost) and a /usage view.
  4. Add an OpenAI-compatible /v1/chat/completions endpoint + SSE streaming passthrough.
  5. (Real) swap MockAdapter for OpenAIAdapter (+ one more) behind the same interface.

Deliverables

  • Routing demo (easy→cheap, hard→premium) + a fallback demo (provider down → served).
  • Per-tenant usage/cost report.
  • Streaming passthrough working.
  • A note on key custody (real keys server-side; scoped internal keys) (Phase 14.03).

Why it matters (interview)

The platform-engineer centerpiece — and component 1–3,6,8 of the Capstone (interview-prep 02).