« Phase 04 · Warmup · Track Overview

Lab 01 — The LLM Gateway & Model Abstraction Layer

The problem

Twelve agent teams, six model providers, three regions, two capacity types, and a regulator who will ask where each inference ran. If every team calls providers directly you get: twelve sets of credentials, twelve retry policies (some of which retry payments), no idea what anything costs, no way to migrate off a provider, and no answer to "which model produced this decision?"

The gateway is the fix. It is the platform's choke point in the good sense — one place to enforce, one place to observe, one place to change providers — and it is the component this JD names first in the integrations section.

What you build

Key concepts

Files

Run

pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v
python solution.py

Success criteria

• All 69 tests green against your lab.py.
• ContentFiltered has fall_over = False, and a filtered primary raises rather than trying the next provider.
• A rule naming an unknown deployment fails at construction, and a typo'd classification in a rule fails at construction.
• restricted + eu-west yields no route, and the failure is accounted.
• A token-limit rejection does not consume an RPM slot.
• The quota boundary is inclusive: spend == budget refuses.
• A cache hit calls no provider and reports cost_micros == 0.
• The same prompt for a different tenant is a miss.
• A FinishReason.LENGTH response is never cached.
• A side-effecting request never falls over; a request with no headroom raises BudgetExhausted.
• Two identically-constructed gateways produce identical responses and identical records.

How this maps to the real stack

Honest limits. No streaming (which changes usage reporting and makes cancellation matter), no distributed rate limiting (the hard part at scale), no provider-side prompt-cache accounting, no weighted or learned routing, no request hedging, and no circuit breaker per deployment — that last one arrives in Phase 10 and belongs here too.

Extensions

1. A circuit breaker per deployment. Open after N failures in a window; skip open deployments in candidates(). Then measure how it interacts with the fallback chain.
2. Streaming. Make complete a generator. Discover that usage arrives last (or not at all unless you ask), and that your accounting has to survive a cancelled stream.
3. Distributed rate limiting. Replace the in-process bucket with a shared counter and think about what happens when the shared store is unavailable — fail open or fail shut?
4. Weighted routing and canaries. Split traffic 95/5 to compare a new deployment, with the split deterministic per session so a user does not flip mid-conversation.
5. Hedged requests. After p95, fire a second request to the fallback and take the first response. Then compute what it costs you in tokens versus what it buys in tail latency.
6. Reconciliation. Add a monthly billing export and a job that compares it to Accounting.cost_by("deployment"). The gap is where dropped usage blocks live.

Interview / resume bullets

• "Built the bank's LLM gateway: a normalized request/response and — the part that matters — a normalized error taxonomy across six providers, with retryable and fall_over as separate flags so a provider safety refusal never triggers failover to another model."
• "Made routing a policy over task class, tenant and data classification, with residency and per-deployment classification ceilings as an independent second gate, so restricted data provably cannot leave the region."
• "Made fallback budget-aware: the gateway refuses a fallback that does not fit the caller's remaining latency budget, and refuses any fallback at all for side-effecting requests — which removed the double-execution risk from every agent team at once."
• "Introduced per-tenant token accounting and cost attribution by tenant, agent, deployment and model, including failed calls, plus a failover-rate metric that surfaces a degrading provider before its error rate does."