« Track Overview · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 04 — The LLM Gateway & Model Abstraction Layer

Answers this JD line: "Architect and engineer the platform's LLM gateway and model abstraction layer, providing a unified interface across foundation model providers (Azure AI Foundry, AWS Bedrock, OpenAI, Anthropic, Google Vertex AI, Cohere) with intelligent routing, fallback, retries, prompt and response caching, semantic caching, rate limiting, token accounting, cost attribution, and tenant isolation."

Why this phase exists

This is the component the JD describes in the most detail, and for good reason: it is the only place where a platform can be a platform.

Without a gateway, every agent team holds provider credentials, writes its own retry policy, chooses its own model, and reports nothing. The consequences compound quietly: you cannot answer what anything costs, you cannot migrate off a provider, you cannot enforce residency, you cannot tell a regulator which model produced a decision, and at least one team's retry-on-timeout is sitting in front of a tool that moves money.

With a gateway, all of that becomes one component's job — and that concentration is the point. One place to enforce, one place to observe, one place to change.

Five ideas carry the phase:

1. Normalizing responses is the easy half; normalizing errors is the job. Six providers express "you are rate limited," "I refused on safety grounds," "your request is malformed" and "I broke" in six vocabularies. A caller cannot handle six.
2. retryable and fall_over are different flags. A safety refusal is retryable nowhere and must not trigger failover — trying providers until one answers is shopping for a compliant model, and a regulator will ask about it by name.
3. Route on what the caller is, not on what model it wants. Task class, tenant, data classification, residency. A caller that names a model has hard-coded a vendor decision into an agent, and you will find every one of those references the day you migrate.
4. A fallback must fit the latency budget. Otherwise a partial provider degradation becomes a total SLO breach, because every failover request breaches on its own.
5. Every cache key starts with the tenant. A semantic cache that can cross a tenant boundary is a data breach with an excellent hit rate — and it is the one failure in this phase with no runtime detection.

Concept map

• The abstraction layer: a normalized request (messages, task class, plus tenant, agent, classification, residency, latency budget, side-effecting) and a normalized response (text, finish reason, three-tier usage, deployment, cost, cache status, attempts).
• The error taxonomy: rate limited · timeout · unavailable · content filtered · invalid request · quota exceeded · no route · budget exhausted — each with retryable and fall_over.
• Deployments: provider × model × region × capacity (PAYG / provisioned / self-hosted), each with its own price triple, expected latency, and classification ceiling.
• Routing: ordered rules on task class / tenant / classification, then the constraints a rule cannot express — residency and the deployment's own ceiling.
• Fallback: ordered chain, budget-aware, refused for side-effecting requests, and never on a content filter.
• Rate limiting and quotas: per-tenant RPM and TPM token buckets, plus a monthly spend ceiling that fails closed.
• Caching: exact (hash of the whole request) · semantic (embedding similarity, tenant- partitioned, with a floor) · and provider-side prefix caching, which you influence by ordering the prompt rather than by storing anything.
• Accounting: token accounting by tier, cost attribution by tenant/agent/deployment/provider/ model, cache-hit rate, and failover rate as an early-warning signal.

The lab

Integrated scenario (how this shows up at work)

Tuesday, 14:20. The primary Azure deployment starts returning 429s at 30% of requests.

Without a gateway: twelve teams see errors, four of them retry immediately (making it worse), one of them retries a payment-initiating call, and nobody can say how much of the fleet is affected.

With the gateway: the failover rate metric moves before the error rate does. Requests fall over to the second deployment only if the remaining latency budget fits it — so an interactive agent with 400 ms of headroom fails fast rather than breaching its 3-second SLO, while a batch job with 30 seconds of budget quietly succeeds. Side-effecting calls do not fail over at all. Cost attribution shows the shift to the more expensive fallback in real time, and the quota ledger stops a tenant before the surprise arrives as an invoice.

Every one of those behaviours is a specific decision in this lab.

Deliverables checklist

• Lab 01 green under LAB_MODULE=solution pytest and under your own lab.py.
• You can list the eight normalized error classes and say which two flags each carries.
• You can explain why a content filter must not trigger failover.
• You can explain why a deployment rather than a model is the routing target.
• You can write a routing policy for restricted data with a residency constraint.
• You can do the fallback budget arithmetic on a whiteboard.
• You can state the three cache tiers, their keys, and the two rules for semantic caching.
• You can name four things you attribute cost by, and why failures must be included.

Key takeaways

• The gateway is a policy enforcement point that happens to speak HTTP. Treating it as a proxy is how it ends up owning nothing.
• Error normalization is the abstraction layer. Anyone can normalize a happy path.
• fall_over is a separate decision from retryable, and content filtering is the case that proves it.
• A fallback without a budget check is a way to breach your SLO faster.
• Never fail over a side-effecting call. The gateway cannot know whether the first attempt landed.
• Tenant first in every key, and never cache a truncated or entitlement-dependent answer.
• Account failures. The cost you cannot see is the cost you incur during an incident.