« Phase 04 · Track Overview

Warmup — The LLM Gateway, From Zero

Assumes HTTP and Phase 00's budget arithmetic. Assumes nothing about model providers, token pricing, semantic caching or rate-limiting algorithms.

Table of Contents

• 1. What a gateway is, and why it exists
• 2. The abstraction layer
  • 2.1 The normalized request
  • 2.2 The normalized response
  • 2.3 Normalizing errors — the actual job
• 3. Deployments, not models
• 4. Routing
• 5. Fallback
  • 5.1 The budget arithmetic
  • 5.2 The two absolute refusals
  • 5.3 Retry versus failover
• 6. Rate limiting and quotas
• 7. Caching, three tiers
  • 7.1 Exact
  • 7.2 Prefix (provider-side)
  • 7.3 Semantic
• 8. Token accounting and cost attribution
• 9. Tenant isolation at the gateway
• 10. Lab walkthrough
• 11. Success criteria
• 12. Common mistakes
• 13. Interview Q&A
• 14. References

1. What a gateway is, and why it exists

An LLM gateway (or AI gateway) is a single ingress in front of every model provider. Every model call in the bank goes through it.

The naive objection is that it adds a hop. The response is that it adds the only place a platform-level decision can be made. Without it:

The concentration is the point, and it is also the risk: the gateway becomes a serial dependency for everything. Phase 00's arithmetic applies — it must be highly available, or it caps the platform. In practice that means it is stateless, horizontally scaled, and holds only caches and counters that it can lose.

2. The abstraction layer

2.1 The normalized request

NormalizedRequest(
    messages=(Message("system", "..."), Message("user", "...")),
    task_class=TaskClass.REASONING,
    tenant="wholesale",                 # from the verified token, never the body
    agent_id="payments-investigator",
    max_output_tokens=512,
    temperature=0.0,
    data_classification="restricted",
    residency="uae-north",
    latency_budget_ms=3000,
    side_effecting=False,
    cacheable=True,
)

The top half is what every provider SDK has. The bottom half is why this type exists. Tenant, classification, residency, latency budget and side-effecting are platform concerns that no provider SDK can carry, and each one drives a decision:

A gateway that accepts a provider's native request shape and adds these as HTTP headers works, and is worse: headers are optional by convention, so they get forgotten, and a forgotten data_classification defaults to something.

2.2 The normalized response

Two fields do disproportionate work.

finish_reason. Providers use different strings; the caller must be able to detect LENGTH in particular, because it means the answer is truncated — and a truncated answer must never be cached, never be treated as complete, and usually should be retried with a larger budget. Normalizing this into an enum is what makes "never cache a non-STOP response" a one-line rule.

usage, in three tiers. input_tokens, cached_input_tokens, output_tokens. Providers report cached input differently — some as a separate field, some folded into the input count with the discount applied at billing. Normalizing them is a substantial part of the layer's value, and getting it wrong means your cost model is quietly wrong in exactly the direction that flatters you.

2.3 Normalizing errors — the actual job

The happy path is easy. Here is what six providers actually send you when something goes wrong, conceptually:

• rate limited (429), with or without Retry-After;
• request too large, in tokens, which is a 400 that looks like a limit;
• content filtered by the provider's own safety system, sometimes as a 400, sometimes as a 200 with an empty completion and a finish_reason;
• model overloaded (529, 503), which is transient;
• a genuine 5xx;
• a timeout, which your client raises rather than the provider;
• an authentication failure;
• a model that has been deprecated out from under you.

A caller cannot handle six vocabularies. So the gateway maps them onto a small taxonomy, and — this is the part people miss — each class carries two independent flags:

Why ContentFiltered must not fail over. If deployment A's safety system refuses and the gateway tries B, and then C, you have built a system that tries providers until one of them agrees to produce the content. That is shopping for a compliant model, and it is a sentence a regulator will say back to you. The refusal is a signal, not an obstacle: log it, surface it, count it, and stop.

retryable and fall_over being separate also lets you express "retry the same deployment but do not move on" (rare) and "move on but do not retry here" (common under sustained rate limiting).

3. Deployments, not models

A model is not a routing target. A deployment is: a specific model, on a specific provider, in a specific region, on a specific capacity type.

azure-gpt-uae-ptu    azure · gpt-frontier · uae-north · PROVISIONED · 700 ms
azure-gpt-uae-payg   azure · gpt-frontier · uae-north · PAYG        · 900 ms
anthropic-eu         anthropic · claude-frontier · eu-west · PAYG   · 800 ms
self-hosted-uae      vllm · llama-open · uae-north · SELF_HOSTED    · 1400 ms

The first two are the same model and differ in every way that matters to a routing decision: latency (dedicated capacity does not queue behind other tenants), price, and behaviour under load. The third has a different residency answer. The fourth has a different price by an order of magnitude and a different classification ceiling.

Each deployment carries a price triple (input, cached input, output per 1 000 tokens) rather than one price, because the three tiers differ by an order of magnitude in both directions — output is typically several times input, and cached input is a small fraction of it.

4. Routing

A routing rule matches on what the caller is:

RoutingRule("restricted-must-stay-onshore",
            deployments=("azure-gpt-uae-ptu", "azure-gpt-uae-payg", "self-hosted-uae"),
            classifications=("restricted",), priority=10)
RoutingRule("cheap-classification",
            deployments=("self-hosted-uae", "azure-gpt-uae-payg"),
            task_classes=(TaskClass.CLASSIFICATION,), priority=20)
RoutingRule("default",
            deployments=("azure-gpt-uae-ptu", "anthropic-eu", "azure-gpt-uae-payg"),
            priority=100)

Rules are ordered by priority; the first match whose chain is non-empty wins. Then two constraints that a rule cannot express are applied as a second gate:

• residency — if the request says uae-north, an eu-west deployment is removed;
• the deployment's own classification ceiling — a deployment rated for confidential is removed from a restricted request even if the rule matched.

Two gates rather than one is deliberate and is the Phase 00 defence-ordering rule applied here: a misconfigured rule that lists an offshore deployment for restricted data is still caught. The lab tests exactly that case.

The anti-pattern to name in a review: a request that says model="gpt-4o". It works, and it means an agent has hard-coded a vendor decision. When that model is deprecated — and it will be — you have a code change in twelve repositories instead of a rule change in one.

5. Fallback

5.1 The budget arithmetic

The primary fails at time t. Should you try the next deployment?

$$\text{remaining} = \text{latency_budget_ms} - \text{elapsed} \qquad \text{allowed if } \text{remaining} \ge \text{expected_latency}_{\text{next}}$$

Worked: a 3 000 ms budget, the primary times out after 2 400 ms, the fallback's expected latency is 800 ms. Remaining is 600 ms; 600 < 800, so no fallback — fail fast with a clear error.

That looks like giving up too easily. It is the opposite: attempting the fallback produces a 3 200 ms response that has already breached the SLO, and it does so for every affected request. A provider degradation that would have cost you a partial error rate becomes a total SLO breach across the fleet. Failing fast preserves the budget for the requests that can still succeed.

The corollary from Phase 00: headroom is the fallback decision. If your latency budget has no headroom, you do not have a fallback, and you should know that before the incident rather than during it.

5.2 The two absolute refusals

Side-effecting requests never fail over. If the request will cause the model to emit a tool call that changes state, the gateway cannot know whether the first attempt landed. A timeout is not evidence of non-execution — the request may have been processed and the response lost. So the gateway raises, and the caller (which does have an idempotency key, from Phase 10) decides.

Content filtering never fails over. §2.3.

5.3 Retry versus failover

They are different operations and both need a policy:

• Retry = the same deployment, after a backoff. Correct for a transient blip, and it is where Retry-After matters.
• Failover = the next deployment in the chain. Correct for sustained degradation.

Under sustained rate limiting, retrying the same deployment is actively harmful — you are adding load to something that is already shedding it — so the right shape is usually one fast retry (or none) followed by failover. Exponential backoff with full jitter (sleep = random(0, base * 2^attempt)) is the standard, and the jitter matters more than the backoff: without it, every client retries in lockstep and creates a thundering herd on recovery.

6. Rate limiting and quotas

Rate limit bounds requests per unit time and protects the system. Quota bounds consumption per period and protects the budget. They are different windows, different enforcement points, and different error codes.

The gateway needs both rate dimensions:

• RPM (requests per minute) — protects against a flood of small requests.
• TPM (tokens per minute) — protects against one 100 000-token request.

An RPM-only limit does not protect the provider's capacity; a TPM-only limit does not stop a misbehaving loop. The lab enforces both with token buckets (see Phase 00's cheat sheet for the algorithm).

One subtlety the lab tests: a request rejected on tokens must not also consume an RPM slot. Otherwise a tenant that is over its token budget silently burns its request budget too, and is throttled twice for one attempt. Check both, then consume both.

Quotas fail closed. A tenant at its monthly ceiling is refused, before any provider is called. This is not only a budget control: unbounded consumption by one tenant degrades every other tenant through the provider's shared capacity, so the quota is an availability control wearing a finance costume.

7. Caching, three tiers

7.1 Exact

Key: a hash of everything that could change the answer — tenant first, then deployment, task class, temperature, max output tokens, and the full prompt.

Hit rate depends entirely on traffic shape: near zero for open conversation, high for classification, extraction and batch work. Cheap, safe, and the first thing to add.

Two rules the lab enforces:

• Never cache a non-STOP finish. A truncated or filtered answer is not the answer.
• Never cache when cacheable=False. The caller marks entitlement-dependent answers, and the gateway obeys without arguing.

7.2 Prefix (provider-side)

Not a cache you build — a discount you earn. Providers reuse computation for a shared prompt prefix and bill those tokens at a fraction of the normal rate.

You influence it by ordering the prompt: stable content first (system prompt, tool schemas, policy text), volatile content last (the user's turn, retrieved chunks). That single ordering decision can move a large fraction of an agent's input tokens into the cheap tier, and it costs nothing to implement.

It is also why NormalizedRequest.stable_prefix() exists in the lab: making the prefix an explicit concept means you can measure how much of your prompt is stable, and that number is directly a cost lever.

7.3 Semantic

Key: embedding similarity of the prompt. Enormously effective — a 30% hit rate at near-zero marginal cost is a 30% saving — and the single most dangerous thing in this phase.

Three non-negotiables, and you say all three whenever you propose one:

1. Tenant-partitioned, not tenant-filtered. Entries live in separate partitions. Filtering after ranking is one refactor away from not filtering at all, and the failure mode is that tenant B receives tenant A's correct-looking answer — a 200 OK, a happy user, and a breach discovered months later. It is the only failure in this phase with no runtime detection.
2. A similarity floor validated against negative examples. "Is this payment held?" and "Is this payment not held?" are extremely close in embedding space and have opposite answers. Tune the threshold against pairs you know must not collide, not by picking 0.9 because it sounds right.
3. Never for entitlement-dependent answers. "What is my balance?" is the same question from every user and has a different answer for each. If the answer depends on who is asking, do not cache the answer — cache the retrieval, if anything.

8. Token accounting and cost attribution

$$\text{cost} = \frac{(t_{\text{in}} - t_{\text{cached}}),c_{\text{in}} + t_{\text{cached}},c_{\text{cache}} + t_{\text{out}},c_{\text{out}}}{1000}$$

Computed in integer micro-USD and divided last, so accumulated cost is exact and a month of records sums without float drift.

Attribute by tenant (chargeback), agent (which agent is expensive), deployment (is the fallback costing us), provider (concentration risk), and model (is anyone still on the deprecated one).

Three rules that separate a working cost model from a decorative one:

• Record failures. A call that timed out after the provider generated 400 tokens still cost money, and a cost model that counts only successes under-reports exactly during an incident, which is when you need it.
• Record cache hits with zero cost. Otherwise a hit double-counts the original spend and your savings look like spending.
• Reconcile monthly against the provider's billing export. The gap is where dropped usage blocks, uncounted retries and unit misunderstandings live. It is unglamorous and it catches real bugs.

Two derived metrics worth alerting on:

• Cache hit rate — a sudden drop usually means a prompt changed and broke prefix stability.
• Failover rate — this moves before the error rate does, because failover is what converts a provider's errors into your successes. It is the best early warning the gateway produces.

9. Tenant isolation at the gateway

The gateway is where multi-tenancy is either enforced or lost. Five places the tenant appears, and all five are required:

1. Rate-limit buckets — per tenant, or one tenant starves the rest.
2. Quota ledger — per tenant, or one tenant spends the platform's budget.
3. Cache keys — tenant first, always.
4. Routing — a tenant may have its own deployments (dedicated capacity, or a residency requirement no one else has).
5. Accounting — or you cannot charge back, and cost becomes a tragedy of the commons.

And the rule inherited from every other phase: the tenant comes from the verified token, never from the request body. A tenant field a caller can set is a caller who can read another tenant's cache.

10. Lab walkthrough

Work Lab 01 in this order.

1. NormalizedRequest, Usage, Deployment.cost_micros (§2). Small; the validation tests are free correctness. Divide by 1000 last.
2. The error taxonomy flags (§2.3). Two class attributes per class. Get ContentFiltered right.
3. classification_rank, RoutingRule, Router (§4). Validate rules at construction — unknown deployment and typo'd classification both fail there, not at 3 a.m.
4. TokenBucket (§6). Refill before every decision; never go negative.
5. RateLimiter.admit (§6). Check both buckets before consuming either.
6. QuotaLedger (§6). The boundary is inclusive.
7. hash_embed, cosine, cache_key (§7). Tenant first in the key; handle the all-zero vector without dividing by zero.
8. ExactCache, SemanticCache (§7). Partition by tenant; expire on read.
9. Accounting (§8). Sorted output; refuse an unknown dimension.
10. Gateway._execute_with_fallback (§5). The order inside the loop is the lesson: side-effecting check, then budget check, then attempt.
11. Gateway.complete (§1–§8). Quota → rate limit → route → cache → execute → store.
12. make_scripted_adapter — the seam that makes every failure reproducible.

Then python solution.py and read the seven sections against §§2–8.

11. Success criteria

Without the guide open:

• List the eight error classes and both flags for each.
• Explain why content filtering must not fail over, in a sentence a regulator would accept.
• Explain why a deployment rather than a model is the routing target.
• Write a routing policy with residency and classification, and say why the deployment ceiling is a second gate.
• Do the fallback budget arithmetic and explain why failing fast is the right answer.
• Explain why side-effecting calls never fail over.
• Distinguish retry from failover and say what full jitter is for.
• Explain RPM vs TPM and why a token rejection must not spend a request slot.
• Name the three cache tiers, their keys, and the three semantic-cache rules.
• Explain why prefix caching is a prompt-ordering decision.
• Say what you attribute cost by, and why failures and cache hits both need records.
• Name the five places the tenant must appear.

12. Common mistakes

Normalizing responses but not errors. The caller now handles six vocabularies anyway.

Failing over on a content filter. Shopping for a compliant model.

One retryable flag doing both jobs. You will either retry a filter or refuse to fail over on a 429.

Routing on a model name. A vendor decision hard-coded into twelve repositories.

A fallback with no budget check. Partial degradation becomes a total SLO breach.

Failing over a side-effecting call. Double execution, and the gateway cannot tell.

Retrying hard into a 429. Adding load to something that is shedding load.

No jitter. Every client retries in lockstep; recovery causes a second outage.

RPM without TPM. One request can be 100 000 tokens.

Consuming the request slot on a token rejection. Throttled twice for one attempt.

A cache key without the tenant. The only silent, severe failure in this phase.

Caching a truncated answer. You will serve it for the rest of its TTL.

Counting only successful calls. Your cost model is blind exactly during incidents.

Never reconciling against billing. The gap is real and it is always in the same direction.

13. Interview Q&A

Q: Design our LLM gateway.

A: "Single ingress in front of every provider, and I'd frame it as a policy enforcement point that happens to speak HTTP rather than as a proxy. It owns a normalized request — messages plus the things a provider SDK can't carry: tenant, agent, data classification, residency, latency budget, and whether the call is side-effecting — and a normalized response with a normalized finish reason and three-tier usage. The hard half is normalizing errors: six providers express rate limiting, safety refusal, malformed request and overload in six vocabularies, and each of my classes carries two independent flags, retryable and fall_over. Routing is a policy over task class, tenant and classification, with residency and each deployment's classification ceiling as a second, independent gate — deployments rather than models, because the same model in two regions has two latencies, two prices and two residency answers. Then budget-aware fallback, per-tenant RPM and TPM buckets, a monthly quota that fails closed, three cache tiers, and token accounting that includes failures. And it has to be stateless and horizontally scaled, because it's now a serial dependency for everything and Phase-00 arithmetic says it caps the platform."

Q: A provider's safety filter refuses. What does the gateway do?

A: "Stops. It records the refusal, surfaces it, and counts it — and it explicitly does not try the next provider. That's why fall_over is a separate flag from retryable in my taxonomy: if a content filter triggered failover, I'd have built a system that tries providers until one agrees to produce the content, which is shopping for a compliant model. That's a sentence a regulator will say back to me, and I'd rather never have to answer it. The refusal is a signal, not an obstacle — a rising filter rate is either an attack or a broken prompt, and both need a human."

Q: The primary times out at 2 400 ms of a 3-second budget. Fall over?

A: "No. The fallback's expected latency is 800 ms and I have 600 left, so attempting it produces a 3 200 ms response that has already breached — for every affected request. A provider degradation that would have cost me a partial error rate becomes a total SLO breach across the fleet. So I fail fast with a clear error and preserve the budget for requests that can still succeed. The general form is: headroom is the fallback decision, and if the latency budget has no headroom, I don't have a fallback and I want to know that at design time rather than during the incident. The other absolute refusal is side-effecting requests — if the call will emit a tool call that changes state, a timeout isn't evidence of non-execution, so the gateway raises and lets the caller decide with its idempotency key."

Q: How do you make caching safe in a multi-tenant bank?

A: "Three tiers, three different risk profiles. Exact-match caching is keyed on a hash of everything that could change the answer with the tenant as the first component; it's safe and its hit rate depends entirely on traffic shape. Prefix caching isn't a cache I build — it's a discount I earn by ordering the prompt so stable content comes first and volatile content last, and stable_prefix() exists in my design so I can measure what fraction is stable, because that's directly a cost lever. Semantic caching is the dangerous one and I'd state three conditions whenever I propose it: tenant-partitioned rather than tenant-filtered, because filtering after ranking is one refactor from not filtering; a similarity floor tuned against negative examples, because 'is this payment held' and 'is this payment not held' are close in embedding space with opposite answers; and never for entitlement-dependent answers. And the reason I'm careful is that a semantic cache mis-hit is the only failure in the gateway with no runtime detection — it returns a 200, the user is happy, and you find out months later."

Q: How do you attribute cost?

A: "In integer micro-USD, divided last so a month of records sums exactly, with the three price tiers — fresh input, cached input, output — because they differ by an order of magnitude in both directions. Attribution by tenant for chargeback, by agent to find the expensive one, by deployment to see what failover is costing, by provider for concentration risk, and by model to find who's still on the deprecated one. Three rules people get wrong: record failures, because a call that timed out after generating 400 tokens still cost money and a success-only model is blind exactly during an incident; record cache hits at zero cost, or a hit double-counts and your savings look like spending; and reconcile monthly against the provider's billing export, because the gap is where dropped usage blocks and uncounted retries live. And I'd alert on two derived metrics — cache hit rate, whose sudden drop usually means someone broke prefix stability, and failover rate, which moves before the error rate does because failover is what converts the provider's errors into my successes."

Q: The gateway is now a single point of failure. Defend it.

A: "It is, and Phase-00 arithmetic says it caps the platform, so it has to be built as a data-plane component: stateless, horizontally scaled, holding only caches and counters it can lose, with policy and routing config pushed and cached so a control-plane outage doesn't touch it — fail-static, not fail-open or fail-shut. The alternative isn't 'no single point of failure', it's twelve teams each holding credentials and each being their own single point of failure with none of the observability. I'd rather have one component I can make 99.99% than twelve I can't measure. What I would not do is put anything slow or stateful in it — no synchronous policy lookups, no database on the request path, and the distributed rate limiter has to degrade to a local approximation rather than fail the request."

14. References

• Gateways — Azure API Management's AI-gateway capabilities (token-limit, semantic-caching and emit-token-metric policies); LiteLLM (router, fallbacks, budgets); Kong AI Gateway; Portkey. Read at least two configs; the vocabulary is remarkably consistent.
• Provider docs — each of Azure OpenAI, AWS Bedrock, OpenAI, Anthropic, Google Vertex AI and Cohere: their error/status taxonomies, their usage shapes, and their prompt-caching semantics. This is the material the abstraction layer exists to hide, and you cannot design the hiding without reading it.
• Nygard, Release It!, 2nd ed. — timeouts, circuit breakers, bulkheads, and the failure modes a gateway concentrates.
• Amazon Builders' Library — Timeouts, retries, and backoff with jitter: the canonical statement of why full jitter matters more than the backoff curve.
• Google, The Site Reliability Workbook, Ch. 5 — load shedding and graceful degradation, which is what a gateway does under pressure.
• OWASP Top 10 for LLM Applications — Unbounded Consumption, which is what quotas and token buckets exist to close.