« Phase 04 · Warmup · Track Overview

Core Contributor Notes — How the Real Gateways Work

Table of Contents

• 1. The three shapes of a real gateway
• 2. Azure APIM as an AI gateway
• 3. LiteLLM: the reference open-source router
• 4. Where usage numbers actually come from
• 5. Provider-side prompt caching
• 6. Streaming changes everything
• 7. Sharp edges
• 8. What the miniature simplifies
• 9. References

1. The three shapes of a real gateway

For a bank the honest answer is usually APIM at the edge plus a thin in-house layer inside: APIM gives you the network position, mTLS, WAF integration, and an operating model your platform team already understands; the in-house layer holds the control model (tenant, classification, residency, side-effect semantics) that no product knows about.

The failure mode to avoid is putting the control model in APIM policy expressions. They are XML, they are hard to test, and a routing decision that must be auditable should be code with a test suite — which is exactly what the lab builds.

2. Azure APIM as an AI gateway

APIM ships policies that map almost one-to-one onto this phase:

Two things worth knowing before you build on it:

Backend pools do round-robin or priority with a circuit breaker, which is a load-balancing model rather than a policy model. Expressing "restricted data must stay in-region" is possible but awkward; expressing "route classification tasks to the cheap self-hosted deployment" is awkward too. That is the seam where the in-house layer earns its place.

The semantic-cache policy calls an embedding deployment, which means a cache lookup costs a model call. That changes the economics: a semantic cache is only worth it when the embedding call is much cheaper than the completion it saves, which is true for large completions and false for short ones. Measure before enabling.

3. LiteLLM: the reference open-source router

Worth reading even if you do not deploy it, because it is the most complete open implementation of this phase's ideas:

• Model groups and fallbacks — a named group with an ordered list, fallbacks per group, and context_window_fallbacks for the case where the request is too large rather than the deployment being unhealthy. That second kind is a distinction the lab does not make and production does.
• Routing strategies — simple-shuffle, least-busy, usage-based-routing (by TPM/RPM headroom), latency-based-routing. Note that all four are load strategies; policy routing (classification, residency) is still yours.
• Budgets and rate limits per key, per team, per model — the same two-dimensional idea as the lab's tenant limits.
• Cooldowns — after N failures a deployment is removed from the pool for a period. This is a circuit breaker by another name, and it is the extension the lab flags.

The design decision LiteLLM makes that is worth copying: the OpenAI request shape as the normalized shape. It is not the most elegant abstraction, but it means every client SDK already speaks it, which removes an adoption barrier that a bespoke schema would create. Add your platform fields as headers or an extra_body key and accept the mild ugliness.

4. Where usage numbers actually come from

The lab computes cost from a price table and the adapter's reported usage. Production reads usage from the provider's response, and there are four traps:

Streaming responses may omit usage unless you explicitly ask (OpenAI-compatible APIs need stream_options: {include_usage: true}). A gateway that streams by default and forgets this silently under-reports for exactly the traffic that matters most.

Cached-token accounting is provider-specific. Some report cached input as a separate field, some fold it into the input count with the discount applied at billing, and the definition of a cached token differs. Normalizing these into one Usage shape is a substantial part of what the abstraction layer is for, and the direction of the error when you get it wrong is always flattering.

Failed and cancelled requests still cost. A stream cancelled after 400 tokens was billed for 400 tokens. If your accounting only records completions, your incident-time cost is invisible.

Reconcile monthly against the billing export. Gateway-measured spend and the invoice will disagree. The gap is dropped usage blocks, uncounted retries, and unit misunderstandings — per-1K versus per-1M is a genuinely common one, and it is a factor of a thousand.

5. Provider-side prompt caching

Not a cache you build. Providers reuse computation for a shared prompt prefix and bill those tokens at a fraction of the input rate. Three implementation realities:

• It is prefix-sensitive. A single changed byte early in the prompt invalidates everything after it. This is why prompt assembly order is a cost decision: system prompt, then tool schemas, then policy text, then history, then the user's turn.
• There are minimum lengths and TTLs. Below some prefix length there is no discount at all, and cache entries expire in minutes. A gateway that reorders prompts for readability can accidentally destroy the benefit.
• Some providers require an explicit marker, some do it automatically. Normalizing "did this request get a prefix discount" into one boolean is worth doing, because otherwise you cannot measure the thing you are optimizing.

The measurable KPI: fraction of input tokens billed at the cached rate. If it is low, someone put a timestamp at the top of the system prompt.

6. Streaming changes everything

The lab is request/response. Real gateways stream, and it changes five things:

1. Usage arrives last, or not at all (§4).
2. Failover becomes partial. If the primary fails after emitting 200 tokens, you cannot silently switch — the client has already seen output. Either you buffer until the first token is safe (adding latency) or you accept that failover is only possible pre-first-token. Most gateways choose the latter and it is worth stating explicitly.
3. Cancellation matters. A client that disconnects should stop the upstream call; otherwise you pay for tokens nobody reads. This requires propagating cancellation through the gateway, which is a real piece of work.
4. Caching is harder. You cannot cache a response you have not finished receiving, and you must not cache one that was cancelled mid-stream (it looks like a LENGTH finish that never arrived).
5. Guardrails on output become incremental. Scanning a complete response is easy; scanning a stream means either buffering (defeating the point) or scanning windows and accepting that you may emit a token before you block.

Point 2 is the one to raise in a design review: budget-aware failover and streaming are in tension, and the resolution is usually "failover before the first token, then commit."

7. Sharp edges

Per-1K versus per-1M pricing. Providers publish both. A factor-of-1000 error in a cost model is not subtle in aggregate and is completely invisible per request.

Token estimation is not token counting. The lab's len/4 is a stand-in. Real rate limiting against a TPM quota needs a real tokenizer, and different models tokenize differently — so a gateway that estimates with one model's tokenizer and routes to another is systematically wrong. Estimate conservatively and reconcile from the response.

Retry-After is a hint you should honour. Providers send it on 429. Ignoring it and applying your own backoff is how a 429 storm sustains itself.

Context-window overflow is a 400, not a limit. It looks like a capacity problem and it is a request problem. It needs its own error class and its own fallback (a larger-context deployment), which is LiteLLM's context_window_fallbacks and the lab's most obvious missing case.

Model version pinning. gpt-4o and gpt-4o-2024-11-20 are different contracts. Pin the dated version, gate the change behind evals, and treat an unpinned deployment as a compliance finding — the model will change under you.

Idle connection pools. Providers close idle connections; a pool that does not detect it turns into a burst of first-request failures after a quiet period.

8. What the miniature simplifies

9. References

• Azure API Management — the AI-gateway policy set (llm-token-limit, llm-emit-token-metric, llm-semantic-cache-lookup/-store), backend pools and circuit breakers.
• LiteLLM — router documentation: model groups, fallbacks, context_window_fallbacks, cooldowns, routing strategies, budgets. The most complete open implementation of this phase.
• Kong AI Gateway, Portkey, Helicone — alternative shapes; read at least one other for contrast.
• Provider documentation — for each of Azure OpenAI, AWS Bedrock, OpenAI, Anthropic, Google Vertex AI and Cohere: error/status taxonomy, usage shape, prompt-caching semantics and minimums, streaming usage options, and the deprecation policy. This is the material the abstraction hides.
• Amazon Builders' Library — Timeouts, retries, and backoff with jitter.
• Nygard, Release It!, 2nd ed. — circuit breakers and bulkheads, which the gateway concentrates.
• OWASP Top 10 for LLM Applications — Unbounded Consumption.