« Phase 04 · Warmup · Track Overview
Hitchhiker's Guide — The LLM Gateway
The 30-second mental model
One ingress in front of every provider. A policy enforcement point that happens to speak HTTP, not a proxy.
It owns: normalization (requests, responses, and errors), routing, budget-aware fallback, rate limits, quotas, three cache tiers, and token accounting. It is the only place that can answer "what does this cost", "which model produced this", and "can this data leave the region".
It is also now a serial dependency for everything, so it is stateless, horizontally scaled, and fail-static on config.
The numbers
| Thing | Value |
|---|---|
| Cost | \( \frac{(t_{in}-t_{cached})c_{in} + t_{cached}c_{cache} + t_{out}c_{out}}{1000} \), integer micro-USD, divide last |
| Fallback allowed iff | budget − elapsed ≥ next.expected_latency |
| 3 000 ms budget, 2 400 ms elapsed, 800 ms fallback | no — 600 < 800, fail fast |
| Token bucket | capacity C, refill r/s; burst C, long-run r |
| Rate dimensions needed | two: RPM and TPM |
Semantic cache saving at hit rate h, near-zero hit cost | ≈ h |
| Backoff | sleep = random(0, base · 2^attempt) — full jitter |
The error taxonomy
| Class | retryable | fall_over |
|---|---|---|
RateLimited | ✔ | ✔ |
ProviderTimeout | ✔ | ✔ |
ProviderUnavailable | ✔ | ✔ |
ContentFiltered | ✘ | ✘ |
InvalidRequest | ✘ | ✘ |
QuotaExceeded | ✘ | ✘ |
NoRouteAvailable | ✘ | ✘ |
BudgetExhausted | ✘ | ✘ |
Two flags, not one. Failing over on a content filter is shopping for a compliant model.
The two absolute refusals
- Never fail over a side-effecting call. A timeout is not evidence of non-execution.
- Never fail over a content filter. See above.
The three cache tiers
| Tier | Key | Risk |
|---|---|---|
| Exact | hash(tenant, deployment, task class, temperature, max tokens, prompt) | staleness |
| Prefix (provider-side) | a shared leading prompt — you earn it by ordering the prompt | none |
| Semantic | embedding similarity ≥ floor, within a tenant partition | a silent breach |
Semantic cache, three non-negotiables: tenant-partitioned · a floor tuned on negative examples · never for entitlement-dependent answers.
Never cache a non-STOP finish reason. Never cache when cacheable=False.
One-liners
- Deployment, not model — provider × model × region × capacity. Two of those differ in latency, price and residency for the same model.
- Route on the caller, not the model name — a request naming
gpt-4ois a vendor decision hard-coded into an agent. - Two gates — the rule matches, then residency and the deployment's classification ceiling filter. A misconfigured rule is still caught.
- Retry ≠ failover — retry is the same deployment after a backoff; failover is the next one. Under sustained 429s, retrying hard makes it worse.
- Tenant from the token, never the body.
- Account failures — a timeout after 400 generated tokens still cost money.
- Failover rate moves before the error rate. Best early warning the gateway produces.
- Reconcile monthly against billing. The gap is always in the same direction.
Vocabulary
Deployment · a routing target. PTU / provisioned throughput · dedicated capacity.
PAYG · shared, per-token. Spillover · overflow from dedicated to PAYG. Prefix cache ·
provider-side reuse of a shared prompt prefix. Full jitter · random(0, base·2^n). TPM /
RPM · tokens and requests per minute. Fail static · keep enforcing the last known-good
config. Showback / chargeback · report spend vs bill it.
War stories
The retry that doubled the payments. The gateway retried on timeout. Some of those requests had already emitted a tool call downstream. Nobody had classified the request as side-effecting, because the field was optional.
Shopping for a compliant model. A fallback chain that fired on any error, including the provider's safety refusal. The third provider answered. It was found in a model-risk review, and the question asked was exactly "so your system tries providers until one agrees?"
The fallback that breached the SLO. A 4-second fallback timeout inside a 3-second p95 budget. Every provider blip turned a partial degradation into a fleet-wide breach, and the dashboard showed the SLO failing while every individual component looked healthy.
The cache that leaked. Semantic cache keyed on prompt embedding, tenant applied as a post-filter. A refactor moved the filter. Hit rate 34%, everyone delighted, until a Retail user received a Wholesale answer. A 200 OK, a happy user, and a breach discovered months later.
The invoice nobody could decompose. No attribution. Finance asked which team was responsible for a 40% month-on-month rise and the answer took three weeks and was still an estimate.
The 429 storm. Immediate retries with no jitter. The provider recovered, every client retried in the same second, and it went down again.
Beginner mistakes
- Normalizing responses but not errors.
- One
retryableflag doing two jobs. - Failing over on a content filter.
- Routing on a model name.
- A fallback with no budget check.
- Failing over a side-effecting call.
- Retrying hard into a 429; no jitter.
- RPM without TPM.
- Spending an RPM slot on a token rejection.
- A cache key without the tenant.
- Caching a truncated answer.
- Counting only successful calls.
- Putting a synchronous database lookup on the gateway's request path.
What "good" sounds like
"It's a policy enforcement point that speaks HTTP. Normalized request carrying tenant, classification, residency, latency budget and side-effecting — the fields a provider SDK can't. Normalized errors with
retryableandfall_overas separate flags, because a safety refusal is neither, and failing over on one is shopping for a compliant model. Routing on task class, tenant and classification, with residency and each deployment's ceiling as an independent second gate. Fallback allowed only if the remaining budget fits the next deployment's expected latency, and never at all for side-effecting calls. Per-tenant RPM and TPM, a monthly quota that fails closed, three cache tiers with the tenant first in every key, and accounting that includes failures and cache hits so the numbers survive an incident. Stateless and fail-static, because it's now a serial dependency for the whole platform."