« System Design · Track Overview

Design 02 — The Multi-Provider Model Gateway

"Every agent in the bank calls models through one service. Design it."

The question it turns on: can you make a routing and capacity decision with arithmetic, and defend the single point of failure you just created?

Table of Contents

• 1. Constraints before components
• 2. The request path, and what each stage denies
• 3. Compose the SLO — and the single point of failure
• 4. The latency budget, and why the fallback is the small model
• 5. Routing: the four gates, in order
• 6. Capacity: PTU vs PAYG, with the arithmetic
• 7. Caching: three tiers, three different risks
• 8. Tenant isolation and rate limiting
• 9. Failure modes and blast radius
• 10. Evidence and cost attribution
• 11. What you build first
• 12. What changes at 10×
• 13. The questions you will be asked

1. Constraints before components

The constraint that surprises people: the gateway is now the platform's most critical single service. Every agent depends on it. You have consciously created a single point of failure, and §3 is where you defend that.

2. The request path, and what each stage denies

   agent
     │
   ┌─▼──────────────────────────────────────────────────────────┐
   │ 1  AUTH            denies: an unauthenticated caller;      │
   │                            a token whose tenant ≠ claimed  │
   ├────────────────────────────────────────────────────────────┤
   │ 2  QUOTA           denies: a tenant over TPM or RPM;       │
   │                            a request over the per-call cap │
   ├────────────────────────────────────────────────────────────┤
   │ 3  CACHE           denies nothing — it SERVES              │
   │    exact → prefix → semantic (tenant-scoped keys)          │
   ├────────────────────────────────────────────────────────────┤
   │ 4  ROUTE           denies: classification, residency,      │
   │                            budget, ladder                  │
   ├────────────────────────────────────────────────────────────┤
   │ 5  CALL + RETRY    denies: nothing; it FALLS OVER —        │
   │                            and only if the budget fits     │
   ├────────────────────────────────────────────────────────────┤
   │ 6  ACCOUNT         denies nothing; it MEASURES             │
   │    tokens, cost, latency, attribution                      │
   └─┬──────────────────────────────────────────────────────────┘
     ▼
   provider (Foundry / Bedrock / self-hosted vLLM)

Two design decisions visible in the ordering:

Quota before cache. A tenant over quota does not get a free cached answer — otherwise the cache becomes a quota bypass, and a tenant with a high hit rate silently escapes its ceiling.

Cache before route. A cache hit costs no route, no provider call and no tokens. Routing first would price a request you are not going to make.

3. Compose the SLO — and the single point of failure

The gateway's own availability multiplies into every agent's. Be explicit about it:

$$A_{\text{effective}} = A_{\text{gateway}} \times A_{\text{provider path}}$$

• One provider: \( 0.9995 \times 0.999 = 0.9985 \) → 99.85%
• Two providers: \( 0.9995 \times 0.999999 = 0.99950 \) → 99.95%

The gateway is now the binding constraint, which is the honest answer to "you've built a single point of failure":

"Yes — deliberately. The alternative is every agent implementing routing, residency and cost control itself, which is twelve implementations of the residency check and eleven chances to get it wrong. I make the gateway boring: stateless, no synchronous dependency on the control plane, a local policy bundle, multi-AZ, and a deploy that is a rolling replace with a health gate. Then I measure it — the gateway's own availability is an SLO with an error budget, and the number is higher than any of its dependencies."

Three properties that make that claim true rather than aspirational:

1. Stateless. Cache and quota state live in Redis; losing an instance loses nothing.
2. No synchronous control-plane call. The policy bundle is local, pushed, with a TTL and a staleness alarm. Fail static.
3. The routing table is data, not code. Adding a provider is a config change with a canary, not a deploy of the service every agent depends on.

And the failover must be independent to count. Two deployments of the same model family in the same region share a failure domain: a capacity event takes both. "Independent" means different provider, different region — and then residency constrains which of those you may actually use.

4. The latency budget, and why the fallback is the small model

The gateway inherits 800 ms TTFT from the platform budget (Design 01).

149 ms of headroom is the whole fallback argument. A second frontier model in another region does not reach TTFT in 149 ms — a cross-region call spends most of that on network alone. So:

"The in-budget fallback is the small model in the same region, not the frontier model somewhere else. Falling over to a second frontier provider is a capacity decision measured in minutes, not a request-level decision measured in milliseconds. I run both: a synchronous fallback to the small model, and an operator-triggered (or breaker-triggered) shift of the routing table to a second provider when the primary is degraded for a sustained period."

That distinction — request-level fallback vs fleet-level failover — is the thing that separates a designed gateway from a diagram with a retry arrow.

What is retryable, and what is not. Three different flags, and conflating them is a classic finding:

The safety-refusal row is the one to say out loud. A content filter is neither retryable nor fall-over-able, because falling over on a refusal means the platform is searching for a model that will do the thing your primary model refused. That is an audit finding, and it is one line of code.

5. Routing: the four gates, in order

for route in ROUTES:                      # ordered by preference
    if route.model in excluded:                       continue
    if rank(classification) > rank(route.max_class):  continue   # 1 classification
    if route.region not in residency_regions:         continue   # 2 residency
    if projected_cost(route) > remaining_budget:      continue   # 3 budget
    if ladder.is_shed(route.tier):                    continue   # 4 degradation
    return route
return None                               # exhausted → refuse, with reasons

All four gates live inside the router. This is the design's most important structural decision and the one that prevents the seam nobody tests:

"If residency is checked at the call site for the primary and the fallback is chosen by a separate pick_fallback(), both functions are correct and the composition is not. The day the primary fails over, confidential data goes to whichever region was next in the list — and that is also the day nobody is reading routing logs."

Projected cost, not measured cost. The budget gate must run before the call, which means projecting from the token estimate:

$$\text{projected} = \frac{\hat{t}{\text{in}}}{1000}c{\text{in}} + \frac{\hat{t}{\text{out}}}{1000}c{\text{out}}$$

Estimating output tokens is genuinely hard; use the tenant's measured p90 output length per agent class and re-check against the actual afterwards. Say that — an interviewer who has built one will know the estimate is the weak part.

Exhaustion is the only denial. Reasons collected while skipping routes that were then replaced by an eligible one are diagnostics, not refusals. Logging them as denials turns a successful fallback into a reported failure — a real bug, and a subtle one.

6. Capacity: PTU vs PAYG, with the arithmetic

$$\text{break-even tokens} = \frac{C_p}{c_t}\times 1000 \qquad U_{\text{BE}} = \frac{\text{break-even tokens}}{T}$$

where \( C_p \) is the monthly cost of dedicated capacity, \( c_t \) the blended PAYG cost per 1,000 tokens, and \( T \) the tokens the dedicated capacity actually serves at your token mix.

Three things that make this arithmetic wrong if skipped:

1. Throughput per unit depends on the input/output mix. Prefill and decode cost differently. A workload that is 10% output tokens gets far more throughput per unit than one that is 50% output — enough to move the break-even utilization from ~77% to ~40%. Measure with your own traffic shape; do not use the vendor's example.

2. Latency, not cost, is often the reason. Dedicated capacity removes shared-pool congestion and 429s. If your p95 TTFT is being set by other tenants' bursts, the PTU is buying you a latency SLO and the cost arithmetic is secondary.

3. A reserved commitment is a finance instrument. A one-year commitment on a model family is a bet against deprecation. Price the exit before signing it.

The shape that actually works:

"Size the dedicated floor to p50 demand for the latency-sensitive tier, spill everything above it to PAYG, and put batch on the cheapest thing available. Then measure utilization weekly — a floor at 40% utilization is a floor that is too big, and the decision to shrink it is easier if you stated the target when you bought it."

The self-hosted tier is a third option and belongs in the answer for sovereignty rather than cost: an open-weight model on your own GPUs, in-country, where the data never leaves your tenancy. Its economics are dominated by utilization — a GPU at 20% utilization is more expensive than PAYG, and the KV-cache arithmetic sets the concurrency (Phase 05).

7. Caching: three tiers, three different risks

Three non-negotiables in a bank:

Tenant-scoped keys, tenant first. The tenant is the first component of every key. A cache key built before the tenant is resolved is the tenant-leak bug, and it appears months later under load.

A high similarity floor, tuned with negative examples. "What is our exposure to Zenith?" and "What was our exposure to Zenith?" are close in embedding space and different in answer.

Never cache entitlement-dependent answers. Two users with different clearances asking the same question must not share a cache entry. When in doubt, cache the retrieval, not the answer — the retrieval can be re-filtered per viewer, and the answer cannot.

8. Tenant isolation and rate limiting

Token bucket, per tenant, on two dimensions:

tokens = min(C, tokens + (t - last_refill) * r)
if tokens >= k: tokens -= k; admit
else:           reject with retry-after = (k - tokens) / r

• TPM and RPM both. One request can be 100,000 tokens; an RPM-only limit does not protect the provider quota, and a TPM-only limit does not protect against a storm of tiny calls.
• C = r gives no burst tolerance; C = 60r tolerates a one-minute burst. Pick deliberately.
• Never let the bucket go negative — the classic boundary bug, which silently grants free capacity after one large request.
• Reserve a floor per tenant. A pure shared pool means the noisiest tenant sets everyone's latency. Each tenant gets a guaranteed floor plus fair access to the surplus.

And the tenant comes from the token, never the request body. Anything the caller can set, the caller can forge.

9. Failure modes and blast radius

Fail open on cache, fail closed on quota is the row worth explaining. A cache outage should degrade cost and latency, not availability — serve from origin. A quota outage must not let every tenant through unmetered, because the provider quota is a hard external limit and exceeding it takes down all tenants at once. Different directions for different failures, deliberately.

10. Evidence and cost attribution

Every call emits one record:

{ "trace_id": "...", "tenant": "wholesale", "agent_id": "payments-investigator",
  "user_id": "layla.almansouri", "model": "gpt-frontier-uaenorth", "region": "uaenorth",
  "route_reason": "primary", "input_tokens": 4812, "cached_tokens": 3900,
  "output_tokens": 380, "cost_micros": 3900, "ttft_ms": 612, "total_ms": 1840,
  "classification": "confidential", "cache": "miss", "policy_version": "2026-03-11.4" }

The fields people forget, and why each matters:

• cached_tokens separately from input_tokens — otherwise prompt-cache savings are invisible and nobody can justify the prompt restructuring that produced them.
• route_reason — "primary" vs "fallback:429" vs "fallback:budget". Without it you cannot tell a degraded week from a normal one.
• classification and region together — this pair is the residency evidence. An auditor asking "prove no confidential data left the country" gets a query, not an assertion.
• user_id as well as agent_id — cost attribution to a team is nice; attribution to a human is what makes a runaway agent traceable.

The unit economic is cost per successful action, not cost per call:

$$\text{CPSA} = \frac{\text{cost}_{\text{action}}}{P(\text{success})}$$

A 30% failure rate multiplies effective cost by 1.43 — which is the sentence that turns an evaluation budget into a funded programme.

11. What you build first

1. Auth, routing, one provider, accounting. The narrow waist. Every agent moves onto it before anything clever exists, because migrating agents later is the expensive part.
2. Quota and tenant isolation. Before the second business unit onboards, not after the first incident.
3. The residency gate, inside the router. Cheap now, structural later.
4. The second provider and the fallback. With the budget check, and with a test that the fallback path actually serves production traffic sometimes.
5. Prompt/prefix caching. The highest-value, lowest-risk cache tier.
6. PTU capacity, once you have a month of measured token mix — not before, because the sizing arithmetic needs the mix.
7. Semantic caching. Last, deliberately: highest risk, and it needs the negative-example tuning that only production traffic provides.

12. What changes at 10×

400,000 calls/day.

The gateway becomes latency-critical infrastructure. 40 ms of cache lookup at 40,000 calls is invisible; at 400,000 it is a capacity line item. The semantic-cache embedding call moves onto a local model.

Quota state becomes contended. A single Redis key per tenant is a hot key. Shard by (tenant, bucket) and accept approximate limiting — exactness was never the point.

Provider quota becomes the binding constraint, not your capacity. You are now managing a portfolio of quota across providers and regions, and the routing table becomes a scheduling problem: which tenant gets the frontier model at 09:00 on month-end.

Cost attribution becomes a chargeback. Once cost is charged back, every field in the accounting record is disputed. It must be right, and it must be reconcilable against the provider's own bill — which means storing the provider's request id.

Model deprecation becomes a standing programme. With ten models in the routing table, one is always being deprecated. Version pinning, eval re-baselining and a migration runbook stop being projects and become a monthly cadence.

13. The questions you will be asked

"Why not let agents call providers directly?" — Then residency, cost control, tenant isolation and token accounting are implemented N times. The gateway is the place where a policy is enforced once. The cost is a single point of failure, which I have priced and defended.

"What if the gateway is down?" — Every agent is down. That is why it is stateless, multi-AZ, has no synchronous control-plane dependency, and has a higher availability target than anything it calls. And it is why the routing table is data with a canary — the most likely cause of a gateway outage is a bad routing change, not infrastructure.

"How do you stop one team burning the budget?" — Per-tenant TPM and RPM buckets with a guaranteed floor, plus a per-request projected-cost gate, plus a monthly ceiling that trips a breaker. Three layers, because the first two are about rate and the third is about total.

"Your fallback breached residency." — It cannot, because the residency check is inside the router and every route passes through it. That is the specific bug this design is shaped to prevent, and it is the one nobody's component tests catch.

"Semantic cache — yes or no?" — Yes, last, tenant-scoped, with a high floor, never for entitlement-dependent answers, and with the retrieval cached in preference to the answer. It is the highest-value and highest-risk tier and it should arrive when you have traffic to tune it with.