« Phase 04 · Warmup · Track Overview

Principal Deep Dive — Architecture, Tradeoffs & Blast Radius

Table of Contents

• 1. The three tradeoffs of a gateway
• 2. Defending the single point of failure
• 3. Scaling envelope
• 4. Failure modes and blast radius
• 5. Distributed rate limiting
• 6. Concentration risk and the exit plan
• 7. Decisions that look wrong but are intentional
• 8. What changes at 10×

1. The three tradeoffs of a gateway

Tradeoff 1 — abstraction vs capability. A unified interface across six providers is worth a great deal, and it is a lowest common denominator. Provider-specific features — extended thinking modes, structured-output modes, provider-native tool formats, prompt-cache controls — either do not fit the abstraction or leak through it.

The resolution: normalize the contract, pass through the specifics. The normalized request carries a provider_options escape hatch that the gateway does not interpret. What it must still normalize is anything the platform enforces on — usage, finish reason, errors, cost. A caller using provider_options has knowingly pinned itself to a provider, and the gateway should record that so you can measure how much of your fleet is actually portable.

Tradeoff 2 — enforcement vs latency. Every check adds milliseconds to a path that is a serial dependency for the whole platform. Quota, rate limit, routing, cache lookup, guardrails, accounting — each is small; together they are a budget line.

The resolution: everything on the request path is in-process and in-memory. Policy is pushed and cached, not queried. Counters are local with asynchronous reconciliation (§5). The gateway should add single-digit milliseconds; if it is adding fifty, something on the path is doing I/O that should not be.

Tradeoff 3 — one gateway vs per-domain gateways. One is simpler to operate and gives one observability surface. Several give blast-radius isolation and let a high-risk domain run stricter policy.

The resolution for most banks: one gateway deployment per environment, with per-tenant policy inside it — but built so a second instance is a configuration, not a fork. When a domain eventually needs isolation (a payments-only gateway with its own capacity and its own on-call), you want that to be a deployment decision rather than an architecture project.

2. Defending the single point of failure

The gateway is a serial dependency for every AI request in the bank. Phase 00's arithmetic says its unavailability adds directly to the platform's. This objection comes up in every architecture review and it deserves a prepared answer.

First, the counterfactual is worse. Without a gateway you do not have zero single points of failure — you have twelve, one per team, each with its own retry policy, none instrumented. You have replaced one component you can make highly available with twelve you cannot measure.

Second, it is built as a data-plane component, which means:

Third, the failure modes are asymmetric and you can choose. A gateway that cannot reach its config store keeps serving on the last bundle. A gateway that cannot write accounting records keeps serving and buffers. A gateway that cannot reach any provider is not the gateway's outage. The only genuinely fatal case is the process being unavailable, and that is what replicas are for.

The sentence to have ready: "I'd rather have one component I can make 99.99% than twelve I can't measure — and the only thing on its request path is arithmetic."

3. Scaling envelope

Two worth expanding.

Provider limits are the real ceiling, and they are per-deployment. Your gateway can serve 50 000 rps; Azure will give you a TPM quota. This is why capacity planning (Phase 05) is a procurement activity with weeks of lead time, and why the gateway's most valuable operational signal is headroom against the provider limit, not CPU.

Cardinality, again. gateway_requests_total{tenant, agent, deployment, provider, model, task_class, outcome, cache} at 12 tenants × 200 agents × 6 deployments × 8 outcomes is already 115 000 series before you multiply by the rest. The label budget decision from Phase 00 applies here more sharply than anywhere else in the platform: metrics carry tenant, deployment and outcome; agent and request-level detail live on traces and in the accounting store.

4. Failure modes and blast radius

The pattern to name: everything in this table announces itself as an error rate or a metric shift — except the cache mis-hit. Controls that fail silently and severely deserve prevention (a structural rule: tenant in the partition) rather than detection (a threshold you tune).

The config-push row deserves an operational note. A bad routing rule can send restricted data offshore, and it does so instantly across the fleet. So config is treated like code: validated on load (the lab's constructor-time checks are the miniature of this), rolled out in stages, and reversible in seconds. A gateway with a kubectl edit-shaped config workflow is a compliance incident waiting for a Tuesday.

5. Distributed rate limiting

The lab's bucket is in-process. With N gateway replicas, in-process limits mean each tenant gets N × its limit — which is either fine (if you set the per-replica limit to limit/N) or wrong (because replicas scale and traffic is not evenly distributed).

Three approaches, and the tradeoff is the same one every distributed counter has:

For a gateway that must not add latency, the third is usually right: each replica holds a local bucket sized to its share, reports consumption asynchronously, and periodically receives a revised share. It over-admits briefly after a traffic shift and never adds a round trip.

The decision that matters more than the algorithm: what happens when the shared store is unavailable? Fail open (keep serving on local buckets, possibly over-admitting) or fail shut (refuse). For a rate limiter protecting a provider, failing open risks a 429 storm you cannot control; for one protecting your budget, failing open risks spend. The defensible answer is fail open on the rate limit (the provider will throttle you anyway, which is a survivable outcome) and fail closed on the quota (spend is not recoverable). Splitting the two postures is the principal-level observation.

6. Concentration risk and the exit plan

A regulator will ask about dependence on a single model provider. The answer must be an architecture, not an intention — and the gateway is the architecture. But owning a gateway is not the same as being able to switch, and the gap between them is where the honest answer lives:

The practical control: route a small, continuous percentage of production traffic to the alternative. A failover path that has never carried live traffic is a hypothesis, and the day you need it is the day you discover the prompt behaves differently. This costs a few percent of spend and converts "we could switch" from a claim into a measurement.

The related governance obligation is model deprecation. Providers retire and silently update models. Controls: pin versions explicitly, gate every version change behind the evaluation suite, subscribe to deprecation notices, and keep a tested alternative. That is a Phase 15 conversation, and the gateway is where it is implemented.

7. Decisions that look wrong but are intentional

A cache hit reports cost_micros = 0. Looks like it loses information about what the answer would have cost. It reports what this call cost. Reporting the original double-counts spend, and the savings you are trying to measure would show up as spending.

The cache key uses the primary deployment even when a fallback answered. Looks inconsistent with the recorded serving deployment. It is the only choice that hits: the next identical request also computes the primary as its key. Keying on the responder produces a cache that never hits after a failover — exactly when you want it most. The accounting record still names the real responder, so the evidence trail stays accurate.

Content filtering does not fall over. Looks like a reliability regression. It is a compliance control, and the alternative has a name a regulator will use.

The rate limiter charges cache hits. Looks like throttling the cheap path. The limit protects the gateway too, and a tenant hammering it with cacheable requests is still load. Defensible either way; the point is to decide and document.

Rules fall through when every deployment is inadmissible. Looks like it could silently apply a less appropriate policy. The alternative — a specific rule shadowing a general one it cannot satisfy — produces a hard failure where a valid route existed. Falling through with the second gate still enforcing is the safer failure.

Config validation at construction rather than at use. Looks like it makes hot reload harder. It makes a bad config fail at load, which is the only place you can still roll back cheaply.

8. What changes at 10×

At 3 deployments and 4 tenants, the lab is close to shippable. At 30 deployments, 40 tenants and 10 000 rps:

• Distributed rate limiting (§5) stops being optional, and its failure posture becomes a documented decision.
• The semantic cache needs an ANN index per tenant partition, at which point its memory cost becomes visible and the "one index with a filter" temptation returns. The answer is the same.
• Routing becomes data, not config: a rules service with an API, staged rollout, and an audit trail of who changed what. A YAML file edited by four people is a compliance gap.
• Cost attribution becomes chargeback, which changes team behaviour within a quarter — and requires the reconciliation job to be trustworthy, because now people argue with it.
• Per-deployment circuit breakers appear, because at 30 deployments you cannot manually remove a sick one fast enough.
• Canary and shadow routing become standard: every model or prompt change goes to a small slice first, and the eval suite gates promotion.
• The accounting store outgrows metrics. Records go to a columnar store where you can ask "which agent's cost per successful action rose this week", which is the question that actually drives optimization.

Seams to build now, cheap today: provider_options as an opaque passthrough with a flag recording that it was used; a deployment_version on every accounting record; config validated at load; tenant partitions in every cache; and the failover-rate metric, which will be your first useful alert.