« Phase 00 · Warmup · Track Overview

Principal Deep Dive — Architecture, Tradeoffs & Blast Radius

The principal-engineer lens: what you trade for what, where it breaks at scale, what the blast radius is, and which decisions look wrong until you know why.

Table of Contents

• 1. The three tradeoffs that define this platform
• 2. Layering: what it costs and when to collapse it
• 3. Scaling envelope
• 4. Failure modes and blast radius
• 5. The control-plane dependency problem
• 6. Two SLOs, and the politics of publishing them
• 7. Decisions that look wrong but are intentional
• 8. What changes at 10×

1. The three tradeoffs that define this platform

Tradeoff 1 — layers vs latency and availability. Every enforcement point you add is another serial dependency (worse availability), another network hop (worse latency), and another thing to operate. Every one you remove is a class of bad action that now has one fewer independent denial.

The resolution is not "fewer layers" or "more layers"; it is asymmetric layering by side-effect class. Read actions traverse a short path with cached policy. Money-moving actions traverse the full path with fresh policy, dual control, and idempotency. One platform, two depths, chosen by the tool's declared side-effect class — which is why that classification, introduced in Phase 10, is a platform concept and not a tool-author's opinion.

Tradeoff 2 — isolation vs cost. Silo (per-tenant infrastructure) gives the strongest isolation and the worst economics; pool (shared, isolated in the data path) gives the best economics and concentrates the risk in your code. For an AI platform the decision is usually per component, not per platform: pool the model gateway (isolation via token accounting and quotas), pool the kernel (isolation via per-run budgets and identity), but consider siloing the vector index for tenants under an information barrier, because that is where the leakage risk is highest and the cost of a namespace bug is a regulatory event.

Tradeoff 3 — autonomy vs evidence. Every increment of agent autonomy increases the evidence you must produce. A read-only advisory agent needs citations. An agent that opens a case needs an audit record. An agent that moves money needs the full chain: identity, delegation, policy version, model version, approval, idempotency key, and a tamper-evident link. The design rule: autonomy is granted in bands, and each band has a fixed evidence contract. Teams then choose their band knowingly rather than discovering the obligation at their ORR.

2. Layering: what it costs and when to collapse it

Concretely, each layer costs something measurable:

When to collapse. If a layer contributes latency and unavailability but denies nothing that another layer does not already deny, it is ceremony. The honest test: delete it on paper and enumerate what now gets through. If the answer is "nothing," collapse it. In practice the two candidates for collapse are (a) a separate "orchestration service" between the channel and the kernel, and (b) a "tool proxy" that duplicates the action gateway's contract checks.

When not to collapse. Never collapse the action gateway into the kernel. The kernel executes model-proposed plans; the gateway exists precisely because the kernel's input is untrusted. Putting them in one process means one bug removes both, and the whole architecture rests on their independence.

3. Scaling envelope

The platform's dimensions do not scale together, and the binding constraint moves:

Two of these deserve emphasis because they surprise people:

Observability cardinality is a real scaling limit. A metric labelled by agent, tool, tenant, model, and status at 200 agents × 60 tools × 12 tenants × 6 models × 5 statuses is 4.3 million series. Metrics backends fall over well before that. The mitigation is a deliberate label budget: high-cardinality identifiers live on traces and logs (where they are cheap to store and queryable), and metrics carry only low-cardinality dimensions (tenant, status, and a bucketed tool class). Deciding this in Phase 00 saves an emergency migration in month nine.

Tool count degrades accuracy before it degrades anything technical. A registry with 400 tools is not a scaling problem for the registry; it is a scaling problem for \( p \). The architectural answer is that agents never see the whole registry — capability discovery returns a policy-filtered, task-relevant subset, typically single digits. That makes discovery an authorization-aware operation, which is a Phase 09 mechanism with a Phase 00 justification.

4. Failure modes and blast radius

The last row is the one to internalize: it is the only failure in the table with no runtime detection. Everything else announces itself as an error rate. A cache that returns the wrong tenant's correct-looking answer produces a 200 OK, a happy user, and a regulatory incident discovered months later. Controls that fail silently and severely deserve prevention, not detection — which is why the rule is structural (tenant in the key) rather than statistical (a threshold you tune).

5. The control-plane dependency problem

Every platform team eventually builds a policy service and calls it synchronously from the data path. It works beautifully until the policy service is deployed on a Tuesday.

Three postures, and the reason only one is right:

• Fail open — if policy is unreachable, allow. Availability preserved, security destroyed. In a bank this is not a tradeoff; it is a finding.
• Fail shut — if policy is unreachable, deny. Security preserved, availability destroyed, and worse: the control plane's availability now multiplies into the data plane's, so your carefully composed 99.66% becomes 99.66% × (control plane).
• Fail static — the data plane holds a versioned policy bundle, refreshed asynchronously. If refresh fails, it keeps enforcing the last known-good bundle and raises an alert on bundle staleness. Availability preserved, security preserved, at the cost of a bounded window in which a revocation has not propagated.

Fail-static is correct, and the interesting engineering is in that last clause. Revocation latency is now a designed parameter, not an accident. Typical shape: bundle refresh every 30 s, staleness alarm at 5 minutes, hard-stop at 30 minutes (after which the data plane does fail shut, because a 30-minute-old policy in a bank is worse than an outage). And urgent revocations get a second channel — a kill-switch push that does not wait for the next refresh — because "we suspended an agent but it kept acting for 30 seconds" is a sentence you do not want to say to an examiner.

6. Two SLOs, and the politics of publishing them

The arithmetic says the read path and the action path have different achievable availabilities. Publishing two SLOs is technically obvious and organizationally hard, because someone will ask why the platform "isn't just 99.9%."

The framing that works: an SLO is a promise about the platform's own contribution, plus an honest pass-through of its dependencies. The read path is mostly yours to control, so you commit to a number and hold it. The action path traverses systems you do not own; you commit to your contribution and publish the composed number with its dependency breakdown. This is the same structure a cloud provider uses when it excludes customer-caused outages, and it is defensible because it is auditable.

The failure to avoid: publishing one optimistic number, breaching it, and losing the credibility that makes the error-budget policy enforceable. The error-budget mechanism only works if everyone believes the number. Publish something you will hold.

7. Decisions that look wrong but are intentional

The admission pipeline runs all five layers instead of short-circuiting. Looks wasteful and looks like an information leak. It is a measurement device: defence_depth turns a slogan into an assertion. In production you short-circuit on the serving path and run the full evaluation in shadow, alerting when a money-moving tool has defence_depth == 1.

The budget ledger clamps at zero and reports overspend separately. Looks like it loses information. It preserves the more important information: a negative remainder on one layer must never cancel a positive remainder on another, or the platform total silently lies.

policy_state has no time input. Looks incomplete — surely the state should depend on how much of the window remains? It should inform the conversation, but making the policy a pure function of budget remaining is what makes it enforceable without argument. Time-weighted variants ("we're at 40% with 2 days left, that's fine") reintroduce exactly the negotiation the policy exists to end.

shed_until_fits can return the whole ladder and still not fit. Looks like it should raise. The finding "we shed everything and still cannot afford a fallback" is the most valuable output the function produces — it means the design is infeasible, and that belongs in the caller's report, not in an exception.

The model layer is not a "layer" in the five-layer diagram. Looks inconsistent with the JD's own language. It is cross-cutting: the kernel calls it, the knowledge foundation calls it (for embeddings and reranking), and the guardrails call it (for model-based classification). Drawing it as a horizontal layer implies an ordering that does not exist and hides that a model outage hits three layers at once.

8. What changes at 10×

At 20 agents you can hold the platform in your head. At 200 the following stop being optional:

• Tool versioning with deprecation windows. At 20 agents you can email everyone. At 200 you need a registry that knows who uses what, a major-version pin, and a scheduled deprecation with automated impact analysis.
• Per-tenant capacity contracts. A shared quota works while all tenants are small. Once one tenant is 40% of traffic, you need reserved floors and burst pools, and the routing layer needs to know both.
• Self-service onboarding with a gate. Manual ORRs do not scale past roughly one a week. The gate becomes automated checks (eval suite passing, scopes reviewed, cost ceiling set, runbook present) with a human review reserved for high side-effect classes.
• Evidence generation rather than evidence collection. At 20 agents you can assemble an evidence pack for an examiner by hand in a day. At 200 you cannot, and the design must emit linked artifacts as a by-product of serving. Retrofitting this is the single most expensive remediation in the whole track.
• Cost attribution becomes chargeback. Showback changes nothing at 20 agents because nobody is big enough to care. At 200, unattributed cost becomes a tragedy of the commons within one quarter.
• Cardinality governance. See §3. The migration from "label everything" to a label budget is painful and always happens under pressure.

The principal-level move is to build the seams for these at 20 agents — a version field on every tool, a tenant on every metric and key, an artifact emitted by every control — while deferring the machinery until the volume justifies it. Seams are cheap in Phase 00 and extremely expensive in month nine.