« Phase 17 README · Warmup · Hitchhiker's · Deep Dive · Core Contributor · Staff Notes

Phase 17 — Principal Deep Dive: The Enterprise Agentic Platform

The mechanism view treats the platform as one ordered function. The principal view treats it as a system with a scaling envelope, a failure taxonomy, a cost model, and a set of nested trust boundaries — and asks the question a Staff/Principal system-design interview actually tests: not "can you build a ReAct loop" but "can you reason about the whole thing under load, under attack, and under a budget." This is that view.

The architecture as nested trust rings

Draw the boundaries explicitly, because every security property lives on one. Outside → gateway: the token is untrusted until verified; the question and any pasted content are untrusted forever. Gateway → agent: the agent receives tenant-scoped, guardrail-checked context, may call only allow-listed tools, and its output is scanned before it leaves — the model itself is treated as an untrusted component producing proposals your code validates. Agent → tools/data: least privilege and tenant scoping mean a hijacked agent has a tiny blast radius; it cannot reach another tenant's data or a tool it was not granted. The composed system is a series of rings, each narrowing what the untrusted core can touch. The single most important principal insight is that the model is inside the innermost ring, not trusted machinery — every design decision follows from refusing to trust the model's output.

Scaling each layer independently

The layers have wildly different cost curves, which is why they must scale independently. Auth and authZ are O(1) stateless checks against a JWKS cache and a policy engine — cheap, horizontally trivial, and the correct place to shed load. The token bucket is per-tenant state that wants a fast shared store (Redis-class) so limits are enforced across replicas. Retrieval is I/O- and memory-bound against a vector store whose per-tenant namespacing is both a correctness and a sharding decision. The agent runtime is the expensive, latency-dominant, GPU-bound tier — it is where you route models by difficulty, cache semantically, and enforce a step budget. Eval is sampled precisely because running an LLM-judge on every request would double your model spend. The capacity principle: put the cheap, high-selectivity checks first so the expensive tiers only ever see traffic that has already earned the right to be there.

Failure modes and blast radius

A principal designs by asking "how does each layer break, and what is the blast radius?" Auth fails closed — a bad or expired token is rejected, never allowed; the failure mode is denied service, not granted access. Quota fails closed per-tenant — an over-limit tenant is throttled without affecting others; it also protects the downstream model's own rate limits from being exhausted by one noisy tenant. The input guard has false negatives — which is why least privilege and the output guard exist; no single guard is trusted to be perfect. The eval gate is sampled — so a rare bad answer can slip, which is why output guarding and human review on high-impact paths are the backstops. Retrieval misconfiguration is catastrophic — an unscoped index is a cross-tenant leak with a blast radius of every customer, which is why isolation is threaded through the query and audited, not bolted on. Naming these failure modes, and the fact that no single layer is trusted, is the difference between a design that survives production and a demo.

The "looks wrong but is intentional" decisions

Three choices read as redundant or wasteful until you have operated the system. Redundant guards (input and output and least privilege and sampled eval) are not belt-and-suspenders paranoia; they are defense in depth against the certainty that each layer has a false-negative rate, and the math is that four independent 90%-effective layers leave 0.1% getting through, not 10%. Sampled rather than universal eval looks like a quality gap but is a deliberate cost trade — you buy most of the signal at a fraction of the model spend, and cover the rest with the output guard which runs on 100%. Deriving tenant from the token, never the request looks like distrust of your own clients, and it is — the request body is attacker-controllable and the signed token is not. A design review that flags these as over-engineering is misreading the threat model.

Cross-cutting concerns as first-class, not afterthoughts

Cost/latency are engineered from day one: the platform reports $/resolved-task, not $/request, because a cheap request that fails and retries is expensive; model routing, semantic caching, and a p95 latency budget are instrumented, not aspirational. Observability is the audit-plus-trace substrate — OpenTelemetry-GenAI spans for every agent step, a cost meter per request, and an audit log on every terminal path — so you can answer "what did this agent do, for whom, at what cost?" for any request, including the blocked ones. Multi-tenancy is threaded everywhere identity is used: row-level scoping on the store, per-tenant vector namespaces, tenant-keyed caches, per-tenant secrets and quotas — and the number-one leak remains a shared vector index queried without a tenant filter. Reliability is the compounding-failure math from Phase 00: 0.95^n decides how many autonomous steps you allow before a checkpoint (Phase 08) or a human (Phase 10) — the platform's autonomy budget is a designed number, not an emergent one.

The production mapping

Every miniature in the lab maps to a heavier production component while the lifecycle stays identical: the HMAC token becomes OAuth2/OIDC with short-lived rotated JWTs; lexical retrieve becomes hybrid dense+BM25+rerank on a real vector DB with GraphRAG/RAPTOR; the scripted agent becomes the ReAct/ReWOO runtime calling a real model via MCP tools in a sandbox; the in-memory store becomes Postgres with row-level security and per-tenant namespaces; regex guardrails become Llama-Guard-class classifiers plus the architectural controls; the is_grounded heuristic becomes an LLM-as-judge with a CI regression suite; the Span dataclass becomes exported OpenTelemetry spans; the single process becomes durable workflows, a sandbox, and autoscaling services. The principal point is that only the components get heavier; the order and the boundaries — the actual design — are scale-invariant. That invariance is why the capstone is the right object to rehearse for the senior interview: it is the whole system, reduced to the decisions that do not change with scale.