« Phase 13 README · Warmup · Hitchhiker's · Deep Dive · Core Contributor · Staff Notes

Phase 13 — Principal Deep Dive: Multi-Tenancy as a System

The Deep Dive traced the mechanisms. This doc is about the platform those mechanisms live inside: where the tenant boundary sits in a real production stack, how it scales, what fails and how far the blast radius reaches, and which "looks wrong" decisions are load-bearing. The through-line: multi-tenancy trades physical isolation for logical isolation to earn the economics of SaaS, and your entire job is making the logical boundary as reliable as the physical one you gave up.

The isolation model is a per-resource decision, not a platform-wide one

The AWS SaaS Factory vocabulary — silo (dedicated infra per tenant), pool (shared, isolated in the data path), bridge (hybrid) — is usually taught as if you pick one for the whole system. The principal move is to recognize you pick one per resource, per customer tier. Pool the stateless compute and the cache because they are cheap to share and carry no durable data. Pool the primary datastore too, but back it with row-level security so the tenant predicate cannot be forgotten. Silo the regulated store, or give your top-tier regulated customers a dedicated vector index while everyone else shares one namespaced index. That last sentence is a bridge architecture, and being able to draw it — with the cost and blast-radius math for each resource — is the difference between "we're multi-tenant" and "here is exactly how much isolation each dollar buys." The wrong answer in a review is "pool" or "silo"; the right answer is a table.

The capacity and cost envelope

Pooling is the whole economic argument: one machine at 70% utilization instead of a hundred at 5%, patched and deployed once. But pooling creates two scaling problems the architecture must answer.

• Noisy neighbors. Shared capacity means one tenant's burst can starve everyone. The per-tenant token bucket bounds each tenant's long-run rate to refill_per_tick while allowing bursts to capacity. At platform scale this becomes a distributed limiter (Redis-backed) plus fair-share scheduling plus per-tenant cost budgets ($/month), because a tenant can be within its request quota and still blow your model bill. The three controls answer three different questions: requests/sec (bucket), fairness under contention (scheduler), and spend (budget).
• Per-tenant data skew. Pooled indexes and tables develop hot tenants — one customer with 10× the data or traffic. Namespacing keeps them correct but not balanced; at scale you shard hot tenants onto their own partitions (the pool→bridge migration), which is why namespace-per-tenant is the right primitive even in a pool: it is the seam you shard along later without rewriting queries.

The capacity story is therefore not one number but a composition: pooled base capacity, per-tenant quotas carving it into fair slices, and a documented promotion path (namespace → dedicated shard → silo) for tenants that outgrow the pool.

Failure modes and blast radius

Think in blast radius, because in multi-tenancy the interesting failures are not crashes — they are quiet cross-tenant reads that look like normal operation.

• The trusted request body (BOLA/IDOR). A handler reads tenant_id from the request and filters by it. Nothing errors; the query runs perfectly — against the attacker's chosen tenant. Blast radius: every tenant, immediately, because any authenticated user can now name any tenant. This is OWASP API #1 and the single most common serious API vulnerability in the wild. The architectural guard is that the tenant is an AuthN output threaded into the data path, never an AuthZ input.
• The shared vector index leak. The team nails OAuth, RBAC, and RLS, then dumps every tenant's embeddings into one index because "RAG finds the best chunk." Retrieval surfaces B's document in A's answer whenever it is more similar. Blast radius: any pair of tenants whose data is semantically close, and the leak looks like good recall, so it can run for months undetected. No prompt fixes it; the leak is upstream of the model.
• RBAC that fails open. Default-allow with a deny-list leaks the moment someone forgets a case, and someone always forgets a case. Blast radius: whatever the missing case gated. Default-deny makes the failure mode "access denied" (a ticket) instead of "access granted" (a breach).
• The audit log as the breach. A team dumps full request bodies into the "audit" log for debugging. Now every prompt's PII and every secret sits in a second durable copy with weaker access controls than the primary store. Blast radius: your entire compliance perimeter. Redact on ingest; log metadata, never content.

Cross-cutting concerns

Security. The tenant boundary composes with, but is not, access control. IAM/RBAC decides who may do what; the tenant predicate decides whose data this is; they fail independently and a platform needs both. The verified token is the root of trust for the entire request — which is why its signing key is the platform's crown jewel, why tokens are short-lived and rotated, and why alg is pinned (an attacker who talks your verifier into alg: none owns every tenant).

Cost. Isolation is not free: a per-tenant namespace, bucket, and budget is per-tenant state and per-tenant bookkeeping, and silos multiply your operational surface by N. The cost lever is which resources you silo — silo only what compliance forces, pool the rest, and price the dedicated tier to cover the operational tax.

Observability. In multi-tenancy every metric, log, and trace carries tenant_id as a first-class dimension — not for dashboards but for isolation verification and chargeback. You need to answer "what did tenant A cost, and did any request cross a tenant boundary?" which means the tenant tag has to be stamped from the trusted principal at the edge and propagated on the correlation id (the trace story of Phase 14).

Multi-tenancy vs the AI channels. The genuinely new surface is that an agent has four extra leak channels a web app does not: the vector index (#1), the prompt/semantic cache, agent memory, and co-mingled fine-tunes. The first three are query-time partition problems (namespace them). The fourth is different in kind — it is baked into the weights, unfilterable at query time, so the only mitigation is architectural: per-tenant LoRA adapters or per-tenant models, never one blob trained on the union.

The "looks wrong but is intentional" decisions

• Enforce isolation in the data path, not the handler. Putting the tenant filter in RLS / a scoped query builder looks like over-engineering when one WHERE clause would do. It is exactly right: the handler that forgets the clause is the breach, and there will be a handler that forgets. Enforce where it cannot be forgotten.
• Deny cross-tenant even for a tenant-admin. It reads like an admin restriction bug. It is the point: "admin" is scoped to your own tenant; there is no cross-tenant authority in the model, because cross-tenant admin is precisely the privilege escalation the boundary exists to stop.
• One AuthError for every auth failure. Collapsing "bad signature," "expired," and "malformed" into one type looks like poor error reporting. It denies an attacker an oracle — a verifier that distinguishes them tells the attacker how close a forgery is.
• Redact before store, even though it's redundant with "never log secrets." Belt and suspenders on the one log that becomes catastrophic if it leaks. The backstop is the point.

Where this fits the platform decision

Every serious enterprise agent platform — Citi's multi-tenant agentic ecosystems, Cohere's and Docker's shared platforms, any SaaS in jd.md — solves the same five problems: prove identity, decide permission, isolate data (including the AI channels), meter fairly, and audit immutably. The controls in this phase are the evidence a SOC 2 auditor asks for and the GDPR obligations (right-to-erasure is easier when data is namespaced; data residency pushes you toward regional silos) mapped to concrete mechanisms. The principal signal is not "we use RBAC"; it is naming, per resource, the isolation model, its blast radius, and its cost — and knowing that on an AI platform the vector index is the boundary classic security training forgot.