« Phase 13 README · Warmup · Hitchhiker's · Principal Deep Dive · Core Contributor · Staff Notes

Phase 13 — Deep Dive: The Mechanics of Tenant Isolation

The load-bearing mechanism of this phase is not "add auth." It is a single invariant enforced at every data-access site: the effective tenant of any read or write is principal.tenant, a value that originated inside a signature-verified token and is never sourced from the request. Every data structure in the lab exists to make that invariant impossible to violate — not merely unlikely. This doc traces those structures, the order of operations that keeps them sound, and the exact points where the naive design silently leaks.

The token is a signed byte string, not a session

issue_token builds a JWT-shaped record header.payload.signature. The header pins {"alg": "HS256"}; the payload is the claim set {tenant, sub, roles, iat, exp}; the signature is HMAC-SHA256(secret, base64url(header) + "." + base64url(payload)). The critical property is that the tenant lives inside the signed region. Flip one byte of the payload — acme→globex — and the recomputed HMAC no longer equals the attached tag. That is what lets a stateless verifier trust the tenant with zero server-side lookup: integrity is carried by the message.

verify_token is a strict, ordered pipeline, and the ordering is the security:

  1. Structural split. Split on . and reject anything that is not exactly three parts. A 2-part or 4-part string never reaches the crypto.
  2. Recompute and compare the signature over header.payload using hmac.compare_digest, a constant-time comparison. This runs before any claim is decoded, because until the signature verifies, the payload is attacker-controlled bytes with no more authority than the request body.
  3. Decode the payload, then check exp against an injected now (now >= exp ⇒ expired). The clock is a parameter, not time.time(), so expiry is deterministic and replay-testable.
  4. Construct a Principal(tenant, user, roles) — the only trusted identity in the system.

Two footguns the ordering closes. First, comparing signatures with == leaks, through timing, how many leading bytes matched; compare_digest collapses that side channel by always scanning the full length. Second, trusting claims before verifying is the entire class of JWT breaches — read exp first and an attacker sets exp to the year 3000; read tenant first and you have already lost. The invariant "verify, then trust" is not stylistic, it is the whole contract of a bearer token.

The tenant-derivation invariant, enforced twice

The cardinal rule — tenant from the token, never the request — is defended at two independent layers so a single bug is not a breach:

  • AuthZ gate. authorize compares the request's claimed resource_tenant against principal.tenant and denies any mismatch before it even consults roles. Cross-tenant is rejected first; role permissions are checked only within the caller's own tenant.
  • Data layer. TenantStore.query, VectorIndex.search, SecretStore.get_secret all take a Principal and scope only by principal.tenant. There is no code path where a request-supplied tenant reaches storage — the request never gets to name the tenant it queries.

This is defense in depth in the literal sense: two gates, structurally different, both keyed off the same trusted field. The naive single-gate design ("check the tenant in the handler") fails the day one handler forgets the check; the two-gate design requires both the authz comparison and the data-layer scoping to be wrong simultaneously.

Row-level security as a pure function

TenantStore.query(principal, filters) models Postgres RLS in three rules, and the third is the whole lesson:

  1. Force the predicate row.tenant == principal.tenant on every row — the injected boundary.
  2. Apply the caller's other filters (amount == 100) normally.
  3. Drop any tenant key the caller put in filters. A crafted filters={"tenant": "globex"} from an acme caller is silently discarded; the forced predicate still returns only acme rows.

Rule 3 is what makes the tenant predicate "not the caller's to set." In real Postgres, CREATE POLICY ... USING (tenant_id = current_setting('app.current_tenant')) appends the predicate inside the database regardless of the query text, so a forgotten WHERE is no longer a leak — the engine adds it. The lab reproduces that as an unconditional filter the caller's input cannot widen. Query complexity is O(n) over the tenant's rows; the point is not speed but that scope is a property of the function, not of the caller's discipline.

The #1 leak: vector search is namespaced, not filtered-after

VectorIndex is a dict[tenant → list[(id, vector)]]. add(principal, id, vec) writes into principal.tenant's bucket; search(principal, query, k) computes cosine similarity only against that bucket, sorts descending, breaks ties by id, and returns the top-k. The mechanism that matters is where the tenant filter sits relative to the nearest-neighbor computation.

The naive design keeps one global index and filters results after the ANN search — or worse, doesn't filter at all and "lets RAG find the best chunk." Both fail at the mechanism level: similarity is a geometric property of embeddings and does not respect ownership. When tenant B's document is geometrically closer to tenant A's query than any of A's own documents, a global-then-filter search either (a) returns B's chunk because there's no filter, or (b) burns the top-k slots on B's vectors and hands A a thinner, sometimes empty result while B's text sat one row away in memory. Post-filtering also leaks timing and existence — the search touched B's vectors to score them. Namespacing moves the tenant partition before the distance computation: A's query is never scored against B's vectors at all. The lab proves it by inserting an identical vector under two tenants and showing A's search can never surface B's copy.

The same shape recurs in the other two AI channels. PromptCache keys on the tuple (tenant, prompt), so an identical prompt from a different tenant is a miss, never a cross-tenant hit that would echo B's private answer to A. AgentMemory.recall returns only the caller's partition. In all three the fix is identical: the tenant is part of the key or the partition, evaluated before the lookup, not a filter bolted on after.

The token bucket: two invariants make it correct

RateLimiter.allow(tenant, cost, now) is a lazy token bucket per tenant. State per bucket: capacity, tokens, refill_per_tick, last_tick. On each call:

  1. Lazy refill: tokens = min(capacity, tokens + (now - last_tick) * refill_per_tick), then last_tick = now. Refill is computed on demand from the injected clock, not by a background thread — no timer, fully deterministic.
  2. Spend iff affordable: if tokens >= cost, deduct and allow; else allow nothing, deduct nothing, deny.

Two invariants, both tested: never negative (a denied request spends zero, so a burst of denials can't drive the bucket below zero and delay recovery) and refill capped at capacity (an hour of idleness cannot bank unlimited burst; the next tick starts with at most capacity). The per-tenant part is the isolation: buckets are independent, so tenant A draining its tokens has exactly zero effect on B's — that is noisy-neighbor isolation expressed as a data-structure boundary, not a policy.

The gateway pipeline: fixed order, structured outcomes

Gateway.handle(token, request) composes everything in one non-negotiable order:

verify_token → authorize → rate-limit → _dispatch (tenant-scoped) → audit

Ordering is load-bearing. Verify before authorize (you cannot decide permissions for an unverified identity). Authorize before rate-limit (don't spend a cross-tenant attacker's tokens telling them "denied" — though either order is defensible, the lab denies early). Dispatch keyed off principal so the data layer never sees a request-named tenant. Audit last, always — even on denial — because the security value of the log is precisely the denied and anomalous events. Each failure raises a structured GatewayError/AuthError, never a bare stack trace, and AuthError collapses "bad signature" / "expired" / "malformed" into one type on purpose: a verifier that distinguishes them hands an attacker an oracle.

Audit: redact on the way in, hand back copies

AuditLog.record stamps a monotonic sequence number and appends; there is no update or delete, so the trail is tamper-evident, and entries() returns a copy so a caller cannot mutate history. redact walks the event recursively and masks any field named like a secret before it is stored — defense in depth so that the log built to keep you safe never becomes the leak. Metadata (who, what, tenant, decision, correlation id) is recorded; secret content never is.

What to hold onto

Strip the vocabulary and one mechanism runs through the whole phase: make the trusted tenant a structural property of every key, partition, and predicate, evaluated before the operation, so that violating isolation requires two independent bugs instead of one forgotten WHERE. Verify before you trust; partition before you search; derive, never accept. Every leak in this space is a place where someone let the request name the tenant.