« Phase 13 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor

Phase 13 — Staff Engineer Notes: Owning the Tenant Boundary

Anyone can add OAuth and an RBAC table. The gap between using multi-tenancy and being trusted to own it is judgment about things nobody hands you a default for: which resources you silo versus pool, where the tenant identity is allowed to originate, whether the AI leak channels are closed, and — the one that gets escalated to you — whether the platform can pass an audit and a red team without a cross-tenant read. This doc is about that judgment and the signal that proves you have it.

The decisions a staff engineer actually owns here

Where the tenant identity originates is a one-way door. The single most consequential decision is architectural and invisible in a demo: the tenant is an output of authentication, threaded into the data path, and never an input the request can name. Get this right on line one and isolation is a property of the system; retrofit it and you are grep-ing for tenant_id across a hundred queries under incident pressure. You own the rule that the request body never chooses the tenant, and you own enforcing it in the data path (RLS / a scoped query builder) so no handler can forget it.

The isolation model is a portfolio, decided per resource. The junior instinct is one model everywhere. The staff move is a table: pool the stateless compute and cache; pool the primary store behind RLS; silo the regulated store; give top-tier regulated customers a dedicated vector index while others share a namespaced one. You own the cost-vs-isolation-vs-ops math for each row and the promotion path (namespace → dedicated shard → silo) for tenants that outgrow the pool.

Closing the AI leak channels is your responsibility, because classic security training omits them. A team can pass a traditional pen test and still ship a cross-tenant breach on day one via a shared vector index. You own the mandate that the index, the prompt/semantic cache, and agent memory are all partitioned by the trusted tenant, and that fine-tunes are per-tenant adapters, never one blob on the union — because that last one cannot be filtered at query time.

Metadata filtering as a security control is a trust decision only you should make. Using a tenant tag to isolate a shared index is defensible only if the tag is set by a trusted ingestion path, not by the requester, and only if the engine filters before the search. You own that judgment; getting it wrong turns "isolation" into "please don't peek."

The decision framework, out loud

  • A few large regulated customers, compliance-first, blast radius must be one customer → silo the data store per tenant; accept the cost and ops multiplier, price for it.
  • Many small customers, margin-sensitive, isolation enforceable in code → pool behind RLS + namespaces; the cheapest and most scalable, and the one that punishes a forgotten predicate.
  • A mix — a long tail sharing infra and a few whales who demand dedicated → bridge; pool the cheap stateless tier, silo the sensitive store or the top customers, and document which is which.
  • Right-to-erasure / data-residency is a hard requirement → lean toward namespaced-or-siloed data per region so deletion is a partition drop and residency is a deployment boundary.

Naming the model per resource with its blast radius is the signal. "We're multi-tenant" is the anti-signal.

Code-review red flags

  • tenant_id read from the request body, query param, or header. Broken Object-Level Authorization (OWASP API #1). The tenant comes from the verified token or it is a privilege- escalation bug. Non-negotiable.
  • A shared vector index / prompt cache / agent memory with no tenant key. The #1 AI leak channel and its two runners-up. Demand a namespace or a tenant-keyed cache, evaluated before the lookup.
  • RBAC with no default-deny. Default-allow plus a deny-list fails open the first time someone forgets a case. Fail closed.
  • A tenant check in the handler instead of the data path. The handler that forgets it is the breach. Push it into RLS / the scoped store.
  • A tenant-admin role with cross-tenant reach. "Admin" is scoped to one tenant; cross-tenant admin is the escalation the boundary exists to stop.
  • == on a signature, alg taken from the token, or claims trusted before verification. Any one is a JWT CVE waiting to happen. Constant-time compare, pinned algorithm, verify-then-trust.
  • Full request bodies or secret values in the audit log. The log meant to keep you safe becomes the leak. Redact on ingest; log metadata, not content.
  • README.md chapter links, an app running as the DB table owner, or one over-broad role spanning tenants — small tells the author hasn't internalized the boundaries.

War stories worth carrying

  • The confident recall that was a breach. A support agent surfaced another customer's incident report because every tenant's documents sat in one vector index and RAG "found the best chunk." It looked like good retrieval and ran for weeks. The lesson: similarity ignores ownership; the leak is upstream of the model and no prompt catches it. Namespaces are not optional.
  • The tenant_id in the JSON body. An endpoint filtered by a request-supplied tenant. An authenticated user changed one field and read another tenant. The fix was one line — derive the tenant from the signed token — and the bug was one line, which is exactly why it ships.
  • The noisy neighbor that timed everyone out. One tenant's nightly batch consumed the shared model rate limit and every other tenant's requests failed. The fix was a per-tenant token bucket, not more capacity — the capacity was fine, the fairness was missing.
  • The audit log that was the incident. A team dumped request bodies into "audit" for debugging, and the durable copy of every prompt's PII became the thing the regulator flagged. Redaction on ingest, from day one.

The interview / architecture-review signal

What a reviewer listens for: do you keep identity, permission, and tenancy as three separate boxes, and do you know where the tenant comes from? The winning sentence is specific — "I derive the tenant from the verified token and thread it through every data access: rows, vector namespaces, prompt cache, agent memory, secrets — and default-deny everywhere." Then the follow-through: name the four AI leak channels unprompted (especially the shared vector index), place a design in silo/pool/bridge per resource with a cost and blast-radius argument, walk the JWT verification order with its footguns, and explain why a tenant-admin still cannot cross the boundary. Candidates who "wave at auth" are common; the person who names where the tenant identity originates and the AI-specific failure modes is the one handed "make this platform pass the audit."

Closing takeaways

  1. The tenant comes from the token, threaded through the data path. A request-supplied tenant is privilege escalation; enforce derivation where it cannot be forgotten.
  2. A shared vector index will leak. Namespace the index, key the caches by tenant, keep fine-tunes per-tenant — the AI channels are the ones classic security training forgets.
  3. Isolation is a portfolio decision, per resource. Silo the regulated, pool the cheap, name the blast radius for each.
  4. Default-deny, and keep tenancy separate from permission. Two independent gates so one bug is not a breach; cross-tenant is denied even for admins.
  5. Isolation is designed in, not bolted on. The best multi-tenant platform is the one where nothing interesting ever happens across tenants — because someone threaded the boundary through every layer from the first line.