« Phase 13 README · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 13 Warmup — Secure Multi-Tenant Platform: AuthN/Z, Isolation, Secrets & Quotas

Who this is for: you can write Python and you did the earlier phases. You may have used "log in with OAuth" and heard "multi-tenant SaaS" but never had to build the boundary that keeps one customer's data away from another's inside one shared process. By the end you will have built an agent gateway that authenticates, authorizes, isolates, meters, and audits — and you will be able to threat-model tenancy like a platform engineer. Nothing here needs a GPU, an API key, or a network. It is hmac, a dictionary, and one relentless idea: derive the tenant from the token, not the request.

Table of Contents

  1. Why multi-tenancy exists — and the obligation it creates
  2. AuthN vs AuthZ: two different questions
  3. OAuth2, OIDC, and the bearer token
  4. The JWT under the hood: header, payload, signature
  5. The cardinal rule: never trust tenant_id from the request
  6. RBAC, ABAC, and default-deny
  7. Isolation models: silo, pool, and bridge
  8. Row-level security: threading the tenant through every query
  9. The AI-specific leak channels (the shared vector index is #1)
  10. Secrets and key custody
  11. Quotas and the noisy-neighbor problem
  12. Audit logs: append-only, metadata not content
  13. Encryption and compliance: SOC 2, GDPR, data residency
  14. Common misconceptions
  15. Lab walkthrough
  16. Success criteria
  17. Interview Q&A
  18. References

1. Why multi-tenancy exists — and the obligation it creates

A tenant is a customer boundary: a company, a business unit, a team — a set of users and data that must be kept logically separate from every other set. Multi-tenancy means one running system serves many tenants on shared infrastructure: the same servers, the same database, the same vector index, often the same process and memory.

Why share at all? Cost and operations. Running a dedicated stack per customer (a "single- tenant" deployment each) means paying for idle capacity times the number of customers, and patching, deploying, and monitoring N copies of everything. Pooling lets you buy one big machine running at 70% utilization instead of a hundred small ones at 5%, and you deploy once. This is the core economic engine of SaaS; it is why your platform can charge $20/seat and still profit.

But sharing creates an obligation: because tenant A's request executes in the same process that holds tenant B's data, the isolation is entirely your code's responsibility. There is no operating system, no VM, no network segment automatically keeping them apart — you removed those walls on purpose to save money. In a single-tenant world, a bug leaks one customer's data to themselves. In a multi-tenant world, the same bug leaks it to a different customer, which is a breach. The whole of this phase is the discipline of earning the cost savings without paying the breach.

The one-sentence framing. Multi-tenancy trades physical isolation for logical isolation to save money; your job is to make the logical isolation as reliable as the physical isolation you gave up.

2. AuthN vs AuthZ: two different questions

Two words that sound alike and mean different things, and confusing them is a classic security bug:

  • Authentication (AuthN) — who are you? The system verifies the caller's identity, usually by checking a credential (a password, a signed token, a client certificate). Output: a trusted identity — in our lab, a Principal with a tenant, a user, and roles.
  • Authorization (AuthZ) — are you allowed to do this? Given a known identity, the system decides whether a specific action on a specific resource is permitted. Output: a boolean decision (allow / deny).

The order is fixed: authenticate first, then authorize. You cannot decide what someone may do until you know who they are. And the failure modes differ: an AuthN failure means "I don't believe you are who you claim" (a forged or expired token); an AuthZ failure means "I believe you, but you may not do that" (a viewer trying to write, or anyone reaching across the tenant boundary).

A subtle but critical point for tenancy: the tenant is an AuthN output, not an AuthZ input you take from the caller. The verified token tells you the tenant; you do not ask the request which tenant it wants to be. Section 5 is entirely about this.

3. OAuth2, OIDC, and the bearer token

OAuth 2.0 (RFC 6749) is the industry-standard framework for delegated authorization — it lets a user grant an application limited access to their resources without sharing a password. Its sibling OpenID Connect (OIDC) layers authentication on top of OAuth 2.0, adding an ID token that proves who the user is. When a JD says "OAuth, authorization," this is the vocabulary: authorization servers, clients, scopes, access tokens, and the flows (authorization- code, client-credentials) that mint them.

The unit that actually travels on each request is the bearer token: a string the client puts in the Authorization header (Authorization: Bearer <token>). "Bearer" means whoever holds it, wields it — there is no further proof of identity, so a stolen bearer token is a stolen identity. That is why bearer tokens are short-lived (minutes to an hour), sent only over TLS, and never logged. A scope is a coarse permission the token carries (read:invoices, write:reports); scopes and roles together bound what the token can do.

We do not run a real OAuth server in this lab — no network — but we build the piece that every OAuth deployment ultimately relies on: a signed, self-describing token you can verify offline. That token is almost always a JWT.

4. The JWT under the hood: header, payload, signature

A JSON Web Token (JWT, RFC 7519) is a bearer token with a specific shape: three base64url- encoded parts joined by dots — header.payload.signature. Written out (the dots are literal):

  eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9   .   eyJ0ZW5hbnQiOiJhY21lIiwic3ViIjoiYWxpY2Ui...}   .   3f9a...c2
  └──────────── header ────────────┘       └──────────────── payload (claims) ───────────┘      └ signature ┘
  • Header — metadata: {"alg": "HS256", "typ": "JWT"}. alg names the signing algorithm.
  • Payload — the claims: JSON key/values the token asserts. Standard claims include sub (subject/user), iat (issued-at), exp (expiry). We add tenant and roles. Claims are not encrypted — only signed. Anyone can base64-decode and read them; the signature only guarantees they were not changed.
  • Signature — a keyed hash over header.payload. With HS256 it is \(\text{HMAC-SHA256}(\text{secret},\ \text{header}.\text{payload})\).

HMAC (Hash-based Message Authentication Code, RFC 2104) is a keyed hash: only someone holding the secret can produce a signature that verifies. Crucially, HMAC is not the same as hash(secret + message) — the naive construction is vulnerable to length-extension attacks, which is exactly why a real primitive exists and why the lab uses hmac.new(...) and not a hand-rolled hashlib.sha256(secret + msg).

Verification is the security-critical dance, and the order is not negotiable:

  1. Split into three parts (reject if not exactly three).
  2. Recompute the signature over header.payload with your secret and compare it to the token's signature using a constant-time comparison (hmac.compare_digest) — a byte-by-byte == leaks, through timing, how many leading bytes matched, which an attacker can exploit to forge a signature. Reject on mismatch. Do this before you trust any claim, because an unverified payload is attacker-controlled bytes.
  3. Only now decode the payload and check exp against the current time — an injected clock in the lab, so tests are deterministic (now >= exp means expired). A wall clock would make the test non-reproducible and, in a durable system, non-replay-safe (Phase 08).

Change any claim — flip tenant from acme to globex — and step 2 fails, because the signature no longer matches. That is how a stateless token proves integrity with no server-side session. In the lab, issue_token mints one and verify_token returns a Principal or raises AuthError; tampered, wrong-key, and expired tokens are all rejected.

alg: none and the confused-algorithm bug. Real JWT libraries have been breached by accepting a token whose header says alg: none (no signature) or by verifying an HS256 token using an RSA public key as the HMAC secret. The lesson: pin the algorithm you accept; never let the token's own header talk you out of verifying it.

5. The cardinal rule: never trust tenant_id from the request

This is the single most important sentence in the phase:

The tenant is derived from the verified token, never read from the request body.

Picture the wrong version. A handler does:

tenant = request.json["tenant_id"]          # WRONG: attacker-controlled
rows = db.query("SELECT * FROM invoices WHERE tenant_id = ?", tenant)

An attacker authenticated as tenant acme simply sends {"tenant_id": "globex"} and reads Globex's invoices. The database dutifully filters — by the wrong tenant, the one the attacker chose. This is Broken Object-Level Authorization (OWASP API Security #1, "BOLA/IDOR"), and it is the most common serious API vulnerability in the wild. The request body is untrusted input; letting it choose the tenant is handing the caller a SELECT on everyone's data.

The right version derives the tenant from the thing the caller cannot forge — the signed token:

principal = verify_token(bearer, secret)     # tenant is inside the SIGNED payload
rows = store.query(principal, filters)        # tenant forced to principal.tenant

In the lab this is enforced twice, on purpose (defense in depth). The gateway's authz gate compares the request's claimed resource_tenant against principal.tenant and denies a mismatch. And independently, the data layer (TenantStore.query, VectorIndex.search, SecretStore.get_secret) is always scoped by principal.tenant and never by the request's claim — so even if authz had a bug, there is no code path where a request-supplied tenant reaches storage. Two independent gates mean one mistake is not a breach.

6. RBAC, ABAC, and default-deny

Once you know who the caller is, you decide what they may do. Two models dominate:

  • RBAC — Role-Based Access Control. Users have roles (viewer, editor, tenant-admin), roles map to permissions (read, write, manage), and a request is allowed if the caller's roles include the needed permission. It is a small, auditable table — the opposite of if user.name == "alice" checks scattered through the code. RBAC (formalized by NIST) is the workhorse of enterprise authorization because it is easy to reason about and to review.
  • ABAC — Attribute-Based Access Control. The generalization: decisions are a function of attributes of the subject, the resource, the action, and the environment (NIST SP 800-162). "A user in the EU region may read documents classified internal during business hours" is an ABAC policy RBAC cannot express. ABAC is more expressive and more complex; most platforms use RBAC for the coarse decisions and ABAC-style attribute checks (tenant, region, sensitivity) for the fine ones. The tenant check is itself an attribute check — it is the one ABAC rule every multi-tenant RBAC system needs.

The posture that matters more than the model is default-deny (a.k.a. fail closed): anything not explicitly granted is denied. The opposite — default-allow with a deny-list — fails open the moment you forget to add a case, and you will forget a case. In the lab, authorize returns True only when the action is in the union of the caller's role permissions and the resource belongs to the caller's tenant; an unknown role grants nothing, an unknown action is denied, and cross-tenant is denied even for a tenant-admin — because "admin" is scoped to your own tenant, not to everyone's.

7. Isolation models: silo, pool, and bridge

How physically separate are the tenants? The AWS SaaS Factory vocabulary gives three points on a spectrum, and naming your design is a Staff-level move in a review:

  • Silo — dedicated infrastructure per tenant. Each tenant gets its own database (or its own whole stack). Strongest isolation, simplest mental model ("their data is in their database"), easiest compliance story — and the most expensive and operationally heavy (N stacks to run and patch). Common for a handful of large regulated enterprise customers.
  • Pool — shared infrastructure, isolated in the data path. All tenants share one database, one index, one process; isolation is enforced by tenant_id on every row and every query (Section 8). Cheapest and most scalable, and the hardest to get right, because one missing WHERE tenant_id = ? is a leak. This is what our lab models.
  • Bridge — hybrid. Some resources pooled, some siloed. A common pattern: pool the stateless compute and the cache, but silo the most sensitive data store (or give your top customers a dedicated index while everyone else shares one). Most real platforms end up here.

There is no universally correct choice — it is a cost vs isolation vs operational-burden tradeoff, decided per resource and per customer tier. The senior answer in an interview is not "pool" or "silo"; it is "pool the cheap stateless tier, silo the regulated data, and here's the cost and blast-radius math for each."

8. Row-level security: threading the tenant through every query

In the pool model, isolation is the query. Every table carries a tenant_id column, and every read must filter on it. The danger is human: one query, written by one tired engineer, that forgets the WHERE tenant_id = ? and returns everyone's rows.

Row-Level Security (RLS) moves that filter out of the application and into the data layer so it cannot be forgotten. PostgreSQL implements it directly:

ALTER TABLE invoices ENABLE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation ON invoices
  USING (tenant_id = current_setting('app.current_tenant')::uuid);

Now every query against invoices — no matter who wrote it — is automatically filtered to the tenant in the session variable, which the connection sets from the verified token. A forgotten WHERE is no longer a leak; the database appends the predicate for you.

Our TenantStore.query is a pure-Python model of exactly this. Three rules:

  1. Always require row.tenant == principal.tenant — the injected predicate, the RLS boundary.
  2. Apply any other filters the caller supplied (amount == 100).
  3. Drop any tenant key in the caller's filters, so a crafted filters={"tenant": "globex"} from an acme caller is silently ignored and still returns only acme rows.

That third rule is the whole lesson in one line: the tenant predicate is not the caller's to set. The lab proves it — a crafted cross-tenant filter returns nothing but the caller's own rows.

9. The AI-specific leak channels (the shared vector index is #1)

Everything so far is classic SaaS security. Now the part that is new to agent platforms, and the reason this phase exists in an agentic curriculum: an agent has four extra places to leak between tenants that a normal web app does not have, and the biggest one is invisible if you only think about SQL.

1. The shared vector index — the #1 leak channel. RAG (Phase 05) embeds every document into a vector index and, at query time, retrieves the nearest vectors by similarity. If that index is shared across tenants, a query from tenant A retrieves the globally-nearest chunks — and similarity does not respect ownership. Whenever tenant B's document happens to be more similar to A's question than A's own documents, B's text is retrieved, injected into the prompt as "context," and paraphrased straight into A's answer. The user sees another company's data in their agent's reply. No prompt prevents this — the leak happens in the retrieval step, before the model runs. The fix is architectural: partition the index by tenant namespace and search only the caller's. In production that is a per-tenant Pinecone namespace, a Weaviate class/tenant, a separate index per tenant, or a WHERE tenant_id = ? metadata filter on the ANN query. The lab's VectorIndex.search looks up principal.tenant's namespace and cannot see any other — proven by adding an identical vector to two tenants and showing tenant A's search never returns tenant B's.

2. The prompt / semantic cache. Caching model responses is a large cost win (Phase 14) — but if the cache key is just the prompt text, tenant A asking the same question as tenant B receives B's cached answer, which may quote B's private data. The fix is one line: key the cache on (tenant, prompt). Identical prompts from different tenants are different cache entries. The lab's PromptCache proves a same-prompt lookup from another tenant is a miss, never a cross hit.

3. Agent memory. Long-term agent memory (conversation history, learned facts — Phase 04) is a store like any other; if it is one pool, agent runs for tenant A can recall facts written by tenant B. Partition it per tenant. The lab's AgentMemory.recall returns only the caller's items.

4. Co-mingled fine-tunes. If you fine-tune one model on many tenants' data, the weights now encode all of them, and the model can regurgitate one tenant's training data to another — a leak you cannot filter at query time because it is baked into the parameters. The mitigation is per-tenant adapters (a LoRA per tenant, loaded for that tenant's requests) or per-tenant models, never one blob trained on the union. (Cross-ref Phase 05 for the index, Phase 04 for memory, Phase 10 for how injected text weaponizes any of these channels.)

Why this is the phase's headline. A team can nail OAuth, RBAC, and RLS and still ship a cross-tenant breach on day one because they put every tenant's documents in one vector index and "let RAG find the best chunk." The shared vector index is the leak that classic security training does not warn you about. Namespaces are not optional.

10. Secrets and key custody

Tenants have secrets: API keys to their own systems, database passwords, the platform's own token- signing key. Two rules survive every design:

  • Scope every secret to a tenant, and fetch it by the trusted tenant. A secret lives at (tenant, name); get_secret(principal, name) looks up (principal.tenant, name), so a caller can only ever resolve its own tenant's secret — a cross-tenant read does not "get denied," it simply does not exist for that caller (not leaking existence is itself a control).
  • Never write a secret to a log, a trace, or an error message. This is where secrets actually leak in practice — not through a clever attack, but through a debug log that dumped a request, an exception that printed a config, an audit event that recorded the payload. The lab's audit layer redacts any field named like a secret before it is stored, as a backstop for exactly this mistake.

In production the store is a vault — HashiCorp Vault, AWS Secrets Manager, a KMS-encrypted table — with short-lived, rotated credentials and an audit trail of every access. The strongest posture is BYOK (bring-your-own-key): the tenant holds the root encryption key, so the platform can decrypt the tenant's data only with the tenant's cooperation, and a platform-side compromise does not expose the tenant's plaintext. Regulated customers ask for BYOK by name.

11. Quotas and the noisy-neighbor problem

Shared infrastructure has a shared-resource problem: one tenant sending a flood of requests can consume all the capacity and starve everyone else. That is the noisy-neighbor problem, and the fix is a per-tenant quota so each tenant's usage is bounded independently.

The standard mechanism is the token bucket. Picture a bucket that holds up to capacity tokens and refills at refill_per_tick tokens per unit time. Each request costs one (or more) tokens; if the bucket has enough, the request proceeds and tokens are deducted; if not, the request is denied (rate-limited). The bucket allows short bursts up to capacity while bounding the long-run rate to refill_per_tick — which matches how real traffic behaves (bursty, but bounded on average). Contrast a fixed window (N requests per minute), which is simpler but allows a double-rate burst across a window boundary; the token bucket is the interview-safe default.

Two invariants make it correct, and the lab tests both:

  • Never negative. Tokens are deducted only when the bucket has enough; a denied request spends nothing. The bucket can sit at 0 but never below — otherwise a burst of denials could "owe" tokens and delay recovery.
  • Refill is capped at capacity. Idle time cannot bank unlimited burst; a tenant that was quiet for an hour still starts the next second with at most capacity tokens.

The per-tenant part is the isolation: each tenant has its own bucket, so tenant A draining its tokens has zero effect on tenant B's. The lab's RateLimiter uses an injected clock (an integer tick counter), so refill-over-time is deterministic and testable — no wall clock. In production this is a per-tenant limiter in the gateway (or a distributed one in Redis), and it pairs with fair-share scheduling and per-tenant cost budgets ($/month) for the full quota story.

12. Audit logs: append-only, metadata not content

When a regulator, a customer, or an incident responder asks "who accessed this data, when, and was it allowed?", the answer is the audit log. Its properties are specific:

  • Append-only. Entries are added, never updated or deleted. Immutability is what makes the log tamper-evident — if an attacker who breaches the system could edit the audit trail, the trail is worthless. In practice this is a write-only sink (an append-only table, an object-lock bucket, a log pipeline to a SIEM). The lab's AuditLog has record and entries but no delete, and hands back a copy so callers cannot mutate the trail.
  • Metadata, not content. The log records who (user, tenant), what (action, resource kind), when (a timestamp / sequence number), and the decision (allow/deny + reason) — not the secret content. You want to know that Alice at Acme read secret stripe_key and was allowed; you must never record the secret's value. The lab redacts any secret-named field before it is stored, as defense in depth against an accidental leak into the very log meant to keep you safe.
  • Correlation IDs. A single logical request fans out across services; a correlation (or trace) id stitches the audit entries back together so you can reconstruct one request's whole path (this is the trace story of Phase 14).

An audit log is not a debug log. Debug logs are verbose, mutable, and short-lived; audit logs are minimal, immutable, and retained for years for compliance. Conflating them — dumping full request bodies into your "audit" log — is how the audit log itself becomes the breach.

13. Encryption and compliance: SOC 2, GDPR, data residency

Two more controls turn a secure design into a compliant platform:

  • Encryption in transit and at rest. In transit: TLS on every hop, so a network tap sees ciphertext. At rest: the database, the vector index, the object store, and the backups are encrypted with managed keys (and, for the strongest posture, per-tenant keys / BYOK — a compromised disk leaks nothing, and revoking a tenant's key cryptographically erases their data).
  • Compliance frameworks name the controls you built. SOC 2 is an audit of your security controls across five "trust service criteria" (security, availability, processing integrity, confidentiality, privacy); the access control, audit logging, and encryption in this phase are literally the evidence a SOC 2 auditor asks for. GDPR governs personal data of EU residents and adds obligations like the right to erasure (you must be able to delete a tenant's data — easier when it is namespaced) and data residency (some data must physically stay in a region, which pushes you toward regional silos or per-region deployments). You do not need to be a lawyer, but a Staff platform engineer knows which technical control each requirement maps to, because the auditor will ask.

None of this is in the lab's code — it is offline and has no disks or network — but it is the frame the lab's mechanisms live inside, and it is what an enterprise interviewer means by "secure, compliant, multi-tenant."

14. Common misconceptions

  • "I filter by the tenant_id in the request, so I'm isolated." You isolated by the attacker's chosen tenant. The tenant must come from the verified token; a request-supplied tenant_id is privilege escalation (BOLA/IDOR).
  • "A shared vector index is fine, retrieval just finds the best match." That is precisely the leak: the best match may be another tenant's document, and it lands in your user's answer. Namespace per tenant or you will leak — this is the #1 multi-tenant AI incident.
  • "RBAC means we're secure." RBAC without default-deny fails open; RBAC without a tenant check lets a tenant-admin roam across tenants. The model is necessary, not sufficient.
  • "We'll add the tenant check in the handler." Put it in the data path (RLS), not in each handler, or the one handler that forgets it is the breach. Enforce it where it cannot be forgotten.
  • "A JWT is encrypted, so I can put secrets in it." A JWT is signed, not encrypted — anyone can read the claims. Never put a secret in a token payload.
  • "Rate limiting is a performance feature." In multi-tenancy it is an isolation feature: it stops a noisy neighbor from starving everyone else. Per-tenant, or it does nothing for fairness.
  • "The audit log is where we dump everything for debugging." No — it is minimal, immutable, and redacted. Full request bodies in the audit log turn the log into the leak.
  • "Signature check with == is fine." Use a constant-time compare (hmac.compare_digest); a plain == leaks match progress through timing and enables signature forgery.

15. Lab walkthrough

Open lab-01-multi-tenant-gateway/ and fill the TODOs top to bottom — the file is ordered to match this warmup:

  1. AuthN — _sign (HMAC-SHA256, base64url), issue_token (build the claims, sign header.payload), verify_token (three parts → constant-time signature check → decode → exp vs injected now → Principal). Tests probe tampered, wrong-key, and expired tokens.
  2. AuthZ — permissions_for (union over roles, unknown role → empty) and authorize (default-deny; cross-tenant denied before the role check).
  3. Isolation — TenantStore.insert (stamp the tenant) and query (force principal.tenant, drop any caller-supplied tenant filter).
  4. AI leak channels — VectorIndex.add/search (per-namespace, descending cosine, ties by id), PromptCache.put/get (key on (tenant, prompt)), AgentMemory.append/recall, SecretStore.get_secret ((principal.tenant, name)).
  5. Quotas — RateLimiter.allow (lazy refill capped at capacity, spend only if affordable, never negative).
  6. Audit — redact (recursive mask of secret-named fields) and AuditLog.record (seq + append).
  7. Gateway — handle (verify → authorize → rate-limit → dispatch → audit) and _dispatch (route to the tenant-scoped layer, always keyed off principal).

Run LAB_MODULE=solution pytest test_lab.py -v first to see green, then make your lab.py match. Finish by reading solution.py's main() output — it walks the whole warmup: tenant A reads its own data, is denied tenant B's, a forged token is rejected, the quota exhausts and refills, and the audit log prints with secrets redacted.

16. Success criteria

  • You can explain AuthN vs AuthZ and why the tenant is an AuthN output, not a request input.
  • You can describe a JWT's three parts and why the signature is verified before the claims are trusted, with a constant-time compare.
  • You can state the cardinal rule (tenant from the token, never the request) and name the attack it prevents (BOLA/IDOR, privilege escalation).
  • You can name the three isolation models (silo/pool/bridge) and place a design in one with a cost/isolation argument.
  • You can name the four AI-specific leak channels and the isolation control for each, and say why the shared vector index is #1.
  • You can whiteboard a token bucket and argue its two invariants (never negative, capped refill) and its per-tenant isolation.
  • All 35 lab tests pass under both lab and solution.

17. Interview Q&A

Q: A handler reads tenant_id from the JSON body and filters the query by it. What's wrong? A: The tenant is attacker-controlled — an authenticated Acme user sends {"tenant_id": "globex"} and reads Globex's data. That's Broken Object-Level Authorization (OWASP API #1). The tenant must be derived from the verified token and threaded through the query; the request body never chooses the tenant. Ideally enforce it in the data path (Postgres RLS / a scoped query builder) so no handler can forget it.

Q: Your platform serves many customers from one vector database. Where's the leak, and how do you fix it? A: A shared vector index is the #1 multi-tenant AI leak channel: RAG retrieves the nearest vectors by similarity, and similarity ignores ownership, so tenant A's query can retrieve tenant B's chunk whenever B's is more similar — and it lands in A's answer. No prompt fixes it; the leak is in retrieval. Fix: partition by tenant namespace (a Pinecone namespace, a Weaviate tenant, a per-tenant index, or a tenant_id metadata filter on the ANN query) and search only the caller's namespace, derived from the token.

Q: Difference between authentication and authorization, and where does the tenant fit? A: AuthN is "who are you" — verify a credential, produce a trusted identity. AuthZ is "may you do this" — a default-deny decision over that identity. The tenant is an output of AuthN (it's in the signed token), and it becomes the most important attribute in every AuthZ check: the resource's tenant must equal the principal's tenant, unconditionally, even for an admin.

Q: Walk me through verifying a JWT. What are the footguns? A: Split into three parts; recompute HMAC over header.payload and compare with a constant-time function before trusting any claim; then decode and check exp against the current time. Footguns: using == instead of hmac.compare_digest (timing side channel), accepting alg: none or letting the token pick the algorithm (algorithm-confusion attacks), trusting claims before verifying the signature, putting secrets in the payload (it's signed, not encrypted), and using a wall clock for exp in a system that needs deterministic replay.

Q: Silo vs pool vs bridge — how do you choose? A: Silo is dedicated infra per tenant: strongest isolation and simplest compliance, highest cost and ops burden — good for a few large regulated customers. Pool is shared infra isolated in the data path: cheapest and most scalable, hardest to get right (one missing tenant filter is a leak). Bridge is hybrid — pool the cheap stateless tier, silo the sensitive data or the top customers. The senior answer is per-resource: I pool stateless compute and cache, silo the regulated store, and I bring the cost and blast-radius numbers.

Q: How does a token bucket give you noisy-neighbor isolation, and what are its invariants? A: Each tenant gets its own bucket of capacity tokens refilling at a fixed rate; requests spend tokens and are denied when the bucket is empty. Because buckets are per-tenant, one tenant draining its tokens can't touch another's — that's the isolation. Invariants: never go negative (deduct only when affordable, so denials cost nothing) and cap refill at capacity (idle time can't bank unlimited burst). It allows bounded bursts while capping the long-run rate.

Q: What goes in an audit log, and what must never? A: Who (user, tenant), what (action, resource kind), when (timestamp/seq), and the decision (allow/deny + reason) — append-only and immutable so it's tamper-evident, with correlation ids to reconstruct a request. What must never: secret content — values, keys, passwords, full request bodies. Redact secret-named fields before they're stored, or the audit log meant to keep you safe becomes the place you leak.

Q: A tenant-admin requests another tenant's resource. Allowed? A: No — never. "Admin" is scoped to one's own tenant; there is no cross-tenant authority in the model. The tenant check runs before the role check and denies the mismatch unconditionally. Cross-tenant admin is exactly the privilege escalation the boundary exists to stop.

18. References

  • OAuth 2.0 Authorization Framework — RFC 6749. https://datatracker.ietf.org/doc/html/rfc6749
  • OpenID Connect Core 1.0 (authentication layer over OAuth 2.0). https://openid.net/specs/openid-connect-core-1_0.html
  • JSON Web Token (JWT) — RFC 7519; JSON Web Signature (JWS) — RFC 7515. https://datatracker.ietf.org/doc/html/rfc7519
  • HMAC: Keyed-Hashing for Message Authentication — RFC 2104. https://datatracker.ietf.org/doc/html/rfc2104
  • NIST, Role-Based Access Control (RBAC) — the model. https://csrc.nist.gov/projects/role-based-access-control
  • NIST SP 800-162, Guide to Attribute Based Access Control (ABAC). https://csrc.nist.gov/pubs/sp/800/162/final
  • PostgreSQL Row Security Policies (Row-Level Security). https://www.postgresql.org/docs/current/ddl-rowsecurity.html
  • AWS SaaS Factory, SaaS tenant isolation strategies (silo / pool / bridge). https://docs.aws.amazon.com/whitepapers/latest/saas-architecture-fundamentals/tenant-isolation.html
  • OWASP API Security Top 10 — API1:2023 Broken Object-Level Authorization (BOLA/IDOR). https://owasp.org/API-Security/editions/2023/en/0xa1-broken-object-level-authorization/
  • OWASP Top 10 for LLM Applications — LLM06 Sensitive Information Disclosure. https://genai.owasp.org/
  • Pinecone namespaces (per-tenant vector isolation). https://docs.pinecone.io/guides/index-data/indexing-overview#namespaces
  • Weaviate multi-tenancy. https://weaviate.io/developers/weaviate/manage-data/multi-tenancy
  • "Token bucket" rate-limiting algorithm. https://en.wikipedia.org/wiki/Token_bucket
  • AICPA SOC 2 (trust service criteria). https://www.aicpa-cima.com/topic/audit-assurance/audit-and-assurance-greater-than-soc-2