« System Design · Track Overview

Design 04 — Authorized Retrieval at Scale

"Forty million documents across three business units. Some are behind information barriers. Every agent needs to search them. Design the knowledge foundation."

The question it turns on: can you make isolation structural rather than a filter?

Table of Contents

• 1. Constraints before components
• 2. Why "filter after retrieval" is the wrong shape
• 3. The topology decision
• 4. The retrieval path, and what each stage denies
• 5. Authorization: three orthogonal dimensions
• 6. Hybrid retrieval, and why RRF
• 7. The graph, and what it is actually for
• 8. Freshness, versions and the forgotten pin
• 9. The latency budget
• 10. Failure modes and blast radius
• 11. Evidence
• 12. What you build first
• 13. What changes at 10×
• 14. The questions you will be asked

1. Constraints before components

The load-bearing constraint is the last-but-one: prove, per record, that entitlement held. Not "we filter results" — demonstrate it for a specific document and a specific user on a specific day. That requirement is what forces the isolation to be structural.

2. Why "filter after retrieval" is the wrong shape

The intuitive design: one index, search it, drop what the user may not see.

   query ──► ANN search over ALL chunks ──► top 50 ──► filter by ACL ──► top 5

It fails in four distinct ways, and naming all four is the answer to this design question:

1. Recall collapses for restricted users. If 90% of the top 50 belongs to other tenants, a user with narrow entitlement gets 5 results where a broad user gets 50. The system is quietly worst for the users with the tightest permissions — usually the ones handling the most sensitive work.

2. The filter is a side channel. Result counts, latency, and "no results found" all leak information about documents the user cannot see. Ask "what do we know about Project Falcon" and get a slower empty answer than for a nonsense string, and you have learned Project Falcon exists.

3. It is one bug from a breach. A filter is a conditional. A refactor, a cache, a new code path that forgets it — and the index cheerfully returns whatever was nearest. Structural isolation has no such conditional to forget.

4. It cannot be proved. "We apply a filter" is an assertion about code. "This user's query ran against an index containing only documents they are entitled to" is a fact about data, and only the second one survives an audit.

The rule: isolation is a property of what you search, not of what you return.

3. The topology decision

Three options, and the answer is a hybrid — but you must be able to defend each.

The decision: index per tenant, partition per classification band within it, and barriers as a separate mechanism.

   ┌─── wholesale-index ─────────────────────────────────────┐
   │   partition: public+internal    (most queries)          │
   │   partition: confidential        (entitlement checked)  │
   │   partition: restricted          (entitlement checked)  │
   └──────────────────────────────────────────────────────────┘
   ┌─── retail-index ────────────────────────────────────────┐  … same shape
   ┌─── group-index ─────────────────────────────────────────┐  … same shape

   ┌─── barrier-vault ───────────────────────────────────────┐
   │   MNPI / deal-scoped documents, separate store,          │
   │   queried ONLY when the viewer holds the barrier         │
   └──────────────────────────────────────────────────────────┘

The query fans out only to partitions the viewer is cleared for. A user with internal clearance never touches the confidential partition — not "gets filtered out of it", never touches it. The side channel closes because the search space itself is different.

Why barriers get their own store. A barrier is not a clearance level; it is an orthogonal, named, time-bounded need-to-know. A Project Falcon memo classified confidential passes a confidential clearance check. Modelling it as a level is the MNPI leak that every individual check reports as working correctly. Separate store, explicit opt-in, and the query only reaches it when the viewer holds the specific barrier.

The honest cost: 3× index overhead and a fan-out query. Say it. The trade is isolation you can prove against storage you can buy.

4. The retrieval path, and what each stage denies

   query + Principal
        │
   ┌────▼─────────────────────────────────────────────────────┐
   │ 1  RESOLVE ENTITLEMENT   from the TOKEN, never the query  │
   │      tenant · clearance · desk · barriers held            │
   │      denies: a tenant claimed in the request body         │
   ├──────────────────────────────────────────────────────────┤
   │ 2  SELECT PARTITIONS     the search space IS the control  │
   │      denies: everything outside it — structurally         │
   ├──────────────────────────────────────────────────────────┤
   │ 3  RETRIEVE (parallel)                                    │
   │      BM25  ∥  dense  ∥  graph expansion                   │
   ├──────────────────────────────────────────────────────────┤
   │ 4  FUSE (RRF, k=60)      score-free rank fusion           │
   ├──────────────────────────────────────────────────────────┤
   │ 5  RERANK (cross-encoder, top 50 → 8)                     │
   │      first thing shed under load — it is NOT a control    │
   ├──────────────────────────────────────────────────────────┤
   │ 6  FINAL ENTITLEMENT CHECK   per document, per viewer     │
   │      denies: anything whose ACL changed mid-flight        │
   ├──────────────────────────────────────────────────────────┤
   │ 7  EMIT   doc ids, VERSIONS, snapshot, classification     │
   └──────────────────────────────────────────────────────────┘

Stage 1 reads entitlement from the verified token. The tenant comes from the token, never the request. Anything the caller can set, the caller can forge — and a tenant_id in a request body is the single most common finding in this design.

Stage 6 looks redundant and is not. The partition selection was made at query time; an ACL can change between then and now, and a document can be reclassified. The final check is cheap (you have ≤ 50 documents) and it closes the race. Defence in depth: stage 2 is structural, stage 6 is verifying, and they fail independently.

Stage 5 is explicitly not a control. Which is why it can be shed under load. Say that out loud — it is the degradation-ladder invariant applied to this subsystem.

5. Authorization: three orthogonal dimensions

The mistake is collapsing them into one "permission" field.

Evaluated in that order, and the order matters:

if document.barrier and document.barrier not in viewer.barriers:   # 1
    remove("behind deal:PROJECT-FALCON")
elif document.desk and document.desk != viewer.desk \
        and document.classification == "restricted":               # 2
    remove("desk-scoped to advisory")
elif rank(document.classification) > rank(viewer.clearance):       # 3
    remove("exceeds the viewer's clearance")

Barrier before classification. The memo is confidential; the viewer holds confidential. Check 3 first and it passes. This ordering bug is invisible in a unit test of either check.

Removals produce reasons, not silence. "falcon-memo is behind deal:PROJECT-FALCON" goes into the evidence pack. "Access denied" starts an investigation; that sentence ends one. And a removal is a control acting, not a request failing — the agent answers from what it may see.

Barriers are time-bounded and audited. A deal closes; the barrier lifts, or converts to an archive restriction. Grants have an expiry and a granting authority, and the list of who held which barrier when is itself regulated evidence.

6. Hybrid retrieval, and why RRF

Financial text defeats pure dense retrieval in a specific way: the queries are full of exact tokens that embeddings smooth over. PMT-771, LEI 5493001KJTIIGC8Y1R12, pain.001.001.09, IFRS 9. A dense retriever returns documents about similar payments; BM25 returns the one with that identifier.

$$\text{BM25}(D,Q)=\sum_{q\in Q}\text{IDF}(q)\cdot\frac{f(q,D)(k_1+1)}{f(q,D)+k_1(1-b+b\frac{|D|}{\text{avgdl}})}$$

with \( k_1 \in [1.2,2.0] \) and \( b=0.75 \). Term frequency saturates; long documents are penalized.

Fuse with Reciprocal Rank Fusion, \( k=60 \):

$$\text{RRF}(d)=\sum_i \frac{1}{k+\text{rank}_i(d)}$$

Why RRF rather than a weighted score blend: BM25 scores and cosine similarities live on incomparable scales that shift with corpus and query. Any weighted blend needs calibration, and the calibration drifts. RRF uses only ranks, so there is nothing to calibrate — which is why it survives contact with a corpus that changes daily.

Worth knowing the shape: a document ranked 1st by one retriever and 10th by the other scores \( 1/61 + 1/70 = 0.0307 \); one ranked 3rd by both scores \( 2/63 = 0.0317 \). Consistent mid-rank beats a single strong opinion — usually right, and occasionally the thing you need to tune around.

Chunking is the other half of retrieval quality and gets less attention than it deserves: structure-aware splits (clause, section, table) rather than fixed windows; ~15% overlap; and the parent document's title and section path prepended to every chunk, because a chunk that reads "the limit is 5 million" is useless without knowing which limit.

7. The graph, and what it is actually for

The vector store answers "what text is relevant?" The graph answers "what is connected?" Those are different questions, and a bank asks the second one constantly.

Modelled on FIBO (Financial Industry Business Ontology) in RDF, with SHACL for validation and SPARQL for query:

   Zenith Supplies FZE ──isSubsidiaryOf──► Zenith Holdings Ltd
                       ──hasAccountAt───► Bank
   Zenith Holdings Ltd ──controlledBy───► [beneficial owner]
                       ──isSubjectTo────► [sanctions designation]

The query the vector store cannot answer: "is this counterparty connected, through any ownership path of length ≤ 4, to a sanctioned entity?" That is graph traversal, not similarity.

Three uses in this design:

1. Grounding expansion — retrieved chunk mentions Zenith Supplies; the graph supplies the ownership chain, and that goes into the context as structured fact rather than retrieved prose.
2. Entity disambiguation — three counterparties named "Zenith"; the graph resolves which one the LEI refers to.
3. Validation — SHACL shapes assert that every counterparty has an LEI, a jurisdiction and a screening date. A violation is a data-quality ticket, not a runtime surprise.

Graph results carry entitlement too. An ownership edge can itself be MNPI. Same three dimensions, applied to triples.

8. Freshness, versions and the forgotten pin

Three different freshness requirements need three pipelines, and pretending otherwise produces either stale policy documents or an absurd bill:

The third row is the design decision worth defending: do not index what changes by the second. Positions and prices are tool calls, not retrieval. An indexed price is a wrong price with a citation, which is worse than no price.

Every chunk carries a version, and the retrieval artifact records doc_id@version — because "the agent read the policy" is unfalsifiable and "the agent read aml-policy@v7" is checkable.

And the retrieval snapshot — the forgotten sixth pin. Pin the model, the prompt, the policy, the tool set and the guardrails; re-run six months later against a re-indexed corpus; get a different answer with five matching pins and no explanation. The snapshot id closes it:

"retrieval_snapshot": "idx-2026-03-11T06:00Z"

9. The latency budget

350 ms p95, from Design 01.

Two consequences:

The rerank is 40% of the budget. Which is exactly why it is rung 1 on the degradation ladder, and why shedding it is invisible to users in a way that shedding retrieval is not.

The graph runs in parallel or not at all. A sequential graph expansion after retrieval blows the budget. Fire it concurrently on the entities in the query; if it does not return in time, proceed without it and flag reduced grounding.

10. Failure modes and blast radius

The embedding-model change is the expensive one and belongs in the answer unprompted: 400M chunks re-embedded is a real cost and a real elapsed time. Dual-write into both vector spaces, query the old, backfill the new, cut over per tenant, then retire. Anyone who has run this once will recognize that you have.

11. Evidence

{ "trace_id": "...", "viewer": "layla.almansouri", "tenant": "wholesale",
  "clearance": "confidential", "desk": "payments", "barriers_held": [],
  "partitions_searched": ["wholesale/public+internal", "wholesale/confidential"],
  "retrieval_snapshot": "idx-2026-03-11T06:00Z",
  "returned": [ {"doc_id": "case-note-991", "version": "v3", "classification": "confidential",
                 "rank": 1, "retriever": "rrf(bm25:2,dense:1)"} ],
  "removed": [ {"doc_id": "falcon-memo", "reason": "behind deal:PROJECT-FALCON"} ],
  "graph_expansion": {"entities": ["zenith-supplies-fze"], "edges_returned": 4} }

partitions_searched is the field that makes the design provable. It is not "we filtered" — it is the search space itself, recorded. The auditor's question "prove this user never had access to Project Falcon material" becomes a query over these records rather than a code review.

removed with reasons is the second one. It shows controls acting, which is what defence depth counts.

12. What you build first

1. Entitlement resolution from the token, and the per-tenant index split. Structural isolation first — retrofitting it into a shared index is a data-migration project, not a refactor.
2. BM25 + dense + RRF. Hybrid from the start; financial identifiers make dense-only retrieval visibly bad on day one.
3. Chunking with structure and parent context. Cheap, and it dominates quality.
4. The retrieval artifact, with versions and the snapshot. Before the first audit conversation.
5. Classification partitions. Once you have real classified data and know the distribution.
6. The reranker. Quality, not correctness; it can wait, and it is the first thing shed anyway.
7. The barrier vault. When the first MNPI use case is real — and it will be, in Wholesale.
8. The graph. Last: the most work, the narrowest query class, and the easiest to defer honestly.

13. What changes at 10×

400M documents, 4B chunks, six tenants.

Sharding within a tenant. One index per tenant stops fitting. Shard by time or entity, and now recall depends on shard routing — a query that must search all shards costs a fan-out, and a query routed to the wrong shard silently loses recall. Measure per-shard recall, not just global.

The ANN index becomes a memory problem. HNSW graphs are RAM-resident; at 4B vectors that is a capacity plan, not a config value. This is where quantization (PQ, or int8) enters, and it costs recall — measure how much on your graded set, not the vendor's.

Re-embedding becomes a standing programme. At 4B chunks you cannot re-embed on a whim, which means the embedding model becomes a pinned dependency with a deprecation runbook.

Freshness and cost collide. Incremental indexing at 2M documents/day is a streaming pipeline with its own SLO. The staleness SLI stops being a nice-to-have and becomes the thing users complain about.

Entitlement resolution becomes a hot path. Cache per session; and when barriers change, the cache must be invalidated per user — which is why grants are events, not table rows.

14. The questions you will be asked

"Why not one index with metadata filtering?" — Four reasons: recall collapses for narrowly entitled users, result counts and latency form a side channel, a filter is one refactor from a breach, and it cannot be proved to an auditor. Isolation is a property of what you search, not of what you return.

"Isn't per-tenant indexing expensive?" — Roughly 3× storage overhead. That is the price of provable isolation, and it is smaller than one incident. The place I would share is the embedding model and the pipeline, not the index.

"How do you know retrieval is good?" — A graded set per tenant, recall@20 and nDCG@10 tracked as a release gate, plus grounding rate in production (what fraction of answers cite a retrieved document). Offline numbers alone are how a retriever quietly degrades.

"What about prompt injection in retrieved documents?" — Retrieval is the delivery mechanism for indirect injection. Scanned before entering the prompt, survivors marked as tainted, and any side-effecting action derived from them requires human approval. Containment, not prevention — Phase 11.

"A user says the agent quoted a document they shouldn't have seen." — Then I pull the retrieval artifact for that trace: viewer, clearance, barriers held, partitions searched, documents returned with versions, and documents removed with reasons. Either the entitlement was wrong, in which case the record shows exactly which dimension, or it was correct and the concern is about the document's classification — a different problem with a different owner.