« Phase 06 · Track Overview

Warmup — The Knowledge Foundation, From Zero

Assumes Python and the platform framing from Phase 00. Assumes nothing about retrieval, embeddings, BM25, vector databases, or why any of this is a security topic.

Table of Contents

• 1. What retrieval is for, and what it is not
• 2. Chunking
  • 2.1 Why the unit of retrieval is not the document
  • 2.2 Fixed-size chunking and what it destroys
  • 2.3 Structure-aware chunking
  • 2.4 Overlap
  • 2.5 Provenance, and why every chunk must be citable
• 3. Embeddings
  • 3.1 What an embedding actually is
  • 3.2 Cosine similarity, derived
  • 3.3 Feature hashing, and why the signs matter
  • 3.4 Embedding strategy and the re-embedding migration
• 4. Lexical retrieval: BM25
  • 4.1 From counting words to TF-IDF
  • 4.2 The two problems TF-IDF has
  • 4.3 BM25, term by term
  • 4.4 Why IDF must be clamped
• 5. Hybrid retrieval
  • 5.1 The two retrievers fail on opposite inputs
  • 5.2 Score fusion is a trap
  • 5.3 Reciprocal Rank Fusion
• 6. Reranking
  • 6.1 Bi-encoders and cross-encoders
  • 6.2 The relevance floor
• 7. Vector store topology
  • 7.1 ANN: HNSW and IVF-PQ
  • 7.2 The filtering cliff
  • 7.3 Silo, pool, bridge
• 8. Authorized retrieval
  • 8.1 The failure with no detection
  • 8.2 Two mechanisms, deliberately
  • 8.3 Information barriers
  • 8.4 Freshness as a contract
• 9. Grounding
• 10. Context engineering
• 11. Lab walkthrough
• 12. Success criteria
• 13. Common mistakes
• 14. Interview Q&A
• 15. References

1. What retrieval is for, and what it is not

A language model knows what was in its training data, approximately, without attribution, and as of some date. A bank needs answers that are current, specific to this institution, and attributable to a source. Retrieval is how you get all three: find the relevant text, put it in the prompt, and generate an answer grounded in it.

What retrieval is not:

• It is not authorization. This is the single most important sentence in the phase. A vector index returns the nearest chunk; "nearest" has no opinion about who owns it. Retrieval that is merely relevant is a data-leak mechanism with excellent recall.
• It is not a fix for a model that reasons badly. If the right passage is in the context and the answer is still wrong, the problem is downstream.
• It is not a memory system. Retrieval finds documents; memory (Phase 01) records what happened. Conflating them produces a store that leaks across sessions.

The quality bound worth internalizing before anything else: recall@k caps everything downstream. If the answer is not in the retrieved set, no prompt, no reranker and no larger model recovers it. Measure recall on a golden set before tuning the generator, or you will spend weeks optimizing a component that is not the constraint.

2. Chunking

2.1 Why the unit of retrieval is not the document

You could embed whole documents. Two reasons not to:

1. A 40-page policy has one embedding, which is the average of forty pages of meaning — close to nothing in particular. Retrieval accuracy collapses.
2. Context is finite and expensive. Injecting 40 pages to answer one question wastes tokens and buries the relevant sentence among thousands of irrelevant ones, which measurably degrades the answer.

So documents are split into chunks: units small enough to be specific, large enough to be self-contained. And how you split is the highest-leverage quality decision in the whole pipeline — and consistently the most under-invested.

2.2 Fixed-size chunking and what it destroys

The default implementation everyone writes first: split every 512 tokens.

...  Releases above AED 100,000 require dual control by two authorised  |CHUNK BOUNDARY|
     officers.  ## Escalation  If a possible match scores above 0.80 ...

The chunk containing "Releases above AED 100,000 require dual control by two authorised" is retrievable and useless. It will match a query about dual control, be returned with high confidence, and support an answer that is missing its predicate.

Worse, and more common in a bank: a boundary through the middle of a table. Half the rows, no header. The model reads column values with no idea what column they are in.

The general failure: fixed-size chunking optimizes a number nobody cares about (uniform chunk size) at the cost of the property that matters (semantic completeness).

2.3 Structure-aware chunking

Documents already tell you where their boundaries are: headings, sections, list items, table rows, clause numbers. Split on structure first, then apply a size limit within each structural unit.

sections = split_on_headings(text)          # structure first
for name, body in sections:
    for piece in split_by_size(body, max_tokens):   # size second, within a section
        yield Chunk(text=piece, section=name, ...)

Two immediate benefits:

• A chunk rarely spans a semantic boundary, so it is self-contained.
• Every chunk carries its section name, which is free context for the model and a component of the citation for a human.

The lab implements exactly this, and its _split_by_size splits on word boundaries — never mid-word — because a chunk ending in "authoris" helps nobody.

For a bank the structural units worth handling explicitly are: headings, numbered clauses, tables (keep the header with every row group), and definition lists. Each one is a small amount of code and a measurable recall improvement.

2.4 Overlap

Even with structure-aware splitting, a fact can span a boundary. Overlap re-includes the last n tokens of the previous chunk at the start of the next.

The trade is explicit:

A rule that generalizes: overlap of 10–20% of chunk size. And an invariant the lab enforces: overlap < max_tokens, or chunks stop advancing and you generate infinitely.

2.5 Provenance, and why every chunk must be citable

Every chunk in the lab carries doc_id, doc_version, section, and a character span, rendered as:

pol-hold-release@v1#Hold release[16:192]

That string is the difference between "the system said so" and "here is the clause, in version 1 of this policy, at these characters." In a regulated platform the second is the only acceptable answer, and it is impossible to reconstruct later — the span must be recorded at ingestion.

doc_version matters more than people expect: a policy changes, the answer changes, and an auditor asks which version applied on 12 March. Without the version on the chunk, that question has no answer.

3. Embeddings

3.1 What an embedding actually is

An embedding is a function from text to a vector of numbers, trained so that semantically similar text produces geometrically close vectors.

That is the entire idea. "The payment is on hold" and "this transfer has been stopped" share almost no words, and a good embedding places them close together. That is what lexical matching cannot do and why dense retrieval exists.

The vector's dimensionality (384, 768, 1536, 3072 are common) is a capacity/cost trade: more dimensions capture more distinctions and cost more to store and compare.

3.2 Cosine similarity, derived

To compare two vectors you want a measure of direction, not magnitude — a long document and a short one about the same topic should score as similar.

The cosine of the angle between vectors \( a \) and \( b \):

$$\cos\theta = \frac{a \cdot b}{|a|,|b|} = \frac{\sum_i a_i b_i}{\sqrt{\sum_i a_i^2}\sqrt{\sum_i b_i^2}}$$

Range: −1 (opposite) to 1 (identical direction). If you L2-normalize every vector at ingestion — divide by its own magnitude so \( |a| = 1 \) — the denominator becomes 1 and cosine similarity is just the dot product:

$$\cos\theta = \sum_i a_i b_i$$

That is why the lab normalizes in hash_embed and cosine is a one-liner. It is faster and it removes a whole class of bug where someone forgets to divide.

3.3 Feature hashing, and why the signs matter

The lab needs a deterministic embedder with no model. It uses the hashing trick: hash each token to a dimension index and accumulate.

index = hash(token) % dimensions
vector[index] += 1

This works, and it has a flaw that is worth understanding because it generalizes. Two different tokens can hash to the same index — a collision. With += 1, collisions always add constructively, so unrelated documents accumulate spurious shared mass and everything looks more similar than it is. Similarity scores drift upward and never go negative.

The fix is a random sign per token, taken from another part of the same hash:

sign = +1 if digest[4] % 2 == 0 else -1
vector[index] += sign

Now a collision between two different tokens adds \( +1 \) and \( -1 \) with equal probability, so collisions cancel in expectation and the dot product remains an unbiased estimator of the true sparse dot product. The lab tests this directly: across 40 unrelated pairs, at least one similarity is ≤ 0 — which is impossible without signs.

This is not just a lab detail. It is the same reasoning behind count-sketch data structures, and it is the kind of thing that separates "I used an embedding library" from "I know why it works."

3.4 Embedding strategy and the re-embedding migration

Choosing an embedding model involves four questions:

1. Quality on your corpus. Benchmarks (MTEB and similar) are a starting point and not an answer — financial documents, with their identifiers and boilerplate, behave differently from the web text most benchmarks use. Measure recall@k on your own golden set.
2. Dimensionality. Directly sets storage and query cost. Some models support truncation (Matryoshka-style) so you can trade quality for cost without changing model.
3. Multilingual coverage. In the UAE this is not optional: Arabic and English in the same corpus, sometimes the same document.
4. Where it runs. A managed embedding API sends your text to a third party — which is a residency and classification question (Phase 15), not just a cost one.

And then the part nobody plans for. Changing the embedding model invalidates every vector in your index. Old and new vectors are not comparable; there is no conversion. Re-embedding a corpus of millions of chunks is a migration, and it must run without downtime:

1. Dual-write — new ingestion writes to both the old and the new index.
2. Backfill — re-embed the existing corpus into the new index, throttled to protect the embedding endpoint's rate limit.
3. Shadow-read — serve from the old index, query both, and log the difference in recall on a golden set.
4. Cut over — when the new index is at least as good, switch reads.
5. Retain the old index for a rollback window, then delete.

Budget it as weeks, not hours, and note that step 2 dominates the cost. Plan this before you need it, because the day you need it is the day a better model has shipped and everyone wants it immediately.

4. Lexical retrieval: BM25

4.1 From counting words to TF-IDF

Start naive: rank documents by how many query terms they contain. Two problems appear at once.

Problem 1 — common words dominate. A query for "the payment hold" matches every document containing "the". Fix: weight each term by how rare it is. Inverse document frequency:

$$\text{IDF}(t) = \log\frac{N}{\text{df}(t)}$$

where N is the number of documents and df(t) how many contain t. A term in every document has IDF 0; a term in one document has high IDF.

Problem 2 — long documents win. A 10 000-word document contains more of everything. Fix: also count how often the term appears, relative to the document — term frequency.

Multiply them and you have TF-IDF, the foundation of lexical search for decades.

4.2 The two problems TF-IDF has

Term frequency grows without limit. A document mentioning "payment" 100 times is not 50× more relevant than one mentioning it twice. Relevance saturates, and raw TF does not.

Length normalization is all-or-nothing. Dividing by document length over-corrects: a genuinely comprehensive long document is penalized for being thorough.

4.3 BM25, term by term

BM25 fixes both with two tunable parameters:

$$\text{BM25}(D,Q)=\sum_{t\in Q}\text{IDF}(t)\cdot\frac{f(t,D),(k_1+1)}{f(t,D)+k_1\left(1-b+b\frac{|D|}{\text{avgdl}}\right)}$$

Read it in pieces:

• \( f(t,D) \) — occurrences of term t in document D.
• \( \frac{f(k_1+1)}{f+k_1} \) — the saturation curve. As \( f \to \infty \) this approaches \( k_1+1 \), so the score is bounded. At \( k_1 = 1.5 \): f=1 scores 1.0, f=2 scores 1.43, f=9 scores 2.14, f=10 scores 2.17. The step from 9 to 10 is 1/14th of the step from 1 to 2. That is saturation, and the lab tests exactly this comparison.
• \( k_1 \) — how fast it saturates. Higher means slower saturation (closer to raw TF). Typical range 1.2–2.0.
• \( \left(1-b+b\frac{|D|}{\text{avgdl}}\right) \) — the length normalization, mixed with parameter b. At b=0 length is ignored entirely; at b=1 it is fully applied; 0.75 is the standard compromise.

The lab implements this directly and tests both behaviours in isolation — saturation with b=0, length normalization by comparing b=1 against b=0 — because with both active they are not separable.

4.4 Why IDF must be clamped

The IDF form BM25 actually uses adds smoothing:

$$\text{IDF}(t)=\log\left(\frac{N-\text{df}+0.5}{\text{df}+0.5}+1\right)$$

The +0.5 terms prevent division by zero and dampen extremes. The +1 inside the log and the outer max(0, …) in the lab both exist for the same reason: without them, a term appearing in more than half the documents gets a negative IDF, and a document containing it scores worse than one that does not.

That is not a rounding artefact; it actively inverts ranking for common domain terms. In a bank corpus, "payment" or "account" may genuinely appear in most documents. The lab has a test asserting no score goes negative, because this is a real bug that ships.

5. Hybrid retrieval

5.1 The two retrievers fail on opposite inputs

They are not two attempts at the same thing. They fail on complementary inputs, which is why combining them beats either — and why "hybrid" is not just "more retrievers is better."

For a bank this is decisive. Payment references, LEIs, IBANs, account numbers, product codes and deal names are exactly what a user asks about and exactly what embeddings blur.

5.2 Score fusion is a trap

The obvious combination is a weighted sum: 0.5 * bm25 + 0.5 * cosine.

It does not work, for a reason worth stating precisely: the two scores live on incomparable scales. BM25 is unbounded and corpus-dependent (the lab's example produces 3.99 and 1.63); cosine is bounded in [−1, 1] (0.249 and 0.149). Normalizing them requires estimating each distribution, and those estimates are corpus-specific, query-specific, and invalidated the moment you change the embedding model or the corpus grows.

Teams that do this end up with a magic weight that someone tuned once, that nobody can justify, and that silently degrades.

5.3 Reciprocal Rank Fusion

RRF avoids the problem entirely by discarding the scores and using only the positions:

$$\text{RRF}(d)=\sum_{i}\frac{1}{k+\text{rank}_i(d)}$$

with \( k = 60 \) conventionally. Rank 1 contributes \( 1/61 \), rank 2 contributes \( 1/62 \), and so on.

Three properties:

No calibration. Positions are comparable across any retrievers; scores are not. Change the embedding model and RRF still works.

Consistent agreement beats one strong signal. A document ranked 1st by one retriever and 10th by the other scores \( 1/61 + 1/70 = 0.03068 \). A document ranked 3rd by both scores \( 2/63 = 0.03175 \) — higher. That is the behaviour you want from hybrid retrieval: two independent methods agreeing is stronger evidence than one being enthusiastic.

k controls flatness. Large k makes rank differences matter less (all contributions approach \( 1/k \)); small k makes the top rank dominate. 60 is empirical and rarely worth tuning.

6. Reranking

6.1 Bi-encoders and cross-encoders

The dense retriever is a bi-encoder: it embeds the query and the document separately and compares vectors. That separation is what makes it fast — every document embedding is precomputed — and it is also its limitation: the document's embedding was produced without any knowledge of the query.

A cross-encoder feeds the query and the document together into a model that outputs a relevance score. It can attend to their interaction, so it is far more accurate. It also cannot precompute anything, so it must run once per (query, document) pair.

That cost difference is the architecture:

retrieve  →  top 50 candidates   (bi-encoder + BM25, fast, over the whole index)
rerank    →  top 5              (cross-encoder, slow, over 50 pairs only)

Never over the index. And because it is the most expensive stage per unit of quality, it is the first thing shed under latency pressure — the Phase 00 degradation ladder, made concrete. The lab makes this explicit: disabling rerank records "rerank" in degraded and still returns an answer, which is what makes retrieval a degradable rather than serial dependency.

6.2 The relevance floor

A subtle and important point. First-stage retrieval always returns something. An ANN index has a nearest neighbour for any query, including one about a topic entirely absent from the corpus. BM25 returns nothing when no term matches, but dense retrieval does not have that property.

So without a floor, "there is nothing relevant here" is indistinguishable from "here are the least-irrelevant chunks I have." The consequences compound:

• the generator is handed noise and asked to answer from it;
• the grounding check is asked to support claims against irrelevant text;
• and the honest answer — "I don't have information about that" — becomes unreachable.

The lab adds min_score to rerank. In production the floor is tuned on a labelled set with known-absent queries, and "no relevant documents" is a first-class outcome that the agent must be able to express.

7. Vector store topology

7.1 ANN: HNSW and IVF-PQ

Exact nearest-neighbour search compares the query against every vector — \( O(N) \), fine at 10 000 chunks and hopeless at 10 million. Production uses approximate nearest neighbour, trading a little recall for orders of magnitude in speed.

HNSW (hierarchical navigable small world) builds a multi-layer graph. Upper layers are sparse and enable long jumps; lower layers are dense and enable fine navigation. Search greedily descends. Fast and high-recall; memory-hungry (the graph is large) and slow to build.

IVF-PQ (inverted file with product quantization) clusters vectors and searches only the nearest few clusters, with vectors compressed into quantized codes. Memory-efficient; lower recall, and recall depends on how many clusters you probe.

The practical decision: HNSW when recall matters and memory is available (most enterprise RAG), IVF-PQ at very large scale or when memory-bound. The lab does exact search, which is correct at its scale and is why the filtering cliff below is not observable in it.

7.2 The filtering cliff

Here is the phenomenon that makes topology a design decision rather than a deployment detail.

You have one index with 10 million chunks across 12 tenants. A query from tenant A:

1. ANN search returns the 100 nearest neighbours across the whole index;
2. you filter to tenant A;
3. roughly 8 survive — and if tenant A is small, zero.

Recall has collapsed, and it collapses harder the more selective the filter is. Adding classification and barrier filters on top makes it worse. The user sees an empty or terrible result set, and the cause is invisible from the application's point of view.

Three fixes:

Note that the correct fix for tenancy — a namespace per tenant — is the same thing that makes isolation structural. The performance argument and the security argument point the same way, which is a rare and useful alignment to have in your pocket during a design review.

7.3 Silo, pool, bridge

The SaaS isolation vocabulary, applied to vector stores:

For a bank the honest default is bridge: pool where the data is genuinely shared (public policy, product documentation), silo where a barrier or a classification demands it. And the decision is per-corpus, not per-platform.

The lab implements namespaces, which is the mechanism underneath all three: silo is one namespace per tenant, pool is one namespace for everyone, bridge is a mixture.

8. Authorized retrieval

8.1 The failure with no detection

Worth stating on its own, because it is the reason this phase is in a bank track at all:

A shared index with a post-hoc entitlement filter returns tenant B's chunk to tenant A when the filter is missing, wrong, or bypassed. The result is a 200 OK with a plausible answer. No exception, no error rate, no alert. The user is satisfied. You find out months later, from someone other than your monitoring.

Compare this to every other failure in the platform — a provider outage, a policy denial, a schema violation — all of which announce themselves as an error rate. This one does not.

Controls that fail silently and severely deserve prevention, not detection. That is the argument for making isolation structural: a namespace is not a rule that can be forgotten; it is a different place to look.

8.2 Two mechanisms, deliberately

The lab enforces tenant isolation and content entitlement by two different mechanisms, and the asymmetry is intentional:

Blast radius drives the choice. A tenant-filter bug crosses a customer boundary; a classification bug does not leave the tenant. So the tenant boundary gets the stronger, structural mechanism, and the finer-grained checks get the filter — applied before ranking, so an ineligible chunk can neither occupy a result slot nor leak its existence through result-set size or timing.

"Filter before you search" is the slogan. visible() in the lab is a conjunction — tenant AND classification AND barrier — and the test asserts that failing any one of the three hides the chunk.

8.3 Information barriers

The bank-specific control, and the one that has no equivalent in a general RAG system.

An information barrier (historically a "Chinese wall") is an enforced separation between businesses that must not share information — classically advisory and trading, because advisory holds MNPI (material non-public information) about deals.

The compliance requirement is old and well understood. What is new is that an agent with retrieval across the corpus creates a barrier crossing silently: no human read the document, no access was logged as unusual, and the information reaches a trader through a summary.

The control is that the barrier is a retrieval constraint, not a policy document: a chunk tagged barrier="advisory" is invisible to any principal not inside that barrier — and invisible means not retrieved, not filtered from the answer. The lab implements exactly this, and Phase 11 adds MNPI detection on the output side as the second, independent gate.

8.4 Freshness as a contract

A retrieved chunk carries a timestamp. Whether it may be relied on is a policy question, and it should be explicit.

The failure without a contract: an agent answers with a policy that was superseded last month, cites it correctly, and is confidently wrong in a way that looks authoritative.

The lab's RetrievalPolicy.max_stale_ticks drops chunks older than the contract allows and counts the exclusions, so "we had no fresh information" is distinguishable from "we had no information". That distinction matters to the agent (it should say so) and to the operator (a rising freshness-exclusion rate means an ingestion pipeline has stalled — one of the few useful leading indicators retrieval produces).

9. Grounding

Grounding is the property that every claim in the answer is supported by a retrieved span, and that you can point at which one.

Two reasons it is not optional in a bank:

1. Evidence. "The system said so" is not an answer to a customer complaint or an audit query. "Clause 4.2 of the sanctions hold-release policy, version 3, says so" is.
2. Detection. A model asked to answer from context sometimes answers from its priors instead — fluently, plausibly, and wrongly. Checking claims against the retrieved set catches it.

The lab's check_grounding maps each claim to its best-matching retrieved chunk and requires an overlap threshold. That threshold is a real trade-off worth naming:

Tune it against a labelled set of (claim, evidence) pairs where you know the answer, not by feel. In production the check is an entailment model rather than token overlap, and the same threshold problem exists with the same solution.

Two design points from the lab worth carrying:

• A claim with only stop-words is trivially supported. "It is the case that…" carries no content; failing it would be noise.
• No evidence means nothing is supported. An empty retrieved set with a confident answer is precisely the case to catch.

10. Context engineering

The last step: assemble instructions, tool schemas, policy, memory, retrieved chunks and the user's question into one prompt within a token budget.

Two rules, both easy to get backwards.

Order for prefix caching. Provider-side prompt caching (Phase 04) discounts tokens shared with a recent request's prefix, and it is invalidated by the first changed byte. So:

instructions  →  tool schemas  →  policy  →  memory  →  retrieved  →  question
└──────── stable across requests ────────┘   └──── varies per turn ────┘

Putting a timestamp or a session id at the top costs you the entire discount, silently. The lab reports cacheable_fraction so the number is visible rather than assumed.

The question is never dropped. When the budget is tight, drop the lowest-ranked retrieved chunks, and report which ones went. An assembler that truncates the user's question to fit more context has inverted its purpose; one that silently drops evidence makes the grounding check and the audit record lie.

A third, subtle one the lab enforces: measure the assembled text, not the sum of its pieces. Segment headers and the separators between blocks are real tokens. A budget computed from the parts over-fills by exactly the amount nobody accounts for — which the lab discovered by testing it.

11. Lab walkthrough

Work Lab 01 in this order.

1. classification_rank, Document, Chunk.citation (§2.5). Validate the classification at ingestion — a typo must fail when the document arrives, not when a query needs it.
2. estimate_tokens, _split_sections, _split_by_size, chunk_document (§2). The overlap < max_tokens guard prevents an infinite loop; the word-boundary rule prevents half-words.
3. hash_embed, cosine (§3.3). The sign is load-bearing — the "non-positive similarity" test fails without it.
4. BM25Index (§4). Write _idf with the clamp, then search. Run the saturation and length-normalization tests before the rest.
5. VectorIndex (§7.3). Namespaced, sorted namespaces, deterministic tie-break.
6. reciprocal_rank_fusion (§5.3). Positions only; sort by (-score, chunk_id).
7. lexical_overlap_reranker, rerank (§6). Do not forget the min_score floor.
8. AuthorizedRetriever (§8). The order in retrieve is the lesson: search the namespace → count considered → pre-filter → fuse → rerank. Filtering after fusion passes some tests and fails the point.
9. check_grounding (§9). Stop-words removed; best match wins; threshold validated.
10. assemble_context (§10). Measure the assembled text; drop lowest-ranked first; never drop the question.

Then python solution.py and read the nine sections against §§2–10.

12. Success criteria

Without the guide open:

• Explain why the unit of retrieval is a chunk, and what fixed-size chunking destroys.
• Justify structure-before-size and name four structural units worth handling in a bank.
• Derive cosine similarity and explain why normalization turns it into a dot product.
• Explain feature hashing and why the random sign is necessary.
• Describe the re-embedding migration in five steps.
• Write BM25 and say what k1 and b each control.
• Explain why IDF must be clamped, with the failure it prevents.
• Give two queries where BM25 wins and two where dense wins.
• Explain why score fusion is a trap and RRF is not.
• Compute an RRF score and show that agreement beats a single strong signal.
• Explain bi-encoder vs cross-encoder, and why rerank runs over the top-k only.
• Explain the relevance floor and what becomes unreachable without it.
• Describe the filtering cliff and three fixes.
• Explain why tenant and classification use different mechanisms.
• Explain what an information barrier is and why an agent crosses one silently.
• State the two rules of context assembly.

13. Common mistakes

Fixed-size chunking. Retrievable, unusable chunks; broken tables.

Overlap ≥ chunk size. Infinite generation.

No provenance on chunks. No citations, and no answer to "which version applied?"

Unsigned feature hashing. Everything looks similar; nothing is ever dissimilar.

Forgetting that changing the embedding model invalidates the index. It is a migration.

Unclamped IDF. Common domain terms actively invert ranking.

Weighted score fusion. A magic constant nobody can justify, silently degrading.

Reranking the whole index. A cross-encoder cannot precompute; you have built a very expensive retriever.

No relevance floor. "Nothing relevant" becomes unreachable and the generator answers from noise.

A shared index with a post-hoc filter. The failure with no detection.

Filtering after ranking. Leaks existence through result-set size; one refactor from leaking content.

Treating an information barrier as a policy document. It is a retrieval constraint.

No freshness contract. Confidently wrong, correctly cited.

Volatile content at the top of the prompt. Silently destroys the prefix-cache discount.

Budgeting the pieces instead of the assembled text. Over-fills every time.

Dropping retrieved chunks silently. The grounding check and the audit record both lie.

14. Interview Q&A

Q: Design the knowledge foundation for a bank's agent platform.

A: "I'd start from the constraint that makes it different from ordinary RAG: retrieval has to be authorized, not merely relevant, because a vector index returns the nearest chunk and 'nearest' has no opinion about who owns it. So tenant isolation is the index namespace — a cross-tenant chunk is never a candidate, not filtered out — and classification and information-barrier checks are a pre-filter inside the namespace, applied before ranking. Two different mechanisms on purpose: blast radius. A tenant-filter bug crosses a customer boundary; a classification bug doesn't leave the tenant, so the tenant boundary gets the structural mechanism. On the quality side: structure- aware chunking with 10–20% overlap and provenance on every chunk including the document version; hybrid BM25 plus dense, because payment references and LEIs are exactly what embeddings blur and lexical matching nails; RRF to merge, score-free so I never have to re-calibrate when the embedding model changes; and a cross-encoder rerank over the top-50 only, with a relevance floor so 'nothing relevant' is expressible. Then grounding — every claim maps to a retrieved span or the answer fails — and a freshness contract, because a correctly-cited superseded policy is the worst kind of wrong."

Q: Why is a shared vector index with a tenant filter dangerous?

A: "Because it's the only failure in the platform with no runtime detection. Every other failure — a provider outage, a policy denial, a schema violation — shows up as an error rate. This one produces a 200 OK with a plausible answer: the user is satisfied, nothing alerts, and you find out months later from someone other than your monitoring. There's also a performance version of the same problem, the filtering cliff: ANN search returns the 100 nearest neighbours across the whole index, you filter to one tenant, and eight survive — or zero if that tenant is small. So the security argument and the performance argument point at the same fix, which is a namespace per tenant. That's a nice position to be in during a design review. And the general principle: a control that fails silently and severely deserves prevention, not detection — a namespace isn't a rule someone can forget, it's a different place to look."

Q: Why hybrid retrieval? Isn't a good embedding model enough?

A: "No, and not because more retrievers is better — because they fail on complementary inputs. Ask for PMT-771 or an LEI and BM25 nails it while dense retrieval is near-useless, since an identifier has no semantics to embed. Ask 'why would a transfer be stopped' and dense wins, because the corpus says 'hold' and 'blocked' and never says 'stopped'. In a bank the identifier case is most of what users actually ask about. For merging I'd use RRF rather than a weighted score sum: BM25 is unbounded and corpus-dependent, cosine is bounded in [−1,1], and normalizing them requires distribution estimates that are invalidated the moment you change the embedding model. RRF reads positions only, so it needs no calibration — and it has a property you want, which is that consistent agreement beats one strong signal. A chunk ranked 3rd by both retrievers scores above one ranked 1st and 10th."

Q: You want to change embedding models. What happens?

A: "It's a migration, not a config change, because old and new vectors aren't comparable and there's no conversion. Five steps: dual-write so new ingestion populates both indexes; backfill the existing corpus into the new one, throttled against the embedding endpoint's rate limit — that's the step that dominates cost and time; shadow-read, serving from the old index while querying both and logging recall differences on a golden set; cut over when the new index is at least as good; and retain the old one for a rollback window. Weeks, not hours, for a large corpus. The reason to plan it before you need it is that the day you need it is the day a better model ships and everyone wants it immediately — and that's the worst moment to be designing a migration."

Q: How do you know your retrieval is any good?

A: "Recall@k on a golden set, measured before I touch the generator — because if the answer isn't in the retrieved set, no prompt and no larger model recovers it, so recall caps everything downstream. Then I'd separate the stages: recall@50 for first-stage retrieval, precision@5 after reranking, and end-to-end grounding coverage — the fraction of claims that map to a retrieved span. Those three tell you where the problem is, which a single end-to-end score doesn't. And I'd track two operational signals that are genuinely useful: the freshness-exclusion rate, which rises when an ingestion pipeline stalls, and the 'no relevant documents' rate, which is only meaningful if you have a relevance floor — without one, first-stage retrieval always returns something and 'I don't know' becomes unreachable."

Q: An agent in the advisory business retrieves a document about a live deal. What's wrong?

A: "Potentially nothing — or an information-barrier breach, depending on who's asking. That's MNPI, and the barrier between advisory and trading exists precisely to stop it moving. What's new with agents is that the crossing happens silently: no human read the document, no access looks unusual, and the information reaches a trader inside a summary. So the barrier has to be a retrieval constraint rather than a policy document — a chunk tagged with a barrier is invisible to any principal outside it, and invisible means not retrieved, not filtered from the answer. I'd also want the second, independent gate on the output side: MNPI detection in the guardrail chain, so a barrier-tagging mistake at ingestion doesn't become a disclosure. Two gates, because one of them will be misconfigured eventually."

15. References

Retrieval fundamentals

• Robertson & Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond, 2009 — the definitive BM25 treatment, including where k1 and b come from.
• Manning, Raghavan & Schütze, Introduction to Information Retrieval, CUP 2008 — free online; chapters 6 (scoring), 8 (evaluation) and 11 (probabilistic retrieval).
• Cormack, Clarke & Büttcher, Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods, SIGIR 2009 — the RRF paper, and the source of k=60.

Dense retrieval and reranking

• Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering, EMNLP 2020.
• Nogueira & Cho, Passage Re-ranking with BERT, 2019 — the cross-encoder reranking pattern.
• Weinberger et al., Feature Hashing for Large Scale Multitask Learning, ICML 2009 — the hashing trick and why the signs matter.
• MTEB (Massive Text Embedding Benchmark) — a starting point for model selection, not an answer for your corpus.

Vector stores and ANN

• Malkov & Yashunin, Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs, 2016 — HNSW.
• Jégou, Douze & Schmid, Product Quantization for Nearest Neighbor Search, 2011 — the PQ half of IVF-PQ.
• pgvector, Azure AI Search, Pinecone, Weaviate and Qdrant documentation — read each one's filtering and multi-tenancy pages specifically; that is where the topology decision is made.

Grounding and evaluation

• Es et al., RAGAS: Automated Evaluation of Retrieval Augmented Generation, 2023 — faithfulness, answer relevance, context precision/recall.
• Gao et al., Retrieval-Augmented Generation for Large Language Models: A Survey, 2023 — a good map of the design space.
• AWS Bedrock contextual grounding checks — a production implementation of the faithfulness idea, worth reading for its API shape.

Isolation

• AWS SaaS Lens / SaaS Factory — the silo / pool / bridge vocabulary, applied here to indexes.
• OWASP Top 10 for LLM Applications — Sensitive Information Disclosure and Vector and Embedding Weaknesses.