« Phase 05 README · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 05 Warmup — Retrieval Foundations

Who this is for: you can write Python and you understand the agent loop (Phases 00–01). You may have called a vector store (db.similarity_search(query)) but never built one, and you can't yet say why your RAG demo cites the wrong document. By the end you will have built the whole retrieval pipeline — chunker, embedder, dense index, BM25, RRF, reranker, and the metrics that prove it works — and you will know, from the math, exactly why hybrid search beats either dense or lexical retrieval alone. No GPU, no API key, no model download; the "embeddings" are a hashing trick you can read.

Table of Contents

1. Why retrieval exists at all
2. The RAG pipeline, end to end
3. Chunking: size, overlap, and the tradeoff
4. Embeddings: what a vector actually means
5. Cosine, dot product, and why we normalize
6. From exact kNN to approximate nearest neighbors
7. BM25: lexical retrieval from first principles
8. Why hybrid beats either retriever alone
9. Reciprocal Rank Fusion: combining without comparing scores
10. Bi-encoder vs cross-encoder: retrieve, then rerank
11. Measuring retrieval: recall@k, precision@k, MRR, nDCG
12. The vector-database landscape
13. Production: filtering, security-trimming, freshness
14. Common misconceptions
15. Lab walkthrough
16. Success criteria
17. Interview Q&A
18. References

1. Why retrieval exists at all

A large language model is a function from a prompt to text. Its knowledge is frozen at two points: what was in its training data (up to a cut-off date), and what you place in its context window on this call. Everything else — your company's runbooks, the ticket filed an hour ago, the customer row that changed this minute, the PDF marked confidential — the model has never seen and cannot know. If you ask about it anyway, a well-tuned model will do the most dangerous thing it can: produce a fluent, confident, wrong answer. That is hallucination, and it is not a bug you can prompt away — the model is doing exactly what it was trained to do (continue the text plausibly) with information it does not have.

Retrieval-Augmented Generation (RAG) is the fix. Before the model answers, your code retrieves the most relevant snippets from a trusted corpus and pastes them into the prompt, so the model answers from the provided text instead of from its frozen memory. Retrieval buys you three things a bare LLM cannot have:

• Grounding. The answer is tied to real source text you control, so it's checkable and far less likely to be invented.
• Freshness. The corpus can change every second; the model doesn't need retraining to know today's facts. Re-index the new document and it's instantly answerable.
• Citations. Because you know which chunks you injected, you can show the user the source — the difference between "the model says" and "page 4 of the security policy says."

"Why not just use a long-context model and paste everything?" You can, and sometimes you should. But it isn't free. Every token in the window is billed on every call and adds latency; a 200k-token prompt at $3/1M input tokens is $0.60 per request before the model writes a word. Worse, quality is not monotone in context length: models exhibit "lost in the middle" — accuracy is highest for facts at the very start or end of a long context and sags for facts buried in the middle (Liu et al., 2023). Retrieving the right 2k tokens is cheaper, faster, and often more accurate than dumping 200k. Retrieval and long context are complements — retrieval decides what deserves the expensive window.

2. The RAG pipeline, end to end

Retrieval splits cleanly into an offline (indexing) phase and an online (query) phase.

  OFFLINE (once per document, or on change)
  documents ──► chunk ──► embed ──► index (dense vectors + BM25 postings)

  ONLINE (once per query)
  query ──► embed(query) ─┐
           tokenize(query)─┴─► dense top-n  +  BM25 top-n
                                      └──► RRF fuse ──► rerank ──► top-k chunks
                                                                      └──► assemble context ──► LLM

This phase owns everything up to "top-k chunks." Assembling those chunks into a budgeted prompt is Phase 04 (context engineering); generating and citing from them is the agent loop (Phase 01). The discipline is that each stage is separately testable: you can measure whether the right chunk was retrieved (this phase's metrics, §11) completely independently of whether the model then wrote a good answer. Teams that conflate the two spend weeks "improving the prompt" when the real bug is that the relevant chunk never made it into the window.

Every stage is a design decision with a cost and a failure mode, and the rest of this warmup walks them in order.

3. Chunking: size, overlap, and the tradeoff

You cannot embed or index a 40-page document as one unit — a single vector for 40 pages is a blurry average that matches everything and retrieves nothing precisely, and you'd inject the whole document when the user needs one paragraph. So you chunk: split each document into retrievable units. Two strategies, both in the lab:

• Fixed-size with overlap (chunk_text). Slide a window of size words forward by size − overlap words. The overlap exists so a fact that straddles a boundary survives intact in at least one chunk — without it, "The threshold is set in config.yaml. It defaults to 30 seconds." can be split so no single chunk contains both the setting and its default.
• Sentence-packed (sentence_chunk). Greedily pack whole sentences up to a token budget, never splitting mid-sentence. This respects natural meaning boundaries far better than an arbitrary word cut. Its cousins in production split on Markdown headings, code blocks, or document structure ("semantic chunking").

The size is a genuine tradeoff you tune per corpus:

The failure mode of chunks that are too large is a diluted embedding (the "average of many topics" that matches everything weakly). The failure mode of chunks that are too small is lost context (the chunk matches but doesn't carry enough to answer). Overlap and sentence boundaries are how you hedge. In the lab you'll prove the invariants: no chunk exceeds size, consecutive chunks share exactly overlap words, and every word is covered.

4. Embeddings: what a vector actually means

An embedding is a function from text to a fixed-length list of numbers — a vector — with one magic property: texts with similar meaning map to vectors that are close together in that space, and unrelated texts map far apart. "How do I reset my password?" and "I forgot my login credentials, help" land near each other even though they share almost no words. That is the whole point of dense retrieval — it matches meaning, not just tokens.

How does a real embedder get that property? It's a neural network (a bi-encoder, §10) trained with a contrastive objective: shown many (query, relevant-passage) pairs, it's pushed to make matching pairs' vectors close and mismatched pairs' vectors far. After training on hundreds of millions of pairs, the geometry of the vector space encodes semantic similarity. Models you'll meet: sentence-transformers (all-MiniLM-L6-v2, 384 dims; bge-large, 1024), Cohere Embed, OpenAI text-embedding-3 (up to 3072 dims). More dimensions ≈ more capacity, at more storage and compute.

The lab's embedder is honest, not learned. We can't ship a trained neural net in a stdlib-only lab, so embed uses the hashing trick: hash each token into one of dim buckets and count occurrences — a bag-of-words vector. It's deterministic (we hash with hashlib, never Python's process-salted builtin hash(), so a persisted vector never drifts) and it captures token overlap, but not learned meaning — it can't tell that "car" and "automobile" are related, because they hash to different buckets. That limitation is deliberate and instructive: it's exactly why, in §8, our "dense" retriever's advantage over BM25 comes from weighting common shared words, not from synonymy. Swap embed for a real bi-encoder (one function) and true paraphrase matching appears; the rest of the pipeline is unchanged.

Matryoshka embeddings, briefly: some modern embedders (OpenAI text-embedding-3, Nomic) are trained so the first d dimensions of the vector are themselves a usable, lower-fidelity embedding. You can truncate a 3072-dim vector to 256 dims to save storage and speed up search, trading a little accuracy — "one embedding, many sizes." Handy when the index gets big.

5. Cosine, dot product, and why we normalize

Given two vectors, how "close" are they? The default in retrieval is cosine similarity — the cosine of the angle between them:

$$\operatorname{cos}(\mathbf{q}, \mathbf{d}) = \frac{\mathbf{q}\cdot\mathbf{d}}{\lVert\mathbf{q}\rVert,\lVert\mathbf{d}\rVert} = \frac{\sum_i q_i d_i}{\sqrt{\sum_i q_i^2},\sqrt{\sum_i d_i^2}}.$$

It ranges in [-1, 1] ([0, 1] for the non-negative bag-of-words vectors in the lab): 1 means same direction (maximally similar), 0 means orthogonal (unrelated), -1 means opposite. The key feature is that it measures direction, not magnitude — a long document and a short one about the same topic point the same way, so cosine ignores length. That's usually what you want: "aboutness," not "how much text."

The dot product \(\mathbf{q}\cdot\mathbf{d}\) is the numerator alone — it does grow with magnitude. Here's the crucial practical fact: if all vectors are L2-normalized (scaled to unit length), cosine similarity and dot product are identical. So production systems normalize once at index time and then use the cheaper dot product for every search. That's why the lab's embed divides each vector by its L2 norm \(\lVert\mathbf{v}\rVert = \sqrt{\sum_i v_i^2}\): after normalization, a fast dot-product index computes cosine for free. (A subtlety: a zero vector has no direction, so cosine with it is undefined — the lab defines it as 0.0 rather than dividing by zero.)

Some embedders are trained for dot product / Euclidean distance instead of cosine; you must match the metric the model was trained with, or recall silently drops. When in doubt, normalize and use cosine — it's the safe default and the one nearly every model supports.

6. From exact kNN to approximate nearest neighbors

Dense retrieval is a k-nearest-neighbors search: embed the query, then find the k stored vectors with the highest cosine. The lab's DenseIndex does this exactly — it scores the query against every vector and sorts:

$$\text{cost} = O(N \cdot \text{dim}) \text{ per query.}$$

For thousands of vectors that's instant. For millions, scanning all of them per query is too slow, and this is where real systems switch to Approximate Nearest Neighbor (ANN) indexes — they return almost the true top-k, trading a few percent of recall for 10–1000× speed. Three ideas power essentially all of them:

• HNSW (Hierarchical Navigable Small World) — the default in FAISS, pgvector, Weaviate, Qdrant, Milvus. Build a multi-layer graph where each vector links to its near neighbors; upper layers are sparse ("express lanes"), lower layers dense. To search, start at the top, greedily hop to the neighbor closest to the query, descend a layer, repeat. You reach the query's neighborhood in O(log N) hops instead of scanning N. The graph is what makes vector search scale, and "HNSW" is a word you should be able to explain in an interview.
• IVF (Inverted File) — cluster all vectors into, say, 4096 centroids (Voronoi cells) with k-means; at query time find the few nearest centroids and scan only those cells. You trade recall (the true neighbor might sit just across a cell boundary you didn't scan — mitigated by scanning nprobe cells) for scanning a small fraction of the corpus.
• PQ (Product Quantization) — compress each vector by splitting it into sub-vectors and replacing each with the id of the nearest entry in a small learned codebook. A 1024-dim float vector (4 KB) shrinks to a few dozen bytes, so the whole index fits in RAM and distance computations are table lookups. Trades precision for memory; usually layered on top of IVF (IVF-PQ).

You won't build these in the lab — the point is that the brute-force DenseIndex you do build has the exact semantics these approximate. When someone says "we moved from flat to HNSW and recall dropped 1% but p99 dropped 40×," you'll know precisely what they traded.

7. BM25: lexical retrieval from first principles

Before embeddings, search was lexical: match the query's words against the documents' words, cleverly weighted. The clever weighting that has won for 30 years is BM25 ("Best Matching 25," Robertson & Sparck-Jones lineage). It scores a document \(d\) for a query \(q\) as a sum over the query terms present in \(d\) of three factors:

$$\operatorname{score}(q, d) = \sum_{t \in q} \operatorname{IDF}(t) \cdot \frac{tf(t,d),(k_1 + 1)}{tf(t,d) + k_1\left(1 - b + b,\dfrac{|d|}{\text{avgdl}}\right)}$$

Read it piece by piece — each factor fixes a real problem with naive word-count matching:

• IDF — inverse document frequency. A word in every document (like "the," or "network" in a networking corpus) tells you nothing about which document is relevant, so it should count for almost nothing. A word in one document (a rare error code, a product SKU) is enormously informative. IDF encodes exactly that — the smoothed, non-negative form (the Lucene variant the lab uses) is

  $$\operatorname{IDF}(t) = \ln!\left(1 + \frac{N - df(t) + 0.5}{df(t) + 0.5}\right),$$

  where \(N\) is the number of documents and \(df(t)\) the number containing \(t\). A term in all \(N\) docs gives \(\operatorname{IDF} \approx \ln(1 + 0.5/N) \approx 0\); a term in one doc gives a large value. This is BM25's superpower: it up-weights rare, discriminating tokens — the exact keywords, IDs, and codes that dense embeddings dilute.

• TF saturation. More occurrences of a query term should raise the score, but with diminishing returns — a document that says "invoice" 50 times isn't 50× more relevant than one that says it once. The \(tf,(k_1+1)/(tf + k_1(\dots))\) form saturates: the 2nd occurrence adds a lot, the 20th almost nothing. \(k_1\) (default 1.5) controls how fast it saturates.

• Length normalization. A long document contains more words by chance, so it would rack up spurious matches; the \((1 - b + b,|d|/\text{avgdl})\) term penalizes documents longer than average and rewards shorter ones. \(b\) (default 0.75) controls how hard; \(b=0\) disables length normalization entirely.

BM25 needs no training and no GPU — it's counting with good weights — which is why after two decades it is still the lexical half of nearly every serious search system (Elasticsearch, OpenSearch, Lucene, Postgres full-text). In the lab you implement idf, score, and search exactly as above, and a test confirms IDF drives a corpus-wide term toward zero.

8. Why hybrid beats either retriever alone

Here is the heart of the phase. Dense and lexical retrieval fail in opposite directions, and that complementarity is why you run both.

• Dense (embeddings) is strong on paraphrase, weak on rare tokens. It matches meaning, so "how do I get onto the shared drive?" finds a doc titled "accessing network file shares" even with no shared words. But a rare identifier like TX4096 is just one dimension of a high-dimensional vector; averaged in with everything else, its signal is diluted, and the dense index happily returns three semantically similar gateway docs that don't actually mention the code.
• Lexical (BM25) is strong on rare tokens, weak on paraphrase. Its IDF loves TX4096 (df = 1 → huge weight), so it nails exact keywords, IDs, error codes, and quoted phrases. But it matches tokens: if the relevant document is phrased entirely in different (or merely common) words, BM25's IDF discards the overlap and misses it.

The lab makes this concrete and deterministic with a crafted corpus:

• Query A ("network drive access kerberos"). The truly relevant doc shares only common words with the query (network/drive/access — near-zero IDF in this corpus), so BM25 misses it: a distractor that owns the rare word "kerberos" wins BM25. The dense embedder, which weights all shared tokens regardless of rarity, ranks the relevant doc #1 — dense catches it.
• Query B ("error code tx4096 connection troubleshooting"). The relevant doc contains the rare identifier tx4096, so BM25 catches it (huge IDF). But tx4096 is one thin bucket in the doc's bag-of-words vector, and a shorter distractor packed with the query's common words wins the cosine — dense misses it.

Run each retriever alone and each gets one query right: dense-only recall@1 = 0.5, bm25-only recall@1 = 0.5. Fuse them and the hybrid gets both → recall@1 = 1.0. That inequality — hybrid ≥ max(dense, bm25), strict on this set — is a passing test in the lab, and it is the entire business case for hybrid search stated as arithmetic. (Note: because our embedder is lexical, the "dense win" here comes from weighting common shared words rather than from true synonymy — a real bi-encoder would add paraphrase wins on top.)

9. Reciprocal Rank Fusion: combining without comparing scores

Once you have a dense ranking and a BM25 ranking, how do you merge them? The naive idea — add the scores — is broken, because the scores live on incompatible scales: cosine is in [0, 1], BM25 is in [0, ∞) and depends on corpus statistics. Adding 0.7 to 12.3 is meaningless, and normalizing them ("min-max the BM25 scores") is fragile and needs tuning per corpus.

Reciprocal Rank Fusion (RRF) sidesteps the whole problem by fusing ranks, not scores (Cormack, Clarke & Büttcher, 2009):

$$\operatorname{RRF}(d) = \sum_{r \in \text{rankings}} \frac{1}{k_{\text{const}} + \operatorname{rank}_r(d)},$$

where \(\operatorname{rank}_r(d)\) is \(d\)'s 1-indexed position in ranking \(r\) (a ranking that omits \(d\) contributes nothing for it). Properties that make it the industry default:

• Scale-free. It never looks at a raw score, so cosine-vs-BM25 scale mismatch simply can't arise. No normalization, no tuning knobs beyond \(k_{\text{const}}\).
• The constant damps the top. \(k_{\text{const}}\) (the paper's default 60) softens the gap between rank 1 and rank 2, so a document ranked moderately in both lists beats a document ranked #1 in one list and absent from the other. That's precisely the behavior you want: a result both retrievers agree on is more trustworthy than one only a single retriever loved. In the lab, a doc appearing in both top-n lists provably outranks any doc in only one.
• Order-independent and deterministic. It's a sum, so the order of the input rankings doesn't matter, and ties are broken by id — the lab tests both.

RRF is what Elasticsearch's rank retriever, Weaviate's hybrid search, and most pgvector-plus-BM25 stacks use under the hood. It is a shockingly simple formula for how well it works.

10. Bi-encoder vs cross-encoder: retrieve, then rerank

Retrieval so far used a bi-encoder: the query and each document are embedded separately, into independent vectors, and compared with a dot product. That separation is what makes it scale — you precompute and index every document vector once, and a query is one embed plus a fast nearest-neighbor lookup. The cost of that separation is accuracy: the query's vector is computed without ever "seeing" the document, so subtle query-document interactions (negation, which entity the pronoun refers to, whether the numbers actually match) are invisible to a dot product.

A cross-encoder fixes that by feeding the (query, document) pair together through the model, which can attend across both and output a single, much more accurate relevance score. The catch: there's nothing to precompute — you must run the model once per candidate document, so scoring a whole corpus is infeasible.

The resolution is the two-stage pipeline and it's the standard everywhere:

1. Retrieve with the cheap bi-encoder (and BM25) — cast a wide net, get the top ~50–200 candidates. Optimize for recall: don't lose the right answer.
2. Rerank those few candidates with the expensive cross-encoder — reorder precisely. Optimize for precision@k: put the best ones on top.

"Retrieve wide and cheap, rerank narrow and precise." The lab models the reranker with an injected token_overlap_scorer (a deterministic stand-in that counts shared query tokens) so the pipeline stays offline and testable — but the shape is identical to production, where you'd drop in Cohere Rerank or a bge-reranker cross-encoder with a one-line change. A good reranker routinely lifts answer quality more than any prompt tweak, because it fixes the ordering the fast retriever got approximately right.

11. Measuring retrieval: recall@k, precision@k, MRR, nDCG

You cannot improve — or safely change — what you don't measure, and retrieval has its own metrics, computed against a labeled set of (query, relevant-chunk-ids) pairs, separately from whether the LLM then wrote a good answer.

• Recall@k — of all the relevant chunks, what fraction appear in the top k?

  $$\operatorname{recall@}k = \frac{|{\text{relevant}} \cap {\text{top-}k\text{ retrieved}}|}{|{\text{relevant}}|}.$$

  This is the metric that matters most for RAG. A relevant chunk that isn't retrieved can never reach the LLM — recall@k is the ceiling on how good the answer can be. If recall is low, no reranker or prompt can save you.

• Precision@k — of the top k you returned, what fraction are relevant?

  $$\operatorname{precision@}k = \frac{|{\text{relevant}} \cap {\text{top-}k\text{ retrieved}}|}{k}.$$

  Precision matters when the context budget is tight: every irrelevant chunk you pack in wastes tokens and can distract the model (lost-in-the-middle again).

• MRR — Mean Reciprocal Rank — averages \(1/\operatorname{rank}\) of the first relevant result across queries:

  $$\operatorname{MRR} = \frac{1}{|Q|}\sum_{q \in Q}\frac{1}{\operatorname{rank of first relevant}_q}.$$

  It rewards putting a right answer high — the reranker's whole job — and is the classic single-number summary for question-answering retrieval.

• nDCG — normalized Discounted Cumulative Gain — the metric for graded relevance (some chunks are "perfect," some "partially relevant"). It sums each result's relevance discounted by a \(\log\) of its position (rank 1 counts full, rank 10 counts little), then normalizes by the best possible ordering so scores are comparable across queries. Use it when relevance isn't binary; recall@k and MRR (binary) are enough for the lab.

The load-bearing idea: evaluate retrieval and generation separately. When your RAG system gives a bad answer, recall@k tells you instantly whether the retriever failed (the chunk wasn't there) or the generator failed (the chunk was there and the model still botched it). Teams that only eval the final answer spend weeks debugging the wrong half. A recall@k gate in CI (Phase 11) catches a chunking or embedding regression before a user does.

12. The vector-database landscape

The lab's DenseIndex + BM25 is a toy vector database. Production picks one off the shelf, and the Citi JD literally lists five — here's the map, and which JD wants which:

The interview-ready summary: pgvector for "vectors beside my SQL data," Pinecone for managed, Weaviate/Qdrant for open-source hybrid, FAISS for in-process, Chroma for prototypes, Milvus for massive scale, Neo4j when the graph is the point. They differ in ops model and features, but under the hood they all run the HNSW/IVF/PQ ideas from §6 with metadata filtering bolted on.

13. Production: filtering, security-trimming, freshness

The lab retrieves from a trusting, static corpus. Production has three concerns the lab doesn't, and they're where retrieval meets the rest of the platform:

• Metadata filtering. Real queries are "find docs about VPNs authored by IT, updated this year, in English." The vector index stores metadata alongside each vector and filters during search (where team = 'IT' and lang = 'en'). Doing it before the ANN search (pre-filtering) preserves recall but is harder to index; doing it after (post-filtering) is easy but can empty your top-k. Every serious vector DB supports this because pure similarity is rarely the whole query.
• Security-trimming / ACLs. This is the one that gets people fired. A retriever that ignores who is asking will happily surface a chunk from a document the user isn't allowed to read — the model then summarizes confidential data into the answer. You must filter by the caller's permissions, and a shared vector index across tenants is the #1 multi-tenant leak channel (Phase 13): tenant A's query must never retrieve tenant B's vectors. Enforce it with per-tenant namespaces/collections or a mandatory tenant-id filter on every search — architecturally, not by hoping the prompt asks nicely.
• Freshness and rebuilds. The corpus changes, so the index must too. Adding/updating a document means re-chunk, re-embed, upsert. Two traps: stale vectors (a doc changed but its old embedding lingers — you must delete-then-insert, keyed by a stable chunk id) and re-embedding on model change (switching embedding models invalidates every stored vector; you must re-embed the whole corpus, often behind a versioned index and a dual-read migration). Budget for full rebuilds — they're routine, not exceptional.

None of these change the algorithms you built; they wrap them. But they're the difference between a demo and a system, and they're exactly what the enterprise JDs (Citi's multi-tenant platform) are hiring you to get right.

14. Common misconceptions

• "Vector search is all you need; BM25 is legacy." No — they fail in opposite directions (§8). BM25 catches the rare keyword/ID/exact-phrase queries that dense search dilutes, and it's free to run. Hybrid beats either, provably. The best modern systems run both.
• "Bigger chunks give the model more context, so they're better." Bigger chunks make blurrier embeddings that match everything weakly and waste tokens when injected. Chunk size is a tuned tradeoff, not "bigger is better."
• "I can just add the cosine and BM25 scores." They live on incompatible scales; adding them is meaningless. Fuse by rank (RRF), not score.
• "Reranking is optional polish." A cross-encoder reranker often lifts answer quality more than any prompt change, because it fixes the ordering a fast bi-encoder only got approximately right. Retrieve wide, rerank narrow.
• "Cosine and dot product are different metrics." On L2-normalized vectors they're identical — which is why you normalize and then use the cheaper dot product.
• "If the RAG answer is wrong, improve the prompt." Check recall@k first. If the relevant chunk wasn't retrieved, the prompt is irrelevant — the bug is in the retriever, and no prompt can inject a chunk that isn't there.
• "A shared vector index is fine for multiple tenants." It's the #1 cross-tenant data-leak channel. Isolate by namespace or a mandatory tenant filter (Phase 13).

15. Lab walkthrough

Open lab-01-hybrid-retriever/ and fill the TODOs top to bottom — the file is ordered to match this warmup:

1. Chunking — chunk_text (guard overlap < size; slide a window stepping by size − overlap) and sentence_chunk (pack whole sentences, never split one). The tests check the overlap-seam and coverage invariants.
2. Embedding — embed (bag-of-words over _bucket, then L2-normalize; empty text → zero vector) and cosine (guard the zero vector). Determinism comes from hashlib, which is given.
3. Dense index — DenseIndex.search (cosine against all, sort by (-score, id), top-k; k beyond the corpus returns everything).
4. BM25 — idf (the smoothed formula), score (IDF · TF-saturation · length-norm), and search (positive scores only, sorted, empty query → []).
5. RRF — rrf (sum 1/(k_const + rank); sort by (-score, id); order-independent).
6. Rerank — token_overlap_scorer (distinct shared tokens) and rerank (stable sort by score, so fusion order breaks ties).
7. Pipeline — HybridRetriever.retrieve (dense top-n + BM25 top-n → RRF → rerank → top-k).
8. Metrics — recall_at_k, precision_at_k, reciprocal_rank, mrr.

Run LAB_MODULE=solution pytest -v first to see green, then make your lab.py match. Finish by reading solution.py's main() — it prints the two miss cases and the recall@1 table where hybrid = 1.0 beats dense = 0.5 and bm25 = 0.5.

16. Success criteria

• You can explain, from the formulas, why BM25 catches rare tokens (IDF) and dense catches common-word overlap / paraphrase — and therefore why hybrid beats either.
• You can derive cosine similarity and say why L2-normalization makes it equal the dot product.
• You can state the BM25 formula and what k1, b, and IDF each control.
• You can explain why RRF fuses by rank, not score, and what k_const does.
• You can describe bi-encoder vs cross-encoder and where each runs.
• You can define recall@k / precision@k / MRR and say which to optimize for RAG, and why you eval retrieval separately from generation.
• You can map pgvector / Pinecone / Weaviate / FAISS / Chroma / Milvus / Neo4j to a use case.
• All 31 lab tests pass under both lab and solution.

17. Interview Q&A

Q: Dense retrieval or BM25 — which do you use? A: Both, fused. They fail in opposite directions: BM25's IDF makes it excellent at rare keywords, IDs, error codes, and exact phrases but blind to paraphrase; dense embeddings match meaning/paraphrase but dilute a rare token into one dimension. I run both, fuse with Reciprocal Rank Fusion, and rerank the top candidates. I can show a crafted set where each alone gets recall@1 = 0.5 and the hybrid gets 1.0 — hybrid ≥ max(dense, bm25) is the whole argument, and it's testable.

Q: How do you combine a cosine score and a BM25 score? A: You don't add them — they're on incompatible scales ([0,1] vs [0,∞)). I fuse by rank with RRF: score(d) = Σ 1/(k + rank_r(d)) over the rankings, k≈60. It's scale-free, needs no normalization, is order-independent, and its k constant makes a doc that's strong in both lists beat one that's #1 in only one — which is exactly the consensus behavior you want.

Q: What's the difference between a bi-encoder and a cross-encoder, and why do you have both? A: A bi-encoder embeds query and document separately, so document vectors are precomputable and indexable — fast and scalable, but the query never "sees" the document. A cross-encoder scores the (query, document) pair together — much more accurate, but you must run it per candidate, so it can't scan a corpus. So: retrieve wide with the bi-encoder (optimize recall), then rerank the top ~100 with the cross-encoder (optimize precision@k). Retrieve cheap and wide, rerank precise and narrow.

Q: Why chunk, and how do you pick the size? A: You can't embed a 40-page doc as one blurry vector, and you don't want to inject the whole thing for one paragraph. Size is a tradeoff: small chunks give precise, on-topic matches but can lose the context a fact needs and blow up the index count; large chunks carry more context but average multiple topics into a vague embedding and waste tokens. I use overlap (or sentence/structure boundaries) so a fact straddling a boundary survives, and I tune size against recall@k on a labeled set.

Q: Your RAG system gives a wrong answer. How do you debug it? A: First I check recall@k on the labeled query: was the relevant chunk even retrieved? If not, the bug is in the retriever (chunking, embedding, or fusion) and no prompt change helps — a chunk that isn't in the window can't be used. If the chunk was retrieved and the model still botched it, that's a generation problem (prompt, model, or lost-in-the-middle). Evaluating retrieval and generation separately is what makes this a five-minute diagnosis instead of a two-week guess.

Q: How does ANN differ from the exact kNN, and what do you trade? A: Exact kNN scans every vector — O(N·dim) — fine to thousands, too slow at millions. ANN (HNSW graph descent, IVF centroid cells, PQ compression) returns almost the true top-k, trading a few percent of recall for 10–1000× speed and much less memory. You tune the recall/latency knob (ef_search, nprobe) per SLO. FAISS, pgvector, Pinecone, Weaviate all run these; the semantics are the brute-force search, approximated.

Q: A shared vector store serves multiple customers. What's your first concern? A: Tenant isolation — a shared index is the #1 cross-tenant leak channel. Tenant A's query must never retrieve tenant B's vectors, so I isolate by namespace/collection per tenant or enforce a mandatory tenant-id filter on every search, plus security-trimming by the caller's ACLs. That's architectural, not a prompt instruction (Phase 13).

18. References

• Robertson & Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond (2009) — the definitive BM25 reference. https://www.staff.city.ac.uk/~sbrp622/papers/foundations_bm25_review.pdf
• Cormack, Clarke & Büttcher, Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods (SIGIR 2009) — RRF. https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf
• Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering (DPR, 2020) — the bi-encoder for retrieval. https://arxiv.org/abs/2004.04906
• Malkov & Yashunin, Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs (HNSW, 2016). https://arxiv.org/abs/1603.09320
• Jégou, Douze & Schmid, Product Quantization for Nearest Neighbor Search (2011) — PQ. https://inria.hal.science/inria-00514462/document
• Nogueira & Cho, Passage Re-ranking with BERT (2019) — the cross-encoder reranker. https://arxiv.org/abs/1901.04085
• Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023). https://arxiv.org/abs/2307.03172
• Kusupati et al., Matryoshka Representation Learning (2022) — truncatable embeddings. https://arxiv.org/abs/2205.13147
• pgvector — Postgres vector extension. https://github.com/pgvector/pgvector
• FAISS — a library for efficient similarity search (HNSW/IVF/PQ). https://faiss.ai/
• Pinecone learning center — hybrid search & vector DB concepts. https://www.pinecone.io/learn/
• Weaviate hybrid search docs. https://weaviate.io/developers/weaviate/search/hybrid