« Track Overview · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 05 — Retrieval Foundations: Chunking, Embeddings, Hybrid Search & Reranking

Answers these JD lines: Citi's "RAG and vector database experience: Pinecone, Weaviate, FAISS, pgvector, ChromaDB" and "semantic search," plus the enterprise-RAG language in Cohere (agent infrastructure, enterprise use cases) and Wolters Kluwer (AI platform & agents). This phase is the retrieval half of every "build a RAG system over our documents" requirement in jd.md; Phase 06 handles the advanced graph/hierarchical retrieval (GraphRAG / LightRAG / RAPTOR / Neo4j).

Why this phase exists

An LLM knows only what it was trained on and what you put in its context window. Everything the model doesn't know — your private docs, this morning's ticket, the row that changed a minute ago — has to be retrieved and pasted in. Retrieval is how you give a stochastic parrot grounding (answers tied to real sources), freshness (facts newer than the training cut-off), and citations (a human can verify). And it is the part teams most consistently under-invest in: they reach for a one-line VectorStore.similarity_search, ship it, and then wonder why the agent confidently cites the wrong policy. RAG quality is dominated by retrieval quality, and retrieval quality is an engineering discipline — chunking, embeddings, indexing, fusion, reranking, and measurement — not a library call.

Four ideas carry the phase:

1. Retrieval is grounding, freshness, and citation — often cheaper than a giant context. You can stuff 200k tokens into a long-context model, but it costs tokens on every call, adds latency, and past some point degrades quality ("lost in the middle"). Retrieving the right 2k tokens is cheaper, faster, and often more accurate.
2. Lexical and dense retrieval fail in opposite directions, so you run both. BM25 (lexical) nails rare keywords, IDs, error codes, and exact phrases but is blind to paraphrase. Dense (embeddings) catches paraphrase and meaning but dilutes a rare token into one dimension of a vector. Hybrid search fuses them and beats either alone — the central, testable result of the lab.
3. Fusion must be score-scale-free. Cosine lives in [0, 1]; BM25 lives in [0, ∞). You cannot add them. Reciprocal Rank Fusion combines ranks, not scores, so no normalization or tuning is needed.
4. You cannot ship what you cannot measure. Recall@k, precision@k, MRR, nDCG — retrieval has its own offline metrics, evaluated separately from generation, so you know whether a change helped before it reaches a user.

Concept map

• The RAG pipeline: documents → chunk → embed → index (offline) then query → retrieve → (fuse) → rerank → assemble context → generate (online). This phase owns everything up to "assemble context" (which is Phase 04).
• Chunking: fixed-size + overlap vs sentence/semantic packing; the size ↔ recall ↔ precision ↔ cost tradeoff; a fact that straddles a boundary must survive in some chunk.
• Embeddings: text → vector; cosine vs dot product; why L2-normalize; the bi-encoder that makes vectors indexable; Matryoshka (truncatable) embeddings, briefly.
• ANN vs exact: brute force is O(N·dim); HNSW (greedy graph descent), IVF (coarse centroid cells), PQ (compressed vectors) trade a little recall for huge speed.
• BM25: IDF · TF-saturation · length-norm, summed over query terms — the lexical workhorse.
• Hybrid + RRF: run dense and BM25, fuse by rank; why it beats either alone.
• Bi-encoder vs cross-encoder: retrieve wide with a bi-encoder, rerank narrow with a cross-encoder.
• Metrics: recall@k / precision@k / MRR / nDCG; retrieval eval vs generation eval.
• The vector-DB landscape: pgvector, Pinecone, Weaviate, FAISS, Chroma, Milvus, Neo4j — and which JD wants which.

The lab

Integrated scenario (how this shows up at work)

You're building an enterprise document assistant — an agent that answers employee questions over a corpus of IT runbooks, HR policies, and incident post-mortems. A user asks "how do I fix error TX4096 on the payments gateway?" Pure dense retrieval returns three plausible-looking runbooks about gateway connectivity but not the one that actually names TX4096, because that rare token is one thin dimension in the embedding — the model then hallucinates a fix. Pure lexical retrieval nails the TX4096 runbook but, on the next question ("how do I get onto the shared network drive?"), misses the relevant doc because it's phrased entirely in common words that IDF discards. You ship hybrid: BM25 for the rare-keyword questions, dense for the paraphrase questions, RRF to fuse, a reranker to sharpen the top, and recall@k in CI so a chunking change can't silently regress. Same corpus, same agent — the retrieval architecture is the difference between "cites the right runbook" and "makes one up." That is this phase.

Deliverables checklist

• Lab 01 green under LAB_MODULE=solution pytest and your own lab.py.
• You can state, from the formulas, why BM25 catches rare keywords and dense catches paraphrase — and why RRF fuses them without normalizing scores.
• You can explain bi-encoder vs cross-encoder and where each runs in the pipeline.
• You can name recall@k / precision@k / MRR / nDCG and say which to optimize for RAG.
• You can map pgvector / Pinecone / Weaviate / FAISS / Chroma / Milvus / Neo4j to a JD.

Key takeaways

• RAG is mostly retrieval, and retrieval is engineering, not a library call. The one-line similarity_search is the demo; the pipeline is the product.
• Run both retrievers. Lexical = rare tokens/IDs/exact phrases; dense = paraphrase/meaning. Hybrid + RRF beats either, and you can prove it with recall@k.
• Retrieve wide and cheap, rerank narrow and precise. The bi-encoder finds candidates; the cross-encoder orders them.
• Measure retrieval separately from generation. A recall@k gate in CI catches the regression before the user does.
• The trust boundary applies here too. Metadata filtering and security-trimming (a shared index is the #1 multi-tenant leak channel, Phase 13) are retrieval concerns, not afterthoughts.