Lab 01 — Hybrid Retriever (BM25 + Dense + RRF + Rerank)

Phase 05 · Lab 01 · Phase README · Warmup

The problem

RAG is roughly 80% retrieval quality: if the right chunk never makes it into the context window, no prompt, no model, and no amount of "temperature tuning" recovers the answer. So the retriever is the part worth building by hand. This lab builds the whole pipeline from stdlib — no numpy, no FAISS, no embedding API — so every mechanism is visible:

1. Chunk a document into retrievable units (fixed-size with overlap, or sentence-packed).
2. Embed each chunk with a deterministic hashing bag-of-words embedder; compare with cosine.
3. Index the vectors in a brute-force dense cosine kNN.
4. Score the same chunks with BM25 from scratch (IDF + TF saturation + length norm).
5. Fuse the two rankings with Reciprocal Rank Fusion (RRF) — no score normalization.
6. Rerank the fused shortlist with an injected cross-encoder stand-in (token overlap).
7. Measure it: recall@k, precision@k, MRR over a labeled set.

The payoff is a deterministic demonstration of the single most important fact about retrieval: hybrid beats either retriever alone. On a crafted corpus, BM25 alone misses a match whose only shared words are common (its IDF zeroes them out), and the dense index alone misses a rare identifier (its bag-of-words vector spreads too thin) — yet the hybrid catches both, so hybrid recall strictly beats max(dense, bm25). That inequality is the business case for hybrid search, and here it's a passing test, not a slide.

What you build

File map

Run it

pytest test_lab.py -v                       # your lab.py (red until you implement)
LAB_MODULE=solution pytest test_lab.py -v   # the reference (green)
python solution.py                          # the worked example

Success criteria

• chunk_text honors overlap < size, and consecutive chunks share exactly overlap words; sentence_chunk never splits a sentence.
• embed is deterministic across runs (you used hashlib, not the salted builtin hash()) and L2-normalized; cosine(v, v) == 1.
• BM25.idf drives a term that appears in every document toward 0, and search ranks a doc containing a rare query term first.
• rrf is order-independent, breaks ties deterministically, and a doc appearing in both lists beats a doc that is #1 in only one.
• On the crafted demo set, hybrid recall@1 = 1.0 > 0.5 = dense-only = bm25-only, and you can explain which mechanism makes each baseline miss.
• All 31 tests pass under both lab and solution.

How this maps to the real stack

• Chunking is what LangChain's RecursiveCharacterTextSplitter, LlamaIndex's node parsers, and Unstructured do — with the same overlap tradeoff. Production tunes chunk size to the embedder's context and the corpus's structure (headings, code blocks, tables).
• The hashing embedder is a stand-in for a trained bi-encoder: sentence-transformers (all-MiniLM, bge-*), Cohere Embed, or OpenAI text-embedding-3. Those place paraphrases near each other; ours only captures token overlap in hash space. The interface (text → vector, compare by cosine) is identical — you'd swap one function.
• DenseIndex is the exact version of what FAISS, pgvector, Pinecone, Weaviate, Milvus, and Chroma do approximately with HNSW / IVF / PQ. Real indexes trade a little recall for orders-of-magnitude speed and add metadata filtering; the nearest-neighbor semantics you built here are exactly what they approximate.
• BM25 is what Elasticsearch / OpenSearch / Lucene and pgvector's ts_rank / Postgres full-text give you. Your idf/score are their scoring function, defaults k1=1.5, b=0.75 and all.
• RRF is the fusion Elasticsearch's rank retriever, Weaviate hybrid search, and pgvector-plus-BM25 stacks use to combine dense and lexical results without normalizing incompatible score scales. k_const=60 is the paper's default and most systems' too.
• Rerank is where Cohere Rerank, bge-reranker, and cross-encoder models slot in — run on the top-N a fast retriever proposed, exactly as here.

Limits of the miniature. A hashing bag-of-words embedder cannot match true synonyms (car ≈ automobile) the way a trained bi-encoder can — so our "semantic miss" is driven by BM25's IDF discarding common shared words, not by learned meaning. Real ANN indexes are approximate (recall < 100%), real rerankers are learned models, and real corpora need metadata filtering and security-trimming (Phase 13). The pipeline shape and the why-hybrid argument are exactly the production ones.

Extensions (your own machine)

• Swap the hashing embedder for a real bi-encoder (pip install sentence-transformers, one function change) and re-run the eval — watch true paraphrase matches appear.
• Replace DenseIndex with faiss or hnswlib and measure the recall/latency tradeoff vs the brute-force baseline on 100k vectors.
• Add metadata filtering (a where predicate) and a security-trim step (drop chunks the caller isn't authorized to see) before fusion — the #1 production concern.
• Add nDCG@k with graded relevance and compare rankers on a bigger labeled set.

Interview / resume signal

"Built a hybrid retriever from scratch — overlapping/sentence chunking, a hashing bag-of-words embedder with cosine, a brute-force dense kNN, BM25 (IDF + TF saturation + length norm), Reciprocal Rank Fusion, and a cross-encoder-style reranker — and proved on a crafted corpus that hybrid recall strictly beats either dense or lexical alone, because lexical catches rare keywords/IDs that dense dilutes and dense catches common-word overlap that BM25's IDF discards. Evaluated with recall@k / precision@k / MRR."