Lab 01 — Hybrid Retrieval Engine

Phase: 04 — Semantic Retrieval & Search | Difficulty: ⭐⭐⭐⭐☆ | Time: 5–7 hours

BM25 and dense retrieval fail on opposite query classes — IDs vs paraphrases. This lab builds both from scratch, fuses them with RRF, and proves the complementarity with adversarial tests where each parent fails and the hybrid survives.

What you build

  • BM25Index — inverted index + the full scoring function, with property tests pinning each term's behavior: IDF rarity ordering, TF saturation (6× occurrences scores < 3× — sublinearity verified), and length normalization (b=0.75 penalizes padding; b=0 provably doesn't)
  • DenseIndex — L2-normalized vectors, exact cosine top-k, with a scale-invariance test (cosine's defining property)
  • rrf_fuse — rank-based fusion: scale-free (a test rescales scores 10,000× and asserts identical output), agreement-rewarding
  • The eval harness — recall@k, MRR, macro-averaged evaluate() over qrels, verified on hand-computed cases

Key concepts

ConceptWhat to understand
Two-stage architectureFirst-stage recall@k bounds the whole system — measure it separately
BM25's three termsEach fixes a failure: IDF (common-word dominance), k1 (linear TF), b (length bulk)
Analyzer parityOne tokenize, two call sites — the lexical version of training-serving skew
RRFRanks not scores → no calibration across incommensurable scorers
Adversarial slicesAggregate recall hides which retriever fails where; the ID-query and paraphrase-query tests are slices

Files

FilePurpose
lab.pySkeleton with # TODO markers
solution.pyComplete reference; python solution.py demos both query classes
test_lab.py12 tests incl. the hybrid-survives-both construction
requirements.txtnumpy, pytest

Run

pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v

Success criteria

  • All 12 tests pass — test_hybrid_survives_both_adversarial_query_classes is the thesis
  • You can predict the effect of k1→0 and b→0 before re-running the property tests with those values (then verify)
  • You can state what RRF's c=60 does and what happens at c=0

Extensions

  • Weighted RRF with per-retriever weights tuned on a dev query set — measure whether it beats plain RRF on held-out queries (the tuning-data honesty check)
  • Phrase/exact-field support for the ID class (never stem a SKU)
  • Swap the toy vectors for a real sentence-transformer and re-run the paraphrase slice (mind the query/passage prefixes!)

References

  • Robertson & Zaragoza (2009) — BM25's derivation
  • Cormack et al. (SIGIR 2009) — RRF
  • WARMUP.md chapters 2–6, 8