Phase 04 — Semantic Retrieval & Search
Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 2 weeks (35–45 hours) Roles Supported: the JD's "search … semantic retrieval" — and the candidate- generation stage of every ranking/recsys/RAG system in phases 05, 06, and 10
Why This Phase Exists
Retrieval is the universal first stage: search ranks what retrieval found, recommenders rank retrieved candidates, RAG generates from retrieved passages. A senior MLE owns retrieval as an engineered system — lexical and dense scorers, fusion, indexes, and above all an evaluation harness with labeled queries, because retrieval quality upper-bounds everything downstream and is invisible without measurement.
This phase is the search-engineering counterpart of the LLM track's RAG phase (its Phase 07 covers RAG assembly and ANN internals): here you build the scorers and the harness from scratch and own the evaluation methodology.
Concepts
- The retrieval problem: queries, documents, relevance; recall-oriented first stage
- BM25 from first principles: IDF, term-frequency saturation (k1), length normalization (b) — implemented, not imported
- Dense retrieval: embeddings, cosine/dot equivalence under normalization, bi-encoder asymmetry (the LLM track covers training; here we operate them)
- Hybrid fusion: reciprocal rank fusion (RRF) and why rank-based fusion wins
- Index structures at a working level: inverted index, HNSW/IVF trade-offs
- Retrieval evaluation: qrels, recall@k, MRR, precision@k; building a labeled set
- Query understanding lite: normalization, analyzers, synonyms — where lexical systems are actually won
Labs
Lab 01 — Hybrid Retrieval Engine (flagship, implemented)
| Field | Value |
|---|---|
| Goal | Build BM25 (inverted index + scoring) and a dense index from scratch, fuse with RRF, and evaluate with recall@k/MRR on a labeled query set — demonstrating the case where hybrid beats both parents |
| Concepts | IDF/saturation/length-norm mechanics, cosine top-k, rank fusion, qrels-based evaluation |
| Steps | 1. Tokenizer + inverted index; 2. BM25 scoring with k1/b; 3. dense index (normalized vectors, top-k); 4. RRF fusion; 5. eval harness (recall@k, MRR); 6. the lexical-gap and paraphrase-gap test cases where each parent fails and hybrid survives |
| How to Test | pytest test_lab.py — 12 tests incl. BM25 property tests (rarity, saturation, length) and the hybrid-beats-parents construction |
| Talking Points | Why does IDF use a log? What does b=0 do? Why fuse ranks instead of scores? What does recall@k of the first stage bound? |
| Resume Bullet | Implemented a hybrid lexical+dense retrieval engine (BM25 and cosine indexes from scratch, RRF fusion) with a qrels evaluation harness; demonstrated hybrid recall gains over both single retrievers on adversarial query classes |
→ Lab folder: lab-01-hybrid-retrieval/
Extension Project A — ANN Benchmark (spec)
Index 100K+ real embeddings (any open set) with hnswlib and FAISS-IVF; measure recall@10-vs-QPS curves across efSearch/nprobe; reproduce the recall-latency trade curve and write the "which index when" memo.
Extension Project B — Query-Set Construction (spec)
Build a 50-query labeled set for a corpus you care about (docs, wiki dump): sample real-ish queries, label relevant docs, compute inter-annotator agreement on a subset — then measure your lab's engine on it. The artifact every retrieval team needs and few have.
Guides in This Phase
Deliverables Checklist
- Lab 01 implemented; all 12 tests pass
- Extension A curves + memo
- Extension B labeled set + baseline numbers