« Phase 05 README · Warmup · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes
Phase 05 — Hitchhiker's Guide
The compressed practitioner tour. Read the WARMUP for the derivations; this is the stuff you say in the meeting.
30-second mental model
RAG is ~80% retrieval quality, and everyone under-invests in it. The retriever is a pipeline,
not a similarity_search call: chunk → embed → index (dense) + BM25 → RRF fuse → rerank →
top-k → measure. Dense and lexical fail in opposite directions — dense catches paraphrase
and dilutes rare tokens; BM25 catches rare tokens/IDs and misses paraphrase — so you run both
and fuse them, and hybrid provably beats either alone. Retrieve wide and cheap (bi-encoder +
BM25), rerank narrow and precise (cross-encoder). And you measure retrieval separately from
generation (recall@k), or you'll debug the wrong half at 2 a.m.
The numbers & defaults to tattoo on your arm
| Thing | Value / rule |
|---|---|
| BM25 defaults | k1 = 1.5 (TF saturation), b = 0.75 (length norm) |
| BM25 IDF | ln(1 + (N − df + 0.5)/(df + 0.5)) — rare term → big, corpus-wide term → ~0 |
| RRF | Σ 1/(k + rank), k = 60; fuses rank, not score; a doc in both lists beats a #1-in-one |
| cosine = dot | only on L2-normalized vectors — so normalize, then use the cheaper dot |
| exact kNN cost | O(N · dim) per query — fine to ~10⁵, use ANN beyond |
| the metric for RAG | recall@k — an un-retrieved chunk can never reach the LLM |
| two-stage | retrieve top ~100 (recall), rerank to top ~5 (precision) |
| chunking | size is a tradeoff; add overlap so a boundary-straddling fact survives |
Framework one-liners
- pgvector = vectors inside Postgres, next to your SQL + row-level security. The enterprise
default.
CREATE INDEX ... USING hnsw. - Pinecone = managed/serverless vectors, zero ops. Weaviate / Qdrant = open-source with built-in hybrid + filtering. FAISS = a library (HNSW/IVF/PQ), not a server. Chroma = the SQLite of vector DBs (prototypes). Milvus = billions of vectors. Neo4j = graph + vectors → the door to GraphRAG (Phase 06).
- ANN = HNSW (greedy graph descent, the default), IVF (centroid cells), PQ (compress vectors) — trade a little recall for 10–1000× speed.
- Rerankers = Cohere Rerank,
bge-reranker— cross-encoders you run on the shortlist. - Embedders =
all-MiniLM/bge/ Cohere Embed / OpenAItext-embedding-3— bi-encoders; swap one function to replace the lab's hashing stand-in.
War stories
- The gateway agent that hallucinated a fix. Pure dense retrieval returned three plausible
gateway runbooks but not the one naming error
TX4096— the rare token was one diluted vector dimension. Adding BM25 + RRF put the exact-match runbook on top. Rare identifiers are a lexical job. - The "improve the prompt" death march. Two weeks tuning the prompt for a RAG bot that kept missing answers. First recall@k check: the relevant chunk was never retrieved — chunks were 2000 tokens of averaged mush. Halving chunk size fixed it in an afternoon. Measure retrieval first.
- The score-addition bug. Someone "fused" dense and BM25 by adding
cosine + bm25_score. BM25 scores dwarfed cosine, so it was BM25-only with extra steps. Switched to RRF (rank-based) and recall jumped. You cannot add incompatible scales. - The cross-tenant leak. One shared index, no tenant filter; a demo query surfaced another customer's doc into the answer. Namespaces per tenant, mandatory tenant-id filter. This one ends careers — see Phase 13.
Vocabulary
Chunk (retrievable unit) · Embedding / vector (text → numbers, meaning ≈ closeness) · Bi-encoder (embed separately, indexable) · Cross-encoder (score the pair together, rerank) · Cosine / dot (equal when normalized) · kNN / ANN (exact vs approximate nearest-neighbor) · HNSW / IVF / PQ (the ANN tricks) · BM25 (IDF · TF-saturation · length-norm) · IDF (rare = informative) · Hybrid (dense + lexical) · RRF (fuse by rank) · recall@k / precision@k / MRR / nDCG (retrieval metrics) · security-trimming (filter by who's asking).
Beginner mistakes
- Treating retrieval as one
similarity_searchline instead of a tuned pipeline. - Using dense-only and being blind to rare keywords/IDs (or BM25-only and blind to paraphrase).
- Adding cosine and BM25 scores instead of fusing by rank (RRF).
- Chunking too big (blurry embeddings) or with no overlap (boundary-straddling facts lost).
- Skipping the reranker, then blaming the model for bad ordering.
- Debugging the prompt when recall@k would have shown the chunk was never retrieved.
- Sharing one vector index across tenants with no isolation filter.
- Forgetting that switching embedding models invalidates every stored vector.