Warmup Guide — Retrieval, RAG & Agents

Zero-to-expert primer for Phase 07: grounding LLMs in external knowledge — sparse and dense retrieval, chunking, vector indexes, the RAG pipeline's failure points, and the agent loop — assuming Phase 02's embedding foundations.

Table of Contents

• Chapter 1: Why Retrieval — The Knowledge Problem
• Chapter 2: Sparse Retrieval — BM25, Still Undefeated
• Chapter 3: Dense Retrieval — Bi-Encoders and Their Training
• Chapter 4: Approximate Nearest Neighbors — HNSW and IVF
• Chapter 5: The RAG Pipeline — Chunking to Generation
• Chapter 6: Where RAG Fails — A Debugging Taxonomy
• Chapter 7: Rerankers and Hybrid Search
• Chapter 8: Agents — The Loop, Tools, and Failure Containment
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Why Retrieval — The Knowledge Problem

An LLM's knowledge is frozen at training time, stored diffusely in weights, unattributable, and expensive to update (Phase 06 Ch. 7: fine-tuning injects behavior, not reliable facts). Retrieval-Augmented Generation splits the job: a retriever finds relevant text from a controllable corpus; the LLM reads it in-context and synthesizes. What this buys, precisely: updatable knowledge (re-index, done), attribution (citations to real passages), access control (retrieve only what this user may see), and a hallucination reduction (not elimination — Ch. 6). The price: you now operate a search engine in front of your LLM, and the search engine's quality bounds the system — garbage retrieved, garbage generated. This phase is mostly about the search engine.

Chapter 2: Sparse Retrieval — BM25, Still Undefeated

BM25 is TF-IDF (Phase 02 Ch. 2) with two refinements, and it remains the baseline that embarrasses fancy systems:

$$\text{BM25}(q, d) = \sum_{t \in q} \text{IDF}(t) \cdot \frac{tf(t,d),(k_1 + 1)}{tf(t,d) + k_1\left(1 - b + b,\frac{|d|}{\text{avgdl}}\right)}$$

• Term-frequency saturation ($k_1 \approx 1.2$): the 10th occurrence of a term adds far less than the 2nd — relevance isn't linear in counts.
• Length normalization ($b \approx 0.75$): long documents accumulate matches by bulk; penalize proportionally.

Why it endures: exact lexical match carries intent — IDs, error codes, function names, rare entities ("XK-2447 timeout") are precisely what dense embeddings blur and exactly what users search for. Zero training, interpretable scores, mature infrastructure (Lucene/Elasticsearch). The professional default: BM25 is the baseline every dense system must beat on your corpus, and hybrid (Ch. 7) usually beats both.

Chapter 3: Dense Retrieval — Bi-Encoders and Their Training

Dense retrieval embeds queries and documents into one vector space (Phase 02's machinery, industrialized): a bi-encoder encodes each independently — documents offline into an index, the query at request time — similarity = dot product.

What makes a retrieval embedding model (vs generic sentence similarity):

• Trained contrastively on (query, relevant-doc) pairs: InfoNCE loss pulls true pairs together against in-batch + hard negatives (BM25-retrieved near-misses — the ingredient that most improves quality; DPR's central lesson).
• Asymmetry: queries are short questions, documents are long prose — models use instruction prefixes ("query: …" / "passage: …", or task instructions in modern e5/bge/gte models) to condition the encoder per side. Omitting the prefix at inference silently costs recall — a top-3 production bug.
• Pooling: CLS or mean over tokens → one vector; L2-normalized so dot = cosine (Phase 02 Ch. 3's equivalence, now operational).
• The bi-encoder's structural weakness: query and document never interact during encoding — one vector must anticipate every question a document answers. That ceiling is what rerankers fix (Ch. 7).

Chapter 4: Approximate Nearest Neighbors — HNSW and IVF

Exact top-k over N vectors is O(N·d) per query — fine to ~1M, then you buy speed with recall:

• HNSW (the default): a multi-layer skip-list-like proximity graph — search greedily descends from sparse upper layers to the dense bottom layer. Sub-millisecond at ~95–99% recall@10. Knobs: M (edges/node — memory vs recall), efConstruction (build quality), efSearch (query-time recall vs latency — the one you tune live). Costs: RAM-resident (graph + vectors), slow builds, deletes are awkward (tombstones + rebuild).
• IVF (inverted file): k-means the corpus into nlist cells; search probes the nprobe nearest cells only. Cheaper memory, faster builds, natural for disk/segmented systems; recall cliff if the true neighbor sits in an unprobed cell. Usually paired with PQ (product quantization — compress vectors ~16–64×; the quantization worldview of the model-accuracy track, applied to the index itself) for billion-scale.
• The two facts to keep: recall@k of the ANN layer upper-bounds end-to-end RAG quality (measure it against brute force on a sample — it's one of the lab's checks), and filtering interacts badly with ANN (metadata-filtered search degrades graph traversal; engines differ wildly here — ask this question of any vector DB you evaluate).

Chapter 5: The RAG Pipeline — Chunking to Generation

The full assembly (Lab 02 builds it end-to-end):

1. Ingest & chunk: split documents into retrieval units. The tension: small chunks (200–400 tokens) embed precisely but lose context; large chunks (1–2K) keep context but blur the embedding and waste prompt budget. Structure-aware splitting (headings, paragraphs, code blocks) with ~10–15% overlap is the sane default; chunk-expansion (retrieve small, hand the LLM the surrounding section) decouples the two needs and is the single highest-leverage upgrade.
2. Embed & index: batch-embed chunks (with the passage prefix!), store vectors + text + metadata (source, position — you need provenance for citations).
3. Retrieve: embed query (query prefix!), ANN top-k (k≈20–50, over-retrieve for the reranker).
4. Rerank (Ch. 7) → top 3–8 into the prompt.
5. Generate: a prompt that instructs grounding ("answer from the passages; say 'not found' if absent; cite [1][2]") — the instruction measurably matters.
6. Evaluate (the part everyone skips): retrieval metrics (recall@k, MRR against a labeled set — even 50 hand-labeled queries transform your ability to iterate) and generation metrics (faithfulness: are claims supported by the retrieved text? LLM-as-judge with spot-checking is the pragmatic tool — Phase 08 formalizes).

Chapter 6: Where RAG Fails — A Debugging Taxonomy

Diagnose by stage — each failure has a distinct signature (the lab plants several):

The meta-rule (same as every debugging chapter in this curriculum): bisect by stage with fixed inputs — log query → retrieved chunks → prompt → answer for every request, and you can localize any complaint in minutes; skip the logging and every incident is archaeology.

Chapter 7: Rerankers and Hybrid Search

• Hybrid search: run BM25 and dense in parallel, fuse with Reciprocal Rank Fusion: $\text{RRF}(d) = \sum_r 1/(60 + \text{rank}_r(d))$ — rank-based, no score calibration needed between incommensurable scorers, robust default. Hybrid covers Ch. 6's lexical gap and the dense model's paraphrase strength simultaneously; it's the standard production answer.
• Cross-encoder rerankers: feed (query, document) jointly through a transformer → relevance score. Full token-level interaction — far more accurate than bi-encoders — but O(pairs) inference, so it only reranks the top 20–100. The two-stage pattern (cheap recall → expensive precision) is the same architecture as every ranking system from web search to ads; recognizing it as such is the senior framing. Late-interaction models (ColBERT) sit between: token-level vectors with cheap MaxSim scoring.

Chapter 8: Agents — The Loop, Tools, and Failure Containment

An agent is an LLM in a loop with tools:

while not done: thought → tool call → observation → (repeat) → answer

(ReAct's pattern, now native via structured tool-calling APIs — the model emits JSON conforming to tool schemas; the runtime executes and returns observations.)

What the engineer actually owns:

• Tool design dominates agent quality: few, orthogonal tools with crisp descriptions, typed arguments, and informative error returns (the model reads errors and retries — a good error message is agent UX). RAG-as-a-tool ("search the KB") composing with this loop is the standard enterprise assistant shape.
• Failure containment: max-iteration caps (loops happen), timeouts, idempotent or confirmation-gated side-effecting tools, sandboxed execution, and observability — full traces of every thought/call/observation, because debugging an agent without traces is impossible.
• Honest economics: each loop iteration is an LLM call with a growing context — latency and cost compound; multi-step agents amplify single-step error rates (0.95⁵ ≈ 0.77). Use the dumbest sufficient pattern: a fixed pipeline beats an agent when the workflow is known; agents earn their cost when the path is genuinely dynamic.

Lab Walkthrough Guidance

Lab 02 — RAG Pipeline:

1. Corpus + labeled eval queries first (~30–50 questions with known source passages). Building eval before the pipeline is the discipline that makes every later step measurable.
2. BM25 baseline; record recall@5/MRR. This number is the bar.
3. Dense path: embed with prefixes, exact search first (no ANN) — measure; then add the ANN index and measure the recall delta at several efSearch values (Ch. 4's upper-bound check, made personal).
4. Hybrid RRF; then a cross-encoder reranker on top-30 — by now you have a five-row ablation table (BM25 / dense / hybrid / +rerank / oracle), which is the lab's deliverable.
5. Wire generation with grounding instructions + citations; run the faithfulness check on 20 answers; deliberately break one stage (wrong prefix, tiny efSearch) and confirm your logging localizes it (Ch. 6's bisection drill).

Success Criteria

You are ready for Phase 08 when you can, from memory:

1. Write BM25's formula and explain saturation and length normalization; state why lexical match still matters.
2. Describe bi-encoder training (InfoNCE, hard negatives, prefixes) and its structural ceiling vs cross-encoders.
3. Compare HNSW and IVF-PQ on memory/build/recall/deletes, and name the filter-interaction caveat.
4. Defend a chunking strategy including the small-vs-large tension and chunk expansion.
5. Recite six rows of the failure taxonomy with signatures.
6. Explain RRF (why rank-based) and the two-stage retrieve-rerank economics.
7. State the agent loop, two containment mechanisms, and the error-compounding arithmetic.

Interview Q&A

Q: Your RAG system answers correctly for verbatim queries but fails on paraphrases — yet your embedding benchmark scores are great. Diagnose. Suspects in order: (1) missing instruction prefixes at inference (benchmarks used them, your service doesn't — silent recall loss); (2) domain shift — the benchmark isn't your corpus; build the 50-query labeled set and measure your recall@k; (3) chunking that splits answers (the retrieval is "right" but partial); (4) ANN recall loss masking as model failure (compare against brute force). The structure — measure per stage against your own labels before swapping models — is the answer being graded.

Q: When is RAG the wrong tool? When the knowledge is behavioral (format, style, procedures → fine-tune, Phase 06); when the corpus fits comfortably in context (long-context stuffing beats a retrieval pipeline's complexity below ~50–100K tokens of stable docs); when queries need aggregation over many documents ("how many customers complained about X" — that's analytics/SQL, not top-k retrieval); and when latency budgets can't fit embed+search+rerank+generate. Naming retrieval's aggregation blindspot is the differentiator.

Q: How would you evaluate a RAG system end to end? Layered: ANN recall vs brute force (index health); retrieval recall@k/MRR on labeled query→passage pairs (the bottleneck metric); reranker NDCG on the same; generation faithfulness (claims supported by context — LLM-judge with human spot-checks) and answer correctness on a QA set; plus abstention quality (does it say "not found" when the corpus lacks the answer — measured with deliberately unanswerable queries). One labeled set of ~100 queries powers all of it; refusing to operate without one is the senior move.

References

• Robertson & Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond (2009)
• Karpukhin et al., Dense Passage Retrieval (2020) — arXiv:2004.04906
• Malkov & Yashunin, HNSW (2016) — arXiv:1603.09320
• Jégou et al., Product Quantization for Nearest Neighbor Search (2011)
• Lewis et al., Retrieval-Augmented Generation (2020) — arXiv:2005.11401
• Liu et al., Lost in the Middle (2023) — arXiv:2307.03172
• Cormack et al., Reciprocal Rank Fusion (SIGIR 2009)
• Khattab & Zaharia, ColBERT (2020) — arXiv:2004.12832
• Yao et al., ReAct: Synergizing Reasoning and Acting (2022) — arXiv:2210.03629
• Anthropic: Building effective agents — the use-the-dumbest-sufficient-pattern argument, from production experience