RAG Evals

Phase 12 · Document 03 · Evaluation Prev: 02 — LLM-as-Judge · Up: Phase 12 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

RAG is the most common production LLM pattern (Phase 9), and it fails in a way that's invisible without the right eval: a confident, fluent answer that's ungrounded (hallucinated) or based on the wrong retrieved chunks. This doc applies the unified eval discipline (00–02) to RAG, with one move that defines RAG eval: separate retrieval from generation. When an answer is wrong you must know whether retrieval missed the chunk or generation ignored it — they're fixed in completely different places. (The mechanics live in Phase 9.09; here we frame RAG eval as an instance of the Phase 12 discipline — golden sets [01], judges [02], the harness [08] — and consolidate the metric set.)

2. Core Concept

Plain-English primer: evaluate the two halves separately

RAG = retrieval + generation, so RAG eval has two halves, and keeping them separate is the core skill (Phase 9.09):

• Retrieval eval — did we fetch the right chunks? (independent of the generator)
• Generation eval — given those chunks, was the answer faithful, relevant, complete? (independent of whether retrieval was perfect)

A wrong answer is either a retrieval miss or a generation failure (or both), and the fix differs — conflating them ("answer wrong → try a bigger model") is the #1 RAG-tuning mistake when the real problem was retrieval (Phase 9.00: retrieval dominates quality).

The RAG metric quadrant (Ragas framing)

Four metrics, two per half (Phase 9.09):

Read the quadrant to localize: low recall → chunking/embeddings/hybrid (Phase 9.02/9.03/9.05); low precision → reranking (Phase 9.06); low faithfulness → grounding/model (Phase 9.08); low answer-relevancy → prompt/packing (Phase 9.07).

Scorers: programmatic IR metrics + judge

Mapping to the Phase 12 scorer hierarchy (00):

• Retrieval = programmatic (when you have gold relevant chunks in the golden set [01]): recall@k, precision@k, MRR, nDCG — objective, cheap. recall@k is the ceiling — if the right chunk isn't retrieved, no model can fix it.
• Generation = LLM-as-judge (02): faithfulness and answer-relevancy are about meaning, so a (calibrated) judge scores them (Ragas/TruLens automate this). Calibrate against humans — judge bias applies here too.

The golden set for RAG

The golden set (01) for RAG carries extra fields: the question, gold relevant chunk IDs (for retrieval metrics), a ground-truth answer, and — crucially — unanswerable questions (to test refusal/hallucination, Phase 9.08). Build it from real queries + hard cases; grow it from production failures (the flywheel, 01).

The 3-step debugging loop

The daily RAG-eval tool (Phase 9.09): for a wrong answer —

1. Was the right chunk retrieved? (recall@k / inspect context) → if no, fix retrieval.
2. If yes, is the answer faithful? → if no, fix grounding/model.
3. If faithful but unhelpful → fix answer-relevancy (prompt/packing). This localizes every RAG failure to a stage.

Offline + online

Same as the discipline (00): offline run the golden set in CI, gate on faithfulness + recall@k when you change chunking/embedder/reranker/prompt (08); online sample faithfulness on live traffic + use thumbs/citation-clicks/escalations, feeding failures back (Phase 7.08, Phase 9.10).

3. Mental Model

   RAG = retrieval + generation → EVAL THE TWO HALVES SEPARATELY (the defining move)
   wrong answer = retrieval miss OR generation failure → fixed in DIFFERENT places [9.00 retrieval dominates]

   QUADRANT:  retrieval → context RECALL (misses?) + PRECISION (noise?)   [programmatic, needs GOLD chunks]
              generation → FAITHFULNESS (hallucination?) + ANSWER-RELEVANCY   [LLM-judge, calibrate [02]]
   localize: recall↓→chunk/embed/hybrid · precision↓→rerank · faithfulness↓→grounding/model · relevancy↓→prompt/pack

   SCORERS: retrieval = recall@k/precision@k/MRR/nDCG (recall@k = CEILING) ; generation = judge (calibrated)
   GOLDEN SET [01]: question + GOLD chunks + ground-truth answer + UNANSWERABLE ; OFFLINE gate + ONLINE feedback
   3-STEP DEBUG: retrieved? → faithful? → relevant?

Mnemonic: separate retrieval from generation — the metric quadrant (recall/precision · faithfulness/relevancy) localizes every failure. recall@k is the ceiling; retrieval = programmatic, generation = calibrated judge. Debug: retrieved? → faithful? → relevant?

4. Hitchhiker's Guide

What to look for first: are you measuring retrieval and generation separately, and do you have gold relevant chunks (for recall@k) + unanswerable cases? Those make RAG eval diagnostic instead of opaque.

What to ignore at first: exotic trajectory metrics. Start with recall@k + faithfulness + answer-relevancy on a small golden set; expand later. (Full mechanics: Phase 9.09.)

What misleads beginners:

• Eval'ing only the final answer. You can't tell why it failed — split retrieval vs generation (Phase 9.09).
• No gold chunks. Without labeled relevant chunks you can't compute recall@k — the retrieval ceiling stays invisible.
• Treating "hallucination" as a model bug. Most are retrieval misses — check recall first (Phase 9.00).
• Trusting the faithfulness judge. Calibrate it (02).
• No unanswerable cases. You never test refusal; hallucination hides (Phase 9.08).

How experts reason: they build a RAG golden set (01) with gold chunks + unanswerable items, measure retrieval (programmatic IR metrics) and generation (calibrated judge) separately, localize failures via the quadrant + 3-step loop, gate changes in CI (faithfulness + recall@k), and feed online failures back (Phase 9.10).

What matters in production: retrieval recall (the ceiling), faithfulness (no hallucination), per-stage localization, and the regression gate catching chunking/embedder/reranker/prompt changes (08).

How to debug/verify: run the 3-step loop — retrieved? faithful? relevant? — to localize; if recall@N is low, fix retrieval (Phase 9.05/9.06) before touching the generator.

Questions to ask: do we eval retrieval and generation separately? gold chunks for recall@k? faithfulness judge calibrated? unanswerable cases included? CI gate + online faithfulness sampling?

What silently gets expensive/unreliable: answer-only eval (can't localize), no gold chunks (invisible retrieval ceiling), uncalibrated faithfulness judge, and missing unanswerable cases (hidden hallucination).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Evaluate retrieval and generation separately — the defining move; a wrong answer is a retrieval miss or a generation failure, fixed differently (Phase 9.09).
• The quadrant: context recall + context precision (retrieval); faithfulness + answer-relevancy (generation) — localizes failures to a stage.
• recall@k is the retrieval ceiling — if the right chunk isn't retrieved, no model can recover it (Phase 9.00).
• Retrieval = programmatic IR metrics (recall@k/precision@k/MRR/nDCG) with gold chunks; generation = calibrated LLM-judge (faithfulness/relevancy) (02).
• Most "hallucination despite RAG" is a retrieval miss — check recall first.
• The RAG golden set needs gold chunks + ground-truth answers + unanswerable cases (01, Phase 9.08).
• The 3-step loop (retrieved? → faithful? → relevant?) localizes every failure.
• Gate offline (faithfulness + recall@k) + monitor online, feeding failures back (Phase 9.10).

9. Observations from Real Systems

• Ragas (the quadrant) and TruLens (the RAG triad) are the standard RAG-eval tools — the unified discipline (00) applied to RAG.
• The decisive debugging moment is discovering a "hallucination" was a retrieval miss — only the split reveals it (Phase 9.09).
• Teams that gate CI on faithfulness + recall@k ship far fewer RAG regressions when swapping embedders/chunking/rerankers (Phase 9.02–9.06).
• Online signals (thumbs, citation clicks, escalations) are mined into new golden examples — the flywheel (Phase 9.10).
• Faithfulness judges need calibration — uncalibrated, they miss subtle ungrounded claims (02).

10. Common Misconceptions

11. Engineering Decision Framework

EVALUATE A RAG SYSTEM:
 1. GOLDEN SET [01]: question + GOLD relevant chunks + ground-truth answer + UNANSWERABLE cases.
 2. RETRIEVAL (programmatic): recall@k / precision@k / MRR / nDCG (recall@k = ceiling) [9.05/9.06].
 3. GENERATION (calibrated judge [02]): faithfulness + answer-relevancy (Ragas/TruLens).
 4. LOCALIZE via quadrant + 3-step loop: recall↓→chunk/embed/hybrid · precision↓→rerank ·
    faithfulness↓→grounding/model · relevancy↓→prompt/pack [9.02–9.08].
 5. OFFLINE gate (faithfulness + recall@k) in CI; ONLINE sample faithfulness + thumbs/citations → feed back [9.10].

12. Hands-On Lab

Goal

Build a RAG eval that scores retrieval and generation separately and localizes a failure via the 3-step loop — the Phase 12 discipline applied to RAG.

Prerequisites

• A RAG pipeline (Phase 9); pip install ragas; a golden set (01) with gold chunks + ground-truth answers + unanswerable cases.

Steps

1. Retrieval metrics: run the retriever; compute recall@k + MRR/nDCG vs gold chunks (the ceiling) (Phase 9.05).
2. Generation metrics: run the full pipeline; score faithfulness + answer-relevancy with a calibrated judge (Ragas, 02); verify unanswerable questions trigger refusal.
3. Localize: pick a wrong answer; apply the 3-step loop (retrieved? → faithful? → relevant?) and bucket it as retrieval vs generation.
4. Calibrate: hand-label faithfulness on ~8 examples; compare to the judge; note agreement (02).
5. Gate: change chunk size (or remove reranking); re-run; show the metric that moves (the regression gate, 08).

Expected output

A retrieval + generation eval report (quadrant), a localized failure (retrieval vs generation), a judge-calibration note, and a regression-gate before/after.

Debugging tips

• Faithfulness high, users unhappy → check answer-relevancy + recall (right but unhelpful, or missing info).
• recall@k low everywhere → fix retrieval (hybrid/rerank, Phase 9.05/9.06).

Extension task

Compare two pipeline configs (dense-only vs hybrid+rerank) across the full quadrant (Phase 9.05/9.06).

Production extension

Wire the quadrant into a CI gate + online faithfulness sampling feeding the golden set (Phase 9.10, 08).

What to measure

recall@k/precision@k/MRR/nDCG; faithfulness/answer-relevancy; refusal correctness; judge-human agreement; per-change deltas.

Deliverables

• A retrieval + generation eval report (the quadrant).
• A localized failure (retrieval vs generation) + a regression-gate before/after.

13. Verification Questions

Basic

1. Why evaluate RAG retrieval and generation separately?
2. Name the quadrant metrics and which half each is.
3. Why is recall@k the retrieval ceiling?

Applied 4. Map each quadrant metric to the pipeline stage you'd fix. 5. Which metrics are programmatic vs judge-scored, and why?

Debugging 6. An answer hallucinates despite good docs existing. First diagnostic step? 7. Right chunk retrieved but answer still wrong. Where's the bug?

System design 8. Design a RAG eval (golden set with gold chunks, quadrant scorers, CI gate, online feedback).

Startup / product 9. Why does the retrieval-vs-generation split de-risk RAG iteration and product trust?

14. Takeaways

1. Separate retrieval from generation — the defining RAG-eval move; localize failures with the quadrant + 3-step loop.
2. The quadrant: context recall/precision (retrieval) · faithfulness/answer-relevancy (generation).
3. recall@k is the ceiling; retrieval = programmatic IR metrics (gold chunks), generation = calibrated judge (02).
4. Most hallucination-despite-RAG is a retrieval miss — check recall first.
5. Golden set needs gold chunks + unanswerable cases; gate offline + feed online failures back (Phase 9.10).

15. Artifact Checklist

• A RAG golden set (gold chunks + ground-truth + unanswerable).
• Retrieval metrics (recall@k/precision@k/MRR/nDCG).
• Generation metrics (faithfulness + answer-relevancy, calibrated judge).
• A localized failure (3-step loop) + a regression-gate before/after.
• A CI gate + online feedback plan.

Up: Phase 12 Index · Next: 04 — Agent Evals