RAG Evaluation

Phase 9 · Document 09 · RAG Prev: 08 — Citations and Grounding · Up: Phase 9 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

RAG has many independently-tunable stages — chunking, embeddings, vector DB, hybrid, reranking, packing, grounding (01–08) — and you cannot improve what you don't measure. The single most valuable capability in RAG engineering is being able to answer: "when an answer is wrong, was it the retrieval or the generation?" — because the fix is completely different. RAG evaluation gives you that split, plus the metrics to tune each stage and a regression guard so a "tweak" doesn't silently break quality. Without it you optimize blind: swap an embedder on vibes, tweak chunking by guesswork, and ship regressions you discover only when users complain. This doc is what turns the rest of Phase 9 from guesswork into engineering, and it connects directly to the broader eval discipline (Phase 12).

2. Core Concept

Plain-English primer: evaluate the two halves separately

RAG = retrieval + generation, so eval has two halves, and keeping them separate is the core skill:

• Retrieval eval — did we fetch the right chunks? (Independent of the generator.)
• Generation eval — given the retrieved chunks, was the answer faithful, relevant, and complete? (Independent of whether retrieval was perfect.)

A wrong answer is either a retrieval miss or a generation failure (or both), and you fix them in different places. Conflating them — "the answer was wrong, let's try a bigger model" — is the most common RAG-tuning mistake when the real problem was a retrieval miss (00).

The RAG metric quadrant

The widely-used framing (popularized by Ragas) is four metrics, two per half:

Read the quadrant to localize a failure: low context recall → fix chunking/embeddings/hybrid (02/03/05); low context precision → add/tune reranking (06); low faithfulness → strengthen grounding/verify citations or change generator (08/Phase 5.05); low answer relevancy → prompt/packing (07).

Classic IR retrieval metrics (when you have labeled relevance)

If you have a labeled set (queries with their known-relevant chunks — "gold"), use standard information-retrieval metrics on the retriever's ranked output:

• Recall@k — fraction of relevant chunks found in the top-k. (The retrieval ceiling — if recall@k is low, no reranker/generator can fix it.)
• Precision@k — fraction of top-k that are relevant.
• MRR (Mean Reciprocal Rank) — 1/rank of the first relevant result, averaged. Rewards putting a relevant chunk high.
• nDCG@k — rank-aware quality that rewards relevant chunks ranked higher (the standard for graded relevance).

These are how you tune chunking/embeddings/hybrid/reranking objectively (02–06).

LLM-as-judge for generation metrics

Faithfulness and answer-relevancy are about meaning, so they're scored by an LLM-as-judge: give a model the question, the retrieved context, and the answer, and ask it to score (e.g., "is every claim supported by the context? 0–1"). Ragas/TruLens automate this. Caveats (Phase 12.02): judges are noisy and biased (position/verbosity/self-preference), so calibrate against human labels on a sample, use a strong judge model, and treat scores as directional — great for catching regressions, imperfect as absolute truth.

The golden dataset

Everything rests on a golden eval set: representative questions, ideally with ground-truth answers and gold relevant chunks. Build it from real user queries + a few hand-crafted hard cases; include unanswerable questions (to test refusal, 08). 50–200 well-chosen examples beat thousands of random ones. This is the same discipline as Phase 12.01 golden datasets and Phase 1.07.

Offline vs online

• Offline — run the golden set in CI on every change; gate merges on no-regression. This is your regression guard for chunking/embedder/reranker/prompt tweaks.
• Online — measure on live traffic: thumbs up/down, citation-click-through, escalation rate, and sampled LLM-judge faithfulness. Feed signals back into the golden set and retrieval improvements (10).

3. Mental Model

   WRONG ANSWER → which half?  (the core RAG-eval skill)
        ┌─────────────── RETRIEVAL ───────────────┐   ┌────────── GENERATION ──────────┐
        context RECALL: did we get ALL needed chunks?   FAITHFULNESS: answer supported by ctx? [08]
        context PRECISION: are retrieved chunks relevant? ANSWER RELEVANCY: addresses the question?
        (IR metrics w/ gold: recall@k · precision@k · MRR · nDCG)   (LLM-as-judge, calibrate vs humans)
        low recall → chunk/embed/hybrid [02/03/05]      low faithfulness → grounding/model [08/5.05]
        low precision → RERANK [06]                     low relevancy → prompt/packing [07]

   GOLDEN SET (real queries + gold chunks/answers + unanswerable) → OFFLINE (CI regression gate)
                                                                   → ONLINE (thumbs/escalation/sampled judge) [10]

Mnemonic: split retrieval vs generation — recall/precision (IR metrics + gold) vs faithfulness/relevancy (LLM-judge). A wrong answer is a retrieval miss or a generation failure; the quadrant tells you which to fix. Gate changes on a golden set.

4. Hitchhiker's Guide

What to look for first: a golden set and the retrieval-vs-generation split. With those, every other Phase 9 tweak becomes measurable; without them you tune blind.

What to ignore at first: perfect judge calibration and huge datasets. Start with ~50 golden examples (incl. unanswerable), recall@k for retrieval, and an LLM-judge faithfulness score; refine later.

What misleads beginners:

• Eval'ing only the final answer. You can't tell why it's wrong — split retrieval vs generation (00).
• No gold chunks. Without labeled relevant chunks you can't compute recall@k — the retrieval ceiling stays invisible.
• Trusting the judge as truth. LLM judges are noisy/biased — calibrate against humans and use for direction/regressions (Phase 12.02).
• No unanswerable questions. Then you never test refusal, and hallucination hides (08).
• Tuning without a regression gate. A chunking "improvement" can silently hurt other queries.

How experts reason: they build a representative golden set (with gold chunks + unanswerable cases), measure retrieval (recall@k/nDCG) and generation (faithfulness/relevancy) separately, localize failures via the quadrant, calibrate the judge against human labels, and run eval offline in CI as a regression gate plus online signals. They treat the golden set as a living asset fed by production failures (10).

What matters in production: the regression gate (no merge that drops faithfulness/recall), online faithfulness sampling, the retrieval ceiling (recall@k) being high enough, and a feedback loop turning failures into golden examples + fixes.

How to debug a wrong answer: (1) Was the right chunk retrieved? (recall@k / inspect context) — if no, fix retrieval (05/06). (2) If yes, is the answer faithful? — if no, fix grounding/model (08/Phase 5.05). (3) If faithful but unhelpful, fix relevancy (prompt/packing, 07). This three-step is the entire RAG-debugging loop.

Questions to ask: do we have a golden set with gold chunks + unanswerable? do we measure retrieval and generation separately? is the judge calibrated? is eval gating CI? what online signals feed back?

What silently gets expensive/unreliable: no golden set (blind tuning), answer-only eval (can't localize), uncalibrated judge (false confidence), no regression gate (silent regressions), and no online loop (drift unseen).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Evaluate retrieval and generation separately — a wrong answer is a retrieval miss or a generation failure, fixed in different places.
• The metric quadrant: context recall + context precision (retrieval); faithfulness + answer relevancy (generation) — popularized by Ragas.
• recall@k is the retrieval ceiling — if it's low, no reranker/generator can recover (05/06).
• Use IR metrics (recall@k/precision@k/MRR/nDCG) with gold relevant chunks to tune retrieval objectively.
• Faithfulness/relevancy are LLM-as-judge metrics — calibrate against humans; judges are noisy/biased (Phase 12.02).
• A golden set (real queries + gold chunks/answers + unanswerable cases) is the foundation — 50–200 good examples beat thousands of random.
• Run offline eval as a CI regression gate + online signals (thumbs/escalation/sampled judge), feeding failures back (10).
• The 3-step debug loop: right chunk retrieved? → faithful? → relevant? localizes any failure.

9. Observations from Real Systems

• Ragas and TruLens are the standard RAG-eval tools; the quadrant / RAG triad framing is ubiquitous in production.
• The decisive moment in most RAG debugging is discovering a "hallucination" was actually a retrieval miss — only the split reveals it (00).
• Teams that gate CI on a golden set ship far fewer regressions when swapping embedders/chunking/rerankers (02–06).
• Online signals (thumbs, citation clicks, escalations) are mined into new golden examples — the feedback loop (10).
• LLM-judge calibration matters — uncalibrated judges have mis-ranked configs; teams validate against human labels on a sample (Phase 12.02).

10. Common Misconceptions

11. Engineering Decision Framework

EVALUATE RAG:
 1. GOLDEN SET: real queries + gold relevant chunks + ground-truth answers + UNANSWERABLE cases (50–200).
 2. RETRIEVAL: recall@k / precision@k / MRR / nDCG on the retriever (needs gold chunks).  [05,06]
 3. GENERATION: faithfulness + answer relevancy via LLM-judge (CALIBRATE vs human labels). [08, 12.02]
 4. LOCALIZE via the quadrant:
       low recall → chunk/embed/hybrid [02/03/05] · low precision → rerank [06]
       low faithfulness → grounding/model [08/5.05] · low relevancy → prompt/pack [07]
 5. OFFLINE: run in CI; GATE merges on no-regression (faithfulness + recall@k).
 6. ONLINE: thumbs/escalation/citation-clicks + sampled judge → feed failures back into the golden set. [10]

12. Hands-On Lab

Goal

Build a RAG eval harness that scores retrieval and generation separately, localizes a failure via the quadrant, and gates a change.

Prerequisites

• A working RAG pipeline (00–08); pip install ragas datasets; a golden set (~30–50 questions with gold chunks + answers, incl. some unanswerable).

Steps

1. Build the golden set: for each question, record the gold relevant chunk id(s) and a ground-truth answer; include a few unanswerable questions.
2. Retrieval metrics: run the retriever; compute recall@k and MRR/nDCG vs the gold chunks. This is your retrieval ceiling (05/06).
3. Generation metrics: run the full pipeline; score faithfulness and answer relevancy with Ragas (LLM-judge); check the unanswerable questions trigger refusal (08).
4. Localize a failure: pick a wrong answer; apply the 3-step: was the gold chunk retrieved (recall)? if yes, is the answer faithful? bucket it as retrieval vs generation and propose the matching fix.
5. Calibrate the judge: hand-label faithfulness on ~10 examples; compare to the LLM-judge; note agreement and adjust trust (Phase 12.02).
6. Regression gate: make a change (e.g., smaller chunks, or remove reranking); re-run; show the metric that moves — demonstrating the gate catches regressions.

Expected output

An eval report: recall@k/nDCG (retrieval) + faithfulness/relevancy (generation) on the golden set; a localized failure (retrieval vs generation); a judge-vs-human calibration note; and a before/after showing a change's metric impact.

Debugging tips

• Faithfulness high but users unhappy → check answer relevancy and recall (right but unhelpful, or missing info).
• Judge scores look random → wrong/weak judge prompt or model; calibrate.

Extension task

Compare two pipeline configs (e.g., dense-only vs hybrid+rerank) on the full quadrant and pick the winner with data (05/06).

Production extension

Wire the harness into CI as a gate (block merges that drop faithfulness/recall) and add online sampling (thumbs + sampled judge) feeding failures back into the golden set (10).

What to measure

recall@k/precision@k/MRR/nDCG; faithfulness/answer-relevancy; refusal correctness; judge-human agreement; per-change metric deltas.

Deliverables

• A golden set (queries + gold chunks/answers + unanswerable).
• A retrieval + generation eval report (the quadrant).
• A localized failure (retrieval vs generation) + a regression-gate before/after.

13. Verification Questions

Basic

1. Why evaluate retrieval and generation separately?
2. Name the four quadrant metrics and which half each belongs to.
3. What does recall@k tell you, and why is it a ceiling?

Applied 4. Map each quadrant metric to the pipeline stage you'd fix if it's low. 5. Why include unanswerable questions in the golden set?

Debugging 6. An answer is wrong. Walk through the 3-step localization. 7. Your LLM-judge faithfulness scores disagree with users. What do you do?

System design 8. Design an offline + online RAG eval system with a CI regression gate and a feedback loop.

Startup / product 9. Why does a golden set + regression gate de-risk RAG product iteration and enterprise trust?

14. Takeaways

1. Evaluate retrieval and generation separately — a wrong answer is a retrieval miss or a generation failure, fixed differently.
2. The quadrant (context recall/precision · faithfulness/answer relevancy) localizes failures to a stage.
3. recall@k is the retrieval ceiling; use IR metrics with gold chunks to tune retrieval; faithfulness/relevancy via calibrated LLM-judge.
4. A golden set (real queries + gold chunks/answers + unanswerable) is the foundation; 50–200 good > thousands random.
5. Gate changes offline in CI + monitor online, feeding failures back — the 3-step debug loop (retrieved? faithful? relevant?) is the daily tool.

15. Artifact Checklist

• A golden set (queries + gold chunks + ground-truth answers + unanswerable).
• A retrieval + generation eval report (the quadrant; recall@k/nDCG + faithfulness/relevancy).
• A localized failure (retrieval vs generation) with the matching fix.
• A judge-vs-human calibration note.
• A CI regression gate + an online-feedback plan.

Up: Phase 9 Index · Next: 10 — Production RAG