Evaluation — Overview

Phase 12 · Document 00 · Evaluation Prev: Phase 11 — AI Coding Platforms · Up: Phase 12 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

You cannot improve, choose, ship, or trust an LLM system you don't measure. "Vibe checks" are not engineering — they don't tell you whether a model swap helped or hurt, whether a prompt change regressed, or whether production quality is drifting. Evaluation is the discipline that turns LLM development from guesswork into engineering, and it's the competitive moat: every model improves, but your golden dataset and eval harness — encoding what good looks like for your task — are uniquely yours and compound over time. You've already met domain-specific eval (RAG Phase 9.09, agents Phase 10.09, code Phase 11.08); this phase is the unified discipline underneath them — golden sets, LLM-as-judge, the eval types, and the model-selection harness that ties it all together.

2. Core Concept

Plain-English primer: measure performance on a defined task

Evaluation measures how well your system performs on a defined set of tasks, so you can answer: Is this change better or worse? Which model should I use? Is production drifting? It's the empirical backbone of Phase 5 model selection and the gate on every change.

The first principle (and the most violated): evaluate on YOUR task, not on benchmarks.

Benchmark vs eval (don't confuse them)

• Benchmark — a standard, public test (MMLU, SWE-bench, MTEB) for comparing models across the field. Useful to shortlist — but it's someone else's task, prone to contamination (models trained on it) and distribution mismatch (≠ your data).
• Eval — your test on your task/data, deciding whether a model/prompt/system works for you. Benchmarks pick candidates; your eval decides. (Phase 11.08, Phase 9.09)

The eval types (when you run what)

Offline gates changes; online catches drift and feeds new examples back. You need both.

The two foundational pieces (everything builds on these)

1. Golden dataset (01) — a curated set of representative tasks (+ expected outputs or criteria), including edge cases and unanswerable items, versioned, and never used for training. The single most valuable eval artifact.
2. The scorer — how you grade an output:
   • Programmatic/objective (preferred when possible): exact match, tests pass, JSON-valid, recall@k. Cheap, reliable. Coding eval leans here (Phase 11.08).
   • LLM-as-judge (02): a strong model scores subjective quality (helpfulness, faithfulness). Scalable but biased/noisy — calibrate against humans.
   • Human — the gold standard for nuance; expensive, so reserve and use to calibrate the judge.

A metric per task (the recurring theme)

There's no single "quality" number — each task type has its own metric(s) (03–07):

• RAG → faithfulness, answer-relevancy, context precision/recall (03)
• Agents → task success, trajectory, safety (04)
• Code → task-resolution, apply-rate, acceptance (05)
• Plus the universal axes: latency, cost (06) and safety (07). The model-selection harness (08) combines these into one weighted decision (Phase 5.09).

Eval is a system, not a one-time check

Mature eval is continuous: offline gate in CI on every change, online monitoring + feedback loop turning production failures into new golden examples, dashboards for trends. The teams that win eval continuously — it's the difference between confident iteration and flying blind.

3. Mental Model

   can't improve/choose/ship/trust what you don't MEASURE → eval = engineering (not vibe checks)
   ★ evaluate YOUR task, not benchmarks:  BENCHMARK (public, compare models, beware contamination) PICKS;
                                          YOUR EVAL (your data/task) DECIDES

   FOUNDATIONS:  GOLDEN SET [01] (representative + edge + unanswerable, versioned, never train)
                 + SCORER: programmatic (preferred) > LLM-JUDGE (calibrate [02]) > human (gold/calibrate)

   A METRIC PER TASK: RAG [03] · agents [04] · code [05] · + latency/cost [06] · + safety [07]
        → combine into a weighted MODEL-SELECTION HARNESS [08] (Phase 5.09)

   TYPES: OFFLINE (CI regression GATE) + ONLINE (drift + feedback loop) + RED-TEAM/safety [07]
   eval is a CONTINUOUS SYSTEM → the MOAT (your golden set + harness compound)

Mnemonic: eval = measure your task (not benchmarks) with a golden set + a scorer (programmatic > judge > human). A metric per task, combined into a weighted selection harness; run offline (CI gate) + online (drift/feedback). The golden set + harness are your moat.

4. Hitchhiker's Guide

What to look for first: a golden dataset for your task and a scorer (programmatic where possible, else a calibrated judge). With those you can gate every change; without them you tune blind.

What to ignore at first: chasing benchmark leaderboards and building elaborate eval infra. Start with 50 representative examples + a scorer + a baseline; expand later.

What misleads beginners:

• Evaluating on benchmarks, not your task. Contamination + distribution mismatch make leaderboard rank ≠ your performance (Phase 11.08).
• Trusting LLM-judge as truth. It's biased/noisy — calibrate against humans, prefer programmatic checks (02).
• One "quality" number. Use a metric per task (+ latency/cost/safety) (03–07).
• Eval as a one-time gate. Quality drifts (data, models, providers) — monitor online + feed back.
• No baseline / no regression gate. You can't tell "better" without a baseline you gate against.

How experts reason: they build a golden set (representative + edge + unanswerable, versioned, never trained on), prefer programmatic scoring and calibrate any judge, use a metric per task combined into a weighted harness (08), gate changes offline in CI, monitor online + feed failures back, and treat the golden set/harness as a compounding asset (the moat).

What matters in production: a regression gate that catches model/prompt/routing changes, online quality + drift monitoring, judge calibration, and the feedback loop growing the golden set (Phase 7.08).

How to debug a quality problem: is it a change (regression gate would catch — re-run golden set vs baseline) or drift (online monitoring — provider/model/version change, Phase 5.10, data shift)? Then localize per the task's metric (e.g., RAG retrieval-vs-generation, 03).

Questions to ask: do we eval on our task (golden set) or just benchmarks? programmatic or judge scoring (calibrated)? a metric per task + latency/cost/safety? a CI regression gate? online drift + feedback?

What silently gets expensive/unreliable: benchmark-only "eval" (false confidence), uncalibrated judges, single-number quality (hidden failures), no regression gate (silent regressions), and no online monitoring (undetected drift).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Evaluate on your task, not benchmarks — benchmarks shortlist (contamination/mismatch); your golden-set eval decides (Phase 11.08).
• The two foundations: a golden dataset (01) + a scorer (programmatic > LLM-judge > human).
• Prefer programmatic/objective scoring; calibrate LLM-judges against humans (02).
• There's no single "quality" — use a metric per task (RAG 03, agents 04, code 05) plus latency/cost (06) and safety (07).
• Combine metrics into a weighted model-selection harness (08, Phase 5.09).
• Eval types: offline (CI gate), online (drift + feedback), regression, red-team — run offline + online.
• Quality drifts (data/model/provider changes) — monitor online + feed failures back (Phase 5.10).
• The golden set + harness are a compounding moat — every model improves; your eval is uniquely yours.

9. Observations from Real Systems

• Teams that eval continuously win — a golden set + CI gate + online feedback is the difference between confident iteration and regressions shipped to users.
• Domain eval is everywhere in this curriculum — RAG (Phase 9.09), agents (Phase 10.09), code (Phase 11.08) — all instances of this unified discipline.
• LLM-as-judge is the dominant scalable scorer for subjective quality (Ragas/TruLens/LangSmith) — but production teams calibrate it (02).
• Benchmark contamination is real — LiveCodeBench / "Verified" variants / private evals exist because models memorize public sets (Phase 11.08).
• Frameworks (OpenAI Evals, Inspect AI, Ragas, Langfuse/LangSmith/Braintrust) provide the harness; the value is your golden set + metrics, not the framework.

10. Common Misconceptions

11. Engineering Decision Framework

BUILD AN EVAL DISCIPLINE:
 1. GOLDEN SET [01]: 50–200 representative tasks (+ edge + unanswerable), versioned, NEVER trained on.
 2. SCORER: programmatic where possible (tests/exact/recall) > LLM-JUDGE (calibrate [02]) > human (gold/calibrate).
 3. METRIC PER TASK: RAG [03] / agents [04] / code [05] + latency/cost [06] + safety [07].
 4. COMBINE for selection: weighted harness (quality·latency·cost·reliability·safety) [08, 5.09].
 5. OFFLINE: run in CI; GATE changes (model/prompt/routing) on no-regression vs baseline.
 6. ONLINE: sample + user/implicit signals; detect DRIFT; FEED failures back into the golden set.
 7. BENCHMARKS only to SHORTLIST models (mind contamination); YOUR eval decides.

12. Hands-On Lab

Goal

Build the core eval loop end-to-end: a small golden set, a scorer (programmatic + judge), a baseline, and a regression gate that catches a change.

Prerequisites

• A task you care about (e.g., support answers, extraction); pip install openai; 2 models or 2 prompts to compare.

Steps

1. Golden set (01): write 15–20 representative examples with expected outputs or criteria; include 2–3 edge/unanswerable cases; version it (a JSON file in git); mark it eval-only.
2. Scorer: implement a programmatic check where possible (exact/contains/JSON-valid) and an LLM-judge (02) with a rubric for subjective items.
3. Calibrate the judge: hand-label ~8 examples; compare to the judge; note agreement and adjust trust (02).
4. Baseline: run the current model/prompt over the golden set; record the score (and latency/cost, 06).
5. Regression gate: make a change (swap prompt or model); re-run; block if the score drops beyond a threshold — the offline gate.
6. (Stretch) weighted decision: if comparing models, combine quality + latency + cost into a weighted score and pick one (08/Phase 5.09).

Expected output

A versioned golden set, a programmatic+judge scorer (with a calibration note), a baseline score, and a regression gate that flags a regressing change — the minimal eval system.

Debugging tips

• Judge scores look random → weak rubric or judge model; add a rubric, prefer programmatic where you can (02).
• "Better" is ambiguous → you lack a baseline/threshold; set them.

Extension task

Add the domain metrics for your system — RAG (03), agent (04), or code (05) — and a safety check (07).

Production extension

Wire the gate into CI, add online sampling (thumbs/accept/escalation) feeding new examples back, and a trends dashboard (Phase 7.08).

What to measure

Golden-set score (baseline vs change), judge-human agreement, latency/cost, regression-gate firing.

Deliverables

• A versioned golden set (representative + edge + unanswerable).
• A programmatic + calibrated-judge scorer.
• A baseline + regression gate that catches a regressing change.

13. Verification Questions

Basic

1. Why is evaluation "engineering vs vibe checks"?
2. What's the difference between a benchmark and an eval?
3. What two foundations does all eval build on?

Applied 4. When do you use programmatic scoring vs LLM-as-judge vs human? 5. Why is there no single "quality" number?

Debugging 6. Production quality dropped 15% with no code change. Change vs drift — how do you tell, and what causes drift? 7. Your eval says model B is better but users disagree. What's likely wrong?

System design 8. Design an eval discipline: golden set, scorers, metric-per-task, offline gate, online feedback.

Startup / product 9. Why are the golden dataset + eval harness a moat (and a fundraising/credibility asset)?

14. Takeaways

1. Eval turns LLM work into engineering — and you evaluate your task, not benchmarks (benchmarks shortlist; your eval decides).
2. Two foundations: a golden dataset + a scorer (programmatic > calibrated LLM-judge > human).
3. A metric per task (RAG/agent/code) plus latency/cost and safety, combined into a weighted selection harness (08).
4. Run offline (CI regression gate) + online (drift + feedback) — quality drifts, so monitor and feed back.
5. The golden set + harness are a compounding moat — the teams that eval continuously win.

15. Artifact Checklist

• A versioned golden dataset (representative + edge + unanswerable, eval-only).
• A scorer (programmatic + calibrated LLM-judge).
• A baseline + regression gate (offline/CI).
• A metric-per-task set (+ latency/cost/safety).
• An online feedback plan growing the golden set.

Up: Phase 12 Index · Next: 01 — Golden Datasets