Phase 12 — Evaluation
The discipline that turns LLM development from vibe checks into engineering: golden datasets, LLM-as-judge, the per-domain metrics (RAG / agent / code), the objective axes (latency / cost), safety/policy red-teaming, and the model-selection harness that combines them into one defensible decision and a regression gate.
Why this phase matters
You cannot improve, choose, ship, or trust an LLM system you don't measure. Evaluation is the empirical backbone of model selection (Phase 5) and the gate on every change — and it's the competitive moat: every model improves, but your golden dataset + eval harness (encoding what good looks like for your task) are uniquely yours and compound over time. The governing principles: evaluate your task, not benchmarks (benchmarks shortlist; your eval decides); a metric per task (no single "quality" number); prefer programmatic scoring, calibrate judges; and safety is a gate, not a weighted axis. The domain eval docs you met in Phases 9–11 (RAG 9.09, agents 10.09, code 11.08) are instances of this unified discipline.
Documents
| # | Document | What you'll be able to do |
|---|---|---|
| 00 | Evaluation Overview | The eval taxonomy; benchmark vs eval; metric-per-task; eval-is-the-moat |
| 01 | Golden Datasets | Build the foundational artifact (representative + edge + unanswerable, never train) |
| 02 | LLM-as-Judge | Score subjective quality; biases, mitigations, calibration |
| 03 | RAG Evals | The quadrant; separate retrieval from generation |
| 04 | Agent Evals | Outcome + trajectory + safety; compounding |
| 05 | Code Evals | A metric per feature; programmatic ground truth; contamination |
| 06 | Latency and Cost Evals | p95-under-load latency; cost per resolved task |
| 07 | Safety and Policy Evals | Red-teaming; violations as hard fails; the refusal balance |
| 08 | Model-Selection Eval Harness | Combine all axes into one decision + a CI regression gate |
How to work through it
Read 00 (the discipline + governing principles) first. 01–02 are the two foundations every eval builds on (golden set + scorer). 03–05 are the per-domain metric sets (consolidating Phases 9.09/10.09/11.08 within the unified lens). 06–07 are the universal axes — objective latency/cost and adversarial safety (a hard gate). 08 is the capstone: the harness that combines quality + latency + cost + safety into one defensible decision and serves as the CI regression gate. Every doc opens with a from-zero plain-English primer and ends with a buildable lab; together they build the eval harness that's your moat.
Phase 12 artifacts
- A versioned golden set (representative + edge + unanswerable) + a calibrated scorer (00, 01, 02).
- A RAG quadrant eval (retrieval vs generation) (03); an agent eval (outcome+trajectory+safety) (04); a per-feature code eval (05).
- A latency (p95-under-load) + cost-per-resolved-task measurement (06).
- A red-team report (violation rates → 0, refusal balance) + a safety gate (07).
- A reusable model-selection harness (all axes, safety-gated, weighted) + a CI regression gate + routing simulation (08).