Cheatsheet 16 — Evaluation Metrics
The metric-per-task map. Full: Phase 12. Lab: labs/lab-05-eval-harness.
Governing principles
- Evaluate YOUR task, not benchmarks (benchmarks shortlist; your eval decides).
- A metric per task — there's no single "quality" number.
- Prefer programmatic scoring > calibrated LLM-judge > human (in that order of preference where feasible).
- Safety is a GATE, not a weighted axis.
- Your golden dataset is the moat — representative + edge + unanswerable cases; never train on it (Phase 12.01).
Scoring methods
| Method | Use when | Caveat |
|---|---|---|
| Programmatic (exact/regex/schema/unit tests) | Verifiable outputs (code, extraction, classification) | Best — deterministic |
| LLM-as-judge | Subjective quality (helpfulness, tone) | Biases (position/verbosity/self); calibrate (12.02) |
| Human | Ground truth / calibration | Slow/expensive; use sparingly |
Metrics by domain
| Domain | Metrics |
|---|---|
| Classification/extraction | accuracy, precision, recall, F1 |
| RAG — retrieval | recall@k, precision@k, MRR, hit rate (9.09) |
| RAG — generation | faithfulness/groundedness, answer relevance, citation accuracy |
| Agents | task success (outcome), trajectory validity, per-step reliability (10.09) |
| Code | pass@k, test-resolution rate, apply-rate (11.08) |
| Latency | TTFT, TPOT, p95/p99 under load (12.06) |
| Cost | cost per resolved task, $/user, gross margin |
| Safety | violation rate (→0, hard gate), over-refusal rate (12.07) |
LLM-as-judge biases (mitigate these)
- Position bias (favors first/last), verbosity bias (favors longer), self-preference (favors own outputs), inconsistency. → randomize order, use rubrics, calibrate against human labels, use pairwise.
The model-selection harness
Combine quality + latency + cost into a weighted score, apply a safety gate, and run it as a CI regression gate so no change ships without passing (Phase 12.08).
Interview line
"I never report one 'quality' score. I split RAG into retrieval vs generation, measure agents by per-step reliability and outcome, prefer programmatic scoring, calibrate any LLM-judge, treat safety as a hard gate, and gate every change on a golden-set regression test."
Next: 17 — Production Readiness · Full: Phase 12