Template 02 — Evaluation Plan
Copy and fill before you scale a feature. Full method: Phase 12.
Evaluation Plan: <feature name>
Author: <name> · Date: <date>
1. What are we evaluating, and why?
- Feature:
<one sentence> - Decision this eval drives:
<ship/no-ship, model choice, prompt change, regression gate>
2. Golden dataset (Phase 12.01)
- Size:
<N>examples · Source:<production / human-labeled / synthetic (validated)> - Composition: representative
<%>+ edge cases<%>+ unanswerable/negative<%> - Rule: never train on it; versioned; disjoint from any training data (no leakage).
3. Metrics (a metric per task — cheatsheet 16)
| Aspect | Metric | Target | Scoring method |
|---|---|---|---|
| Quality | <accuracy / faithfulness / pass@k / ...> | <value> | programmatic / judge / human |
| (RAG) Retrieval | <recall@k / MRR> | programmatic | |
| (RAG) Generation | <groundedness / citation acc> | judge (calibrated) | |
| (Agent) Outcome | <task success> | programmatic | |
| Latency | p95 under load | <ms> | load test |
| Cost | per resolved task | <$> | metering |
| Safety | violation rate | 0 (gate) | red-team |
4. Scoring details
- Programmatic where possible (exact/schema/tests).
- LLM-judge: model
<judge model>, rubric<link>, calibrated against<N>human labels; mitigate position/verbosity/self bias (Phase 12.02).
5. Pass/fail & gating
- Ship if:
<quality ≥ X AND p95 ≤ Y AND cost ≤ Z AND safety = pass AND no regression>. - Regression gate: run in CI on every prompt/model change; block on failure (Phase 12.08).
6. Cadence & ownership
- Run on:
<every PR / nightly / model release>· Owner:<name> - Golden-set growth: add production failures to the set (data-centric loop).
The eval set is your moat. Build it before you scale.