« Phase 11 README · Warmup · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 11 — Hitchhiker's Guide

The compressed practitioner tour. Read the WARMUP for the derivations; this is the stuff you say in the meeting.

30-second mental model

An eval is a dataset + a scorer + an aggregate. The dataset — your golden set — is the moat: curated, versioned, labelled, grown from every incident, measuring your task not a public benchmark. Score with the cheapest reliable rung: programmatic checks (exact/ contains/numeric/rubric) before a calibrated LLM-judge before a human. Grade the path, not just the answer (trajectory eval — tools + order). An LLM-judge is a model that is biased (position/verbosity/self-preference), so validate it against humans with Cohen's κ ≥ 0.6 before it gates anything. Run the golden set on every commit as a CI gate; break out a per-label view because one number hides regressions; keep safety a hard gate, not a term in an average.

The numbers to tattoo on your arm

Framework one-liners

• OpenAI Evals = a registry of datasets + graders (match, includes, fuzzy, model-graded); the open-source shape of run_suite.
• LangSmith / Braintrust = hosted datasets + evaluators (string, trajectory, LLM-judge)
  • experiment tracking + regression diffs across runs.
• ragas = RAG-specific metrics: faithfulness, answer-relevance, context-precision/recall — the retrieval-vs-generation split.
• G-Eval / MT-Bench = the LLM-as-judge recipes (pointwise CoT scoring; pairwise arena).
• CheckList = behavioral testing (invariance / directional / minimum-functionality tests) — the academic root of "re-run the golden set every change."
• All of them are elaborations of for case in golden_set: score(agent(case.input), case.reference).

War stories

• The judge that loved long answers. A team gated releases on an LLM-judge's helpfulness score and "improved" by making answers wordier. Users hated it; the judge had verbosity bias and nobody had checked κ. The fix was one afternoon: control for length, re-validate against humans.
• The 1% improvement that broke refunds. Overall pass rate went up 1% after a prompt tweak; the refund slice had quietly dropped from 94% to 61%. No per-label breakdown, so it shipped. The customers found it before the dashboard did.
• The agent that guessed right. Final-answer eval was green for months. Trajectory eval revealed it was returning cached values without calling the pricing tool — correct by luck, one stale cache away from wrong. Grade the path.
• The safety case averaged away. A "99.7% pass" release leaked an account id on one case. It was one row in a mean, so the aggregate gate let it through. Safety has been a separate hard gate ever since.

Vocabulary

Golden dataset (curated/versioned eval set) · EvalCase (input + reference + expected trajectory + labels) · Scorer (output→[0,1]) · Scoring ladder (programmatic→judge→human) · Rubric (weighted programmatic checks) · Trajectory eval (grade the tool sequence) · Precision/Recall/F1 (set-task metrics) · LLM-as-judge (a model that grades) · Pointwise vs pairwise (grade one vs compare two) · Position/Verbosity/Self-preference bias (the judge's systematic errors) · Cohen's κ (chance-corrected agreement) · Regression gate (fail on a pass-rate drop) · Safety gate (hard-fail on any safety miss) · Offline vs online eval · Feedback loop (thumbs/A-B → new golden cases).

Beginner mistakes

1. Reaching for an LLM-judge when a programmatic check would do.
2. Trusting a judge you never validated against humans (no κ).
3. Reading raw agreement as if it weren't inflated by chance.
4. Grading only the final answer and missing wrong-path bugs.
5. Shipping on one aggregate number with no per-label breakdown.
6. Averaging a safety failure into the mean instead of gating it hard.
7. Evaluating on a public benchmark instead of your own golden set.
8. Treating evals as a pre-launch checklist instead of a standing CI gate + online loop.