Lab 01 — Agent Eval Harness (Trajectory + Judge + Regression Gate)

Phase 11 · Lab 01 · Phase README · Warmup

The problem

"It worked in the demo" is a sample of one from a distribution (Phase 00). To ship an agent you need to know its behavior over a population of inputs, catch the regression the next commit introduces, and prove the safety property never breaks — automatically, on every change. Build the harness that does it: a golden dataset, a programmatic scoring ladder, trajectory evaluation (grade the tool path, not just the answer), an injected LLM-as-judge stand-in with its biases and a Cohen's κ trust check, and a regression gate plus a hard safety gate — evals-as-CI.

What you build

PieceWhat it does
EvalCase + GOLDEN_SETthe curated, versioned, labelled golden dataset — the moat
exact_match / contains / numeric_within / rubric_scorethe programmatic scoring ladder (cheapest, most reliable first)
trajectory_score + precision_recall_f1grade the tool sequence — exact order, soft order (LCS), set-overlap F1
deterministic_judge / judge_with_biasan injected LLM-as-judge stand-in + its position / verbosity / self-preference biases
cohens_kappa / judge_is_trustworthythe human-agreement metric you MUST compute before trusting a judge in CI
run_suite → SuiteResultpass rate + per-label breakdown + trajectory accuracy
regression_gate / safety_gatefail the build on a pass-rate drop; HARD-fail on any safety failure

File map

FileRole
lab.pyyour implementation (fill the # TODOs)
solution.pyreference + a main() that scores the golden set, computes κ, and trips both gates
test_lab.py34 tests: scorers, trajectory order-sensitivity, κ on known agreement/chance, per-label breakdown, gate pass/fail, determinism
requirements.txtpytest only

Run it

pytest test_lab.py -v                       # your lab.py (red until you implement)
LAB_MODULE=solution pytest test_lab.py -v   # the reference (green)
python solution.py                          # worked example

Success criteria

  • You can name the rungs of the scoring ladder (programmatic → calibrated judge → human) and say why you always reach for the cheapest that works.
  • Your trajectory_score gives a wrong-ordered but correct-tool-set path 0.0 in exact mode and 1.0 in set mode — and you can explain why grading the path matters.
  • cohens_kappa returns 1.0 on perfect agreement and ≈0.0 on chance, and you can explain why raw agreement overcounts and why κ ≥ 0.6 gates whether a judge may run in CI.
  • run_suite reports a per-label breakdown, so a small overall drop that is really a safety collapse is visible.
  • regression_gate fails on a pass-rate drop and safety_gate HARD-fails on any safety-labelled failure even at a 95% overall pass rate.
  • All 34 tests pass under both lab and solution.

How this maps to the real stack

  • The golden dataset is a LangSmith / Braintrust / Weights & Biases dataset, an OpenAI Evals YAML, or a ragas test set — curated, versioned, and grown from production incidents. The moat isn't the harness; it's the data.
  • The programmatic scorers are the built-in graders in OpenAI Evals (match, includes, fuzzy) and LangSmith's string evaluators; rubric_score is a code-based grader. Always the first rung.
  • Trajectory evaluation is LangSmith's trajectory evaluators, the OpenAI Agents SDK's run-trace assertions, and ragas/agentevals tool-call metrics — grading the path an agent took, which is unique to agent (vs QA) evaluation.
  • LLM-as-judge is the pattern from Zheng et al.'s MT-Bench / Chatbot Arena (2023) and the G-Eval / ragas / LangSmith llm-as-judge evaluators. The judge_with_bias biases (position, verbosity, self-preference) are the documented failure modes; Cohen's κ against human labels is the calibration step those papers insist on before you trust a judge.
  • The regression + safety gates are evals-as-CI: a GitHub Action that runs the suite on every PR and blocks merge on a drop — the behavioral analogue of a failing unit test, and the discipline behind CheckList (Ribeiro et al., 2020) behavioral testing.

Limits of the miniature. The judge here is a deterministic Jaccard proxy, not a real LLM; the golden set is six cases, not thousands; the biases are modelled as clean additive perturbations rather than emergent behavior. What transfers exactly is the structure: score programmatically first, grade the trajectory, validate the judge with κ before trusting it, and gate CI on a versioned golden set with safety as a hard, separate gate.

Extensions (your own machine)

  • Pairwise judging: add judge_pairwise(a, b, reference) and run both orderings to neutralize position bias; report the win rate (the Chatbot Arena method).
  • Bootstrap confidence intervals: resample the golden set to put an error bar on the pass rate, so regression_gate distinguishes noise from a real drop.
  • Wire a real judge: put a small local model behind the Judge seam, label a human slice, compute κ, and only enable the judge gate if κ ≥ 0.6.
  • Retrieval-vs-generation split (Phase 05): add context-precision / faithfulness scorers so a RAG agent's retrieval and generation are graded separately.

Interview / resume signal

"Built an agent evaluation harness — a versioned golden dataset, a programmatic scoring ladder (exact/contains/numeric/rubric), trajectory evaluation grading the tool sequence (exact order + set-overlap F1), an LLM-as-judge stand-in with its position/verbosity/ self-preference biases validated by Cohen's κ against human labels, and a behavioral regression gate plus a hard safety gate — evals-as-CI over the golden set."