« Phase 11 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Staff Notes

Phase 11 — Core Contributor Notes: Agent Evaluation — LLM-as-Judge, Golden Datasets & Behavioral Regression

Our lab is a stdlib miniature of a real, crowded ecosystem: OpenAI Evals, LangSmith, Braintrust, RAGAS, DeepEval, agentevals, Weights & Biases Weave. This doc is the maintainer's-eye view — how those systems actually implement the pieces the lab models, the non-obvious source-level decisions, the API evolution, and where our miniature deliberately simplifies. Details drift; the patterns are stable, and where I am unsure of an exact field name I describe the pattern rather than invent one.

The dataset abstraction: how the real systems store the moat

Our EvalCase is a frozen dataclass; every real framework has the same record under a different name. OpenAI Evals models a sample as a JSONL row (input messages plus an ideal/reference), registered through a YAML evals/eval_sets registry that points a named eval at a dataset file and a grader class. LangSmith and Braintrust store datasets as versioned collections of examples (inputs + reference outputs + metadata), server-side, with dataset versions you can pin an experiment to. RAGAS builds a test set of (question, contexts, answer, ground_truth) rows.

The non-obvious contributor decision they all converge on is our EvalCase.id: a stable example identifier. LangSmith and Braintrust key example history and cross-run diffs on it; without it you cannot say "case X regressed between run A and run B," only "the aggregate moved." Our labels tuple is their example metadata/tags — the slicing dimension. The lesson the frameworks encode and our lab mirrors: the dataset is the versioned asset, and the id is what makes it diffable.

What our miniature simplifies: GOLDEN_SET is six inline cases; a real dataset store adds versioning, access control, dataset splits, and a UI to add a production trace as a new example in one click (the "every incident becomes case N+1" loop, productized).

Programmatic scorers vs the frameworks' graders

Our exact_match / contains / numeric_within / rubric_score are the "code-based graders." OpenAI Evals ships Match, Includes, FuzzyMatch graders — literally our exact_match, contains, and a normalized fuzzy variant — selected in YAML. LangSmith exposes string evaluators (exact, embedding-distance, regex) and lets you register arbitrary Python evaluators returning a score plus a key. DeepEval frames the same thing as metrics with a threshold and a measure() method.

Our rubric_score over declarative RubricChecks (contains | excludes | regex | min_words | equals) is the pattern behind DeepEval's GEval rubric criteria and the "checklist" style graders — decompose "is this good?" into small assertions you can version and print. A committer's sharp edge our lab gets right: an empty rubric must raise. A rubric that grades everything perfect because it has no checks is the single most dangerous silent default in eval code, and frameworks that let a criteria list default to empty have shipped exactly this bug.

LLM-as-judge: what the real judge actually does

Our deterministic_judge is token Jaccard — a stand-in. The real thing is the MT-Bench / Chatbot Arena method (Zheng et al., 2023): prompt a strong model with the answer, an optional reference, and a rubric, and parse a score or a pairwise verdict. Two implementation lineages worth knowing as a contributor:

• G-Eval (Liu et al., 2023) doesn't just parse the integer the model prints. It uses a chain-of-thought "form-filling" prompt and, in the reference implementation, weights the score by the model's token probabilities over the score tokens — a probability-weighted expectation — to get a smoother, less-quantized score than a raw "7/10." Frameworks that reimplement G-Eval without the logprob weighting get coarser scores and don't always say so.
• RAGAS decomposes a fuzzy question into checkable sub-questions. Faithfulness, for instance, is implemented by extracting the claims in the answer and asking the judge whether each is supported by the retrieved context — a ratio, not a vibe. Answer relevance generates questions from the answer and measures similarity to the original. This is the "split retrieval from generation" discipline our WARMUP §14 names, made concrete: context-precision/recall for retrieval, faithfulness/answer-relevance for generation.

Our judge_with_bias models three documented failure modes as additive perturbations. The real behaviors and their real fixes:

• Position bias — Zheng et al. measured it directly; the production fix, which LangSmith/Braintrust pairwise evaluators implement, is to run both orderings (A,B) and (B,A) and average, canceling it. Our +0.15/−0.15 is the caricature of exactly that.
• Verbosity bias — real judges reliably prefer the longer answer; the fix is to control for length (cap or normalize) and reward concision in the rubric.
• Self-preference — a judge favors its own model family; the fix is cross-model judging, which is why serious pipelines make the judge model different from the model under test.

Cohen's κ: the calibration step frameworks under-emphasize

cohens_kappa is textbook (Cohen, 1960): (p_o − p_e)/(1 − p_e), chance floor p_e = Σ_c P_a(c)·P_b(c), Landis & Koch's 0.6 "substantial" threshold. The honest maintainer's note: most eval frameworks make it easy to run a judge and hard to validate it. They ship the LLM-as-judge evaluator front and center; the κ-against-humans calibration is a doc paragraph, a notebook, or left to you. Our lab inverts that emphasis on purpose — judge_is_trustworthy gates on κ ≥ 0.6, and the worked example computes κ before the judge is allowed near CI. The degenerate branch (both raters one identical category → denom == 0 → return 1.0 by convention) is the kind of edge case a naive (p_o − p_e)/(1 − p_e) crashes on; a careful implementation handles it.

Trajectory evaluation: the newest, least-standardized piece

Grading the path is where the ecosystem is youngest. LangSmith added trajectory evaluators (compare the actual sequence of tool calls / run steps to an expected one — exact-match, superset, and LLM-judged-trajectory variants). The agentevals package and the OpenAI Agents SDK expose run-trace assertions over tool calls. Our three modes map directly: exact is their strict-order match, set is their unordered tool-set overlap, and order (our LCS-over-max-length) is the soft in-between many frameworks don't offer — they tend to give you strict or set, not a graded order score.

The contributor subtlety: real traces are richer than a tuple of tool names — they carry arguments, results, timestamps, retries, nested sub-agent calls. A production trajectory evaluator has to decide what counts as "the path" (tool names only? names + args? args normalized how?), and that decision is where trajectory evals get bespoke. Our miniature grades tool-name sequences only, which is the clean core; the real systems bolt argument-matching and step-level grading on top.

API evolution — why these systems look the way they do

• OpenAI Evals began (2023) as an open-source repo of YAML-registered evals and Python graders you ran locally, contributing new evals via PR. It later grew a hosted Evals API/dashboard so you define datasets and graders as API objects and run them managed. The registry-and-grader shape stayed; the substrate moved from repo to service.
• LangChain's early string evaluators lived in the library; evaluation then migrated into LangSmith as a hosted platform (datasets, experiments, run comparisons, pairwise) because eval needs persistence, versioning, and diffing that a stateless library can't give. The gravitational pull in this whole space is offline-in-a-repo → hosted-experiment-store, for exactly the reasons our Principal doc lays out.
• RAGAS evolved from a fixed metric bundle toward configurable, LLM-judged metrics as people learned the metrics themselves need calibration.

What our stdlib miniature deliberately simplifies

• The judge is a deterministic Jaccard proxy, not an LLM — so no temperature, no prompt-template drift, no logprob weighting, no cost. Real judges add all four, and each is a source of non-reproducibility the real frameworks fight with caching and pinned model+prompt versions.
• Biases are clean additive perturbations, not emergent behavior — you can see each one move a number, which no real judge lets you do.
• The golden set is six cases; trajectories are name-only; there is no online plane, no dataset UI, no CI runner.

What transfers exactly — and it is the whole point — is the structure: versioned labelled dataset with stable ids, programmatic scoring before a judge, trajectory grading of the path, κ-validation before the judge gates, and a per-label suite with safety as a hard separate gate. Learn that shape and every framework above reads as an elaboration of for case in golden_set: score(agent(case.input), case.reference).