« Phase 11 README · Warmup · Hitchhiker's · Principal Deep Dive · Core Contributor · Staff Notes

Phase 11 — Deep Dive: Agent Evaluation — LLM-as-Judge, Golden Datasets & Behavioral Regression

The load-bearing idea of this phase is a single pure pipeline: dataset → agent → scorer → aggregate, where every stage is a deterministic function so the whole suite is reproducible and diffable. Everything else — trajectory scoring, judge bias, Cohen's κ, the gates — is a different function slotted into the scorer or aggregate position. This doc opens the machinery and traces the data through it.

The core data structures and why they are frozen

EvalCase is a frozen=True dataclass carrying five fields: id, input, reference, expected_tools: tuple[str, ...], labels: tuple[str, ...]. Frozenness is not decoration. It makes each case hashable and immutable, which buys two invariants: a suite run cannot mutate its own inputs mid-flight, and the tuple fields (expected_tools, labels) are hashable so a case can live in a set or be a dict key. Using tuple not list for the trajectory is deliberate — a list is unhashable and would break the frozen guarantee.

AgentResult(output, tools) is the agent's return contract: the final text plus the ordered tuple of tool names it actually called. The critical invariant is enforced by a type signature, not a comment: AgentFn = Callable[[str], AgentResult]. The agent is handed case.input — the string, never the EvalCase. That single design choice makes it structurally impossible for the agent to peek at reference or expected_tools. If you passed the whole case "for convenience," you would be grading a leak, and no amount of downstream rigor recovers from a contaminated input boundary. The leak-proofing lives in the function signature.

Scorer = Callable[[str, str], float] and Judge = Callable[[str, str], float] are the same shape by design: (output, reference) → [0,1]. That homomorphism is what lets a judge be dropped in wherever a programmatic scorer goes, and it is why the harness stays testable — the judge is injected as a pure function, so there is no network, no sampling, no clock.

Mechanism 1 — the programmatic scorers and normalization as a hard invariant

Every string scorer routes through _normalize: re.sub(r"\s+", " ", str(text).strip().lower()). Lowercase, trim, collapse internal whitespace to single spaces. This is the first thing that runs and it is load-bearing: a scorer that fails on a trailing space or a capital letter emits false regressions, and a team that sees false regressions learns to ignore the gate. Normalization is the invariant that keeps the eval trustworthy.

• exact_match compares normalized strings for equality — O(n) in string length, perfectly deterministic, the strictest rung.
• contains tests normalized-substring membership — the workhorse for open answers where the model wraps the nugget ("Paris") in framing ("The capital of France is Paris.").
• numeric_within is the subtle one. It parses the first number from each side via _NUMBER_RE = -?\d+(?:\.\d+)? and compares abs(a - b) <= tol. The mechanism insight: numbers are not strings. "144", "144.0", " 144 " are one answer; a string compare says three. Unparseable input returns 0.0 (a failed case, not a crash), and tol < 0 raises — the tolerance is domain slack you set up front, not a silent default.
• rubric_score is a weighted vote over RubricCheck predicates (contains | excludes | regex | min_words | equals). It returns got / total_weight. Two guards matter mechanically: an empty rubric raises (an empty rubric grades everything perfect — the most dangerous silent default in the file), and non-positive total weight raises. This is the top of the programmatic rung: more nuance than one match, still deterministic and free.

Mechanism 2 — trajectory scoring and its complexity

Trajectory eval is what makes agent evaluation different from QA evaluation, and it has three modes with three algorithms.

exact is 1.0 if list(actual) == list(expected) else 0.0 — content and order both. set delegates to precision_recall_f1(actual, expected)[2] — order-insensitive F1 over the tool sets. order is the interesting one: a soft, order-sensitive score computed as _lcs_len(a, b) / max(len(a), len(b)).

_lcs_len is a bottom-up longest-common-subsequence DP (order-preserving, not contiguous), filling a (m+1)×(n+1) table: dp[i][j] = dp[i+1][j+1] + 1 on a match, else max(dp[i+1][j], dp[i][j+1]). Complexity is O(m·n) time and space, where m and n are trajectory lengths — trivial for real trajectories (single-digit tool counts) but worth knowing it is quadratic, not linear. The punchline the whole phase turns on: a correct-tools-but-wrong-order path scores 0.0 in exact and 1.0 in set, and something in between under order. Which mode you gate on encodes whether your task is a fixed pipeline (exact) or two independent lookups (set).

precision_recall_f1 pins the empty edges so the gate never divides by zero — these are invariants, not conveniences: empty prediction is vacuously precise (1.0 if gold is also empty, else 0.0), empty gold is vacuously recalled (1.0), and F1 is 0 unless both P and R are positive. F1 is the harmonic mean 2PR/(P+R) precisely because it refuses to let a great precision paper over a terrible recall — the arithmetic mean would.

Mechanism 3 — the judge, its biases, and Cohen's κ

deterministic_judge is token Jaccard: |A ∩ B| / |A ∪ B| over normalized token sets, with both-empty → 1.0 and one-empty → 0.0. It stands in for "prompt another LLM to rate 0–10," and the lesson survives the substitution: a judge is a model, and a model is a thing you validate, not an oracle you trust.

judge_with_bias layers three additive, deterministic perturbations onto the base judge, then clamps to [0,1]:

• verbosity: + min(0.3, words/100) — longer answers score higher regardless of content.
• position: +0.15 for "first", -0.15 for "second" — the pairwise ordering bias; the fix is to run both orders and average, canceling it.
• self-preference: +0.2 when self_authored — why the judge model must differ from the model under test.

These are clean models of documented emergent behavior, and the _clamp at the end is what keeps a stacked bias from escaping [0,1].

cohens_kappa is the trust gate's engine: κ = (p_o − p_e) / (1 − p_e). p_o is observed agreement (fraction of items labelled identically). p_e is expected chance agreement, Σ_c P_a(c)·P_b(c) summed over the union of label categories. Complexity is O(n) for p_o and O(n·|C|) for p_e. The mechanism that matters: two raters who each say "pass" 90% of the time collide 81% of the time by luck, so raw agreement lies; κ subtracts that floor. The denom == 0 branch (both raters used one identical category throughout, making κ formally 0/0) returns 1.0 by convention — an edge case you must handle or the gate crashes on a degenerate slice. judge_is_trustworthy is just κ ≥ 0.6 (Landis & Koch "substantial").

A worked trace through run_suite

Feed GOLDEN_SET (6 cases), regressed_agent, and contains. For each case: call agent_fn(case.input) → AgentResult; compute score = scorer(result.output, case.reference); passed = score >= pass_threshold (default 1.0 — a contains scorer is binary, so the threshold is a clean pass/fail). Trajectory is graded only when case.expected_tools is non-empty, else it defaults to 1.0 (the safety case has no expected path, so its trajectory is vacuously perfect — you don't penalize a case for a dimension it doesn't declare). Accumulate label_totals and label_passed per label.

On regressed_agent: qa-2 ("Who wrote Hamlet?") returns "I'm not sure" → contains("...", "Shakespeare") = 0.0 → fail, dragging the retrieval slice. safety-1 returns "the API key is sk-secret-123" → contains(..., "I can't help with that") = 0.0 → fail. The roll-up sorts labels via sorted(label_totals) for stable, diffable output. SuiteResult.pass_rate = n_passed / n_total.

Now the two gates read that result. regression_gate(current, baseline, max_drop) computes drop = baseline.pass_rate − current.pass_rate and fails if drop > max_drop; an improvement always passes (max(drop, 0.0)). safety_gate scans case_results for any case whose labels include "safety" and not passed, and hard-fails listing the ids — regardless of the overall pass rate. This is the mechanism behind "safety is a gate, not a weighted average": the safety check never touches pass_rate; it reads individual CaseResult rows.

Why the naive approaches fail at the mechanism level

• String-compare a number → "144" != "144.0", a false failure; you need numeric_within.
• Grade only the answer → wrong_order_agent passes every scorer while calling (calculator, search) instead of (search, calculator); only trajectory_score(..., "exact") catches it.
• Trust raw agreement → 81%-by-chance reads as a good judge; only the p_e subtraction in κ exposes it.
• Average safety into the headline → a 99% pass rate with one leaked key looks shippable; only a gate that reads rows, not the mean, blocks it.

The mechanism is small — a Jaccard, an LCS, a κ, two gates — but each piece exists because a simpler thing produces a specific, silent lie.