« Phase 11 README · Warmup · Hitchhiker's · Deep Dive · Core Contributor · Staff Notes

Phase 11 — Principal Deep Dive: Agent Evaluation — LLM-as-Judge, Golden Datasets & Behavioral Regression

The lab is 600 lines of pure functions. The production system those functions imply is a standing platform with a dataset store, a judge-inference tier, a run/experiment store, a baseline registry, and a CI integration — plus the online loop feeding it. This doc is the architecture around the mechanism.

The reference architecture

Draw four planes.

Dataset plane. The GOLDEN_SET in the lab is a tuple literal; in production it is a versioned artifact in a dataset store (LangSmith/Braintrust dataset, an OpenAI Evals YAML, a ragas test set, or just labelled rows in a git-tracked file). The EvalCase.id is the join key that makes a case survive edits and appear consistently across runs and dashboards — without a stable id you cannot diff two runs or track a single case's history. The dataset is code: PR-reviewed, diffed, and grown from every incident. Everything else in the platform is commodity; this is the moat.

Execution plane. run_suite is the map-reduce: map agent_fn over cases, score each, reduce into SuiteResult. In production this fans out. Two cost centers dominate: the agent-under-test (an LLM policy plus tools) and the judge (another LLM). Both are network calls, both are slow, both cost tokens.

Scoring plane. The scoring ladder is an explicit cost/latency hierarchy: programmatic (~0 cost, instant, deterministic) → calibrated judge (tokens, seconds, non-deterministic) → human (expensive, slow). The architecture decision is to push grading as far down the ladder as it goes and treat the judge tier as a scarce, metered resource.

Gate plane. regression_gate and safety_gate are the CI integration — a GitHub Action that runs the suite on every PR, compares to the stored baseline, and blocks merge. This is the analogue of a failing unit test, but for behavior.

Scaling and the capacity/latency math

The naive suite is a serial loop, and that is fine for six cases and a Jaccard proxy. It falls over the moment the judge is a real LLM over a real golden set.

Do the arithmetic. Suppose 2,000 golden cases, each needing one judge call at ~800 ms and ~1,500 tokens (prompt + rubric + answer + reasoning). Serial, that is 2000 × 0.8 s ≈ 27 minutes per suite run — unacceptable on every PR. Fan out to 50 concurrent judge calls and it drops to ~30 s, but now you are rate-limited by the judge provider, so the real ceiling is provider TPM/RPM, not your loop. Cost: 2000 × 1,500 tokens ≈ 3M tokens per run; at a mid-tier judge model and, say, 30 PRs a day, that is ~90M tokens/day just for the judge, before the agent-under-test's own tokens. This is why the ladder exists: every case a programmatic scorer can grade is a judge call you do not pay for, do not wait on, and cannot be flaked by.

Three architectural levers fall out:

• Cache judge verdicts keyed on (judge_model, prompt_template_version, output, reference). A judge call is a pure function of those inputs; identical inputs across commits should hit cache. This is why the lab keeps the judge a pure (output, reference) → float seam — production adds the cache and the version key around exactly that seam.
• Tier the suite. Run the fast programmatic + safety subset on every commit (seconds, free); run the full judge suite on merge-to-main or nightly. A blocking gate must be fast and deterministic; a judge is neither, so a judge-gated suite runs off the hot path.
• Sample under a confidence interval. pass_rate is an estimate; with 2,000 cases the standard error is meaningful. regression_gate's max_drop is really a noise threshold, and the principled version bootstraps the golden set to put an error bar on the pass rate so the gate distinguishes noise from a real drop rather than guessing a magic 5%.

Failure modes and blast radius

A bad judge gating CI is the highest-blast-radius failure in the system. If the judge has un-validated verbosity bias, the team optimizes the agent to please it — shipping wordier, worse answers with a higher score. The gate is now actively steering the product in the wrong direction, and because it is green, nobody looks. The control is architectural: no judge may gate until cohens_kappa against a human-labelled slice clears 0.6, and κ is re-measured whenever the judge model or prompt template changes (a silent model upgrade on the provider side invalidates yesterday's calibration).

One aggregate number hiding a slice collapse. A change lifts overall pass rate 1% while the refund slice drops 94% → 61%. The mean rose; the product broke. The per_label breakdown in SuiteResult is the mitigation, and it is why EvalCase carries labels — the report must be sliceable or the regression ships. Blast radius: a whole customer segment, invisible on the headline dashboard.

A safety violation averaged into the mean. A 99.7% run leaks one account id. As 1/n of a mean it clears any percentage gate. safety_gate reads individual rows and hard-fails on any safety miss, threshold zero. The design decision — safety is a separate absolute gate, not a weighted term — is the entire reason the golden set is labelled. Blast radius here is a security incident, not a quality dip.

Golden-set contamination / staleness. If the agent can see reference (a case passed as the whole EvalCase), the eval measures a leak and reports fiction. If the golden set never grows, it drifts from a moving production distribution and green stops meaning "good." Both are addressed by discipline the architecture must enforce: input-only agent boundary, and every incident becomes case N+1.

Cross-cutting concerns

Cost. Covered above; the ladder is the cost architecture. The principal move is to treat judge tokens as a budget line and report cost-per-suite-run alongside pass rate.

Observability. The suite emits pass_rate, per_label stats, and trajectory_accuracy as first-class metrics. In production these are time series with alerts on drift, not just a CI pass/fail. trajectory_accuracy is the one people forget to chart — it is how you catch the agent that starts getting right answers via a more expensive path (five tool calls where two would do) before the bill or the latency SLO does.

Security & multi-tenancy. Each team owns its own golden set; a platform serving many teams must isolate datasets and never let one tenant's cases (which may contain real customer data — the adversarial safety cases especially) leak into another's runs or into the judge provider's logs. The safety slice is often the most sensitive data you have.

Offline vs online. Offline evals (this lab: golden set in CI) catch what you thought of. They are a snapshot; production is a moving distribution. The architecture is completed by the online plane — A/B tests against a live metric, guardrail metrics (refusal/exfil/cost/latency), and human-feedback loops (thumbs, edits, escalations). The loop is load-bearing: online surfaces the failure you didn't anticipate, and it becomes offline case N+1. A platform with only offline evals is half a system.

The "looks wrong but is intentional" decisions

• The judge is a pure function, not a real LLM call. Looks like a toy; it is the seam. Determinism makes the harness unit-testable and the verdict cacheable, and it forces the discipline (κ before trust) to the front where it belongs.
• pass_threshold defaults to 1.0. Looks brittle. It is correct for binary scorers (exact/contains return 0 or 1); a soft rubric or judge score wants a lower threshold, set per-suite. The default fails safe toward strictness.
• Trajectory defaults to 1.0 when no expected_tools. Looks like it hides bugs. It correctly declines to penalize a case for a dimension it never declared — the safety case has no path to grade.
• max_drop is non-zero. Looks like tolerating regressions. It is a noise budget for a stochastic system; zero tolerance flaps on sampling noise and trains the team to bypass the gate.
• Safety is a whole separate gate. Looks redundant with the pass rate. It is the opposite: the pass rate is an average, and the entire point is that a bright-line violation must never be averageable.

The architecture is small because the ideas are sharp: version the data, tier the scoring by cost, validate the judge before it gates, slice the aggregate, and make safety un-averageable. Get those five right and the rest is plumbing.