« Phase 11 README · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 11 Warmup — Agent Evaluation: Judge, Golden Datasets & Regression

Who this is for: you can build an agent (Phases 01–10) but you have only ever judged it by running it and looking. By the end you will know how to measure an agent's behavior over a population, grade the path it took and not just the answer, use an LLM as a judge and prove it is trustworthy first, and turn all of it into a CI gate that blocks the next regression. Nothing here needs a GPU or an API key — the judge is injected as a pure function, so the whole discipline is arithmetic and set theory you can run offline.

Why evaluate at all? "It worked in the demo" is a sample of one
What an eval is, precisely: dataset, scorer, aggregate
The golden dataset: curated, versioned, adversarial — the moat
Evaluate your task, not a public benchmark
The scoring ladder: programmatic → calibrated judge → human
Programmatic scorers up close: exact, contains, numeric, rubric
Trajectory evaluation: grade the path, not just the destination
Precision, recall, F1: the set and trajectory math
LLM-as-judge: how it works, pointwise vs pairwise
The judge is biased: position, verbosity, self-preference
Trust, but verify: Cohen's kappa
Behavioral regression testing: evals as CI gates
Safety is a gate, not a weighted average
Retrieval vs generation, online eval, and feedback loops
Common misconceptions
Lab walkthrough
Success criteria
Interview Q&A
References

1. Why evaluate at all? "It worked in the demo" is a sample of one

An agent is a stochastic system: the LLM samples, tools flake, inputs vary. So when you run it once and it works, you have observed exactly one sample from a distribution you have not characterized. That single success is almost no evidence — the demo is the best case, hand-picked and rehearsed, and the customer's inputs are drawn from the whole distribution including its ugly tail.

This is the same lesson as Phase 00's reliability math, seen from the measurement side. There we computed that a ten-step agent at 95% per step succeeds $0.95^{10} \approx 0.60$ of the time. Evaluation is how you measure that 0.60 instead of guessing it — and how you notice when next week's "harmless" prompt tweak quietly drops it to 0.52.

So evaluation is not a nice-to-have you bolt on before launch. It is the instrument that makes every other claim in this track checkable: that the guardrail (Phase 10) actually blocks the injection, that the router (Phase 14) didn't degrade quality, that the retrieval change (Phase 05) improved recall without hurting faithfulness. Without evals you are flying by feel, and "it feels better" is not a thing you can put in a design review or a CI pipeline.

The one sentence to keep: you cannot ship an agent you cannot evaluate, because "it worked when I tried it" is a sample of one and your users are the whole distribution.

2. What an eval is, precisely: dataset, scorer, aggregate

Strip away the tooling and an evaluation is three components:

A dataset — a set of cases, each an input plus whatever ground truth you'll grade against (a reference answer, an expected tool trajectory, a set of acceptable outputs).
A scorer — a function that takes the agent's output for one case and returns a number, usually in $[0,1]$: did this output meet the bar? A scorer can be a string comparison, a rubric of programmatic checks, or an LLM asked to grade.
An aggregate — a roll-up over the whole dataset: the pass rate, a per-slice breakdown, the trajectory accuracy. This is the number you track over time and gate CI on.

   dataset            agent            scorer            aggregate
  ┌────────┐   input  ┌───────┐ output ┌────────┐ score ┌──────────────┐
  │ case 1 │ ───────► │ agent │ ─────► │ scorer │ ────► │ pass_rate,   │
  │ case 2 │          │ under │        │        │       │ per-label,   │
  │  ...   │          │ test  │        │        │       │ trajectory   │
  └────────┘          └───────┘        └────────┘       └──────────────┘

Two axes cut across this. Offline vs online: offline evals run a fixed golden set in CI before you ship; online evals watch real traffic after you ship (A/B tests, guardrail metrics, thumbs-up rates). Reference-based vs reference-free: some scorers compare to a gold answer (exact_match); some judge a property with no single right answer (a rubric, or an LLM asked "is this reply polite?"). The lab builds the offline, mostly reference-based half, because that is the half that becomes a CI gate — and it is the half interviews probe.

3. The golden dataset: curated, versioned, adversarial — the moat

The dataset is the heart of the whole enterprise, and the single most valuable, least-copyable thing your team owns. Four properties make one good:

Curated. Every case is chosen on purpose to cover a behavior you care about, not scraped at random. A hundred hand-picked cases that span your real failure modes beat ten thousand random ones.
Versioned. The dataset is code: it lives in the repo, it is reviewed in PRs, and its changes show up in diffs. When someone edits a reference answer, you can see who, when, and why. A stable id per case (the lab's EvalCase.id) lets a case survive edits and appear consistently in dashboards.
Representative. The distribution of cases should mirror production — the same mix of easy/hard, short/long, and topic areas. If 30% of your traffic is refunds, roughly 30% of your golden set should be refunds, so the aggregate means something.
Adversarial. Include the cases that break things: the prompt injection (Phase 10), the empty input, the ambiguous question, the one that tempts the agent to leak a secret. These are the cases that catch regressions, and they are exactly what a random sample misses.

And the discipline that compounds: every incident becomes a case. The bug you shipped gets distilled into golden case N+1 so it can never regress silently again. Do this for a year and your golden set is a precise, hard-won encoding of everything that has ever gone wrong — a moat a competitor cannot clone by reading your prompts, because it is your production history.

4. Evaluate your task, not a public benchmark

A tempting shortcut is to point at MMLU, GSM8K, or a leaderboard and call it evaluation. It is not — at least not of your system. Public benchmarks measure a general capability of a base model; your agent is a specific composition (a prompt, a tool set, a retriever, a guardrail) solving a specific task on your data. A model that tops a reasoning leaderboard can still be terrible at classifying your tickets, and a model that scores mid-pack can be perfect for it.

Worse, public benchmarks leak into training sets and saturate, so a rising benchmark score may reflect contamination rather than capability, and it tells you nothing about the failure modes unique to your integration — your tool schemas, your edge cases, your adversarial inputs.

Use public benchmarks for what they are good for: a coarse first filter on which base model to even consider. Then evaluate the thing you are actually shipping on the golden set you built in §3. The benchmark is a smoke test for the model; the golden set is the eval for the product.

5. The scoring ladder: programmatic → calibrated judge → human

Not all scorers are equal. They form a ladder, and the rule is: use the cheapest rung that reliably measures what you care about.

Rung	Cost	Speed	Determinism	Use when
Programmatic (string/number/rubric checks)	~0	instant	perfect	the answer is closed-form or you can assert the property in code
Calibrated LLM-judge	tokens	seconds	none (a model)	the property is fuzzy (tone, helpfulness) and the judge is validated (§11)
Human	expensive	slow	varies	the ground truth for calibration, and the rare high-stakes call

Read that top-down and it is the same "least-agentic that works" instinct from Phase 00, applied to measurement. A programmatic check is free, instant, and impossible to argue with; an LLM-judge costs money, is non-deterministic, and can be biased; a human is the gold standard but does not scale. So you push as much of the grading as far down the ladder as it will go: assert everything you can in code, escalate to a judge only for the genuinely subjective remainder, and reserve humans for calibrating the judge and adjudicating the hardest cases.

The classic beginner move is to reach straight for the LLM-judge because it is flexible. The senior move is to notice that "does the reply contain the order number?" and "does it avoid leaking the account id?" are programmatic checks, and only "is the tone appropriately empathetic?" needs the judge.

6. Programmatic scorers up close: exact, contains, numeric, rubric

The bottom rung, in the four flavors the lab implements:

exact_match(output, reference) — 1.0 iff the strings match after normalization (lowercase, trim, collapse whitespace). The strictest scorer; perfect for labels, ids, yes/no. Normalization matters: a scorer that fails on a trailing space produces false regressions and trains the team to ignore the eval.
contains(output, reference) — 1.0 iff the reference appears as a substring. The workhorse for open answers where the model adds framing ("The capital of France is Paris.") around the required nugget ("Paris").
numeric_within(output, reference, tol) — parse the first number from each and compare numerically within a tolerance. Numbers are not strings: "144", "144.0", and " 144 " are the same answer, and tol is the domain slack you decide up front (a physicist's 3.14159 and a napkin's 3.14 agree to two places).
rubric_score(output, checks) — a weighted set of small programmatic assertions ("mentions the refund window" weight 1, "does NOT leak the key" weight 3), returning the weighted fraction that pass. This is the top of the programmatic rung: it captures more nuance than a single string match while staying deterministic and cheap. An empty rubric grades everything as perfect, so the lab raises on it — a silent, dangerous default caught.

The theme: each of these is a pure function of (output, reference), so it is free, instant, and reproducible. You reach past them only when the property genuinely cannot be asserted in code.

7. Trajectory evaluation: grade the path, not just the destination

Here is the idea that makes agent evaluation different from ordinary QA evaluation. A QA system has one output to grade: the answer. An agent has a whole trajectory — the sequence of tools it called, in what order, with what arguments — and the final answer is only its last step. Grading only the answer misses a whole class of bugs:

The agent that returns the right number by luck (it guessed, or a cached value happened to be right) instead of computing it — same answer, no reliability.
The agent that took an expensive detour — five tool calls where two would do — same answer, triple the cost and latency (Phase 00/14).
The agent that skipped a required verification step — same answer this time, a compliance violation the next.
The agent that called tools in the wrong order — searched after it answered, so the answer couldn't have used the search.

So you grade the path. The lab's trajectory_score(actual, expected, mode) offers three strictnesses:

exact — 1.0 only if the tool sequence is identical in content and order. The ground truth for "did it take the path we designed?"
order — a soft, order-sensitive score: the longest common subsequence length over the max length. Rewards getting most of the path right in order; penalizes reordering and gaps.
set — order-insensitive F1 over the tool sets: "did it use roughly the right tools, order aside?" The most forgiving view.

The order-sensitivity is the punchline: a correct-tools-but-wrong-order path scores 0.0 in exact mode and 1.0 in set mode. Which you use depends on the task — a fixed pipeline demands exact; a task where two independent lookups can happen in either order is fine with set. Choosing is the analysis, and being able to express both is why the lab implements all three.

8. Precision, recall, F1: the set and trajectory math

Trajectory set mode, retrieval quality (Phase 05), entity extraction — any task where the answer is a set of things — is graded with precision, recall, and F1. Let $P$ be the set the agent predicted (tools it used, chunks it retrieved) and $G$ the gold set (tools it should have used, chunks that were relevant). The overlap $|P \cap G|$ is the true positives. Then:

$$\text{precision} = \frac{|P \cap G|}{|P|}, \qquad \text{recall} = \frac{|P \cap G|}{|G|}.$$

Precision answers "of the things I did, how many were right?" — it punishes extra tools (wasted work, noise). Recall answers "of the things I should have done, how many did I?" — it punishes missing tools (incomplete work). They trade off: use every tool and recall is 1 but precision tanks; use only the one you're sure of and precision is 1 but recall tanks. F1 is their harmonic mean, which (unlike the arithmetic mean) is dragged down hard by either being low, so a good F1 requires both:

$$F_1 = 2 \cdot \frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}.$$

The harmonic mean is the right average here precisely because it refuses to let a great precision paper over a terrible recall. The lab's precision_recall_f1 also pins down the empty edges so the gate never divides by zero: an empty prediction is vacuously precise, an empty gold set is vacuously recalled, and F1 is 0 unless both are positive. Worked example: predict {search, wrong} against gold {search, calculator} — one true positive, so precision $= 1/2$, recall $= 1/2$, and $F_1 = 0.5$.

9. LLM-as-judge: how it works, pointwise vs pairwise

Some properties genuinely cannot be asserted in code — "is this summary faithful?", "is this reply helpful and appropriately toned?" For those you climb to the next rung: you ask another LLM to grade. This is LLM-as-a-judge, popularized by Zheng et al.'s MT-Bench / Chatbot Arena work, and it comes in two shapes:

Pointwise (single-answer grading): show the judge one output (and optionally a reference and a rubric) and ask for a score — "rate this 1–10" or "does this meet the bar? yes/no." The lab's injected deterministic_judge(output, reference) stands in for this: it returns a similarity in $[0,1]$. Cheap and simple, but the absolute number drifts — a judge's "7" today is not calibrated to its "7" last month.
Pairwise (comparison): show the judge two outputs (A and B) for the same input and ask which is better. This is far more reliable, because relative judgments are easier and more stable than absolute ones — it is how Chatbot Arena ranks models. The cost is that you need something to compare against (a baseline, or the other candidate in an A/B test).

The critical framing, and the reason this phase exists: the judge is a model, so the judge can be wrong. It is the same trust boundary as Phase 00 — you injected an LLM to do the task, now you're injecting an LLM to grade the task, and neither is an oracle. That is why the lab injects the judge as a plain function (so the harness stays deterministic and testable) and why the very next thing you do is measure whether the judge can be trusted at all (§11).

10. The judge is biased: position, verbosity, self-preference

An LLM judge does not just make random errors; it makes systematic ones — documented biases that skew results in a consistent direction. Three you must know cold, each modelled in the lab's judge_with_bias as a clean, deterministic perturbation so you can see it move the score:

Position bias. In a pairwise comparison, the judge tends to prefer whichever answer is shown first (or, for some models, second) — independent of quality. Zheng et al. measured this directly. The fix is mechanical: run the comparison both ways (A-then-B and B-then-A) and average, so the position cancels. The lab models it as +0.15 for position="first", -0.15 for "second".
Verbosity bias. Judges reliably score longer answers higher, mistaking length for thoroughness — even when the extra words add nothing. The lab models it as a length bonus, so a padded answer with the same content beats a terse one. The fix is to control for length (cap it, or normalize) and to have the rubric reward concision explicitly.
Self-preference (self-enhancement) bias. A judge rates outputs from its own model family higher than a different model's, all else equal. The lab models it as a +0.2 bonus when the output is self_authored. The fix is to make the judge model different from the model under test — never let a model grade its own homework.

These biases are not edge cases; they are the default behavior, and they are exactly why a raw judge score is untrustworthy until you have measured and controlled for them. Which is the next section.

11. Trust, but verify: Cohen's kappa

You are not allowed to let a judge gate CI until you have proven it agrees with humans. The metric is Cohen's kappa ($\kappa$), and the reason you can't just use raw agreement is subtle and important: two raters who both say "pass" 90% of the time will agree ~81% of the time by pure luck. Raw agreement overcounts because it gives credit for chance. Kappa subtracts that chance floor.

Let $p_o$ be the observed agreement — the fraction of items the judge and the human labelled identically. Let $p_e$ be the expected agreement by chance — if each rater just threw labels according to their own marginal frequencies, how often would they collide? For each category $c$, the chance both independently pick $c$ is $P_{\text{judge}}(c)\cdot P_{\text{human}}(c)$, so:

$$p_e = \sum_{c} P_{\text{judge}}(c), P_{\text{human}}(c), \qquad \kappa = \frac{p_o - p_e}{1 - p_e}.$$

The numerator is "how much better than chance did they agree," the denominator is "how much better than chance was even possible," so $\kappa$ is the fraction of the non-chance agreement they achieved. Read the endpoints:

$\kappa = 1$: perfect agreement ($p_o = 1$).
$\kappa = 0$: agreement is exactly what chance predicts — the judge is worthless.
$\kappa < 0$: worse than chance (they systematically disagree).

A concrete chance example the lab tests: raters [a,a,b,b] and [a,b,a,b] agree on 2 of 4 items ($p_o = 0.5$), but each is 50/50 so $p_e = 0.5\cdot0.5 + 0.5\cdot0.5 = 0.5$, giving $\kappa = (0.5-0.5)/(1-0.5) = 0$ — the raw 50% agreement was all luck. The rule of thumb (Landis & Koch): $\kappa \ge 0.6$ is "substantial," and the lab's judge_is_trustworthy gates on exactly that. Below 0.6, your judge is a random-number generator in a lab coat, and its scores may not gate a release — only inform one. Compute $\kappa$ on a human-labelled slice before the judge ever runs in CI.

12. Behavioral regression testing: evals as CI gates

Now assemble it into the thing that actually protects you in production. A behavioral regression test re-runs the golden set on every change and compares the result to a stored baseline. It is a unit test for behavior: instead of asserting a function returns 42, it asserts the agent's pass rate did not drop.

The lab's run_suite(cases, agent_fn, scorer) produces a SuiteResult with three things a single number would hide:

the overall pass rate (the headline),
a per-label breakdown (math, retrieval, safety … each with its own rate), and
trajectory accuracy (mean trajectory score — the path, not just the answer).

Then regression_gate(current, baseline, max_drop) fails the build if the pass rate dropped by more than max_drop versus the baseline. That max_drop is your tolerance for noise (a judge or a flaky tool adds a little); a real regression exceeds it. Wire this into a GitHub Action on every PR and you have evals-as-CI-gates: a merge that quietly breaks the refund flow is blocked automatically, before a customer finds it.

Why the per-label breakdown is load-bearing: a change can raise the overall pass rate 1% while collapsing one slice — lift 200 easy cases 2% and tank the 20 refund cases 40%, and the aggregate still goes up. One number hides regressions. Slicing by label is how you see the collapse, and it is the difference between an eval that catches the incident and one that ships it.

13. Safety is a gate, not a weighted average

There is one slice you never average into the headline: safety. If your agent leaks an API key, exfiltrates a customer record (Phase 10), or takes a destructive action without approval, that is not "a case worth 1/200 of the score" — it is a ship-blocker, full stop. A 99.5% pass rate where the 0.5% is "leaked the key" is not a great agent with a rounding error; it is an agent you cannot ship.

So safety is a gate, evaluated separately and absolutely. The lab's safety_gate scans the SuiteResult for any case labelled safety that failed and hard-fails the build if it finds one — regardless of the overall pass rate. This is why the golden set carries labels: so the safety slice can be pulled out and gated on its own, with a threshold of zero failures rather than a percentage.

The mental model: the regression gate asks "did we get worse on average?"; the safety gate asks "did we cross a bright line even once?" You need both, because averaging a bright-line violation into a mean is exactly how bright-line violations ship.

14. Retrieval vs generation, online eval, and feedback loops

Two extensions that round out the picture, both named directly in the JDs.

Split retrieval from generation. A RAG agent (Phases 05–06) has two failure surfaces that demand separate evals. Retrieval is graded with the set metrics of §8 — recall@k, context precision: did we fetch the relevant chunks? Generation is graded for faithfulness (did the answer stay grounded in the retrieved context, or hallucinate?) and answer relevance — the metrics ragas formalizes. The reason to split: an answer can be wrong because retrieval missed the fact or because generation ignored a fact it had, and those are different bugs with different fixes. One blended score can't tell you which.

Offline is not enough — close the online loop. The golden set is a snapshot; production is a moving distribution. So you also run online evals: A/B tests (ship the change to 5% of traffic, compare a live metric to control), guardrail metrics (refusal rate, exfil-scan hits, cost/latency from Phase 14), and human feedback loops — thumbs up/down, escalation rate, edited-response rate. This is Juniper Square's "feedback systems," and the loop is the whole point: production signals surface the failure modes your golden set didn't anticipate, and each one becomes golden case N+1 (§3). Offline evals catch what you thought of; online evals catch what you didn't, and feed it back so next time it's offline.

15. Common misconceptions

"A benchmark score is my eval." A benchmark measures a base model's general capability; your eval measures your composition on your task. They are different questions, and benchmarks leak into training and saturate. (§4)
"Just use an LLM judge, it's flexible." A judge is a model: non-deterministic, costs tokens, and biased (position, verbosity, self-preference). Reach for programmatic checks first, and never trust a judge you haven't validated against humans with $\kappa$. (§5, §10, §11)
"90% raw agreement means the judge is great." Not if both raters say "pass" 90% of the time — most of that agreement is chance. Kappa corrects for it; a $\kappa$ near 0 means the judge is worthless no matter how high the raw agreement. (§11)
"The final answer is what matters." For an agent, the trajectory matters too — a right answer via the wrong tools, wrong order, or an expensive detour is a latent bug. Grade the path. (§7)
"One pass-rate number is enough." A single aggregate hides slice regressions and, worse, averages a safety violation into a mean. Break out per-label, and gate safety separately and absolutely. (§12, §13)
"Evals are a pre-launch checklist." They are a continuous CI gate plus an online feedback loop; the golden set grows from every incident forever. (§12, §14)

16. Lab walkthrough

Open lab-01-agent-eval-harness/ and fill the TODOs top to bottom — the file is ordered to match this warmup:

Scorers — _normalize, then exact_match, contains, numeric_within (parse the first number, compare within tol), and rubric_score (weighted fraction of RubricChecks; raise on an empty rubric). The programmatic rung of §5/§6.
Trajectory — precision_recall_f1 (mind the empty edges), then trajectory_score with its exact / order (LCS) / set modes. The order-sensitivity test is the point (§7/§8).
Judge — deterministic_judge (token Jaccard), judge_with_bias (the three additive biases, clamped), then cohens_kappa (the $(p_o - p_e)/(1 - p_e)$ formula, with the 1 - p_e == 0 sentinel) and judge_is_trustworthy (§9–§11).
Suite + gates — pass_rate, run_suite (score, grade trajectory, per-label roll-up), regression_gate (fail on a drop), safety_gate (hard-fail on any safety miss) (§12/§13).

Run LAB_MODULE=solution pytest test_lab.py -v first to see green, then make your lab.py match. Finish by reading solution.py's main() output — it scores the golden set, computes $\kappa$ between the judge and human labels, and trips both the regression and safety gates.

17. Success criteria

You can explain why a single successful run is a sample of one and what an eval measures instead.
You can name the scoring ladder and justify pushing grading as far down it as possible.
You can explain trajectory evaluation and produce a case that scores 0.0 in exact mode and 1.0 in set mode — and say which mode a given task needs.
You can derive precision/recall/F1 for a set task and say why F1 uses the harmonic mean.
You can state the three judge biases and their fixes, and compute Cohen's $\kappa$, including why raw agreement overcounts.
You can explain evals-as-CI, why the per-label breakdown matters, and why safety is a separate hard gate.
All 34 lab tests pass under both lab and solution.

18. Interview Q&A

Q: Your agent "works great" in the demo. What do you do before shipping it? A: Treat the demo as a sample of one. Build a golden set — curated, versioned, labelled, with adversarial and safety cases drawn from real traffic — and measure the pass rate, a per-label breakdown, and the trajectory accuracy over it. Wire it into CI as a regression gate with safety as a separate hard gate. Now "it works" is a number I can defend and a merge check that protects it.

Q: When would you use an LLM-as-judge, and what has to be true before you trust it? A: Only for properties I genuinely can't assert in code — tone, helpfulness, faithfulness — after I've exhausted programmatic scorers. Before trusting it I label a human slice and compute Cohen's $\kappa$ between the judge and the humans; I require $\kappa \ge 0.6$ before it may gate CI. I also control for its biases: run pairwise comparisons both ways to cancel position bias, control for length against verbosity bias, and use a different model as judge than the one under test to avoid self-preference.

Q: Why grade the trajectory and not just the final answer? A: Because an agent can get the right answer the wrong way — by luck, via an expensive detour, in the wrong tool order, or by skipping a required verification. Those are latent bugs the final answer hides. I grade the tool sequence: exact order for a fixed pipeline, set-overlap F1 when order is genuinely free. A correct-answer-wrong-order run scores 0.0 exact and 1.0 set, and which I gate on is a design decision.

Q: What's wrong with using raw agreement to validate a judge? A: It gives credit for chance. Two raters who each say "pass" 90% of the time agree 81% of the time by luck alone, so 81% raw agreement can mean a worthless judge. Cohen's $\kappa = (p_o - p_e)/(1 - p_e)$ subtracts the chance floor $p_e$; $\kappa = 0$ is exactly-chance, and I gate on $\kappa \ge 0.6$.

Q: A change raises the overall pass rate by 1%. Ship it? A: Not on that number alone. I check the per-label breakdown, because an aggregate can rise while a slice collapses — lift 200 easy cases and tank the 20 refund cases and the mean still goes up. And I check the safety gate independently: if any safety case regressed, it's a hard block regardless of the aggregate. One number hides regressions; that's the whole reason to slice and to gate safety separately.

Q: How do you evaluate a RAG agent? A: Split retrieval from generation. Grade retrieval with set metrics — recall@k, context precision (did we fetch the relevant chunks?). Grade generation for faithfulness (is the answer grounded in the retrieved context or hallucinated?) and answer relevance. A blended score can't tell you whether a wrong answer came from bad retrieval or bad generation, and those need different fixes. (Cross-refs Phase 05/06.)

Q: Offline evals pass but users complain. What's missing? A: The online loop. The golden set is a snapshot; production is a moving distribution. I add online evals — A/B tests against a live metric, guardrail metrics (refusal/exfil/cost/latency), and human feedback (thumbs, edits, escalations) — and I feed every new failure back into the golden set so next time it's caught offline. Offline catches what I thought of; online catches what I didn't.

19. References

Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023) — the LLM-as-judge method, and the position/verbosity/self-enhancement bias measurements. https://arxiv.org/abs/2306.05685
Cohen, J., A Coefficient of Agreement for Nominal Scales (1960) — the original $\kappa$. https://doi.org/10.1177/001316446002000104
Landis & Koch, The Measurement of Observer Agreement for Categorical Data (1977) — the $\kappa$ interpretation thresholds (the 0.6 "substantial" rule of thumb). https://doi.org/10.2307/2529310
Es et al., RAGAS: Automated Evaluation of Retrieval Augmented Generation (2023) — faithfulness / answer-relevance / context-precision, the retrieval-vs-generation split. https://arxiv.org/abs/2309.15217
Ribeiro et al., Beyond Accuracy: Behavioral Testing of NLP Models with CheckList (ACL 2020) — behavioral / regression testing of language models. https://arxiv.org/abs/2005.04118
OpenAI Evals — the open-source eval framework (datasets + graders + registry). https://github.com/openai/evals
Anthropic, Building Effective Agents (2024) and the Anthropic evals guidance — "measure before you optimize," evaluating agentic systems. https://www.anthropic.com/research/building-effective-agents
LangSmith / Braintrust docs — datasets, trajectory evaluators, and LLM-as-judge in a production eval pipeline. https://docs.smith.langchain.com/
Liu et al., G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (2023) — a reference-free LLM-judge with chain-of-thought scoring. https://arxiv.org/abs/2303.16634

Agentic AI Engineer — Agent Infrastructure, Platforms & Production Systems