« Phase 11 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor

Phase 11 — Staff Engineer Notes: Agent Evaluation — LLM-as-Judge, Golden Datasets & Behavioral Regression

Anyone can wire up an eval harness. The line this phase actually trains is between the engineer who runs an eval and the engineer trusted to own the quality bar — to decide whether an agent is good enough to ship, and to defend that decision in a design review when a launch is on the line. That ownership is a set of judgment calls, not a set of functions.

What a staff engineer owns here

Not the harness — the harness is a weekend. You own the golden set (what goes in it, what an incident distills to, what "representative" means for this product), the κ threshold and which judge may gate, the max_drop noise budget, the definition of a safety case, and the escalation policy up the scoring ladder. Every one of those is a decision with a blast radius, and every one is where the quality bar actually lives. The person who owns the golden set owns the release bar, and that is a staff-level responsibility precisely because a wrong call ships a regression or blocks a good launch.

The decision frameworks

Which rung of the scoring ladder? Default to the bottom and justify every step up. "Does the reply contain the order number?" and "does it avoid leaking the account id?" are programmatic — assert them in code, free and deterministic. Only "is the tone appropriately empathetic?" earns a judge, and only after you have exhausted rubric checks. The junior instinct is to reach straight for the LLM-judge because it is flexible; the staff instinct is to notice how much of "good" is actually a set of code-checkable assertions and push grading as far down the ladder as it goes. A human is for calibrating the judge and adjudicating the rare hard case — never the default.

Which trajectory mode? This is a task-shape decision, not a preference. A fixed compliance pipeline (lookup_order → check_policy → draft_reply) demands exact — wrong order is a violation. Two genuinely independent lookups are fine with set. When most of the path is fixed but some steps are interchangeable, order (the soft LCS score) is the honest middle. Being able to express and defend the choice is the signal; gating a free-order task on exact trains the team to ignore trajectory noise.

When may a judge gate CI? Only after κ ≥ 0.6 against a human-labelled slice, re-measured whenever the judge model or prompt changes. Below that bar the judge informs, it does not gate. A staff engineer never lets an un-validated judge into the merge path — that is the difference between a team that does evals and a team that has an eval dashboard.

Offline vs online. Offline (golden set in CI) catches what you thought of; online (A/B, guardrail metrics, thumbs/edits/escalations) catches what you didn't. You own both, and you own the loop that turns every online failure into offline case N+1.

Code-review red flags — reject the PR on these

  • The agent is handed the whole EvalCase instead of case.input. It can now read reference; you are grading a leak. Hard reject.
  • A new LLM-judge gate with no κ measurement in the same PR. "Trust me, the scores look right" is a random-number generator in a lab coat.
  • A number compared as a string (output == "144") where numeric_within belongs — a false-regression factory.
  • A rubric with zero checks, or a scorer that silently returns 1.0 on empty input. Empty-means-perfect is the most dangerous default in eval code.
  • A single aggregate pass rate with no per-label breakdown. One number hides the slice that collapsed.
  • A safety case folded into the weighted score instead of a separate gate. Averaging a bright-line violation is exactly how bright-line violations ship.
  • max_drop = 0 on a judge-scored suite. It will flap on sampling noise and the team will learn to bypass the gate — worse than no gate.
  • A judge that is the same model as the agent under test. Self-preference bias, guaranteed. Never let a model grade its own homework.

Production war stories

  • The judge that loved long answers. A team gated releases on a helpfulness judge and "improved" by making answers wordier. Users hated it; the judge had verbosity bias and nobody had checked κ. The green gate was steering the product the wrong way. Fix: control for length, re-validate against humans — one afternoon.
  • The 1% that broke refunds. Overall pass rate rose 1% after a prompt tweak; the refund slice had quietly dropped 94% → 61%. No per-label view, so it shipped, and customers found it before the dashboard did.
  • The agent that guessed right. Final-answer eval was green for months. Trajectory eval revealed it was returning cached values without calling the pricing tool — correct by luck, one stale cache from wrong.
  • The safety case averaged away. A "99.7% pass" release leaked one account id. It was one row in a mean, so the aggregate gate let it through. Safety has been a separate hard gate ever since.

Each of these is a specific silent lie a simpler design tells you, and each is a story that lands in a design review because it is concrete.

The interview signal — what the room is actually listening for

The four JDs this phase answers (see jd.md) split by what "eval" means to them, and the signal differs by room:

  • Docker — "LLM-as-judge, behavioral regression testing, golden datasets." Bring the whole CI-gate story: golden set in the repo, regression gate on every PR, safety gate separate. They said the words; say them back with the mechanism.
  • Anthropic — "agent prompts & evals," frontier eval judgment. Lean into judge calibration, the three biases, and κ. They will ask why you trust a judge; "I measure its agreement with humans and gate on κ ≥ 0.6" is the answer that lands.
  • Juniper Square — "evaluation frameworks" and "feedback systems." That is the online loop: A/B, thumbs, and feeding production failures back into the golden set.
  • Wolters Kluwer — "AI-agent evaluation," enterprise discipline: versioned datasets, per-slice reporting, auditable gates.

The universal tell an interviewer or architecture review listens for: do you treat a single green run as evidence, or do you say "a demo is a sample of one from a distribution I haven't characterized"? Do you reach for a judge first, or last? Do you know why 81%-by-chance agreement is worthless without κ? Do you grade the path, not just the answer? Say "I also grade the trajectory, because an agent can be right via the wrong tools, wrong order, or an expensive detour" and you have signaled you have been paged by an agent, not just demoed one.

Closing takeaways

  1. The golden set is the moat, not the model. The model is swappable in an afternoon; the few hundred hard-won cases encoding every way your product has broken are the asset. Own it, version it, grow it from every incident.
  2. Climb the ladder from the bottom. Programmatic before judge before human — always. Every case a code check can grade is a judge call you don't pay for and can't be flaked by.
  3. A judge is a model you must validate. Never gate CI on a judge below κ ≥ 0.6, and control its biases (swap positions, cap length, cross-model judge). An un-validated judge is worse than no judge — it steers with authority in the wrong direction.
  4. Grade the path, not just the destination. A lucky right answer is still a bug. Trajectory mode is a task-shape decision you own.
  5. Two gates, two questions. Regression asks "did we get worse on average?"; safety asks "did we cross a bright line even once?" Safety is a gate with threshold zero, never a term in a mean.
  6. One number hides regressions. Slice by label, every time. The scary regression always looks fine in aggregate.