« Phase 33 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor

Phase 33 — Staff Engineer Notes: Eval-Driven Development

There is a wide gap between an engineer who runs an eval and one trusted to own the eval program that gates a team's releases. The first produces a number. The second owns the definition of "done," decides what the gate blocks on, defends the release to a skeptical VP, and takes the pager when a regression ships anyway. This document is about that second person: the judgment calls they own, the decision frameworks they carry, the red flags they catch in review, the war stories that formed those instincts, and the exact signal an interviewer or an architecture review is listening for.

The judgment that separates owner from user

Anyone can point a harness at a dataset. The owner's judgment shows in four decisions a user never makes. What "done" means — turning "make the PR agent good" into "resolves the ticket (execution), doesn't break existing tests (PASS_TO_PASS), stays in scope (safety), under $0.50 (budget)," each naming a grader and a slice. What the gate blocks on — and specifically that safety is a veto and not a term in a weighted average, a decision that will be argued with every quarter by someone who wants to ship at 99% with one leaked secret. When to trust the number — knowing that a 2-point move on 50 tasks is a hypothesis, not a result, and refusing to ship on it. And when the eval itself is wrong — the discipline to suspect the grader when the score and reality disagree, rather than trusting a green gate over a complaining customer. The user optimizes the number; the owner is responsible for whether the number means anything.

The grader decision framework: execution vs judge vs human

This is the single most-tested judgment call in the phase, and the answer is a decision tree, not a preference.

  • Reach for execution when an execution oracle exists. For code that means: is there a test, or can you write one? If yes, run it — it is crisp, unbiased, reproducible, and free of model bias. This is why HumanEval and SWE-bench run the code. Default here for anything functional.
  • Reach for assertions when the output has structure but no full test. Schema conformance, a required substring, a numeric tolerance, "one sentence ending in a period." Cheap, deterministic, and it covers more than people expect if you frame criteria concretely.
  • Reach for an LLM-judge only for genuinely fuzzy outputs — tone, helpfulness, explanation quality, open-ended prose — where no execution or assertion oracle exists. And when you do, you owe the org a validation: measure the judge against human labels with Cohen's κ before it gates anything, prefer pairwise ("is A or B better") over absolute 1-to-5 scoring, and keep a rotating human sample to re-check it. A judge you haven't validated is a random-number generator in a lab coat.
  • Reach for human review for the high-stakes and the un-mechanizable — and to periodically validate the cheaper rungs above it.

The staff-level failure is inversion: using a judge to grade code because it's easy to wire up, when an execution oracle was sitting right there. "We'll have a model grade the patches" instead of running the tests is a tell that someone hasn't shipped a coding agent.

Building the error-analysis flywheel as an org practice

The highest-leverage activity in applied LLM work is not modeling — it is reading your failures and clustering them. Making that an org practice rather than a heroic individual act is a staff responsibility, and it has a shape: a standing weekly ritual where someone owns pulling the week's failures (every thumbs-down, every escalation, every incident, plus a representative sample), open-codes each with a short "why it failed" label, ranks the failure modes by count, fixes the top bucket, and promotes representatives of it into the golden set (deduped) so it can never silently return. That last step is what turns error analysis into a flywheel: the ranked report becomes a roadmap ("one fix — stop editing generated files — recovers a third of failures"), and the promoted cases become permanent regression tests. A team that does this weekly is climbing a gradient; a team staring at a single accuracy dial is guessing. The moat is not the model — it is the golden set that encodes every mistake your system has made in production, which no competitor and no public benchmark has.

Code-review red flags

In review of an eval PR or an eval-gated change, these are the things that should stop the merge:

  • Evals written after the agent. They are shaped by the solution you already built — you're grading the target you already hit. The whole premise is first.
  • No held-out set. Iterating against the dev set with nothing held back is training on the test set; the dev number climbs while reality doesn't, and you have no instrument to catch it.
  • Moving the goalposts. Editing a case's reference so it passes, loosening a threshold to make the gate green, or dropping a failing task "because it's flaky." The dataset version should make this visible — a reference change moves the content hash — and review should treat a goalpost move as a regression, not a fix.
  • One aggregate number, no slices, no safety gate. A single scalar hides a collapsed slice, a flaky case, and a safety regression. If the PR reports "82%" with no per-slice breakdown, it's a rumor.
  • Gaming pass@1 with retries. A gate that flips green on re-run has quietly become pass@many; someone hitting "re-run until green" is defeating the flake policy.
  • A judge grading code, or an unvalidated judge grading anything. No κ, no human sample, gating a release — block it.

Production war stories

  • The 2-point win that was noise. A team celebrated 74% → 76% on 50 tasks and shipped. The standard error at N=50 is about 6 points; the "improvement" sat inside a ±12-point band. A bootstrap CI and a paired test would have said "no signal." They chased noise for a sprint. The lesson: never call a delta an improvement without a CI or a paired comparison.
  • The score went up and customers got angrier. Months of prompt-tuning lifted the dev set 8 points; a held-out set — which they didn't have — would have shown it dropping. Textbook overfitting: they trained on the test set. A held-out slice plus the flywheel fixed it in two weeks. The lesson: the dev gate and the overfitting detector catch different failures, and you need both.
  • The coding agent that "fixed" the bug and broke prod. The patch greened the new test; nobody ran the existing tests. FAIL_TO_PASS green, PASS_TO_PASS untested. Adding the pass-to-pass split turned a silent regression into a blocked merge. The lesson: "the feature works" is not the verdict — "the feature works AND nothing broke" is.
  • The contamination that inflated the number. A benchmark-derived slice scored suspiciously high because the model had seen those exact problems in training. The real capability was lower; the eval was measuring memorization. The lesson: your own golden set, grown from your production failures, is the number you can trust — public benchmarks measure something adjacent.

The signal interviewers and architecture reviews listen for

They are not listening for whether you can define pass@k — that's table stakes. They are listening for whether you treat the eval as an instrument you distrust by default. The tells that you own eval programs rather than run evals: you say "on dataset v7" without being asked, because you know a score without a dataset version is a rumor. You reach for a paired comparison unprompted, because you know a +2% aggregate can hide "fixed five, broke four." You put safety on a separate gate and refuse to average it, and you can say why (a mean is lossy; safety is the signal you can't lose). You name execution before judgment for code, and you name κ the moment a judge appears. And you can state where the TDD analogy breaks — fuzzy oracle, stochastic subject, overfittable suite, open distribution — because naming the limits of your own analogy is the clearest seniority signal there is.

Closing takeaways

  1. Own the definition of "done," not just the number. The eval is the spec; whoever owns it owns what the team is allowed to ship.
  2. Distrust the instrument. A green gate over a complaining customer means the eval is wrong — a bad grader, a stale set, or an overfit suite — and finding which is the job.
  3. Execution beats judgment for code; a judge you haven't validated is noise. Climb from the cheapest reliable rung, and earn any judge's right to gate with κ.
  4. Safety is a veto; never a term in a mean. One safety failure blocks the release regardless of the aggregate. This is non-negotiable and you will have to defend it repeatedly.
  5. The flywheel is the moat. Read failures, cluster, fix the top bucket, promote it into the set, keep a rotating held-out slice. Curation beats volume, and your grown golden set is knowledge no one else has.
  6. The senior one-liner: "I don't ship agent changes I can't defend with a number — a versioned golden set, an execution grader, a paired comparison, and a gate where safety never averages out."