« Phase 33 README · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 33 Warmup — Eval-Driven Development: Building Agentic Systems for Software Delivery

Who this is for: someone who has already built the agent loop (Phases 00-17), framework internals (18-24), and — critically — the eval mechanisms of Phase 11 and the coding-agent harness of Phase 15, and now needs the methodology that puts evals at the center of how a team actually builds and ships an agentic system for software delivery. By the end you will be able to explain, from first principles, why traditional testing breaks for LLM systems and what survives; how to turn an ambiguous product goal into an eval; how to design graders for code (execution) versus prose (judges); the statistics that keep a 50-task eval set from lying to you (pass@k, bootstrap, the overfitting trap); and how to wire all of it into a CI-gated delivery pipeline with an error-analysis flywheel. No API keys, no GPUs, no network — everything in the labs is mechanism.

Table of Contents

  1. What Eval-Driven Development actually is
  2. Why traditional testing breaks for LLM systems — and what carries over
  3. Framing success: from an ambiguous goal to a checkable criterion
  4. The golden task set: build it FIRST
  5. The eval taxonomy for coding and software-delivery agents
  6. Grader design: the scoring ladder
  7. pass@k and the statistics of small eval sets
  8. Bootstrap, significance, and the two-point-delta trap
  9. Error analysis: the highest-leverage activity in applied LLM work
  10. The data flywheel: production telemetry becomes eval cases
  11. The regression gate and eval-gated delivery in CI
  12. Held-out sets and the overfitting trap
  13. Latency and cost as first-class eval dimensions
  14. Real-world grounding: SWE-bench, HumanEval, and the tooling landscape
  15. EDD vs TDD: the honest analogy and where it breaks
  16. Common misconceptions
  17. Lab walkthrough
  18. Success criteria
  19. Interview Q&A
  20. References

1. What Eval-Driven Development actually is

Eval-Driven Development (EDD) is the discipline of building an LLM/agentic system the way Test-Driven Development builds software: you define success as an executable eval before you build the thing, you measure every change against that eval, and you never ship a change that the eval doesn't bless. The eval set is not a report you generate at the end of a project — it is the specification you write at the start, the artifact that decides what "done" means, what to build next, and whether a change is an improvement or a regression dressed up as one.

Read that against the two Cohere Forward-Deployed-Engineer JD lines this phase answers. "Build robust evaluation frameworks, moving well beyond trial and error, to measure agent accuracy, safety, and latency" — "beyond trial and error" is the entire thesis: trial-and-error is opening a playground, trying five prompts, and shipping the one that felt best; an eval framework replaces "felt best" with a number, a confidence interval, and a gate. And "translate high-value, ambiguous business problems into well-framed agentic workflows with clear success criteria and evaluation methodologies" — that is EDD's first step (§3): the skill of turning "make our engineers faster without lowering quality" into a golden set with graders.

The reason this is a methodology and not just "run some evals" is the ordering and the loop. Anyone can bolt an eval onto a finished agent. EDD says the eval comes first and drives the build, and that a small set of steps repeats forever: frame success → build the golden set → write graders → baseline → iterate against the evals → analyze the failures → gate every change → sample production and grow the set. This phase builds the machinery for that loop (Labs 01-03) and this Warmup is the theory behind it.

One sentence to anchor everything: for an agentic system, "write the spec" and "write the eval" are the same act.

2. Why traditional testing breaks for LLM systems — and what carries over

A unit test rests on three assumptions that an LLM system violates:

Assumption 1 — determinism. assert add(2, 3) == 5 passes or fails the same way every run. An LLM samples: the same prompt at temperature > 0 yields different outputs, and even at temperature 0 a model version bump, a different GPU kernel, or a batching change can shift the output. A single green run is therefore one draw from a distribution you have not characterized — it tells you the agent can succeed, not that it will. This is why §7's pass@k exists: you must estimate the probability of success from multiple samples, not assert a single outcome.

Assumption 2 — a single correct output. add has exactly one right answer. "Summarize this diff," "reply to this ticket," "refactor this function" have a space of acceptable outputs and a fuzzy boundary. assertEqual(output, reference) is hopeless — the model's perfectly good summary that happens to use different words fails. You need graders that accept a set of good answers: substring/semantic checks, rubric judges, or — the escape hatch for code — an execution grader where the space of correct programs is huge but "does it pass the tests" is crisp again (§6).

Assumption 3 — an enumerable behavioral surface. You can list the branches of a pure function. You cannot list the inputs to a coding agent; the space of GitHub issues is effectively infinite, and the agent composes tools, plans, and edits in ways you didn't anticipate. So you can never "cover" the surface — you can only sample it representatively and grow the sample from production (§10). Coverage is replaced by a curated, versioned, ever-growing dataset.

And the failure mode when teams don't internalize this: vibe-checking. Someone runs three prompts in a playground, they look good, ship it. Vibe-checking feels like testing and scales to exactly zero — it has no denominator, no slices, no regression detection, and no memory. It is the thing "beyond trial and error" is contrasted against.

What carries over is more than people expect, and saying so signals maturity:

  • Fixtures. The golden dataset is a fixture set — reviewed, diffed, versioned in Git.
  • CI. Evals run in the pipeline on every change, exactly like tests (§11).
  • Regression discipline. "Don't merge a change that breaks a passing case" is unchanged; only the definition of "passing" got statistical.
  • Determinism where you can get it. You inject and stub the model (this whole track's trick), seed any randomness, and freeze dataset versions — so the harness is deterministic even though the model isn't. That is why the labs stub the agent as a pure function.

The one-liner: an LLM eval is a test whose oracle is fuzzy and whose subject is stochastic — so you replace exact assertions with graders, single runs with distributions, and coverage with a curated growing dataset, but you keep fixtures, CI, and regression discipline.

3. Framing success: from an ambiguous goal to a checkable criterion

The hardest part of EDD is not the statistics — it is turning "make the PR-review agent good" into something a machine can score. This is the Cohere "translate ambiguous business problems into clear success criteria" skill, and it is the step most teams skip on the way to a demo.

The move is to decompose a vague goal into observable, checkable criteria, each of which becomes one or more evals. For a feature-delivery / PR-review agent, "good" decomposes into:

  • Resolution — does the change actually accomplish the ticket? (an end-to-end eval: apply the patch, run the repo's tests — Lab 02's fail_to_pass.)
  • No regression — did it break something that worked? (Lab 02's pass_to_pass — the single most-forgotten criterion.)
  • Scope / safety — did it edit only files it was allowed to, avoid leaking secrets, avoid destructive actions? (a trajectory/safety eval — Lab 01/02's allowed_files and safety tag.)
  • Spec conformance — does the output obey the stated contract (naming, API shape, style)? (a conformance eval — assertion or judge.)
  • Cost & latency — did it land under the token/dollar/step budget? (a first-class dimension — §13.)

Notice each criterion names a grader type and a slice. That is not an accident: a well-framed criterion already tells you how to grade it and how to slice the report. "Be helpful" is not a criterion; "resolves the ticket (execution grader), on the backend and frontend slices, without editing generated files (safety grader), under $0.50 (cost grader)" is. The output of §3 is a list like that — and it is simultaneously the product spec and the eval design.

A practical tip from applied practice: write the criteria with the domain expert or PM, in their language, then translate each into a grader. If you can't state how you'd check a criterion, you haven't framed it — you've restated the vague goal.

4. The golden task set: build it FIRST

The golden set is a curated, versioned, representative-plus-adversarial collection of tasks, each with the inputs, a way to grade the outcome, and slice tags — and in EDD you build it before the agent is any good. This is Phase 11's "the golden dataset is the moat," with the EDD-specific insistence on timing: first.

Why first? Three reasons.

  1. It forces §3. You cannot write 30 golden tasks without deciding what success means. The dataset is where the vague goal goes to become concrete.
  2. It gives you a baseline. Run the dumbest agent that could work against the set and you have a number — 40% resolved — instead of a vibe. Every subsequent change is measured against that baseline (Lab 01's paired comparison).
  3. It prevents the worst anti-pattern: building the agent, then writing evals that happen to pass — grading the target you already hit. Evals written after the fact are shaped by the solution and measure nothing.

Composition matters more than size. A 40-task set that covers your real distribution plus the adversarial edges beats a 400-task set of near-duplicates. Curate deliberately: representative common cases, known hard cases, past incidents, and safety probes. Tag every case with slices (backend/frontend, bugfix/feature/refactor, safety) so the report can break the score down — because a single aggregate hides everything that matters (§6, §8).

Versioning is non-negotiable. The set grows (§10), so a score is only meaningful against a named version. Lab 01 and Lab 03 hash the dataset content: change a case and the version changes, and every report is pinned to the exact dataset that produced it. "We're at 82%" is a rumor; "we're at 82% on dataset v7" is a fact you can reproduce.

Numbers to hold: a useful starting golden set is 20-50 tasks; it grows into the hundreds as the flywheel turns; you almost never need thousands unless you're benchmarking, and past a point curation beats volume every time.

5. The eval taxonomy for coding and software-delivery agents

Not all evals are the same shape. For a coding/delivery agent, five types do the work, from cheapest-and-narrowest to richest-and-most-expensive:

Capability evals (single-turn). "Given this signature and docstring, write a correct function." The grader runs the function against hidden tests — this is the HumanEval / MBPP shape, scored with pass@k. Cheap, fast, deterministic-given-the-code, and the right first rung: if the agent can't write a correct standalone function, nothing downstream matters. Lab 02's add-bug and shout-feature are these.

Trajectory evals. Grade the path, not just the destination:did the agent take sensible steps, call the right tools in a defensible order, and stay in bounds? Phase 11 built the trajectory-scoring mechanisms (exact/LCS/set-overlap); here the delivery-specific one is scope: did the agent edit only allowed_files? A right answer produced by editing secrets.py is a failure. Lab 02's route-fix safety trap is a trajectory eval.

End-to-end task evals. The SWE-bench shape: a whole repo at a base commit, a real issue, and the agent's job is to produce a patch that makes the repo's tests pass. The grader is execution over the full repo with the FAIL_TO_PASS / PASS_TO_PASS split (§6, Lab 02). This is the closest proxy to "did it actually deliver the feature," and the most expensive to run.

Spec-conformance evals. Does the output obey a stated contract — function named as specified, JSON matching a schema, style rules followed? Often assertion-graded, sometimes judge-graded. This is where Phase 15's "spec is the contract and the eval" lives.

Safety / regression evals. Two hard gates that are never averaged into the main score: did any previously-passing case regress (§11), and did any safety property fail (secret leak, destructive command, out-of-scope edit)? A safety failure blocks the release regardless of how good the aggregate looks — Lab 01 and Lab 03 make this a non-configurable gate.

And crosscutting all five: latency and cost budgets (§13) as dimensions every task is also scored on. A patch that resolves the issue after 40 tool calls and $12 is not a pass on a budget of 6 calls and $0.50.

The taxonomy is a ladder of fidelity and cost: capability evals are cheap and run per-commit; end-to-end evals are expensive and run pre-release. Choosing which runs when is §11.

6. Grader design: the scoring ladder

A grader turns an output into a score. The cardinal rule, straight from Phase 11, is climb from the cheapest reliable rung first:

  1. Execution / programmatic graders. For code, run it. Apply the patch, run the tests, score resolved iff they pass. This is the gold standard for code and the reason "the judge model can grade everything" is wrong: a program is either correct against its tests or it isn't, and no amount of model judgment substitutes for execution. Lab 02 is built entirely on this. The critical refinement is the FAIL_TO_PASS / PASS_TO_PASS split: the fail-to-pass tests prove the new behavior works (capability); the pass-to-pass tests prove nothing broke (regression). A patch that satisfies fail-to-pass but breaks a pass-to-pass test is not a fix — Lab 02's clamp-bug regression trap makes this concrete, and it is the mistake most naive coding agents make.

  2. Assertion / predicate graders. Not everything is a full test suite. exact_match, contains_all, a numeric tolerance, a regex, a custom predicate ("the summary is one sentence and ends with a period"). Lab 01's exact / contains_all / predicate graders are these — and note the discipline: a predicate that raises (a failed assert) is a structured failure, not a harness crash. The grader catches it and records a FAIL with the exception in the detail.

  3. Rubric-based LLM-judge. For genuinely fuzzy outputs — tone, helpfulness, explanation quality — you ask a model to score against a rubric. This is powerful and dangerous: an LLM-judge is a model, and a model is biased (position bias, verbosity bias, self-preference) and can simply be wrong. Phase 11's rule is binding here: validate the judge against human labels with Cohen's κ before you let it gate anything. Lab 01's rubric grader takes an injected, deterministic judge returning per-criterion scores precisely so the harness stays testable — but in production that judge is an LLM call whose calibration you must earn.

  4. Pairwise comparison. Instead of "score this output 1-5" (hard to calibrate), ask "is A or B better?" (easier and more reliable). Pairwise is how you compare two agent versions when there's no reference answer, and it underlies preference data and Elo-style rankings. Its cost is that it's relative, not absolute — it tells you A beats B, not whether either is good.

  5. Human-in-the-loop. The most expensive, slowest, most reliable rung — reserved for the cases genuinely too fuzzy or too high-stakes to mechanize, and for periodically validating the cheaper rungs (the κ measurement in rung 3 needs human labels).

The engineering point: most of your evals should be on rungs 1-2, a curated few on rung 3, and a small sample on rung 5 to keep rung 3 honest. A suite that leans on an unvalidated judge for everything is slow, expensive, and quietly wrong.

7. pass@k and the statistics of small eval sets

Because the agent samples (§2), you estimate a probability of success per task, not a single outcome. The standard metric is pass@k: the probability that at least one of k independent samples is correct.

The naive estimator — sample k times, check if any passed, repeat — is high-variance and wasteful. The unbiased estimator (Chen et al., 2021, the OpenAI Codex / HumanEval paper) is: draw n ≥ k samples per task, count c correct, and compute

$$ \text{pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}} $$

Read it as: 1 − P(all k drawn samples are failures). \(\binom{n-c}{k}\) is the number of ways to choose k samples entirely from the n − c failures; over \(\binom{n}{k}\) total ways to choose k, that's the probability a size-k draw is all-failures; one minus it is "at least one passes."

Concrete values you should be able to produce on a whiteboard (Lab 01 and Lab 02 test these exactly):

  • n=5, c=2, k=1 → \(1 - \binom{3}{1}/\binom{5}{1} = 1 - 3/5 = 0.4\). (With one draw, pass@1 is just the base rate c/n.)
  • n=5, c=2, k=2 → \(1 - \binom{3}{2}/\binom{5}{2} = 1 - 3/10 = 0.7\).
  • n=5, c=4, k=2 → only 1 failure exists, so any pair contains a pass → 1.0. (Whenever n − c < k, the estimator is 1.)

Why not the naive 1 − (1 − c/n)^k? Because c/n is itself an estimate from a finite sample, and plugging it into that formula is biased upward for small n — you'd systematically overstate pass@k. The combinatorial estimator corrects for the finite sample. This is a favorite interview probe precisely because it separates people who've run code evals from people who've read about them.

What pass@k means for your product number. pass@1 is the honest "one-shot" quality — what a user gets on a single greedy generation, and usually the number you report as the product metric. pass@k for k > 1 characterizes the distribution and matters when your agent can retry, self-verify, or sample-and-rank (which a good coding agent does — it runs the tests and tries again). A large gap between pass@1 and pass@10 says "the model can do it but isn't reliable" — which points you at verification and retry, not at the base model.

8. Bootstrap, significance, and the two-point-delta trap

Here is the mistake that quietly wastes months: you tweak a prompt, the eval goes from 74% to 76% on your 50-task set, you declare victory and ship. On 50 tasks, a two-point move is almost certainly noise. The standard error of a proportion is \(\sqrt{p(1-p)/N}\); at p≈0.75, N=50 that's about 0.061 — a 95% confidence interval of roughly ±12 points. Your "improvement" is a rounding error inside the noise band.

Two tools fix this:

Bootstrap confidence intervals. Resample your N tasks with replacement B times (say B=1000), recompute the pass rate each time, and take the 2.5th and 97.5th percentiles of those B values as a 95% CI. It needs no distributional assumptions and directly shows the uncertainty in your headline number. Report the CI, not just the point estimate — "76% (95% CI 64-86)" tells the truth that "76%" hides.

Paired, per-task comparison. When comparing two agent versions, don't compare the two aggregate rates — compare them per task. Because the same tasks are used for both, a paired analysis cancels task difficulty and has far more power: count wins (candidate right, baseline wrong), losses (the reverse), and ties, and look at the net. Lab 01's compare_runs does exactly this, and the reason is the killer scenario: a candidate that improves the aggregate by 2 points might have fixed five tasks and broken four — the aggregate says "+2%," the paired view says "you regressed four things," and the four regressions are what page you at 2 a.m. A paired McNemar-style view (and a sign test on the discordant pairs) is the statistically honest way to say "is B actually better than A."

The overfitting version of the same trap is §12: if you tune against the eval set long enough, you'll improve it without improving reality. That's not noise — it's leakage, and it needs a held-out set to catch.

The discipline: never call a delta an improvement without a confidence interval or a paired test, and never trust an aggregate without the per-slice breakdown. A 2-point move on 50 tasks is a hypothesis, not a result.

9. Error analysis: the highest-leverage activity in applied LLM work

If you take one practice from this phase into your job, take this one. Applied-AI practitioners (Hamel Husain's evals writing, Eugene Yan's essays) are near-unanimous: the single highest-leverage activity in improving an LLM system is reading your failures and categorizing them. Not adding more eval cases, not swapping models, not prompt-golfing — reading the actual failures and clustering them into failure modes.

The process, concretely:

  1. Run the eval, collect every failure with its input, the agent's output, and enough trace to see what happened.
  2. Open-code the failures — read them and assign each a short label describing why it failed: "off-by-one on the boundary," "edited a generated file," "hallucinated an endpoint that doesn't exist," "correct logic, wrong output format." Early on this is human work; later a classifier or an LLM can assign the label. Lab 03's Failure.signature and injected signer model this step.
  3. Count and rank the modes. Now you have a ranked failure-mode report: "38% of failures are the generated-file mode, 22% are format, 15% are boundary." Lab 03's analyze_failures produces exactly this. Suddenly "we're at 72%" becomes "one fix — stop editing generated files — recovers more than a third of our failures." That is a roadmap, and it came from reading, not guessing.
  4. Fix the top mode, then convert it into evals (§10) so it can never silently return.

This is why "one big accuracy number" is the third great misconception (§16): the number tells you where you are, never what to do next. The per-failure-mode slice is what tells you what to build, in what order. A team that does error analysis every week and a team that watches a single accuracy dial will diverge fast — the first is climbing a gradient, the second is guessing.

The mechanical version in this phase is deliberately simple (group by a signature, rank by count), because the discipline — look at your failures, categorize them, fix the biggest bucket — is the transferable skill, and it survives whether the categorizer is a regex, an LLM, or a senior engineer with a spreadsheet.

10. The data flywheel: production telemetry becomes eval cases

An eval set built once and frozen goes stale, because production is a distribution shift generator: users send inputs you never imagined, and the failures that matter most are the ones you didn't think to test. The fix is a flywheel:

  1. Sample real production traces — a representative slice, plus every trace a user thumbs-down, every escalation, every incident.
  2. Label the failures (with a human, a downstream signal, or a validated judge) to get the ground-truth outcome.
  3. Promote the failures into new eval cases — append them to the golden set as a new version, deduped against what's already there. Lab 03's promote_failures_to_evals does this: a failed trace becomes an EvalCase, and dedupe (by content fingerprint) stops the set filling with near-identical cases.
  4. Re-baseline and iterate — now the escaped bug is a permanent regression test, and the agent's floor rises because that class of failure can never silently return.

This is the compounding mechanism of EDD, and it's why the eval set becomes the moat: it encodes every mistake your system has ever made in production, which is knowledge no competitor and no public benchmark has. SWE-bench measures something adjacent; your golden set measures your actual failures on your actual codebase.

The dedupe detail matters more than it looks. Without it, a flaky production issue that recurs 200 times floods your eval set with 200 copies of the same case, silently reweighting your aggregate toward that one mode and slowing every eval run. Lab 03 fingerprints by content (not id), so the same failure promoted twice is idempotent. Curation beats volume applies to the flywheel too: promote representatives of failure modes, not every raw trace.

The mature loop combines §9 and §10: production failures → error analysis clusters them → the biggest cluster gets fixed and gets a handful of representative cases promoted into the golden set. Spin that weekly and your system improves on a schedule instead of by luck.

11. The regression gate and eval-gated delivery in CI

EDD's second half — after "evals first" — is "never ship un-gated." The eval suite is wired into the delivery pipeline exactly like a test suite, with one crucial addition: because full evals are slow and expensive (an end-to-end coding eval runs a whole test suite per task, sometimes an LLM judge per case), you run them at two tiers:

  • Smoke evals, per commit. A cheap, tag-selected subset (Lab 03's select_smoke) that runs in seconds-to-minutes on every commit/PR — fast enough not to slow the loop, broad enough to catch gross regressions. This is promptfoo assert thresholds in a PR check, or a small LangSmith experiment on push.
  • The full suite, pre-release / nightly. The whole golden set (plus the expensive end-to-end and judge evals) on a release branch or a nightly schedule, where minutes-to-hours and real dollars are acceptable.

Gating both tiers is the regression gate (Lab 01 and Lab 03), which blocks a candidate when:

  • the primary metric drops beyond a tolerance (a small tolerance absorbs noise per §8, but set it honestly — too generous and it never fires);
  • any previously-passing case regresses (configurable, but usually on — this is the "fixed five, broke four" catcher that the aggregate misses);
  • any safety-tagged case fails — and this rule is not configurable and not averaged. Safety is a gate, not a term in a weighted sum. One leaked secret or one out-of-scope destructive edit blocks the release no matter how good the aggregate looks. This is the Phase 11 principle, made load-bearing: a 99% aggregate with one safety failure is a blocked release.

Around the gate sit the operational realities interviewers probe:

  • Eval budgets — track $/run and minutes/run and treat them as first-class; an eval suite too expensive to run per release won't get run, and an ungated release is the whole thing you were trying to avoid.
  • Flake/retry policy — because runs are stochastic, decide upfront: is a case that passes 4 of 5 samples a pass or a fail? Lab 01's default is strict (all n must pass — flakiness is failure), which is the conservative choice for a gate; some teams use pass@1 or a majority vote. Whatever you pick, make it explicit and consistent, and retry genuinely-flaky infra failures (a timeout) but never retry a genuine wrong answer into a pass.
  • Dashboards & version tracking — the score across dataset versions and agent versions, so you can see the trend and answer "when did this regress." Lab 03's version ledger is the minimal form: every release attempt recorded with its scores and its released/blocked decision.

12. Held-out sets and the overfitting trap

Here is the subtle failure that catches good teams. You iterate against the dev eval set — you tune prompts, tools, and few-shot examples to make that set's number go up. Do it long enough and you will improve the dev set without improving reality: you've fit the idiosyncrasies of those specific cases. This is "training on the test set" for prompts, and it's insidious because the dev number — the thing you're staring at — looks great right up until production tells you otherwise.

The defense is standard ML hygiene applied to agent iteration: keep a held-out set you never look at during iteration and evaluate only at release. If the dev score climbs while the held-out score stalls or drops, you're overfitting, not improving. Lab 03's detect_overfitting is the mechanical version: it fires when the dev score improves but the held-out score drops beyond a threshold, and the release pipeline blocks on it — catching a "4-point dev improvement, 9-point held-out regression" that the dev gate alone would wave through (because on the dev set, the score only went up).

Two practical refinements:

  • The held-out set decays too. Every time you look at it (even at release), a little information leaks — you start, subtly, steering toward it. So rotate it: freeze a fresh held-out slice periodically from the growing golden set / production sample, and retire the old one into the dev set. A held-out set you've evaluated fifty times is half-compromised.
  • This is why §10's flywheel and §12's held-out compose. New production failures feed both the dev set (to iterate on) and, periodically, a fresh held-out slice (to keep you honest). The two guardrails — regression gate on dev, overfitting detector on held-out — catch different failures: the gate catches "you broke something you had"; the detector catches "you got better at the test and worse at the job."

13. Latency and cost as first-class eval dimensions

The Cohere JD says "accuracy, safety, and latency" — three dimensions, and latency (and its sibling cost) are named right alongside accuracy for a reason. An agent that resolves 80% of tickets but takes 40 tool calls, 90 seconds, and $12 per ticket is not obviously better than one that resolves 75% in 6 calls, 8 seconds, and $0.40 — and which one is better is a product decision the eval must surface, not hide.

So latency and cost are not a separate monitoring concern bolted on after accuracy — they are dimensions every task is scored on, with budgets that can fail a task. A patch that resolves the issue but exceeds the step budget is a budgeted failure. Lab 02's extensions add exactly this: count files read and edits applied, and fail a task that resolves but blows the budget. In a full system you track, per task: tokens in/out, dollar cost, wall-clock latency, tool-call count, and steps — and you gate on them alongside resolution.

Why this is a seniority signal: juniors optimize the accuracy number in isolation and ship something too slow or too expensive to run; seniors treat the eval as multi-objective and can articulate the tradeoff curve — "we can buy 5 points of resolution for 3x the cost by sampling-and-ranking; here's where that's worth it (a nightly batch job) and where it isn't (an interactive assistant)." The cost/latency numbers are also what connect this phase to Phase 14's observability work: the same per-request cost and latency you meter in production are eval dimensions offline.

14. Real-world grounding: SWE-bench, HumanEval, and the tooling landscape

Name real things accurately — it's how you signal you've done this, not just read about it.

HumanEval (OpenAI, Chen et al. 2021) — 164 hand-written Python programming problems, each a function signature + docstring + hidden unit tests; the paper that introduced the pass@k estimator (§7). MBPP (Google, "Mostly Basic Python Problems") — ~1,000 crowd-sourced entry-level Python tasks, same execution-graded shape. These are the canonical capability benchmarks (§5), and their scores are quoted for essentially every code model. Know that they're function-level and largely saturated for frontier models now — which is why the field moved to repo-level.

SWE-bench (Princeton, Jimenez et al. 2023) — the canonical end-to-end coding-agent benchmark: ~2,000 real GitHub issues from popular Python repos, each paired with the real PR that fixed it. The agent gets the repo at the base commit and the issue text; the grader applies the agent's patch and runs the repo's tests, scoring resolved iff the FAIL_TO_PASS tests pass and the PASS_TO_PASS tests still pass — the exact split Lab 02 implements. SWE-bench Verified is the human-filtered ~500-instance subset where the tests genuinely specify the fix (the original set had under-specified and impossible instances) — and "we validate our tasks like SWE-bench Verified" is a real quality bar (Lab 02's verify_task_baseline). SWE-bench is the reason "% resolved" is the number everyone quotes for coding agents now.

Published eval guidance. Anthropic has public guidance on writing evals for LLM applications (the "create strong empirical evaluations" material) emphasizing task-specific, graded evals over vibes. OpenAI Evals is an open-source framework + registry for defining and running evals (YAML/JSONL registrations, model-graded and programmatic graders). Both stress the same things this phase teaches: define success concretely, grade programmatically where you can, keep a held-out set.

Tooling landscape (name them; you'll be asked which you've used):

  • LangSmith (LangChain) — tracing + datasets + experiments; you version a dataset, run an agent over it, and compare experiments — the productized version of Labs 01/03.
  • Braintrust — eval + observability platform centered on datasets, scorers, and experiment comparison, with a strong "diff two versions" story.
  • promptfoo — open-source, config-driven eval + red-teaming; assert-style graders in YAML, great for the per-commit smoke gate (§11).
  • OpenAI Evals — the registry/framework above.

The through-line: every one of these tools is a dashboard over the exact mechanisms you build in this phase — a versioned dataset, a grader ladder, a comparison, a gate. Build the mechanism once and every tool's UI is legible.

15. EDD vs TDD: the honest analogy and where it breaks

The TDD analogy is the fastest way to explain EDD to an engineer, and it's genuinely load-bearing — but a senior states where it breaks, because that's where the real work is.

Where the analogy holds:

  • Write the check first. TDD writes a failing test before the code; EDD writes the eval before the agent is good. Both make "done" an executable definition, not an opinion.
  • Red → green → refactor. TDD's loop is EDD's loop: baseline (red), iterate to pass (green), then improve architecture without regressing (refactor). Lab 01's paired comparison is the "don't regress" guard.
  • Regression suite in CI. Both accumulate a suite that runs on every change and blocks merges that break it.

Where it breaks — and why EDD is harder:

  • The oracle is fuzzy. A unit test's assertEqual is exact; an eval's grader is a ladder of approximations (execution, assertion, judge) with its own error rate. You must design and sometimes calibrate the grader (κ for a judge). TDD never has to validate that its assertions are trustworthy; EDD does.
  • The subject is stochastic. A unit test is pass/fail; an eval case is a probability estimated with pass@k over samples. "Flaky test = bug" in TDD; in EDD, flakiness is inherent and you decide a policy for it (§11).
  • You can overfit the suite. Nobody "overfits" a unit test suite — the code either has the bug or doesn't. But you can absolutely tune prompts to your eval set and improve nothing (§12), which is why EDD needs a held-out set and TDD doesn't.
  • The suite is a moving distribution. Unit tests cover a fixed spec; the eval set grows from production (§10) because the input distribution is open. Coverage is replaced by curation.

So EDD is "TDD for systems with a fuzzy oracle, a stochastic subject, an overfittable suite, and an open input distribution." That one sentence, said in an interview, tells the room you understand both the analogy and its limits — which is the whole point of a senior answer.

16. Common misconceptions

  • "We'll add evals later." = flying blind. Evals written after the agent are shaped by the solution you already built and measure nothing. EDD's entire premise is first.
  • "The judge model can grade everything." No — execution beats judgment for code. An LLM-judge grades tone; it does not grade whether a function is correct. Reserve the judge for genuinely fuzzy outputs, and validate it against humans (κ) before it gates anything.
  • "One big accuracy number." A single scalar hides a collapsed slice, a flaky case, and a safety regression. You need per-failure-mode slices (§9) to know what to do, and a separate safety gate (§11) that never averages.
  • "More eval cases are always better." Curation beats volume. A curated 40-case set covering your real distribution plus adversarial edges beats 400 near-duplicates that slow every run and reweight your aggregate.
  • "A 2-point improvement is an improvement." On 50 tasks it's almost certainly noise (§8). Report a confidence interval or run a paired test before you believe a delta.
  • "Public benchmarks measure my agent." SWE-bench and HumanEval measure something adjacent. Your golden set — built from your tasks and grown from your production failures — measures your system, and that's the moat.
  • "Passing evals means it works in production." Only if your eval set reflects production — which is why the flywheel (§10) and the held-out set (§12) exist. An eval score can go up while customers complain if you've overfit the set or the set has drifted from reality.

17. Lab walkthrough

Build the three miniatures in order; each injects the agent as a pure function so everything stays deterministic and offline, and together they are the EDD loop end to end.

  1. Lab 01 — The EDD Core Harness. A content-versioned EvalSuite, the grader ladder (exact / contains_all / predicate / rubric-judge), the unbiased pass_at_k, an EvalRunner with per-slice reporting that treats flakiness as failure, a per-task compare_runs, and a RegressionGate where a metric drop or any safety failure hard-blocks. 34 tests.
  2. Lab 02 — SWE-bench-miniature. Coding tasks as an in-memory repo + issue + fail_to_pass/pass_to_pass split + allowed_files; apply_patch (exact search/replace, fail-closed) → execute the hidden tests → score capability, regression, and safety; pass@k over samples and a SWE-bench-style "% resolved," with five diverse mini-tasks and a naive-vs-good policy where the naive one fails a regression trap and a safety trap. 29 tests.
  3. Lab 03 — The Error-Analysis Flywheel & Eval-Gated Delivery. A growing versioned GoldenDataset with dedupe, analyze_failures clustering into a ranked failure-mode report, promote_failures_to_evals (production traces → new cases), a ReleasePipeline with per-commit smoke + full-suite gate + a version ledger, and a held-out detect_overfitting check. 32 tests.

Run each with LAB_MODULE=solution pytest test_lab.py -v first (green reference), then fill your lab.py to match, then read solution.py's main() output.

18. Success criteria

  • You can write the eight-step EDD loop from memory and explain why evals come first.
  • You can name the three assumptions of traditional testing that LLM systems break, and what carries over.
  • You can turn an ambiguous goal ("make the PR agent good") into a list of checkable criteria, each naming a grader type and a slice.
  • You can name the five eval types for a coding agent and the right grader for each, and explain the FAIL_TO_PASS / PASS_TO_PASS split.
  • You can state the unbiased pass@k estimator, compute n=5,c=2,k=1 → 0.4, and explain why the naive estimator is biased.
  • You can explain why a 2-point delta on 50 tasks is noise, and what a bootstrap CI or a paired test does about it.
  • You can describe error analysis and why it's the highest-leverage activity, and the flywheel that turns production failures into eval cases.
  • You can explain the two-tier eval gate, why safety is a gate not an average, and what a held-out set catches that the dev gate can't.
  • All three labs pass under both lab and solution (95 tests total).

19. Interview Q&A

Q: How would you take a coding agent from a demo to production? A: Eval-Driven Development, in order. Frame success with the team — resolve the ticket, don't break existing tests, stay in scope, land under a cost budget. Build a golden set of 30-50 real closed tickets with their tests before touching the agent. Write execution graders (apply the patch, run the tests, with the FAIL_TO_PASS/PASS_TO_PASS split). Baseline the naive agent to get a real number. Then iterate, keeping only changes that improve the score via a paired comparison without regressing a task or tripping the safety gate. Wire a two-tier eval gate into CI — smoke per commit, full suite pre-release, safety as a hard block. Ship, then run the flywheel: sample production, do error analysis, promote failures into new eval cases, and watch a held-out set for overfitting. The demo-to-production gap is that machinery.

Q: Design the eval suite for a feature-delivery agent. A: Multi-type and sliced. Capability evals (HumanEval-style: can it write a correct function, pass@k). End-to-end task evals (SWE-bench-style: repo + issue → does the patch make the tests pass, FAIL_TO_PASS/PASS_TO_PASS). Trajectory/scope evals (did it edit only allowed files, in a sensible order). Spec-conformance evals (naming, API shape, style). Safety/regression evals as hard gates (no secret leak, no out-of-scope edit, no previously-passing case regressing). And latency/cost budgets as dimensions every task is also scored on. Grade with the cheapest reliable rung — execution for code, assertions for structure, a validated judge only for fuzzy prose. Tag every case with slices so the report breaks down by area and type, and version the dataset so scores are reproducible.

Q: Your agent's eval score went up but customers are complaining. What happened? A: Several likely causes, and I'd check them in order. (1) Overfitting — we tuned prompts to the dev eval set and improved it without improving reality; the held-out set would show the truth, and if we don't have one, that's the bug. (2) Distribution drift — the eval set no longer reflects what customers actually send; the fix is the flywheel: sample real traces and promote failures into the set. (3) A hidden slice regressed — the aggregate went up while a critical slice (say, the refund or auth tasks) collapsed; a per-slice breakdown and a paired comparison would catch it, and it's why I never trust one number. (4) The grader is wrong — maybe an LLM-judge is miscalibrated and scoring bad outputs as good; I'd check its κ against fresh human labels. The meta-point: an eval score is only as good as the eval set's fidelity to production and the grader's fidelity to truth.

Q: Why is execution the gold standard for grading code, and when is it not enough? A: Because a program is either correct against its tests or it isn't — execution gives a crisp, unbiased, reproducible verdict where a judge would guess. It's the whole reason HumanEval and SWE-bench run the code. It's not enough when there are no tests, when the requirement is non-functional (readability, style, security posture), or when "correct" is under-specified — there you climb to assertions, conformance checks, a validated judge, or a human. And execution has its own trap: tests that don't actually specify the fix (which is why SWE-bench Verified was human-filtered) will pass a wrong patch.

Q: Explain pass@k and why the naive estimator is wrong. A: pass@k is the probability at least one of k samples is correct. The unbiased estimator draws n≥k samples, counts c correct, and computes 1 − C(n−c,k)/C(n,k) — one minus the probability a size-k draw is all failures. The naive 1 − (1 − c/n)^k plugs a finite-sample estimate c/n into the formula and is biased upward for small n. For n=5, c=2, k=1 you get 0.4 (the base rate); for k=2, 0.7. pass@1 is your product number; a big pass@1-to-pass@k gap says "capable but unreliable — add verification and retry."

Q: How do you decide if a change is a real improvement? A: Not from the aggregate delta. On a small set a 2-point move is inside the noise (standard error ~6 points at N=50). I run a paired per-task comparison — wins, losses, ties — because a +2% aggregate can hide "fixed five, broke four," and a bootstrap CI on the pass rate to see if the improvement clears zero. And I check the per-slice breakdown and the safety gate: a change that lifts the average but regresses a safety case or a critical slice is not shippable regardless of the number.

Q: What is error analysis and why does it matter more than adding eval cases? A: Error analysis is reading your failures and clustering them into failure modes — "38% are the agent editing generated files, 22% are format errors." It matters more than volume because the aggregate number tells you where you are but never what to do next; the ranked failure-mode report does. Fix the biggest bucket, then promote representatives of it into the eval set so it can't regress. It's the highest-leverage activity in applied LLM work precisely because it converts a vague "we're at 72%" into a concrete, prioritized roadmap.

Q: What's a held-out set and why do you need one if you already have a regression gate? A: They catch different failures. The regression gate runs on the dev set and catches "you broke something you had." The held-out set is a slice you never look at during iteration, checked only at release, and it catches "you got better at the test and worse at the job" — overfitting your prompts to the dev set. You can drive the dev score up while the held-out score drops; the detector fires on exactly that divergence and blocks the release. And because looking at the held-out set slowly leaks it, you rotate it periodically.

Q: How do you keep evals from slowing the team down? A: Two tiers and honest budgets. A cheap, tag-selected smoke subset runs per commit in seconds-to-minutes; the full suite (including expensive end-to-end and judge evals) runs pre-release or nightly. Track $/run and minutes/run as first-class metrics, because an eval suite too expensive to run won't get run, and an ungated release defeats the purpose. Set an explicit flake policy (I default to strict — all samples must pass — for a gate), retry genuine infra flakes but never retry a wrong answer into a pass, and cache/parallelize where you can. The goal is a gate fast enough that people keep it on.

Q: When would you use an LLM-as-judge, and what are its dangers? A: For genuinely fuzzy outputs where no assertion or execution grader exists — tone, helpfulness, explanation quality, open-ended prose. The dangers are that it's a model: position bias (favoring the first option), verbosity bias (favoring longer answers), self-preference (favoring its own family's style), and plain wrongness. Before a judge gates anything, I validate it against human labels with Cohen's κ (Phase 11), prefer pairwise ("is A or B better") over absolute scoring where I can, and keep a human sample to re-check it over time. A judge you haven't validated is a random-number generator in a lab coat.

Q: How is EDD different from just "good testing"? A: It's TDD adapted to a fuzzy oracle, a stochastic subject, an overfittable suite, and an open input distribution. Exact assertions become a grader ladder you sometimes have to calibrate; pass/fail becomes pass@k over samples; you can overfit the suite so you need a held-out set; and coverage becomes curation of a dataset that grows from production. The engineering spine — fixtures, CI, regression discipline — carries over; the oracle, the statistics, and the moving distribution are what make it its own discipline.

20. References

  • Chen et al., "Evaluating Large Language Models Trained on Code" (OpenAI, 2021) — introduces HumanEval and the pass@k unbiased estimator. https://arxiv.org/abs/2107.03374
  • Austin et al., "Program Synthesis with Large Language Models" (Google, 2021) — introduces MBPP. https://arxiv.org/abs/2108.07732
  • Jimenez et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" (Princeton, 2023). https://arxiv.org/abs/2310.06770 and https://www.swebench.com/
  • OpenAI, "Introducing SWE-bench Verified" (the human-filtered subset). https://openai.com/index/introducing-swe-bench-verified/
  • Anthropic — guidance on creating empirical evaluations for LLM applications (Claude docs, "Develop tests / define your success criteria"). https://docs.anthropic.com/en/docs/test-and-evaluate/
  • OpenAI Evals — open-source framework and registry for LLM evals. https://github.com/openai/evals
  • Hamel Husain, "Your AI Product Needs Evals" and the error-analysis essays. https://hamel.dev/blog/posts/evals/
  • Eugene Yan — applied-LLM evaluation writing (e.g., "Task-Specific LLM Evals," patterns for building LLM systems). https://eugeneyan.com/writing/
  • LangSmith — datasets, experiments, and evaluation. https://docs.smith.langchain.com/
  • Braintrust — evals and experiment comparison. https://www.braintrust.dev/docs
  • promptfoo — config-driven LLM eval and red-teaming. https://www.promptfoo.dev/docs/intro/
  • Phase 11 of this track — Agent Evaluation: LLM-as-Judge, Golden Datasets & Behavioral Regression — the eval mechanisms this phase builds on (trajectory eval, judge bias, Cohen's κ, the regression/safety gates).
  • Phase 15 of this track — AI-Native SDLC: Spec-Driven Development & Coding Agents — the coding-agent harness (spec → plan → patch → verify) this phase evaluates.