« Phase 33 README · Warmup · Hitchhiker's · Principal Deep Dive · Core Contributor · Staff Notes

Phase 33 — Deep Dive: Eval-Driven Development

An eval harness is not a script that prints a percentage. It is a pipeline over an immutable dataset, and once you see it as a pipeline — dataset → runner → grader → aggregate — every design rule in this phase falls out of the mechanism instead of being a rule you memorize. This document is about that mechanism: the data structures that flow through each stage, the invariants each stage must hold, and a step-by-step trace of one candidate output from raw string to a number you can gate on. The naive approaches ("let a model eyeball the answer," "run it once and read the accuracy") do not fail because they are lazy; they fail at specific, nameable points in this pipeline, and naming those points is the deep skill.

The four stages as data structures

Stage 1 — the dataset is content-addressed, not id-addressed. An EvalCase is a record: (id, inputs, reference, grader_spec, slices, tags). The suite that holds them (EvalSuite in Lab 01) computes a version by hashing the canonicalized content of every case, not by an incrementing integer. That single decision is load-bearing. A score is a function of two things — the code under test and the dataset — and if the dataset can change without the version changing, every stored score becomes a lie. Content-addressing makes the version a cryptographic commitment: "82% on v=a3f9…" is reproducible because the hash pins the exact 40 cases, their exact references, and their exact grader specs. Change one reference string and the hash moves; the old number is now explicitly attached to a dataset that no longer exists. Id-addressing (a hand-bumped v7) breaks the moment someone edits a case and forgets to bump.

Stage 2 — the runner is a sampler, not an executor. The runner's job is not "call the agent"; it is "draw a distribution of outcomes per case." Because the subject is stochastic (§2 of the Warmup), the runner takes n samples per case and stores every one — not a collapsed pass/fail. The output of the runner is therefore a CaseResult holding samples: list[Sample], each Sample carrying the raw output, the grader verdict, and the trace. Collapsing to a single bit here is the first place naive harnesses throw away the information the rest of the pipeline needs: you cannot compute pass@k, a bootstrap CI, or a flake policy from a bit. In the labs the agent is injected as a pure function precisely so the runner is deterministic even though a real model is not — the stochasticity is stubbed, but the shape (n samples per case) is preserved so the estimator math is real.

Stage 3 — the grader is a total function (output, case) → Verdict. A Verdict is (passed: bool, score: float, detail: str). "Total" is the invariant that separates a harness from a crash: a grader must never propagate an exception as a harness failure. In Lab 01 the predicate grader runs candidate-supplied logic that can assert and raise; the grader catches that and records a structured FAIL with the exception text in detail. A raised predicate is a failing case, not a broken run. Miss this and one malformed case takes down the whole suite — the eval becomes the flaky thing it was supposed to measure.

Stage 4 — the aggregate is a reduction that must not lose structure. The final stage folds CaseResults into metrics. The mistake is to fold to a single scalar. The correct reduction emits a structured aggregate: pass_at_1, pass_at_k, per-slice pass rates, a flake count, and — kept entirely separate — a safety verdict that is a boolean gate, not a term in the mean. Safety is not averaged because averaging is a lossy reduction and safety is exactly the signal you cannot afford to lose in the average (one leaked secret across 99 clean cases is 99% "good" and one shipped incident).

The grader ladder, mechanically

The ladder (execution → assertion → judge → pairwise → human) is ordered by oracle reliability, and execution sits at the top for a mechanical reason: its oracle is the program's own test suite, which has an error rate near zero relative to a model's. For code the grader is an execution grader, and its internals are the sharp part of this phase.

apply_patch → run_hidden_tests → verdict. The patch applier (Lab 02) is fail-closed: an exact search/replace that finds no match returns "not applied," which scores as unresolved rather than silently editing nothing and calling it a pass. The execution step runs two test sets and the verdict is a conjunction, not a score:

resolved  ==  all(FAIL_TO_PASS pass)  AND  all(PASS_TO_PASS still pass)  AND  edits ⊆ allowed_files

FAIL_TO_PASS proves the new behavior exists (capability); PASS_TO_PASS proves nothing regressed. They are separate sets because they answer separate questions, and the conjunction is why "the feature works" is not the verdict — a patch that greens the new test but reds an old one is unresolved by construction. That is the clamp-bug trap in Lab 02, and it is the single most common failure of a naive coding agent: it optimizes the capability oracle and never sees the regression oracle because it never ran it.

The pass@k estimator, and why the naive one is biased

Because the runner samples, each case has a probability of success, not an outcome. pass@k is the probability that at least one of k independent samples is correct. The estimator you must be able to derive on a whiteboard (Chen et al., 2021 — the HumanEval paper): draw n samples with n at least k, count c correct, and compute

pass@k  =  1 − C(n−c, k) / C(n, k)

Read it right-to-left. C(n−c, k) counts the ways to choose a size-k sample entirely from the failures (there are n−c of them). Over C(n, k) total ways to choose k, that ratio is the probability that a k-draw is all-failures. One minus it is "at least one passes." When the number of failures is smaller than k (i.e. n−c is below k), C(n−c, k) is zero and the estimator is exactly 1 — any k-draw must contain a pass.

Worked values (Lab 01 and Lab 02 assert these exactly):

n=5, c=2, k=1 → 1 − C(3,1)/C(5,1) = 1 − 3/5 = 0.4. With one draw, pass@1 collapses to the base rate c/n.
n=5, c=2, k=2 → 1 − C(3,2)/C(5,2) = 1 − 3/10 = 0.7.
n=5, c=4, k=2 → only one failure exists, so every pair contains a pass → 1.0.

Why not the naive 1 − (1 − c/n)^k? Because c/n is itself an estimate from a finite sample, and raising an estimate to the k-th power compounds its bias — the plug-in estimator systematically overstates pass@k for small n. The combinatorial form integrates over the finite sample exactly instead of pretending c/n is the true probability. This is a favorite interview probe because it is the exact line between someone who ran code evals and someone who read the abstract.

A worked trace: one candidate, grade to gate

Follow a single coding task through the pipeline. Task add-bug: repo at a base commit, an issue, FAIL_TO_PASS = {test_negative}, PASS_TO_PASS = {test_positive}, allowed_files = {calc.py}.

Runner draws n=5 candidate patches from the injected agent for this case. Each is a Sample.
For sample 1 the patch applier matches the search block in calc.py and applies it → applied. Sample 4's search block does not match (the agent hallucinated context) → not-applied → immediate unresolved for that sample, no execution needed.
Execution on the applied samples: run test_negative (was failing) and test_positive (was passing). Sample 1: both green → FAIL_TO_PASS satisfied, PASS_TO_PASS intact.
Scope check: sample 1 edited only calc.py ⊆ allowed_files → in-scope. Verdict for sample 1: resolved = True.
Suppose across the 5 samples, c=3 are resolved. The aggregate computes pass@1 = 3/5 = 0.6 and, say, pass@2 = 1 − C(2,2)/C(5,2) = 1 − 1/10 = 0.9. The gap (0.6 → 0.9) is the mechanical signal "capable but unreliable — add verification/retry," not "improve the base model."
Safety: had any sample edited secrets.py, that sample trips the safety tag; the aggregate carries safety_failed = True and the gate hard-blocks regardless of the 0.6/0.9 numbers. The safety verdict never entered the mean; it sits beside it as a veto.

Why the naive approaches fail — at the mechanism

"Let a model eyeball it." This replaces the Stage-3 oracle (the program's tests, error rate ≈ 0) with a second stochastic model (error rate unknown and biased — position, verbosity, self-preference). You have now measured your agent with an instrument you never calibrated. For code, where an execution oracle exists for free, this is strictly worse at the mechanism level.
"Run it once, read the accuracy." This collapses Stage 2 to a single sample, discarding the distribution. One green run is one draw; you cannot recover pass@k, a CI, or a flake rate from a bit. The number is real but its uncertainty is invisible, so a 2-point move on 50 tasks (inside a ±12-point band) reads as progress.
"One aggregate number." This over-reduces Stage 4. A collapsed slice, a flaky case, and a safety regression all vanish into the mean. The structure the reduction throws away is exactly the structure that tells you what to fix.

The through-line: the harness is a pipeline, each stage has an invariant (content-addressed version, preserved sample distribution, total grader, structure-preserving aggregate with safety as a veto), and every "vibe-check" shortcut is a specific violation of one of those invariants. Hold the pipeline in your head and you can derive the rest.