« Phase 33 README · Warmup · Hitchhiker's · Deep Dive · Core Contributor · Staff Notes

Phase 33 — Principal Deep Dive: Eval-Driven Development

The Deep Dive treats the eval harness as a pipeline in isolation. This document treats it as a production subsystem that gates deploys, spends real money, holds statistical power it can run out of, and fails in ways that ship regressions with a green checkmark on them. The Lab 01 RegressionGate and Lab 03 ReleasePipeline are the toy; the architecture question is what it takes to run that gate against your main branch for two years without it either rubber-stamping bad releases or blocking good ones so often the team routes around it. Both failure modes end the same way: an ungated release, which is the exact thing you built the gate to prevent.

EDD as a delivery gate, not a report

An eval report is something you read; an eval gate is something that blocks a merge. The architectural consequence of that verb is everything. A report can be slow, flaky, and approximate. A gate sits on the critical path of every engineer's PR, so it inherits the constraints of a CI check: it must be fast enough that people keep it on, deterministic enough that a re-run doesn't flip the result, and its failures must be actionable — "blocked: 3 backend cases regressed, safety clean" — not "score went down." A gate that emits a bare number gets overridden; a gate that names the regressed cases gets fixed.

This forces the two-tier design (Warmup §11), and the tiers are a capacity decision, not a preference. Full evals run a whole test suite per task, sometimes an LLM judge per case, times k samples, times N tasks. That is minutes-to-hours and real dollars. You cannot pay it per commit, so you split: a smoke tier (tag-selected subset, seconds-to-minutes) on every push, and the full tier (the whole golden set plus the expensive end-to-end and judge cases) on release branches and nightly. The smoke tier's job is to catch gross regressions cheaply; the full tier's job is to be the release oracle. Getting the smoke subset wrong in either direction is a real bug: too thin and it waves through regressions the full suite would catch (and now the release tier is your first line of defense, hours after the offending commit); too fat and the per-commit cost pushes people to skip it.

The scaling math you must be able to do live

The cost of one full run is roughly k × N × (model_calls_per_task) × cost_per_call + sandbox_minutes × compute_rate. For an end-to-end coding suite this is dominated by two terms: the sandbox execution (a container spun up per task, dependencies installed, a test suite run) and the k samples multiplier. Two levers move it:

• Parallelism over sandboxed execution. Tasks are embarrassingly parallel — each is an isolated repo at a base commit — so the wall-clock is total_task_seconds / concurrency, bounded by your executor pool. The architecture is a work queue of (task, sample) units fanned out to a pool of ephemeral sandboxes; the collector reassembles CaseResults. This is why real harnesses are container-per-instance: isolation and parallelism come from the same decision.
• Caching what's deterministic. The base repo state, the dependency install, and — critically — a (code_version, dataset_version, task_id, sample) result are cacheable. If neither the agent nor the dataset changed for a task, its result is fixed; re-running it is waste. A content-addressed dataset (Deep Dive, Stage 1) is what makes this cache safe: the cache key includes the dataset hash, so editing a case correctly invalidates exactly the affected entries.

The k multiplier is where principals earn their title. k samples buy you statistical power and a pass@k signal, but they cost k×. The right k is not a constant; it is large on the small capability tier (cheap) and small (often 1) on the expensive end-to-end tier, with the power you lose there bought back by having more tasks instead of more samples.

Statistical power: how many tasks you actually need

The gate is a hypothesis test, and it can be underpowered. The standard error of a pass rate is sqrt(p(1−p)/N); at p≈0.75, N=50 that is about 0.06, a 95% CI near ±12 points. A gate on a 50-task set literally cannot detect a real 5-point improvement — it is inside the noise. So the architecture must (a) report a bootstrap CI, not a bare point estimate, and (b) gate on a paired per-task comparison rather than the aggregate delta, because pairing cancels task difficulty and recovers most of the power. Lab 01's compare_runs is the mechanism: count wins, losses, ties, and gate on the net and the regression list. The killer scenario a bare aggregate hides is "fixed five, broke four, net +2%" — the four regressions are what page you, and the paired view is the only thing that surfaces them before merge. If you need to prove a small improvement, the answer is not a fancier test on 50 tasks; it is more tasks. Power comes from N.

Dataset governance: contamination is the silent score inflator

The most dangerous failure of an eval program is not flakiness; it is a number that is high and wrong. The mechanism is contamination — overlap between what the model saw in training and what your eval measures. Public benchmarks are the cautionary tale: HumanEval and MBPP are function-level and largely saturated for frontier models partly because their problems have leaked into training corpora, and SWE-bench's own issue was under-specified and solution-leaked instances, which is exactly why SWE-bench Verified — a human-filtered subset where the tests genuinely specify the fix — exists. The governance rules that follow:

• Held-out means never-looked-at. A slice you evaluate at every release slowly leaks; each look steers you toward it. Rotate it: freeze a fresh held-out slice periodically from the growing set, retire the old one into dev. A held-out set you've read fifty times is half-compromised.
• Provenance per case. Record where each case came from (hand-authored, promoted from a production trace, imported from a benchmark) so you can quarantine a source if it turns out to be contaminated or mislabeled.
• Your golden set is the moat precisely because it isn't public. It encodes your production failures on your codebase — knowledge no benchmark and no competitor has, and nothing a model could have trained on.

Blast radius of a bad grader

Rank the failure modes by blast radius, because that ranking dictates where you spend review.

1. A grader that passes wrong outputs (false-positive oracle). This is the worst failure in the system. It ships regressions with a green gate — the gate actively vouches for the bug. The sources are under-specified tests (the SWE-bench-Verified problem), an LLM judge that scores fluent-but-wrong as good, and a patch applier that "succeeds" on a no-op. Mitigation: fail-closed graders, execution over judgment wherever an execution oracle exists, and a periodic human sample to keep the judge honest (validate against labels with Cohen's κ before it gates anything).
2. A grader that fails right outputs (false-negative oracle). Less dangerous — it blocks good releases — but it erodes trust until the team disables the gate, and a disabled gate has the blast radius of #1 by other means.
3. A flaky grader. A gate that flips on re-run trains everyone to hit "re-run" until green, which silently converts your strict policy into "pass@many." Decide the flake policy explicitly (Lab 01 defaults to strict: all n must pass), retry genuine infra flakes (a sandbox timeout) but never retry a wrong answer into a pass.

Cross-cutting concerns of an eval platform

• Cost and latency as first-class, gated dimensions. The Cohere JD names "accuracy, safety, and latency" together. An agent that resolves 80% in 40 tool calls and $12 is not obviously better than 75% in 6 calls and $0.40; the eval must surface that tradeoff, not hide it. Score tokens, dollars, wall-clock, and step count per task and let a budget fail a resolving task (Lab 02's budgeted failure). This is the same meter you run in production observability, pointed offline.
• Multi-tenancy and isolation. If the platform runs many teams' evals, each task's sandbox is untrusted code (the agent's patch, the repo's tests) and must be isolated — no shared filesystem, no network egress, resource limits, per-tenant cost accounting. The execution grader is a code-execution service, and it inherits every security concern of one.
• Observability of the eval itself. The version ledger (Lab 03) is the minimal form: every release attempt recorded with (code_version, dataset_version, scores, released|blocked, reason). That ledger is what answers "when did the auth slice regress and which commit did it" — the question you will be asked in the incident review.

"Looks wrong but is intentional"

Three decisions read as bugs to a junior and are the point to a principal. Flakiness counts as failure — treating a case that passes 4-of-5 as a pass would make the gate lie about reliability, so strict is correct for a gate even though it fails cases that "mostly work." Safety is a veto, not a weighted term — it looks like an un-averaged outlier distorting the score, but a mean is a lossy reduction and safety is exactly the signal you refuse to lose in the average. And the eval set grows forever — an unbounded, ever-changing test suite offends the instinct that says a spec should be fixed, but the input distribution is open, so coverage is impossible and curation-that-grows is the only honest substitute. Each is a deliberate trade of a comfortable property (stable suite, single score, mostly-passing tolerance) for the one thing a release gate must never do: vouch for a regression.