« Phase 33 README · Warmup · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 33 — Hitchhiker's Guide

The compressed practitioner tour. Read the WARMUP for the mechanism; this is the stuff you say in the meeting.

30-second mental model

Eval-Driven Development is TDD for agents: write the eval before you build the thing, measure every change against it, ship nothing the eval doesn't bless. The eval set isn't a report you run at the end — it's the spec you write at the start. Traditional testing assumes a single right answer, a deterministic system, and an enumerable surface; an LLM system breaks all three (it samples, the "right" answer is a space, the surface is infinite), so exact assertions become a grader ladder, single runs become pass@k distributions, and coverage becomes a curated, growing, versioned dataset. What carries over: fixtures, CI, regression discipline. The loop: frame success → build the golden set first → write graders → baseline a naive agent → iterate with paired comparisons → error analysis (read failures, cluster into modes, promote into new evals) → regression-gate every change → sample production and grow the set. The flywheel (error analysis + promotion) is the moat.

The things to tattoo on your arm

The distinctions that signal seniority

• EDD vs Phase 11 vs Phase 15 → Phase 11 is the eval mechanisms (trajectory, judge, κ, gate); Phase 15 is the coding-agent harness (spec→plan→patch→verify); EDD is the development methodology that puts evals at the center of building and shipping. Don't conflate the tool with the discipline.
• Execution vs judgment → run the tests for code; a judge for prose. Saying "we'll have GPT grade the patches" instead of running them is a tell that you haven't shipped a coding agent.
• Capability vs end-to-end evals → HumanEval (write a correct function) vs SWE-bench (make the repo's tests pass). Different fidelity, different cost, different tier in CI.
• Aggregate vs per-slice → one accuracy number hides a collapsed slice, a flake, and a safety regression. The per-failure-mode breakdown is what tells you what to build next.
• Regression gate vs overfitting detector → the gate (dev set) catches "you broke something you had"; the held-out detector catches "you got better at the test and worse at the job."
• Safety as a gate vs a term → a 99% aggregate with one safety failure is a blocked release. Safety never averages out.

The one-liners

# pass@k, the unbiased estimator (Chen et al. 2021)
pass@k = 1 - C(n-c, k) / C(n, k)     # n samples, c correct; naive 1-(1-c/n)^k is biased

# the SWE-bench verdict
resolved  ==  FAIL_TO_PASS all pass  AND  PASS_TO_PASS all still pass  AND  in-scope edits only

# the two-tier CI gate
per-commit:  run tag-selected SMOKE subset   (seconds-minutes)
pre-release: run FULL suite + gate           (minutes-hours, real $)
gate blocks on:  metric drop | any case regression | ANY safety failure (never averaged)

War stories

• The 2-point "win" that was noise. A team celebrated 74%→76% on a 50-task set and shipped. Standard error at N=50 is ~6 points; the "improvement" was inside a ±12-point CI. A bootstrap CI and a paired test would have said "no signal." They'd been chasing noise for a sprint.
• The eval score went up and customers got angrier. Months of prompt-tuning lifted the dev set 8 points; a held-out set (which they didn't have) would have shown it dropping. Classic overfitting — they'd trained on the test set. Adding a held-out slice and the flywheel fixed it in two weeks.
• The coding agent that "fixed" the bug and broke prod. The patch made the new test pass; nobody ran the existing tests. FAIL_TO_PASS green, PASS_TO_PASS untested. Adding the pass-to-pass split to the grader turned a silent regression into a blocked merge.
• The judge that graded confidently and wrong. An LLM-judge scored fluent-but-incorrect summaries as great; nobody had measured its κ against humans. It was a random-number generator in a lab coat until someone validated it and swapped half the cases to execution/assertion grading.

Vocabulary

EDD · golden set (curated/versioned/grown) · grader ladder (execution → assertion → judge → pairwise → human) · execution grading · capability eval (HumanEval/MBPP) · end-to-end eval (SWE-bench) · trajectory/scope eval · spec-conformance · FAIL_TO_PASS / PASS_TO_PASS · % resolved · pass@k (unbiased estimator) · bootstrap CI · paired comparison · error analysis / failure modes · the flywheel (promote failures → evals) · smoke vs full suite · regression gate · safety gate · held-out set · overfitting detector · eval budget ($/run, min/run) · flake/retry policy · version ledger · LLM-as- judge + Cohen's κ (see Phase 11).

Beginner mistakes

1. Writing evals after the agent — grading the target you already hit.
2. Trusting one aggregate accuracy number with no slices, no CI, no safety gate.
3. Using an LLM-judge for code instead of running the tests.
4. Forgetting the PASS_TO_PASS split — shipping a fix that breaks a passing test.
5. Believing a 2-point delta on 50 tasks; no bootstrap, no paired test.
6. No held-out set, then overfitting prompts to the dev evals and wondering why prod got worse.
7. Never doing error analysis — staring at the number instead of reading the failures.
8. An eval suite so slow/expensive nobody runs it, so releases go out un-gated.