Lab 01 — The EDD Core Harness: Suites, Graders, pass@k & the Regression Gate

Phase 33 · Lab 01 · Phase README · Warmup

The problem

Eval-Driven Development says: define success as an eval before you build the agent, measure every change against it, and never ship a change that isn't eval-gated. That is a slogan until you have the machinery to run it. This lab builds the machinery — the four pieces every eval platform (LangSmith, Braintrust, OpenAI Evals, promptfoo) has underneath the dashboard, so that when one of them mis-scores a run at 2 a.m. you know exactly which mechanism to look at.

The trap this lab is designed to kill is "one big accuracy number." A single scalar hides a collapsed slice, a flaky case, and a safety regression all at once — the exact failure the Phase 11 integrated scenario walks through. So the harness reports per-slice breakdowns, treats flakiness as failure (strict pass over n samples), computes pass@k the unbiased way, does a per-task paired comparison between two agent versions, and enforces a regression gate where safety is a hard block, not a term in an average.

Where Phase 11 built the scoring mechanisms (LLM-judge bias, Cohen's κ, trajectory scoring) and Phase 15 built the coding-agent harness (spec→plan→patch→verify), this lab builds the development-methodology substrate: the versioned suite + gate that turns "we have some evals" into "evals are the release gate."

What you build

File map

Run it

pytest test_lab.py -v                       # your lab.py (red until you implement)
LAB_MODULE=solution pytest test_lab.py -v   # the reference (green)
python solution.py                          # worked example

Success criteria

• exact / contains_all / predicate / rubric all grade correctly; a predicate whose assert fires returns a FAIL with the exception name in detail, never propagates.
• rubric weights the injected judge's per-criterion scores and passes only at/above the threshold; weights actually change the outcome.
• EvalSuite.version is deterministic, order-independent, and changes when any case's input, expected value, or tags change.
• pass_at_k(5, 2, 1) == 0.4, pass_at_k(5, 2, 2) == 0.7, all-pass → 1.0, none-pass → 0.0, and it validates 0 ≤ c ≤ n, 1 ≤ k ≤ n.
• EvalRunner.run produces per-slice stats, treats a case that passes fewer than all n samples as not passed, and pins the report to the suite version.
• compare_runs finds the regressed task even when the aggregate pass rate is unchanged, and refuses to compare reports from different suite versions.
• RegressionGate blocks on a metric drop beyond tolerance, on any previously-passing case regressing (when enabled), and always on a safety-tagged case failing.
• All 34 tests pass under both lab and solution.

How this maps to the real stack

• The versioned suite is what LangSmith calls a Dataset (with versions), Braintrust calls a dataset + experiment, and OpenAI Evals encodes as a registry YAML + JSONL. Real platforms version datasets so a run is reproducible; our content-hash version is that idea in one property. The lesson — never report a score without the dataset version that produced it — is identical.
• The grader ladder is the real scoring ladder from Phase 11: programmatic/assertion graders are cheapest and most reliable, and you escalate to a model-based judge only for the genuinely fuzzy parts. Our rubric judge is an injected pure function — in production it's an LLM call, but its bias and calibration (Cohen's κ against humans) are Phase 11's subject; here it is stubbed so the harness stays deterministic.
• pass@k is exactly the HumanEval estimator (Chen et al., 2021, Evaluating Large Language Models Trained on Code). Real code-generation leaderboards report pass@1/pass@10/pass@100 with this estimator precisely because the naive one is biased upward for small n.
• The regression gate is the CI eval gate: LangSmith/Braintrust "compare experiments," promptfoo's assert thresholds, an OpenAI Evals score check in a GitHub Action. Real gates block a merge on a pass-rate regression and hard-fail on safety — our GateDecision.reasons is the PR comment that tells the author why the build is red.

Limits of the miniature. The judge is a deterministic stub, so none of the judge-bias / κ-calibration machinery of Phase 11 lives here — it is assumed. "Strict pass = all n samples" is one policy (the conservative one); real suites sometimes use pass@1 or a majority vote as the per-case verdict, which you'd make configurable. The content hash is sha256 of a JSON projection — good enough to detect edits, not a Merkle-tree dataset store.

Extensions (your own machine)

• Add a bootstrap confidence interval on pass_rate: resample cases with replacement B times, report the 2.5/97.5 percentiles — the honest way to say "is this 2-point delta real?"
• Add a held-out split to EvalSuite and a check that flags when the dev slice improves but the held-out slice regresses (the eval-overfitting detector — built for real in Lab 03).
• Make the per-case verdict policy pluggable (all / any / majority over n samples).
• Emit a JUnit-XML or Markdown report from a RunReport so it renders in a CI run summary.

Interview / resume signal

"Built the core of an eval-driven-development harness: a content-versioned eval suite, a grader ladder (exact/contains/assertion/rubric-judge) where a failing assertion is a structured result not a crash, the unbiased pass@k estimator, per-slice reporting that treats flakiness as failure, a per-task paired comparison, and a regression gate where a metric drop or any safety-case failure hard-blocks the release — the CI machinery that turns 'we have some evals' into 'evals are the release gate.'"