« Phase 33 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Staff Notes

Phase 33 — Core Contributor Notes: Eval-Driven Development

Our three labs are miniatures of real, load-bearing harnesses that thousands of models and papers are scored against. This document is the maintainer's-eye view of those real systems — HumanEval, SWE-bench, and the eval-framework layer (OpenAI Evals, EleutherAI's lm-evaluation-harness, Inspect) — with attention to the non-obvious source-level decisions, why the APIs evolved, the sharp edges a committer knows, and precisely what our stdlib miniature simplifies. Where I am not certain of an exact field name or number I describe the pattern rather than invent a specific, and I cite no URLs.

HumanEval: the capability harness, and the sandbox warning nobody reads

HumanEval (the OpenAI human-eval repository, from the 2021 Codex paper) is smaller than people expect: on the order of 164 hand-written Python problems. Each problem is a record with a prompt (a function signature plus docstring), a canonical solution, an entry_point (the function name), and a test — a block of Python assertions that calls the completion. The dataset ships as a gzipped JSONL of these records. The harness pattern is: for each problem, concatenate the model's completion onto the prompt, append the test block and a call to check(candidate), and execute the whole program; a problem is correct iff that program runs to completion without an assertion error, under a timeout.

Two source-level details are the ones that matter and get missed:

• check_correctness runs untrusted code, and the repo says so in capital letters. The reference implementation guards execution behind a disabled-by-default flag and wraps the run in a subprocess with a timeout, faulthandler, and a set of neutered builtins (it monkeypatches destructive calls like os.system, file removal, and process kill into no-ops or exceptions). The maintainers' explicit position is that this is not a real sandbox — it is a guard rail — and you should run it inside a container or gVisor-class isolation. Anyone who runs HumanEval on their laptop with the flag flipped is executing arbitrary model output as themselves. Our Lab 02 sidesteps this entirely by never running arbitrary code: the "execution" is a controlled in-memory test model over a stubbed repo, so there is no untrusted-code surface. That is the single biggest simplification we make, and it is deliberate — the phase teaches the grading logic, not sandbox operations.
• The pass@k estimator lives in the harness, not just the paper. The repo ships the numerically stable reference implementation — the one that computes 1 − prod((n−c−i)/(n−i)) over i in 0..k−1 rather than forming the binomials directly, to avoid overflow and precision loss on large n. Our Lab 01 pass_at_k mirrors the mathematics (1 − C(n−c,k)/C(n,k)) and asserts the same canonical values; a production port should use the product form for stability. Knowing why the product form exists — big-n binomials overflow and lose precision — is a real committer detail.

The sharp edge HumanEval maintainers live with: the benchmark is saturated and contaminated. It is public, tiny, and years old, so frontier models have effectively seen it. High HumanEval is now table stakes and near-meaningless as a discriminator, which is the field's lived reason for moving to repo-level, harder-to-leak benchmarks.

SWE-bench: the end-to-end harness, and why it is Docker-per-instance

SWE-bench (Princeton, 2023) is the end-to-end shape Lab 02 mirrors. A task instance is a real GitHub issue paired with the real merged PR that fixed it. The fields that do the work: the repo and the base_commit (the state before the fix), the problem_statement (issue text), the gold patch (the human fix, used to derive and validate the tests, not given to the model), a test_patch (the tests the PR added or changed), and — the crux — two named test sets, FAIL_TO_PASS (tests that fail at base and must pass after a correct patch) and PASS_TO_PASS (tests already passing that must stay passing). Our Lab 02 lifts these names verbatim because they are the grading contract: resolved iff all FAIL_TO_PASS pass and all PASS_TO_PASS still pass.

The architectural decision a maintainer defends is one Docker image per instance. It looks heavyweight — thousands of images — but it is the only way to make grading reproducible. Every repo at every base commit has its own dependency pins, its own build quirks, its own flaky tests; baking each instance into an image freezes that environment so a run in 2026 grades identically to a run in 2023. The harness applies the model's patch into the container, runs the specified test IDs, parses the test runner's output, and computes the pass/fail split. The parsing is a real sharp edge: pytest, unittest, and repo-custom runners emit different output, so the harness carries per-repo log parsers, and a parser that misreads output is a false verdict — the false-positive-oracle failure the Principal Deep Dive ranks worst.

The evolution every candidate should be able to narrate is SWE-bench Verified. The original set had instances that were effectively ungradable: the FAIL_TO_PASS tests were under-specified (a wrong patch could pass them) or the issue was impossible to solve from the given context, or the test environment simply didn't build. Reported "resolved rates" were therefore contaminated by noise in the benchmark itself, not just the models. Verified is the human-filtered subset (on the order of 500 instances) where annotators confirmed the tests genuinely specify the fix and the instance is solvable — a direct admission that your grader is only as good as your tests actually specifying the behavior. Lab 02's verify_task_baseline is the miniature of exactly this discipline: before you trust a task, confirm the gold patch resolves it and the naive/empty patch does not — a task that "passes" with no change is a broken task, not a solved one.

What we simplify from SWE-bench, on purpose: no Docker, no real repos, no log parsing, and the "tests" are deterministic predicates over an in-memory repo. We keep the part that is the transferable idea — the two-set split, the conjunction verdict, the scope check, the baseline verification — and drop the operational mass (container orchestration, environment resolution) that would bury the concept.

The framework layer: OpenAI Evals, lm-evaluation-harness, Inspect

Above the individual benchmarks sit the frameworks that make evals declarable and runnable at scale. Three worth knowing as a set:

• OpenAI Evals popularized the pattern of an eval as a registered, data-driven spec: a YAML registration pointing at a JSONL of samples plus an eval class, with two families of grader — programmatic "basic" evals (exact/includes/fuzzy match against an ideal) and model-graded evals where a second model scores against a rubric described in a template. The design lesson our labs echo is the separation of the dataset from the grader from the runner — you swap graders without touching data. The sharp edge is that the model-graded path is only as trustworthy as the grader prompt, which is why our Warmup insists on validating a judge against human labels (κ) before it gates.
• EleutherAI's lm-evaluation-harness is the de-facto standard for academic benchmark numbers, and its hard-won lesson is that prompt formatting and answer extraction dominate reported scores. The same model on the same benchmark can move several points based on how choices are presented and how the answer is parsed out of the generation — which is why the harness is fanatical about versioned, pinned task definitions. The transferable idea: an eval number without its exact harness version and prompt template is not reproducible. Our content-addressed dataset version is the same instinct applied to the data half.
• Inspect (UK AI Safety Institute) is the more recent shape and the one to name for agentic evals: it models an eval as a Task of dataset + solver + scorer, where the solver can be a full tool-using agent loop and the scorer can be programmatic or model-based. It reflects where the field went — from single-turn completions to graded trajectories — and its solver/scorer split is nearly one-to-one with our runner/grader split.

What our miniatures deliberately omit, and why it's the right call

The through-line across all three real systems is that most of their code is operational — sandbox isolation, container orchestration, dependency resolution, log parsing, distributed execution, result storage — and comparatively little is the grading logic that is the actual concept. Our labs keep the concept dense and drop the ops: the agent is injected as a pure function so there is no model call, the "execution" is a deterministic in-memory model so there is no untrusted-code sandbox, and everything is offline so there is no network. That is not a shortcut around the hard part; it is a decision to teach the part that transfers (the two-set split, the pass@k estimator, the content-addressed dataset, the safety-as-veto gate, the flywheel) unobscured by the part that is a different job (running untrusted code safely at scale). When you port any of this to production, the missing 90% is real and is exactly what HumanEval's sandbox warning, SWE-bench's Docker-per-instance, and lm-eval's version pinning are about — and now you know which corners those systems are cutting nowhere, and why.