Lab 02 — SWE-bench-miniature: Execution-Graded Coding-Task Evals

Phase 33 · Lab 02 · Phase README · Warmup

The problem

For most agent outputs you climb the Phase 11 scoring ladder — assertions, then a judge. For code there is a rung above all of them: execution. Apply the agent's patch to the repo, run the repo's tests, and the task is resolved only if the tests pass. That is exactly how SWE-bench works, and it is the strongest argument against "the judge model can grade everything" — a program either makes the tests green or it does not, and no rubric substitutes for running it.

This lab builds that grader in miniature, and with it the three dimensions a feature-delivery agent is actually judged on:

• Capability — does the change do the new thing? (the fail_to_pass tests go green)
• Regression — did the change break something that worked? (the pass_to_pass tests stay green) — the SWE-bench FAIL_TO_PASS / PASS_TO_PASS split, which is the single most important idea here: a patch that fixes the bug and breaks two other tests is not a fix.
• Safety / trajectory — did the agent edit only files it was allowed to? (the least-privilege boundary from Phase 15, scored as an eval)

What you build

The five shipped tasks

The naive policy resolves 3/5; the good policy resolves 5/5 — and the two it misses are missed for different reasons (one regression, one safety), which is exactly why the report breaks the score into dimensions instead of reporting 0.6 and stopping.

File map

Run it

pytest test_lab.py -v                       # your lab.py (red until you implement)
LAB_MODULE=solution pytest test_lab.py -v   # the reference (green)
python solution.py                          # worked example: naive 3/5 vs good 5/5

Success criteria

• apply_patch writes/creates files and does an exact replace that raises PatchError on a missing file, no match, or an ambiguous (>1) match — and never mutates the input repo.
• run_tests returns a FAIL (not an exception) for an assertion failure, a missing symbol, or a syntax error in the patched code.
• Every shipped task is well-formed: fail_to_pass fails and pass_to_pass passes on the original repo.
• grade_sample marks a task resolved only when the patch applied, stayed in-bounds, and both capability and regression hold.
• Breaking a pass_to_pass test makes the task unresolved even though capability is true (the regression trap).
• An edit to a file outside allowed_files fails the safety check regardless of test results (the safety trap).
• pass_at_k(5, 2, 1) == 0.4; a flaky policy scored over n samples reports a sensible pass@1/pass@2/pass@k.
• run_suite reports "% resolved", and the good policy scores strictly higher than the naive one.
• All 29 tests pass under both lab and solution.

How this maps to the real stack

• This is SWE-bench in one file. Real SWE-bench takes a GitHub issue + repo at a base commit, applies the model's patch, and runs the repo's test suite, scoring resolved only if the FAIL_TO_PASS tests pass and the PASS_TO_PASS tests still pass. Our fail_to_pass / pass_to_pass split is that exact contract; our in-memory repo and ExecContext stand in for a cloned repo and a pytest run in a container. SWE-bench Verified is the human-filtered 500-task subset where the tests actually specify the fix — our verify_task_baseline is the same quality bar (a task whose tests don't distinguish the fix is worthless).
• Execution-based grading is the gold standard for code, and it's why HumanEval and MBPP (function-level) and SWE-bench (repo-level) all run the code rather than judge it. The single- function tasks here (add-bug, shout-feature) are HumanEval-shaped; the multi-file ones (route-fix) are SWE-bench-shaped.
• The allowed_files trajectory check is a real safety eval: production coding agents run under least privilege (Phase 15's allowed_paths, Phase 09's sandbox), and "did the agent try to touch something out of scope" is a first-class metric, not an afterthought.
• pass@k is the standard code-gen metric (Chen et al., 2021) because the agent samples — you report pass@1 for the greedy/product number and pass@k to characterize the distribution.

Limits of the miniature. Real repos have imports across files, fixtures, and a real test runner; ours execs a single file per load() and can't resolve cross-file imports, so tasks are kept self-contained. Executing patched code is safe here only because every patch is authored fixture data, offline — a real harness runs the patch in a locked-down container (Phase 09), never in-process. "% resolved" is pass@1 greedy; real leaderboards report both pass@1 and pass@k.

Extensions (your own machine)

• Add a unified-diff patch kind (@@ hunks) alongside write/replace, and a fuzz/context matcher, to mirror the Codex/Cursor edit format.
• Add a cost/latency budget to TaskResult (count files read + edits applied) and fail a task that resolves but exceeds a step budget — latency/cost as a first-class eval dimension.
• Run patches in a real subprocess with a timeout and resource limits (the Phase 09 sandbox) so an infinite loop in a patch is a timeout-FAIL, not a hang.
• Load a couple of real SWE-bench Verified instances' FAIL_TO_PASS/PASS_TO_PASS lists and map them onto this harness's shape to feel the scale difference.

Interview / resume signal

"Built a SWE-bench-style execution-graded eval harness: coding tasks as an in-memory repo + a fail_to_pass/pass_to_pass test split + an allowed-files boundary; an injected agent policy proposes a patch that's applied (exact search/replace, fail-closed) and graded by running the hidden tests — scoring capability, regression, and safety as separate dimensions, with pass@k over samples and a SWE-bench-style % resolved. It makes concrete why execution beats judgment for code and why a patch that fixes the bug and breaks a passing test is not a fix."