Code Evals

Phase 12 · Document 05 · Evaluation Prev: 04 — Agent Evals · Up: Phase 12 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Code is the best domain in all of LLM evaluation because it has objective, programmatic ground truth: code compiles or it doesn't, tests pass or they don't, an edit applies or it doesn't. This makes code eval cheap, reliable, and judge-free — the ideal the Phase 12 discipline points toward (00, 02: prefer programmatic). It's also where evaluation most directly drives product decisions (model routing Phase 11.06, apply mechanism Phase 11.04). The catch: each coding feature needs a different metric (autocomplete → acceptance, apply → apply-rate, agent → task-resolution), and benchmarks suffer contamination. (Mechanics: Phase 11.08; here we frame code eval within the unified discipline — golden sets [01], the harness [08].)

2. Core Concept

Plain-English primer: a metric per feature, mostly programmatic

A coding system isn't one thing, so it isn't one eval — each subsystem has its own programmatic success signal (Phase 11.08):

The agent headline is task-resolution (the SWE-bench question: did it fix the bug?); the per-feature metrics localize where it fails.

Coding's superpower: executable ground truth

Most LLM eval reaches for an LLM-judge (02); code can usually execute instead — the Phase 12 "prefer programmatic" principle in its purest form (00):

• Task-resolution: does the change make the hidden/held-out tests pass? (SWE-bench-style) — the gold metric, no judge needed.
• Apply-rate: does the edit apply cleanly? (programmatic, Phase 11.04).
• Compile/lint/type pass: does the result build/type-check?
• pass@k: of k samples, does ≥1 pass the tests? (function-synthesis benchmarks).

Reserve LLM-judge for subjective code quality (style, explanation clarity) and calibrate it (02).

Benchmarks vs your-codebase eval

• Public benchmarks — SWE-bench / SWE-bench Verified (resolve real GitHub issues), HumanEval/MBPP (function synthesis, pass@k), Aider's edit/apply leaderboard, LiveCodeBench (contamination-resistant). Use to shortlist models and track the field.
• Contamination is acute for code — public benchmarks leak into training corpora, inflating scores; "Verified"/fresh/private variants exist for this reason (01 no-leakage rule).
• Your-codebase eval (decides) — a task set from your repo: real issues/PRs with their tests, common edit requests, autocomplete holes from your code. Benchmarks pick models; your eval decides (00, Phase 11.08).

Localizing failures (the subsystem split)

When agent task-resolution is low, per-feature metrics + traces (Phase 10.08) tell you where — the code analog of RAG's retrieval-vs-generation split (03):

• right code not retrieved? → indexing/retrieval (Phase 11.02/11.03).
• edit didn't apply? → apply-rate (Phase 11.04).
• applied but tests fail? → wrong edit → execution model/planning (Phase 11.06).
• too slow (autocomplete)? → latency tier (Phase 11.05).

Online signals are unusually rich for coding

Production gives strong implicit signals (Phase 10.08): autocomplete acceptance/retention, edit accept/reject, did the user keep the change, did CI pass after the agent's PR, time-to-completion. Mine these into your golden set (01) — code eval has a tight, objective feedback loop.

Safety and cost

A code agent that resolves the task but ran an unapproved destructive command (Phase 10.05) or cost $20/task failed. Track safety violations (=0) and cost/resolved-task (06/07) alongside resolution — eval the sandboxed execution too.

3. Mental Model

   code = the BEST eval domain: EXECUTABLE ground truth (compile/tests/apply) → objective, cheap, JUDGE-FREE [00/02]
   A METRIC PER FEATURE (don't conflate):
     autocomplete → ACCEPTANCE [11.05] · apply → APPLY-RATE [11.04] · retrieval → recall@k [11.02] ·
     agent → TASK-RESOLUTION (tests pass) [10.06] · all → latency/cost/SAFETY [06/07]

   BENCHMARKS (SWE-bench/HumanEval/Aider/LiveCodeBench) pick MODELS — mind CONTAMINATION (leakage [01])
        YOUR-CODEBASE task set (issues+tests) DECIDES it works [11.08]
   LOCALIZE low resolution via subsystem metrics + traces [10.08]: retrieved? applied? tests pass? fast?
   ONLINE signals rich: acceptance/retention · edit accept/reject · CI-passed · kept → feed back

Mnemonic: code has executable ground truth — prefer programmatic checks (tests/apply), one metric per feature (acceptance/apply-rate/recall/task-resolution). Benchmarks pick models (beware contamination); your-repo eval decides; subsystem metrics localize; online signals are rich.

4. Hitchhiker's Guide

What to look for first: the right metric per feature (don't grade autocomplete by task-resolution) and a your-codebase task set with programmatic checks (tests). Those make code eval meaningful and objective.

What to ignore at first: public-leaderboard rank. Use benchmarks to shortlist; build a small your-repo eval to decide what ships. (Mechanics: Phase 11.08.)

What misleads beginners:

• One metric for all. Each subsystem needs its own (Phase 11.04/11.05/10.06).
• Benchmark = your reality. Contamination + distribution mismatch (01) mean SWE-bench rank ≠ your-repo performance.
• Judge when you could compile/test. Code has programmatic ground truth — prefer it (02).
• Eval'ing the final diff only. Use subsystem metrics + traces to localize (Phase 10.08).
• Ignoring latency/cost/safety. A slow/expensive/unsafe resolution isn't a win (06/07).

How experts reason: they eval each feature on its metric, prefer programmatic checks (tests/compile/apply), build a your-codebase task set + use benchmarks only to shortlist (mind contamination), localize via subsystem metrics + traces, gate changes in CI (resolution + apply-rate), mine online signals, and require safety=0 + bounded cost/task.

What matters in production: autocomplete acceptance/latency, apply-rate, agent task-resolution on your repo, cost/resolved-task, safety violations (0), and a regression gate that catches model/routing changes (Phase 11.06).

How to debug/verify: low task-resolution → check (retrieved? applied? tests pass?); low acceptance → autocomplete model/context/latency; use traces (Phase 10.08). Trust your-repo eval over benchmark rank.

Questions to ask: a metric per feature? programmatic checks (tests/apply)? a your-codebase task set (not just benchmarks)? CI gate on resolution+apply-rate? online acceptance/kept signals? safety=0 + cost/task?

What silently gets expensive/unreliable: one-metric-for-all (hidden subsystem failures), leaderboard-chasing (benchmark ≠ your repo), judge-when-you-could-test (noisy), and ignoring latency/cost/safety.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Code is the best eval domain — executable ground truth (compile/tests/apply) makes eval objective, cheap, and judge-free (00).
• A metric per feature: autocomplete → acceptance/retention, apply → apply-rate, retrieval → recall@k, agent → task-resolution — don't conflate (Phase 11.08).
• Task-resolution (held-out tests pass) is the agent headline (Phase 10.06); pass@k for function synthesis.
• Prefer programmatic checks over LLM-judges; reserve judges for subjective quality and calibrate (02).
• Benchmarks (SWE-bench/HumanEval/Aider/LiveCodeBench) pick models; a your-codebase task set decides — beware contamination (leakage, 01).
• Localize low resolution via subsystem metrics + traces: retrieved? applied? tests pass? fast? (Phase 10.08).
• Online signals are rich (acceptance/retention, edit accept/reject, CI-passed, kept) — mine them (Phase 10.08).
• Safety (=0) + cost/resolved-task are part of "good" (06/07).

9. Observations from Real Systems

• SWE-bench (Verified) reoriented the field from "plausible code" to actual issue-resolution — the agent-mode standard (Phase 10.06).
• Aider publishes apply/edit-format leaderboards per model — apply-rate as a first-class, model-dependent metric (Phase 11.04).
• GitHub Copilot publishes acceptance/retention studies — the autocomplete north-star (Phase 11.05).
• Contamination is a documented, serious problem — LiveCodeBench and "Verified" variants and private evals exist because models memorize public sets (01).
• The decisive debugging move is the subsystem split — "low resolution" is usually a retrieval or apply failure, not model IQ (Phase 11.08).

10. Common Misconceptions

11. Engineering Decision Framework

EVALUATE A CODING SYSTEM:
 1. PER-FEATURE METRIC: autocomplete→acceptance+p95 [11.05] · apply→apply-rate+tests [11.04] ·
    retrieval→recall@k [11.02] · agent→TASK-RESOLUTION (held-out tests) [10.06].
 2. PREFER PROGRAMMATIC (tests/compile/apply) over judges; calibrate any judge for subjective quality [02].
 3. YOUR-CODEBASE task set (real issues+tests, edit requests, autocomplete holes); benchmarks only to SHORTLIST (mind contamination [01]).
 4. LOCALIZE low resolution: retrieved? [11.02/03] · applied? [11.04] · tests pass? [10.06] · fast? [11.05] — via traces [10.08].
 5. GATE in CI (resolution + apply-rate); decide "cheaper routed model good enough?" here [11.06].
 6. INCLUDE safety (=0) + cost/resolved-task; mine online signals (acceptance/CI-passed/kept) back into the set.

12. Hands-On Lab

Goal

Build a per-feature code eval — apply-rate + agent task-resolution (programmatic) + autocomplete acceptance — on a your-repo task set, and gate a routing change.

Prerequisites

• A code agent (Phase 10.06/Phase 11.04); a small repo with tests; ~10 tasks (failing tests to fix), autocomplete holes, edit requests.

Steps

1. Task-resolution (agent): for each failing-test task, run agent mode and check whether held-out tests pass — programmatic, objective (Phase 10.06). Record resolution rate.
2. Apply-rate (edit): for edit requests, measure how often the edit applies cleanly (Phase 11.04).
3. Acceptance proxy (autocomplete): mask lines; measure how often the completion matches/would be accepted, with p95 latency (Phase 11.05).
4. Localize: for unresolved tasks, use traces (Phase 10.08) to bucket the failure — retrieval miss vs apply miss vs wrong edit (tests fail).
5. Routing gate: swap the execution model to a cheaper one (Phase 11.06); re-run; decide if resolution + apply-rate hold — the production decision.
6. Cost/safety: record cost/resolved-task and confirm no unapproved destructive actions (06/07).

Expected output

A per-feature eval report (acceptance, apply-rate, task-resolution, cost/task, safety), a failure-localization, and a routing-change gate decision (cheaper model good enough?).

Debugging tips

• Task-resolution low → localize (retrieval/apply/edit) before blaming the model.
• Great benchmark, poor your-repo eval → contamination/mismatch; trust your-repo eval (01).

Extension task

Add an LLM-judge for subjective quality (style/explanation) and calibrate it (02); compare to programmatic metrics.

Production extension

Wire the eval into CI as a regression gate (resolution + apply-rate + acceptance), mine online signals into the task set, and track cost/task + safety (Phase 10.08, 08).

What to measure

Task-resolution, apply-rate, acceptance + p95 latency, recall, cost/resolved-task, safety violations; routing-change before/after.

Deliverables

• A per-feature code eval (acceptance / apply-rate / task-resolution) on a your-repo set.
• A failure-localization (subsystem split).
• A routing-change CI gate decision.

13. Verification Questions

Basic

1. Why is code the best domain for objective evaluation?
2. Give the primary metric for autocomplete, apply, retrieval, agent mode.
3. What is task-resolution, and why is it the agent headline?

Applied 4. Why isn't a high SWE-bench rank sufficient to ship a model in your product? 5. How do you decide whether a cheaper routed execution model is "good enough"?

Debugging 6. Agent task-resolution is low. How do you localize the failing subsystem? 7. A model looks great on HumanEval but poor in your IDE. Likely cause?

System design 8. Design a per-feature code eval + CI gate (autocomplete/apply/retrieval/agent + cost/safety).

Startup / product 9. Why are per-feature metrics + a your-codebase eval the basis for trustworthy iteration and routing decisions?

14. Takeaways

1. Code is the best eval domain — executable ground truth makes eval objective, cheap, judge-free.
2. One metric per feature — acceptance / apply-rate / recall / task-resolution — don't conflate.
3. Prefer programmatic checks; reserve calibrated judges for subjective quality (02).
4. Benchmarks pick models (beware contamination); your-codebase task set decides (01).
5. Localize via subsystem metrics + traces; gate in CI (resolution + apply-rate); include safety=0 + cost/task (Phase 11.06).

15. Artifact Checklist

• A per-feature code eval (acceptance / apply-rate / task-resolution / recall) on a your-repo set.
• Programmatic task-resolution (held-out tests pass).
• A failure-localization (subsystem split) for unresolved tasks.
• A routing-change CI gate decision (cheaper model good enough?).
• Cost/resolved-task + safety (=0) tracked; online signals fed back.

Up: Phase 12 Index · Next: 06 — Latency and Cost Evals