Evaluating Code Agents

Phase 11 · Document 08 · AI Coding Platforms Prev: 07 — BYOK and Provider Routing · Up: Phase 11 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

This is the capstone of Phase 11: a coding platform has many tunable subsystems — indexing/retrieval (02/03), apply (04), autocomplete (05), model routing (06) — and you can't improve, choose models for, or ship any of them without evaluation. Coding is the best domain to evaluate because it has objective, programmatic signals (tests pass, code compiles, the edit applied), yet each feature needs a different metric: autocomplete → acceptance rate, apply → apply-rate, agent → task-resolution. It builds on agent evaluation (Phase 10.09) and the eval discipline (Phase 12), specialized for code — and it's how you decide "is the cheaper routed model good enough?" (06).

2. Core Concept

Plain-English primer: a different metric per feature

A coding platform isn't one thing, so it isn't one eval. Each subsystem has its own success signal, and conflating them hides problems:

The headline for agent mode is task-resolution (the SWE-bench question: did it fix the bug?), but you also need the per-subsystem metrics to localize failures.

Coding's superpower: programmatic, objective eval

Unlike most LLM tasks, code has executable ground truth — run the tests, compile, lint, type-check. So coding eval can be objective and cheap (Phase 9.09 preferred-checker idea, Phase 10.09):

• Task-resolution: does the agent's change make the hidden/held-out tests pass? (SWE-bench-style.) This is the gold metric — no judge needed.
• Apply-rate: does the edit apply cleanly to the file? (Programmatic, 04.)
• Build/lint/type pass: does the result compile/lint/type-check?

Lean on these over LLM-judge wherever possible; reserve judges for subjective quality (code style, explanation quality) and calibrate them (Phase 12.02).

Benchmarks vs your-codebase eval

• Public benchmarks — SWE-bench / SWE-bench Verified (resolve real GitHub issues), HumanEval/MBPP (function synthesis), Aider's edit/apply benchmark, LiveCodeBench (contamination-resistant). Useful for comparing models and tracking the field — but beware contamination (models trained on the benchmark) and distribution mismatch (the benchmark ≠ your repo/conventions).
• Your-codebase eval (the one that matters): build a task set from your own repo — real issues/PRs with their tests, common edit requests, autocomplete holes from your code. Public benchmarks pick models; your eval decides if it works for you (Phase 9.09 "eval on your data").

Localizing failures (the subsystem split)

When agent task-resolution is low, the per-subsystem metrics tell you where (Phase 10.08 traces):

• right code not retrieved? → indexing/retrieval (02/03).
• edit didn't apply? → apply-rate (04).
• applied but tests fail? → the edit was wrong → execution model/planning (06).
• too slow (autocomplete)? → latency tier/model (05). This is the retrieval-vs-generation split (Phase 9.09) extended across all subsystems.

Online + offline (and human signal)

• Offline: run the task set in CI; gate model/prompt/routing changes on no-regression in task-resolution + apply-rate (Phase 10.09) — this is how you safely answer "is the cheaper routed model good enough?" (06).
• Online: real signals are unusually rich for coding — autocomplete acceptance/retention, edit accept/reject, did the user keep the change / did CI pass, thumbs, time-to-completion. Mine these into your task set (Phase 10.08).

Safety and cost are part of "good"

A code agent that resolves the task but ran an unapproved destructive command or cost $20/task failed (Phase 10.05/10.09). Track safety violations (must be 0) and cost/resolved-task (Phase 7.09) alongside resolution.

3. Mental Model

   a coding platform = many subsystems → ONE metric per feature (don't conflate):
     autocomplete → ACCEPTANCE/retention [05] · apply → APPLY-RATE [04] · retrieval → recall@k [02/03]
     agent → TASK-RESOLUTION (tests pass, SWE-bench) [10.06] · all → latency/cost/SAFETY [05/7.09/10.05]

   ★ coding has PROGRAMMATIC ground truth (tests/compile/apply) → objective, cheap eval (prefer over judges) [10.09]
   BENCHMARKS (SWE-bench/HumanEval/Aider/LiveCodeBench) pick MODELS (mind contamination/mismatch)
        but YOUR-CODEBASE task set decides if it works for YOU [9.09]
   LOCALIZE low resolution via subsystem metrics + traces [10.08]: retrieved? applied? tests pass? fast?
   OFFLINE gate (CI) + ONLINE signal (acceptance, edit accept/reject, CI-passed, kept) → feed back

Mnemonic: one metric per feature (acceptance / apply-rate / recall@k / task-resolution), all gated by latency/cost/safety. Coding's executable ground truth makes eval objective — use benchmarks to pick models, your-codebase task set to decide it works, and subsystem metrics to localize failures.

4. Hitchhiker's Guide

What to look for first: the right metric per feature (don't eval autocomplete by task-resolution or agents by acceptance) and a your-codebase task set with programmatic checks. Those make eval meaningful.

What to ignore at first: chasing public-benchmark leaderboard rank. Use benchmarks to shortlist models; build a small your-repo eval to decide what ships.

What misleads beginners:

• One metric for all. Each subsystem needs its own signal (05/04/Phase 10.06).
• Benchmark = your reality. Contamination + distribution mismatch mean SWE-bench rank ≠ performance on your repo (Phase 9.09).
• Judge when you could compile/test. Coding has programmatic ground truth — prefer it over noisy judges (Phase 10.09).
• Eval'ing only the final answer. Use subsystem metrics + traces to localize why (Phase 10.08).
• Ignoring latency/cost/safety. A resolved task that's slow, expensive, or unsafe isn't a win (Phase 10.05/7.09).

How experts reason: they eval each feature on its metric (acceptance/apply-rate/recall/task-resolution), prefer programmatic checks (tests/compile/apply), build a your-codebase task set (real issues + tests) and use benchmarks only to shortlist models, localize failures via subsystem metrics + traces, gate changes in CI, mine online signals (acceptance, CI-passed, kept), and require safety=0 + bounded cost/task.

What matters in production: autocomplete acceptance/latency, apply-rate, agent task-resolution on your repo, cost/resolved-task, safety violations (0), and a regression gate that catches model/routing changes (06).

How to debug/verify: low task-resolution → check (retrieved? applied? tests pass? — 02/04/Phase 10.06); low acceptance → autocomplete model/context/latency (05); use traces (Phase 10.08).

Questions to ask: is each feature on its own metric? programmatic checks (tests/apply)? a your-codebase task set (not just benchmarks)? CI gate on resolution+apply-rate? online acceptance/kept signals? safety=0 + cost/task tracked?

What silently gets expensive/unreliable: one-metric-for-all (hidden subsystem failures), leaderboard-chasing (benchmark ≠ your repo), judge-when-you-could-test (noisy), and ignoring latency/cost/safety in "success."

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Each coding feature has its own metric: autocomplete → acceptance/retention, apply → apply-rate, retrieval → recall@k, agent → task-resolution — don't conflate them.
• Coding has programmatic ground truth (tests/compile/apply) → objective, cheap eval; prefer it over LLM-judges (Phase 10.09).
• Task-resolution (tests pass, SWE-bench-style) is the agent headline metric (Phase 10.06).
• Public benchmarks (SWE-bench/HumanEval/Aider/LiveCodeBench) pick models; a your-codebase task set decides if it works for you — beware contamination + distribution mismatch (Phase 9.09).
• Localize low resolution via subsystem metrics + traces: retrieved? applied? tests pass? fast? (Phase 10.08).
• Gate changes offline in CI (resolution + apply-rate) — how you decide "is the cheaper routed model good enough?" (06).
• Online signals are rich for coding — acceptance/retention, edit accept/reject, CI-passed, change-kept — mine them (Phase 10.08).
• Safety (=0 violations) and cost/resolved-task are part of "good" (Phase 10.05/7.09).

9. Observations from Real Systems

• SWE-bench (Verified) reoriented the field from "plausible code" to actual issue-resolution — the agent-mode standard (Phase 10.06).
• Aider publishes apply/edit-format leaderboards per model — apply-rate as a first-class, model-dependent metric (04).
• GitHub Copilot publishes acceptance/retention studies — the autocomplete north-star (05).
• Contamination is a real problem — LiveCodeBench and "Verified" variants exist because models memorize public benchmarks; teams build private, your-repo evals (Phase 9.09).
• The decisive debugging move is the subsystem split — "low resolution" is usually a retrieval or apply failure, not a model-IQ failure (Phase 10.09).

10. Common Misconceptions

11. Engineering Decision Framework

EVALUATE A CODING PLATFORM:
 1. PER-FEATURE METRIC:
       autocomplete → acceptance/retention + p95 latency [05]
       inline edit/apply → apply-rate + correctness (tests) [04]
       retrieval → recall@k of the right code [02/03]
       agent → TASK-RESOLUTION (held-out tests pass) [10.06]
 2. PREFER PROGRAMMATIC checks (tests/compile/apply) over judges; calibrate any judge [10.09/12.02].
 3. YOUR-CODEBASE task set (real issues+tests, edit requests, autocomplete holes) — benchmarks only to SHORTLIST models (mind contamination).
 4. LOCALIZE low resolution: retrieved? [02/03] · applied? [04] · tests pass? [10.06] · fast? [05] — via traces [10.08].
 5. GATE changes in CI (resolution + apply-rate); decide "cheaper routed model good enough?" here [06].
 6. INCLUDE safety (=0) + cost/resolved-task; mine ONLINE signals (acceptance, CI-passed, kept) back into the task set.

12. Hands-On Lab

Goal

Build a per-feature eval for your coding assistant — autocomplete acceptance, apply-rate, and agent task-resolution on a your-repo task set — and use it to gate a routing change.

Prerequisites

• The assistant/agent from 00/Phase 10.06; a small repo with tests; ~10 tasks (failing tests to fix), autocomplete holes (masked lines), and edit requests.

Steps

1. Task-resolution (agent): for each failing-test task, run agent mode and check whether the held-out tests pass after its change — programmatic, objective (Phase 10.06). Record resolution rate.
2. Apply-rate (edit): for edit requests, measure how often the produced edit applies cleanly (04).
3. Acceptance proxy (autocomplete): mask lines and measure how often the FIM completion matches/would be accepted, with p95 latency (05).
4. Localize: for unresolved tasks, use traces (Phase 10.08) to bucket the failure — retrieval miss (02/03) vs apply miss (04) vs wrong edit (tests fail).
5. Gate a routing change: swap the execution model to a cheaper one (06); re-run the eval; decide if task-resolution + apply-rate hold — i.e., is the cheaper model good enough? This is the production decision.
6. Cost/safety: record cost/resolved-task and confirm no unapproved destructive actions (Phase 7.09/Phase 10.05).

Expected output

A per-feature eval report (acceptance, apply-rate, task-resolution, cost/task, safety), a failure-localization for unresolved tasks, and a routing-change gate showing whether the cheaper model is acceptable.

Debugging tips

• Task-resolution low → localize (retrieval/apply/edit) before blaming the model.
• Benchmark looks great but your-repo eval is poor → contamination/distribution mismatch; trust your-repo eval.

Extension task

Add an LLM-judge for subjective quality (code style/explanation) and calibrate it against your judgment (Phase 12.02); compare to the programmatic metrics.

Production extension

Wire the eval into CI as a regression gate (resolution + apply-rate + autocomplete acceptance), mine online signals (acceptance/retention, edit accept/reject, CI-passed, kept) into the task set, and track cost/task + safety dashboards (Phase 10.08).

What to measure

Task-resolution, apply-rate, autocomplete acceptance + p95 latency, retrieval recall, cost/resolved-task, safety violations; routing-change before/after.

Deliverables

• A per-feature eval (acceptance / apply-rate / task-resolution) on a your-repo task set.
• A failure-localization (subsystem split) for unresolved tasks.
• A routing-change gate decision (is the cheaper model good enough?).

13. Verification Questions

Basic

1. Give the primary metric for autocomplete, apply, retrieval, and agent mode.
2. Why is coding unusually good for objective evaluation?
3. What is task-resolution, and why is it the agent headline metric?

Applied 4. Why isn't a high SWE-bench rank sufficient to ship a model in your product? 5. How do you decide whether a cheaper routed execution model is "good enough"?

Debugging 6. Agent task-resolution is low. How do you localize the failing subsystem? 7. Autocomplete feels good in demos but acceptance is low. What do you check?

System design 8. Design a per-feature eval + CI gate for a coding platform (autocomplete/apply/retrieval/agent + cost/safety).

Startup / product 9. Why are per-feature metrics + a your-codebase eval the basis for trustworthy iteration (and model-routing decisions)?

14. Takeaways

1. One metric per feature — autocomplete → acceptance, apply → apply-rate, retrieval → recall@k, agent → task-resolution — don't conflate.
2. Coding has programmatic ground truth (tests/compile/apply) — prefer it over LLM-judges (Phase 10.09).
3. Benchmarks pick models; your-codebase task set decides it works — beware contamination/mismatch (Phase 9.09).
4. Localize failures via subsystem metrics + traces (retrieved? applied? tests pass? fast?).
5. Gate changes in CI (resolution + apply-rate) — including "is the cheaper routed model good enough?" — and include safety (=0) + cost/resolved-task (06/Phase 10.05).

15. Artifact Checklist

• A per-feature eval (acceptance / apply-rate / task-resolution / recall) on a your-repo task set.
• Programmatic task-resolution (held-out tests pass).
• A failure-localization (subsystem split) for unresolved tasks.
• A routing-change CI gate decision (cheaper model good enough?).
• Cost/resolved-task + safety (=0) tracked; online signals fed back.

Up: Phase 11 Index · Next: Phase 12 — Evaluation