Code-Editing Agents

Phase 10 · Document 06 · Agents and Tools Prev: 05 — Sandbox and Approval · Up: Phase 10 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Code-editing agents are the most successful production agents to date — Claude Code, Cursor agent mode, GitHub Copilot agents, OpenAI Codex, and the SWE-bench leaderboard all live here. They work because coding has something most domains lack: a fast, automatic correctness signal — code compiles or it doesn't, tests pass or they don't — so the agent can verify and self-correct in a loop (03 reflection). Studying them teaches the best-understood agent pattern (retrieve context → edit → run tests → fix), the metrics that matter (apply-rate, test-pass / task-resolution rate), and the safety model for an agent that mutates your codebase (05). It's also the bridge to Phase 11 (AI coding platforms), which builds the IDE/product around this agent.

2. Core Concept

Plain-English primer: an agent loop with a correctness signal

A code-editing agent is the standard loop (01/03) specialized for a repository, with tools to see the code, change it, and verify the change:

goal ("fix the failing test in auth/") →
  RETRIEVE context: grep/glob/read relevant files (+ symbol/AST or embeddings) [Phase 9, Phase 11]
  → EDIT: propose a change (apply-patch / search-replace / whole-file)
  → APPLY: harness applies the edit to the workspace
  → VERIFY: run tests / build / lint / type-check  → results fed back [01]
  → if fail: read the error, fix, re-verify  (reflection loop [03])  → repeat
  → done when tests pass (or stop condition)  → present a diff for review [05]

The verification loop is what makes coding agents work: unlike a chatbot, the agent gets an objective "did it work?" every iteration and can iterate to a passing state.

Reading context: the retrieval problem for code

The agent can't fit the repo in context, so it retrieves the relevant slice (this is RAG for code, Phase 9, Phase 11.02–03):

• Agentic/tool-based retrieval (what Claude Code does): the model uses grep/glob/read(offset,limit) to pull just the needed files/spans on demand (what-happens §3.4). Simple, exact, scales.
• Index-based retrieval: embeddings + symbol graph / AST over the repo for semantic "where is X?" (Phase 11.02–03). Most strong coding agents combine: keyword/symbol exactness (Phase 9.05 hybrid) + on-demand reading.

Applying edits: the apply problem (and apply-rate)

How the agent expresses a change matters enormously for reliability:

• Whole-file rewrite — simplest, but expensive (re-emits the file) and risks unintended changes.
• Search/replace (diff) blocks — the model emits old_string → new_string; the harness finds and replaces. Token-efficient and targeted, but fails if old_string doesn't match exactly (whitespace, drift).
• Unified-diff / patch — the model emits a patch the harness applies.
• Apply model — some platforms use a second, fast "apply" model to merge the edit reliably (Cursor's approach), decoupling "decide the change" from "apply it precisely."

Apply-rate — the fraction of proposed edits that apply cleanly — is a first-class metric: a brilliant edit that won't apply is worthless. Exact-match edit formats fail on mismatched context; robust agents re-read before editing and handle apply failures by feeding the error back (01).

The verification loop (why coding agents self-correct)

Coding's superpower is the executable signal: run the tests/build/linter/type-checker and feed results back. This is reflection with a real signal (03) — far stronger than self-critique. The agent:

• runs the relevant tests, reads failures, edits, re-runs — converging to green;
• uses the build/type-checker to catch errors the model "thought" were fine;
• ideally works test-first (write/identify the failing test, then fix until it passes).

This loop is why agents now resolve a large fraction of real GitHub issues (SWE-bench) — not raw model IQ, but edit → verify → fix iteration.

Safety: it's mutating your codebase

A code agent runs commands and changes files — squarely the 05 safety model:

• Work on a branch / in a workspace, never directly on main/prod; present a diff for human review before merge.
• Sandbox command execution (tests/build run in isolation, egress-limited) — generated code is untrusted (05).
• Approval for risky commands (rm, network installs, deploys, git push).
• Rollback is natural via version control (branch/revert).

Metrics that matter

• Apply-rate — edits that apply cleanly.
• Test-pass / build-pass rate — does the change work?
• Task-resolution rate (e.g., SWE-bench) — did it actually solve the issue end-to-end? This is the real metric (09).
• Diff quality / review-acceptance — humans accept the change (no unintended edits, follows conventions).
• Steps/tokens/cost per resolved task — efficiency (Phase 7.09).

3. Mental Model

   CODE AGENT = agent loop [01/03] + a CORRECTNESS SIGNAL (compile/tests) → can self-correct

   RETRIEVE (grep/glob/read on-demand [9] + symbol/AST/embeddings [11]) →
   EDIT (whole-file | search-replace diff | patch | apply-model) →  ← APPLY-RATE matters
   APPLY to workspace → VERIFY (run tests/build/lint/types) → results fed back [01] →
   if fail: read error → fix → re-verify (reflection w/ REAL signal [03]) → green → DIFF for review [05]

   SAFETY [05]: branch/workspace not prod · sandbox command exec (untrusted code) · approve risky cmds · VC rollback
   METRICS: apply-rate · test-pass · TASK-RESOLUTION (SWE-bench) · diff/review-acceptance · cost/resolved-task [09]

Mnemonic: a code agent is the loop plus an executable correctness signal: retrieve → edit → apply → run tests → fix until green, then show a diff. Apply-rate and task-resolution are the metrics; sandbox + branch + review are the safety.

4. Hitchhiker's Guide

What to look for first: is there a verification loop (the agent runs tests/build and iterates on failures)? That, plus apply-rate, is what makes a code agent actually work — not the model alone.

What to ignore at first: fancy repo-wide embedding indexes for a small project — agentic grep/read is often enough. Add indexing when the repo is large (Phase 11.02).

What misleads beginners:

• No verification loop. A code agent that edits without running tests is just autocomplete with extra steps — the executable signal is the whole point (03).
• Ignoring apply-rate. Great edits that won't apply (exact-match misses) silently fail — re-read before editing, handle apply errors (01).
• Editing on main/prod. Work on a branch/workspace; present a diff for review (05).
• Unsandboxed command execution. Generated code/tests are untrusted — sandbox them (05).
• Measuring "looks good" not resolution. The real metric is task-resolution (tests pass, issue solved), not plausible diffs (09).

How experts reason: they build the agent around the edit→verify→fix loop with a robust apply mechanism (and measure apply-rate), retrieve context via grep/read + symbol/index, run tests/build/types in a sandbox, work on a branch with diff review, and evaluate on task-resolution rate (SWE-bench-style) and cost/resolved-task — not vibes (09).

What matters in production: task-resolution rate, apply-rate, diff quality/review-acceptance, safe command execution (sandbox + approval), and cost/latency per resolved task. Reliability comes from the loop + tooling, not just the model (Phase 5.06/5.03).

How to debug/verify: read the trajectory (08) — did it retrieve the right files? did the edit apply (apply-rate)? did it run tests and read failures? did it converge? Evaluate on a held-out issue set (09).

Questions to ask: is there a verify loop (tests/build)? what's the edit format + apply-rate? how is context retrieved (grep vs index)? is command execution sandboxed + on a branch? is it evaluated on task-resolution?

What silently gets expensive/unreliable: no verification loop (unverified edits), low apply-rate (silent failed edits), unsandboxed execution (security), editing prod directly (no rollback), and optimizing plausible diffs instead of resolution.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Code agents are the most successful production agents because coding gives a fast, automatic correctness signal (compile/tests) enabling self-correction (03).
• The core loop is retrieve → edit → apply → verify (run tests/build) → fix → repeat until green (01).
• Apply-rate matters — an edit that won't apply cleanly is worthless; exact-match formats fail on context drift; re-read before editing, or use an apply model.
• Context is retrieved (agentic grep/read + symbol/AST/embeddings), not stuffed (what-happens §3.4, Phase 11).
• Reflection with the executable signal (tests/build/types) is why agents resolve real issues — not raw model IQ.
• Safety = branch/workspace (not prod) + sandboxed command execution + diff review + approval for risky commands + VC rollback (05).
• The real metric is task-resolution rate (SWE-bench-style), plus apply-rate, diff/review-acceptance, and cost/resolved-task (09).
• Reliability comes from the loop + tooling, not just the model (Phase 5.03/5.06).

9. Observations from Real Systems

• Claude Code, Cursor agent, Copilot agents, OpenAI Codex all implement edit→verify→fix loops with retrieval and approval — the canonical pattern (Phase 11).
• Cursor uses a dedicated "apply" model to merge edits reliably (high apply-rate), separating "decide the change" from "apply it precisely."
• Aider's search-replace edit formats and repo map are a well-documented open implementation of the edit/apply problem.
• SWE-bench drove the field's focus from "plausible diff" to actual task-resolution (tests pass on real GitHub issues) (09).
• The biggest reliability gains came from better loops and tooling (verification, apply, retrieval), not just bigger models (Phase 5.03).

10. Common Misconceptions

11. Engineering Decision Framework

BUILD A CODE-EDITING AGENT:
 1. RETRIEVE context: agentic grep/glob/read on demand [9]; add symbol/AST/embeddings for large repos [11].
 2. EDIT format: search-replace/patch (token-efficient) + RE-READ before editing; consider an APPLY MODEL; track APPLY-RATE.
 3. VERIFY loop: run tests/build/lint/type-check; feed failures back; iterate to green (reflection w/ real signal [03]).
 4. SAFETY [05]: work on a BRANCH/workspace (not prod); SANDBOX command execution; APPROVE risky cmds; VC rollback; DIFF review.
 5. EVALUATE on TASK-RESOLUTION (held-out issues, SWE-bench-style) + apply-rate + diff acceptance + cost/resolved-task. [09]
 6. CHOOSE model for apply/test-pass on YOUR repo [5.03]; reliability from loop+tooling > model IQ.

12. Hands-On Lab

Goal

Build a minimal code-editing agent with a verification loop on a small repo, and measure apply-rate and task-resolution.

Prerequisites

• A small Python repo with a failing test; the agent from 01; a sandbox for running tests (05); pytest.

Steps

1. Tools: grep(pattern), read(path, offset, limit), edit(path, old_string, new_string) (search-replace), run_tests() — all operating on a git branch/workspace, tests run in a sandbox (05).
2. Loop: give the goal "make the failing test pass." The agent retrieves (grep/read) → proposes an edit → harness applies (record apply success/failure) → run_tests() → feed results back → iterate until green or stop (03).
3. Apply-rate: over several tasks, count edits that applied cleanly vs failed (exact-match misses); add re-read before edit and show apply-rate improve.
4. Task-resolution: on a held-out set of ~5 failing-test tasks, measure the fraction the agent fully resolves (tests pass) — the real metric (09).
5. Safety: confirm edits land on a branch (not main), tests run sandboxed, and the agent presents a diff before you "merge"; gate a destructive command (e.g., rm) behind approval (05).
6. No-verify contrast: run once without the test loop (edit and stop); show resolution drops sharply — proving the executable signal is the point.

Expected output

A working code agent that iterates to green on a real failing test, with measured apply-rate and task-resolution rate, a branch+diff+sandbox safety setup, and a with/without-verification comparison.

Debugging tips

• Edits fail to apply → exact-match mismatch; re-read the file right before editing, or use a more robust edit format.
• Agent loops without converging → not reading test failures, or no stop condition (00).

Extension task

Add a symbol/keyword index for retrieval on a larger repo (Phase 11.02); add a separate apply model step and compare apply-rate.

Production extension

Run on real GitHub issues (SWE-bench-style), report task-resolution + cost/resolved-task, wire diff review + branch protection + sandboxed CI (05, Phase 7.09); see Phase 11 for the platform.

What to measure

Apply-rate, test-pass rate, task-resolution rate, steps/tokens/cost per resolved task, with vs without verification loop.

Deliverables

• A code-editing agent with retrieve→edit→verify→fix loop (branch + sandbox).
• An apply-rate measurement (+ improvement from re-read/apply model).
• A task-resolution rate on held-out issues + a with/without-verification comparison.

13. Verification Questions

Basic

1. Why are coding agents the most successful production agents?
2. What is the verification loop, and why does it enable self-correction?
3. What is apply-rate, and why does it matter?

Applied 4. Compare whole-file vs search-replace vs apply-model edit strategies. 5. How is repo context retrieved without stuffing the whole codebase?

Debugging 6. Edits keep failing to apply. Cause and two fixes. 7. The agent writes plausible code that doesn't fix the issue. What metric/loop is missing?

System design 8. Design a safe code-editing agent: retrieval, edit/apply, verification loop, branch/sandbox/diff-review, eval.

Startup / product 9. Why does reliability in coding agents come more from the loop + tooling than from the model, and what does that imply for building one?

14. Takeaways

1. Code agents work because of an executable correctness signal — the retrieve → edit → apply → verify (tests/build) → fix loop self-corrects (03).
2. Apply-rate is first-class — robust edit formats / re-read / an apply model make edits actually land.
3. Retrieve repo context (grep/read + symbol/index), don't stuff it (Phase 9/11).
4. Safety = branch/workspace + sandboxed execution + diff review + approval + VC rollback (05).
5. Measure task-resolution (SWE-bench-style), apply-rate, and cost/resolved-task — reliability is the loop + tooling, not model IQ (09).

15. Artifact Checklist

• A code-editing agent with a retrieve→edit→verify→fix loop (branch + sandbox).
• An apply-rate measurement + an improvement (re-read / apply model).
• A task-resolution rate on held-out failing-test issues.
• A with/without-verification comparison.
• Diff review + sandboxed execution + approval for risky commands.

Up: Phase 10 Index · Next: 07 — Browser and Research Agents