« Phase 15 README · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 15 Warmup — AI-Native SDLC: Spec-Driven Development & Coding Agents

Who this is for: you can write Python, you built the agent loop in Phase 01, and you have used Claude Code / Codex / Cursor / Aider but never built one. By the end you will have built the plan→patch→verify loop that sits inside all of them, and you will know exactly why a coding agent edits with a patch, why the spec is the eval, and why verification — not volume — is the point. No framework, no API key: the model is a function you inject.

What AI-native SDLC means and why coding agents are the flagship
Why we inject the model, again
What a coding agent actually is
The in-memory repo and why snapshot and restore matter
Apply-patch part 1: the three edit formats
Apply-patch part 2: why exact search-replace fails closed
Spec-Driven Development: the spec is the contract and the eval
Acceptance checks: verify runs callables, never eval
The plan-patch-verify loop, stated precisely
Feeding failures back and the max_iters guard
The least-privilege guardrail on file edits
Reusable commands, skills, and subagents
Verification as ground truth: SWE-bench and agentic-coding evals
Quality, not volume, and human-in-the-loop review
Common misconceptions
Lab walkthrough
Success criteria
Interview Q&A
References

1. What AI-native SDLC means and why coding agents are the flagship

The software development life cycle (SDLC) is the loop every team runs: understand a requirement, plan a change, implement it, test it, review it, ship it. "AI-native" means an LLM-driven agent participates in that loop as a first-class actor — not a chat box off to the side, but a thing that reads the repo, proposes a diff, runs the tests, and hands a reviewable change to a human. The Temporal JD names the goal precisely: "build trusted agentic coding systems" and "use AI to increase quality, not just output volume."

Of every kind of agent you could build, the coding agent is the flagship, for three reasons. First, it is the one with the most users right now — Claude Code, Codex, Cursor, and Aider are touched by millions of engineers daily. Second, it has a clean reward signal that most agents lack: tests either pass or they don't. A customer-support agent's "success" is fuzzy; a coding agent's is a green suite. Third, it exercises every idea in this track at once — the loop (Phase 01), tool calling (Phase 02), context assembly (Phase 04), retrieval over a codebase (Phase 05/06), sandboxed execution (Phase 09), evaluation (Phase 11). If you understand the coding agent, you understand the agent stack.

This phase builds the core of a coding agent — the spec → plan → apply-patch → verify loop — from scratch, in memory, deterministically, so the mechanism is naked and yours to debug.

2. Why we inject the model, again

Same reason as every other lab in this track (see the Lab Standard): the LLM is the only non-deterministic part, so we replace it with an injected policy — a plain function. For a coding agent there are two:

a planner (spec, current_verify) -> list of step descriptions, and
a patcher (spec, step, current_verify, repo) -> a patch (the model's proposed edit).

In tests both are scripted, pure functions of their inputs, so the whole loop is reproducible and we can assert the exact iteration count. This is not a toy shortcut — it is how coding-agent teams actually test: record/replay fixtures over real transcripts, FakeLLM stubs, golden diffs. And it enforces the deepest rule of the craft: your loop's correctness must not depend on the model's first patch being right. The lab proves this by scripting a patcher that gets the edit wrong the first time; the loop must catch it with verification and recover. If your harness only works when the model is right, it is not a harness — it is a demo.

3. What a coding agent actually is

Strip away the UI and a coding agent is the ReAct loop of Phase 01, specialized:

context  := read the relevant files (retrieval / codebase indexing)
repeat up to max_iters:
    plan  := model plans the change from the spec + last failures   # reason
    for each step in plan:
        patch := model proposes an edit                             # act
        apply_patch(repo, patch)  (roll back if it doesn't apply)   # the tool
    result := verify(repo, spec)   # run the acceptance checks/tests # observe (the REWARD)
    if result.passed: return       # done
return "did not converge"          # the max_iters guard fired

The one structural difference from a generic agent is the observation is a verdict, not a lookup. A search tool returns text the model interprets; the verifier returns pass or fail against the spec — an objective reward. That is why coding agents work better than most agent categories: the loop closes on a ground truth (the tests), not on the model's own judgment. Every box maps to a section below: plan (§7 spec, §9 loop), apply-patch (§5–6), verify (§8), iterate on failure (§10), bound (§10 max_iters), and contain (§11 guardrail).

4. The in-memory repo and why snapshot and restore matter

The lab's Repo is a dict of path → text with read, write, exists, paths, and — the important pair — snapshot and restore. We use a dict, not the real filesystem, for the same reason we inject the model: determinism and safety. A test that writes real files is slow, order- dependent, and can escape its sandbox; a dict is instant and hermetic.

snapshot/restore are the rollback mechanism, and they are not optional decoration. A coding agent applies edits that sometimes don't apply cleanly — the file moved, the context changed, two hunks overlap. If a failed edit leaves the file half-written, the next verify runs against a corrupted tree and the agent chases a ghost. So the loop snapshots before each apply and restores on failure, guaranteeing an invariant: after any failed apply, the repo is exactly what it was before. Real agents get this from git (git stash / a scratch branch / git checkout -- file) or a copy-on-write overlay; the mechanism is the same, and the invariant is the same: an edit either lands whole or not at all. (Strings are immutable in Python, so a shallow dict(self._files) copy is already a deep copy — a nice property the lab relies on.)

5. Apply-patch part 1: the three edit formats

How should an agent change a file? The naive answer — "have the model regenerate the whole file" — is what early tools did, and it is bad: it burns tokens re-emitting unchanged code, it silently drops or rewrites lines the model forgot to copy, and the resulting diff is unreviewable (the whole file looks changed). Real coding agents use one of three edit formats, and knowing all three is table stakes for these interviews:

Exact search/replace (edit block). The model emits a path, an old snippet, and a new snippet; the tool finds old verbatim in the file and swaps it for new. This is Aider's default and Claude Code's edit format (a fenced block with a search half and a replace half). Tiny, precise, and reviewable — the diff is exactly old → new.
Unified diff (@@ hunks). The model emits a standard diff with a few lines of surrounding context and +/- lines; the tool applies it by locating the context. This is what Codex/GPT and some Cursor paths use. It is compact and familiar to patch/git apply, but brittle: if the context lines drifted, the hunk won't apply.
Whole-file write (path, content). Create a new file or do a heavy rewrite where a hunk would be more confusing than a replacement. Every agent keeps this as a fallback.

The lab implements search/replace ({path, old, new}) and whole-file ({path, content}), the two that make the safety property clearest. (The unified-diff form is a listed extension.) The point of the section is that an edit is a structured, minimal, reviewable operation — never "here is the whole file again, trust me."

6. Apply-patch part 2: why exact search-replace fails closed

Here is the property that makes search/replace safe, and the single most important line in the lab: an edit block applies only if old is found exactly once.

If old is not found, the tool returns an error and writes nothing. The file the model imagined does not match the file on disk (stale context, a wrong path), and applying anyway would either corrupt the file or silently do the wrong thing. Fail closed.
If old is ambiguous — it appears more than once — the tool also refuses. It cannot know which occurrence you meant, and guessing is worse than stopping. Imagine old = "return 0" in a file with ten functions that return 0; replacing "the first one" is a coin flip that looks like it worked. So the rule is: exactly one match, or fail.

This is the same discipline as parse in Phase 01 and validation in Phase 02: a bad edit is a structured observation the agent recovers from, not a crash and not a silent corruption. In the loop, a failed apply rolls back (§4) and is recorded in the transcript; the agent can widen the old context and try again. The failure modes to name in an interview: not-found (stale/ wrong context), ambiguous (too little context), overlapping hunks (two edits touch the same lines), and whitespace/indentation mismatch (the classic — the model's old has four spaces, the file has a tab). Every one of these is why "regenerate the file" seems easier and is actually worse: it hides these failures instead of surfacing them.

7. Spec-Driven Development: the spec is the contract and the eval

Spec-Driven Development (SDD) is the practice of writing a verifiable specification first and letting the agent work against it. The key insight — and the reason it matters for agents specifically — is that a good spec is two things at once:

the contract: what "done" means, in enough detail that an agent (or a junior engineer) can implement it without guessing; and
the eval: the acceptance checks that decide whether it's done — the same checks the agent verifies against and the same checks a human trusts at review.

In the lab a Spec is (description, acceptance_checks), where each check is a named predicate over the repo: file_exists("util.py"), defines_function("util.py", "greet"), file_contains("util.py", "Hi {name}"). Those checks are the spec — the prose description is for humans; the checks are what the loop optimizes. This is exactly test-first development with one twist: the agent reads the spec to learn what to build, and reads the failing checks to learn what to fix. GitHub's Spec-Kit (specify) formalizes this into spec.md / plan.md / tasks.md files; the AGENTS.md and CLAUDE.md conventions are the always-on spec ("here is how this repo works, here are the commands, here are the rules"). The resume phrase for this is "Spec-Driven Development, prompt standards."

Why does SDD beat "just tell the agent what you want"? Because an underspecified request has no stopping condition. If the only spec is a sentence, the agent doesn't know when to stop, and you review by reading code. If the spec is a set of checks, the agent stops when they pass and you review by reading the checks and the diff. SDD turns "looks good to me" into "the contract is satisfied."

8. Acceptance checks: verify runs callables, never eval

verify(repo, spec) runs each acceptance check and returns a VerifyResult(passed, failures). Two design decisions are load-bearing.

First: the checks are provided callables, not eval of the agent's output. This is the Phase 09 discipline — "the checks are provided callables, not eval of agent output." The agent proposes text; your code decides whether it's correct by running your predicates. The lab's checks are static: file_exists and file_contains are string/membership tests, and defines_function uses Python's ast module to parse the file and look for a FunctionDef — it never executes the code. A model could write import os; os.system("rm -rf /") in util.py and the AST check would happily confirm it "defines greet" without running a thing. In production the acceptance check is the test suite, but it runs inside a sandbox (Phase 09), never in your agent's process. Never let verification become an arbitrary-code-execution hole.

Second: a check that raises is a failure, not a crash. file_contains("gone.py", "x") reads a missing file, which raises FileNotFoundError; defines_function on a syntactically broken file raises SyntaxError. Both are caught inside Check.run and turned into False. This is "trust but verify" done defensively: the verifier is robust to the agent producing garbage, because the agent will produce garbage sometimes and the loop has to survive it to recover.

9. The plan-patch-verify loop, stated precisely

Here is the loop with nothing hidden:

result := verify(repo, spec)         # maybe it already passes (iterations stays 0)
iterations := 0
while iterations < max_iters and not result.passed:
    iterations := iterations + 1
    plan := planner(spec, result)                 # a list of step descriptions
    for step in plan:
        for patch in patcher(spec, step, result, repo):
            snap := repo.snapshot()
            outcome := apply_patch(repo, patch, allowed_paths)
            if outcome.ok:   record "applied"
            else:            repo.restore(snap); record "apply failed"   # roll back
    result := verify(repo, spec)                   # re-run the checks — the REWARD
    record "verify", passed = result.passed
return RunResult(result.passed, iterations, result, transcript)

Five things are doing the work: the planner (reason), the patcher (act), apply_patch (the tool that crosses into the repo), verify (the observation, which is a reward), and max_iters (the guard). The transcript records every plan, apply, and verify — that is the diff and the trail a human reviews. Notice the loop starts with a verify: if the spec is already satisfied, it does zero work, which is the correct behavior and a nice determinism test. Notice too that each patch applies under its own snapshot, so one failed edit in a multi-edit step doesn't poison the others.

10. Feeding failures back and the max_iters guard

The loop's power is in the two lines that most people skip. Feeding failures back: after a verify, the failures list (the names of the checks that failed) flows into the next planner and patcher call. The agent doesn't re-plan blind — it plans knowing what's still broken. In the lab, the scripted patcher writes the buggy Hey {name} on its first touch of util.py; verify fails on the Hi {name} check; on the next iteration the patcher (seeing the file now exists and the check failed) writes the corrected Hi {name}; verify passes. The whole run takes two iterations, and a test asserts iterations == 2. That is the single most important behavior of a coding agent: it does not have to be right the first time, because verification catches it and the failure tells it what to fix.

The max_iters guard: exactly like max_steps in Phase 01, it is a safety guard, not a target. A model can propose a "fix" that doesn't fix anything forever. Without a cap that is an infinite loop burning tokens; with it, the run stops after max_iters with passed = False and a partial transcript. The lab tests both directions: a patcher that always writes the buggy version runs exactly max_iters times and reports passed = False. If a task genuinely needs more iterations than the budget, that is a signal to improve the spec or decompose the task — not to raise the cap so a confused agent fails more expensively.

11. The least-privilege guardrail on file edits

An agent editing your repo is executing on your side of the trust boundary (Phase 00), and text it read — a ticket, a doc, a dependency's README — might carry an injected instruction (Phase 10). So the same control from Phase 09 applies: least privilege on what it can touch. In the lab, apply_patch takes an allowed_paths set; a patch whose path is not in it is rejected before any write. allowed_paths = {"util.py", "test_util.py"} means the agent can never write secrets.py or .github/workflows/deploy.yml, no matter what it proposes — the check happens first, and a rejected edit rolls back like any other failed apply.

This is an allow-list, and it fails closed: anything not explicitly granted is denied (the Phase 09 lesson — allow-lists beat deny-lists everywhere). In a real coding agent this shows up as scoped file permissions, a read-only vs writable path split, protected paths (you don't let the agent edit CI config or secrets unattended), and the whole thing running in an ephemeral sandbox so even a successful malicious edit can't reach the network. The guardrail plus the sandbox plus human review of the diff is defense in depth: no single layer is trusted alone.

12. Reusable commands, skills, and subagents

The last piece turns a coding agent from a tool into a platform: reusable, parameterized workflows. When your team keeps re-typing "add an endpoint that does X with a test and an OpenAPI entry," that prompt should be a command you invoke by name with arguments — not prose you retype and get slightly wrong each time. The lab models this with a CommandRegistry: a command is (name, required_params, build) where build(args) expands into a Spec. Invoking /add-function with {module, name, returns} produces a spec whose acceptance checks are "the file exists, the function is defined, the body contains the return expression." A missing required parameter is a structured error, never a silently wrong spec.

This maps directly to the real stack: Claude Code slash-commands (.claude/commands/*.md files parameterized with $ARGUMENTS), skills (packaged capabilities the agent loads on demand), subagents (a named agent with its own prompt and tools you delegate to), and Cursor rules. The AGENTS.md / CLAUDE.md file is the repo-wide always-on version. The resume phrase is "reusable agent workflows / reusable commands to support requirement discovery, implementation planning, code review, and test generation" — and the reason it matters is consistency: a command encodes the team's good process once so every run does review, tests, and the OpenAPI entry, instead of depending on whoever typed the prompt remembering all three.

13. Verification as ground truth: SWE-bench and agentic-coding evals

How do you know a coding agent is any good? Not by vibes. SWE-bench is the field's answer: a benchmark of 2,294 real GitHub issues from popular Python repos, where the agent is given the repo at the buggy commit and the issue text, and must produce a patch. The patch is accepted only if the repository's own test suite passes after applying it — the exact plan→patch→verify loop this lab builds, scored on real code. SWE-bench Verified is a human-filtered subset; leaderboards report the percentage of issues resolved, and the numbers climbing from single digits to the high tens of percent over two years is the clearest measure of coding-agent progress there is.

The per-run metric to know is pass@k: run the agent $k$ times and count the task solved if any run's tests pass. The unbiased estimator (from the Codex paper), given $n \ge k$ samples of which $c$ pass, is

$$\text{pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}$$

pass@1 is "right on the first try" (what a user feels); pass@10 is "solvable with retries" (what an agent with a verifier can reach, because the verifier picks the run that passes). The gap between them is exactly the value of the verify loop: verification lets you spend more samples and keep only the correct one. This is the Phase 11 material — golden datasets, task success, trajectory eval — applied to code, where the reward is unusually clean because tests are objective.

14. Quality, not volume, and human-in-the-loop review

Now the reframing that the Temporal JD makes explicit and that separates a senior answer from a junior one: "using AI to increase quality, not just output volume." The failure mode of coding agents is treating them as code fire-hoses — more lines, more PRs, faster. That is a liability: unreviewed AI code accrues the same debt as unreviewed human code, but faster, and a bigger diff is harder to review, not easier. The senior use of a coding agent is to raise quality:

generate the tests, not just the code (the agent is great at the tedious cases you skip);
review the diff — an agent as a critic on a human's PR, or a human as the gate on the agent's (this lab's transcript is that reviewable diff);
refactor under a green suite — the tests are the safety net that makes aggressive cleanup safe;
verify — the whole point of §7–§9 is that the output is checked, not just produced.

And the non-negotiable: human-in-the-loop. The model proposes; a human merges. The agent's job is to hand a human a small, verified, reviewable diff with its acceptance checks green — so the review is fast and the human is deciding on a known-good change, not auditing a wall of generated code. "Increase quality, not volume" is not a platitude; it is the design constraint that decides whether your coding-agent platform is an asset or an incident generator. Build for the verified diff, not the line count.

15. Common misconceptions

"A coding agent is autocomplete." Autocomplete predicts the next token in your editor; a coding agent runs a loop — plan, patch, run tests, iterate — with tests as the reward. Different category entirely.
"Just have the model rewrite the whole file." That hides the failures search/replace surfaces (not-found, ambiguous, whitespace drift), burns tokens, and produces an unreviewable diff. Real agents edit with minimal, structured patches.
"If old matches twice, replace the first one." No — that is a silent-corruption bug. Exactly one match or fail. Ambiguity is the agent's problem to fix (more context), not the tool's to guess.
"More code shipped = more productive." The JD says the opposite: quality, not volume. An unverified big diff is negative value. The metric is verified change.
"The spec is just a comment for the human." In SDD the spec's acceptance checks are the eval the agent optimizes and the reviewer trusts. Prose is for humans; checks are the contract.
"Verify can just run the model's code." Never eval agent output in your process — that is an RCE hole. Checks are your provided callables, and real test runs happen in a sandbox (Phase 09).
"The agent should have write access to the whole repo." Least privilege: an allowed_paths allow-list, protected paths, a sandbox, and human review of the diff. Defense in depth, because the text it read might be adversarial (Phase 10).
"Frameworks make this obsolete." Claude Code / Codex / Aider make it ergonomic. When one corrupts a file, edits the wrong path, or loops without converging at 2 a.m., the person who built the plan→patch→verify loop is the one who fixes it.

16. Lab walkthrough

Open lab-01-coding-agent-harness/ and fill the TODOs top to bottom:

Repo.snapshot / Repo.restore — the rollback primitives. A snapshot must be an independent copy (mutating the repo afterward must not change the snapshot). Everything else depends on this.
apply_patch — path guard first; full-file write; then search/replace with the count-the-matches logic: 0 → not found, >1 → ambiguous, exactly 1 → replace once. Never write on a failure. This is the core mechanism; get the ambiguity check right.
Check.run and verify — run each predicate, turn an exception into False, collect the failures list, set passed.
CodingAgent.run — the loop of §9: verify, then while not passed and under max_iters, plan → apply each patch under a snapshot (roll back on failure) → re-verify → record. Confirm the greet task passes on iterations == 2.
CommandRegistry.run_command — unknown command and missing-param are ValueErrors; otherwise expand to a Spec.

Run LAB_MODULE=solution pytest -v first to see green, then match it. Finish by reading solution.py's main() — it runs the wrong-once-then-fixed loop, shows an apply-patch rollback, and expands a reusable command into a spec.

17. Success criteria

apply_patch replaces an exactly-once match and fails without writing on a not-found or ambiguous old; you can explain why guessing is worse than failing.
A failed apply leaves the repo byte-for-byte unchanged (snapshot/restore).
verify reports failures, and a check that raises becomes a failure, not a crash.
CodingAgent.run fixes the greet spec on iteration 2 and stops at max_iters when it never passes.
The allowed_paths guardrail rejects an out-of-scope edit before any write.
A reusable command expands to a Spec and runs end-to-end; a missing param is a structured error.
All 30 tests pass under both lab and solution.

18. Interview Q&A

Q: How does a coding agent edit a file safely — and why not just regenerate the file? A: It uses a structured patch, usually an exact search/replace edit block (old → new) that applies only if old is found exactly once, or a whole-file write for new files / heavy rewrites. Exact match means the edit fails closed on stale context and refuses when old is ambiguous, so it never silently corrupts the file, and the diff is minimal and reviewable. Regenerating the whole file hides those failures, burns tokens, and produces an unreviewable diff. Under the hood the agent snapshots before applying and rolls back on failure, so an edit lands whole or not at all.

Q: What is Spec-Driven Development and why does it matter for agents? A: You write a verifiable spec first, and its acceptance checks are simultaneously the contract (what "done" means) and the eval (how you know). The agent reads the spec to learn what to build and reads the failing checks to learn what to fix; the reviewer trusts the same checks. It gives the loop a stopping condition — the checks pass — instead of "looks good to me." GitHub's Spec-Kit and AGENTS.md / CLAUDE.md are the productized versions.

Q: Walk me through the plan→patch→verify loop and where the reliability lives. A: Verify the spec; while not passed and under max_iters, plan the change from the spec and last failures, apply each patch under its own snapshot (roll back a failed apply), then re-verify. The reliability is in the guards and the reward: max_iters bounds an agent that never converges, snapshot/restore guarantees a failed edit doesn't corrupt the tree, and verify is an objective reward (tests pass or not) that lets the loop recover from a wrong first patch — which is exactly what makes coding agents work better than fuzzier agent categories.

Q: How do you keep the verifier from becoming a security hole? A: The acceptance checks are provided callables, never eval of the agent's output. In the lab they're static — string checks and an ast parse that never executes code. In production the check is the real test suite, but it runs inside a sandbox (Phase 09), never in the agent's process, and the agent gets an allow-listed, least-privilege view of the repo so it can't touch secrets or CI config. Defense in depth: guardrail plus sandbox plus human review of the diff.

Q: "Use AI to increase quality, not just volume" — what does that mean concretely? A: Don't measure a coding agent by lines or PRs shipped; a bigger unverified diff is negative value and harder to review. Use it to raise quality: generate the tests you'd skip, review diffs (agent as critic, human as gate), refactor under a green suite, and verify every change against its spec. The deliverable is a small, verified, reviewable diff a human merges — human-in-the-loop is non-negotiable because the model proposes and a person ships.

Q: What is pass@k and why does a verify loop change the number you can hit? A: pass@k is the probability a task is solved by at least one of k samples; the unbiased estimator is $1 - \binom{n-c}{k}/\binom{n}{k}$. pass@1 is first-try; pass@10 is solvable-with-retries. A verifier (the tests) lets you generate several patches and keep only the one that passes, so with verification your effective success rate moves from pass@1 toward pass@k. That gap is the entire value of closing the loop on an objective reward — and it's why SWE-bench, which accepts a patch only if the repo's tests pass, is the field's ground-truth benchmark.

19. References

Anthropic — Claude Code docs (edit format, slash-commands, skills, subagents, CLAUDE.md). https://docs.anthropic.com/en/docs/claude-code/overview
Anthropic — Building Effective Agents (2024). https://www.anthropic.com/research/building-effective-agents
OpenAI — Codex / codex-cli and the OpenAI Codex agent. https://openai.com/index/introducing-codex/
Cursor — Agent and edit/apply model. https://docs.cursor.com/
Aider — edit formats (search/replace blocks, unified diff, whole-file). https://aider.chat/docs/more/edit-formats.html
Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023). https://arxiv.org/abs/2310.06770 · SWE-bench Verified: https://openai.com/index/introducing-swe-bench-verified/
Chen et al., Evaluating Large Language Models Trained on Code (Codex; the pass@k estimator, 2021). https://arxiv.org/abs/2107.03374
GitHub — Spec-Kit (Spec-Driven Development toolkit, specify). https://github.com/github/spec-kit
AGENTS.md — the open convention for agent instructions in a repo. https://agents.md/
Temporal — Staff Software Engineer, AI Foundations JD ("agentic coding skills for tools like Codex and Claude Code"; "using AI to increase quality, not just output volume") — jd.md §11.

Agentic AI Engineer — Agent Infrastructure, Platforms & Production Systems