Selecting Coding Models

Phase 5 · Document 03 · Model Selection Prev: 02 — Local vs Cloud · Next: 04 — Reasoning Models

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Coding is the highest-value, most-measured LLM use case — and selecting a coding model well is subtle, because "writes code that looks right" is very different from "produces a clean diff that applies and passes tests in a real repo." The wrong choice gives you plausible-but-broken code, sloppy rewrites instead of surgical edits, or an agent that can't reliably call tools. This doc applies the selection framework to coding: what to optimize for, which benchmarks actually predict usefulness, the candidate tiers, and how to eval on your codebase. It underpins the AI coding platform phase (Phase 11).

2. Core Concept

Plain-English primer

• Code generation vs editing — generation = write new code from a description; editing = change existing code (the harder, more common product need). A great generator can still be a poor editor.
• Diff / patch quality — does the model produce a small, correct diff (just the lines that change) rather than rewriting the whole file? Clean diffs apply cleanly and are reviewable.
• pass@k — coding benchmark metric: fraction of problems solved within k attempts. Prefer pass@1 (one shot) as it reflects real usage (Phase 4.03).
• SWE-bench vs HumanEval — HumanEval = write a small function to pass unit tests (toy); SWE-bench = fix a real GitHub issue so the repo's tests pass (realistic). SWE-bench predicts product usefulness much better.
• Agentic coding — the model works in a loop: read files, propose edits, run tests/tools, iterate. Requires reliable tool calling (Phase 5.06/Phase 10).

What to optimize for (the coding-specific axes)

1. Correctness — does the code run and do the right thing? (Measured by tests, not eyeballing.)
2. Edit quality — clean, minimal diffs that apply; follows the requested output format (e.g. unified diff, search/replace blocks).
3. Instruction & format following — respects constraints (language, style, "only change function X"); critical for apply-patch reliability (Phase 11).
4. Context utilization — can it use a large codebase/long files in context and stay coherent (Phase 1.01, 2.01 recall caveat)?
5. Tool calling (agentic) — reliable, valid tool calls for read/edit/test loops (06).
6. Latency — for autocomplete, TTFT/TPOT must be tiny; for chat/agent, quality matters more.

Benchmarks to weigh (skeptically)

Prefer SWE-bench (real repo fixes) and LiveCodeBench (contamination-resistant, fresh problems) over HumanEval/MBPP (toy, contaminated). Always report pass@1 and confirm on your codebase — launch coding-benchmark wins are heavily gamed (Phase 4.03).

Candidate tiers (verify current — these churn)

• Frontier (hard edits, agentic): top Claude/GPT/Gemini coding tiers — best correctness + tool use.
• Budget (autocomplete, simple edits): fast mid/small tiers (cheap Claude/Gemini/GPT minis).
• Open-weight (self-host/private): Qwen-Coder, DeepSeek-Coder, Codestral, Code-Llama lineage.
• Routing is standard: small fast model for autocomplete; strong/reasoning model for planning + hard edits (Phase 11).

3. Mental Model

"Writes code" ≠ "ships a clean diff that APPLIES and PASSES TESTS in MY repo."

OPTIMIZE: correctness (tests!) · edit/diff quality · format-following · context use · tool calling · latency
BENCHMARKS: SWE-bench / LiveCodeBench (real, fresh) > HumanEval (toy, contaminated) ; report pass@1
DECIDE on YOUR codebase eval (apply-rate, test-pass-rate), not the leaderboard.

ROUTE:  autocomplete → small/fast ; planning + hard edits → strong/reasoning ; private → open coder model

4. Hitchhiker's Guide

What to optimize first: test-pass rate and apply rate (does the diff apply cleanly) on your code — not benchmark scores.

What to ignore at first: HumanEval leaderboard position; it's toy and contaminated.

What misleads beginners:

• Judging by "looks correct" instead of running tests.
• Using HumanEval to pick an agentic coding model (use SWE-bench-style).
• Picking one model for autocomplete and hard edits (route instead).
• Ignoring format-following — a model that ignores the diff format breaks apply-patch.

How experts reason: they build a task-specific coding eval (real issues from their repo, scored by tests + apply-rate), prefer SWE-bench-style signals, and route (fast model for completion, strong/reasoning for planning/edits). They verify tool-calling reliability for agents.

What matters in production: test-pass and apply rates on real changes, format compliance, latency for autocomplete, tool-call reliability for agents, and cost per accepted change.

How to verify: replay 20–50 real changes from your repo through candidates; measure apply-rate, test-pass-rate, and diff cleanliness (Phase 1.07).

Questions to ask: SWE-bench (pass@1) score? Reliable tool calling? Honors diff/patch format? Context window vs my files? Open coder variant for private code?

What silently gets expensive/unreliable: premium model for trivial completions (cost); flaky tool calls (agent failures); models that rewrite whole files (bad diffs, review pain); contaminated-benchmark picks that disappoint on your code.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• "Looks correct" ≠ correct — score coding by tests, not appearance.
• SWE-bench/LiveCodeBench predict usefulness far better than HumanEval (toy + contaminated).
• Report pass@1 — bigger k inflates scores (Phase 4.03).
• Edit/diff quality and format-following determine apply-patch reliability (Phase 11).
• Agentic coding needs reliable tool calling (06).
• Routing is standard: fast/small for autocomplete, strong/reasoning for planning + hard edits.
• Open coder models (Qwen-Coder, DeepSeek-Coder, Codestral) enable private/self-hosted coding.
• Eval on your own codebase — apply-rate + test-pass-rate are the decisive metrics.

9. Observations from Real Systems

• Cursor / Aider / Copilot route models: a small fast model for autocomplete, a strong model for chat/agentic edits — selection-as-routing (Phase 11).
• Aider's leaderboards measure edit-format success and apply-rate, not just code correctness — closer to real usefulness.
• SWE-bench scores track real coding-tool quality, which is why products cite them.
• Open coder models are widely self-hosted for private codebases (data can't leave) — 01/02.
• Format-following failures (model ignores the diff format) are a top cause of broken apply-patch in coding tools.

10. Common Misconceptions

11. Engineering Decision Framework

Define the coding sub-task → choose accordingly:
  AUTOCOMPLETE → tiny TTFT/TPOT; small fast model; quality secondary.
  CHAT/EXPLAIN → mid-tier; good instruction-following.
  HARD EDITS / refactors → strong (maybe reasoning) model; diff-format compliance.
  AGENTIC (read/edit/test loop) → strong model + RELIABLE tool calling (06).
  PRIVATE code → open coder model self-hosted (01/02).

Then run the framework (00): filter (context, tool calling, license) → SCORE on YOUR codebase eval
  (apply-rate, test-pass-rate, diff cleanliness) → spike → route + memo.

12. Hands-On Lab

Goal

Build a coding eval on your code and compare 2–3 models by apply-rate and test-pass-rate (not benchmarks).

Prerequisites

• A repo with tests; 10–20 real change tasks (bug fixes / small features); API/local access to candidates.

Steps

1. Build tasks: for each, capture the prompt (issue + relevant files), the expected behavior, and the tests that must pass.
2. Define edit format: require a unified diff (or search/replace blocks); your harness applies it.
3. Run candidates: for each model, generate the edit, apply it, run tests; record applied?, tests_passed?, and diff size.
4. Score: apply-rate, test-pass-rate, mean diff size (smaller = cleaner), latency, cost per accepted change.
5. Decide + route: pick per sub-task; write a memo (Phase 3.07).

# sketch of the scoring loop
results = []
for task in tasks:
    for model in models:
        diff = generate_edit(model, task.prompt)        # ask for a unified diff
        applied = apply_patch(task.repo, diff)           # did it apply cleanly?
        passed = applied and run_tests(task.repo)        # do tests pass?
        results.append((model, applied, passed, len(diff)))
# aggregate: apply-rate, test-pass-rate, mean diff size per model

Expected output

• A table per model: apply-rate, test-pass-rate, diff cleanliness, latency, cost — your real coding-quality signal.

Debugging tips

• Low apply-rate? The model isn't following the diff format — tighten the prompt or pick a better format-follower.
• Tests pass but diffs are huge? Penalize rewrite-style edits; prefer minimal diffs.

Extension task

Add an agentic task (multi-file, run-tests-and-iterate) and measure tool-call validity + steps to green (06).

Production extension

Wire this into CI as a coding-model regression gate (apply-rate/test-pass thresholds) so model/prompt changes can't silently degrade (Phase 13).

What to measure

Apply-rate, test-pass-rate (pass@1), diff size, latency, cost per accepted change; (extension) tool-call validity.

Deliverables

• A repo-based coding eval harness.
• A 2–3 model comparison table.
• A per-sub-task routing recommendation + memo.

13. Verification Questions

Basic

1. Why is SWE-bench a better signal than HumanEval for product usefulness?
2. What's the difference between code generation and code editing, and which is harder?
3. Why report pass@1?

Applied 4. Design an eval to choose a model for an apply-patch coding feature. What do you measure? 5. Why route autocomplete and hard-edit tasks to different models?

Debugging 6. Your coding agent frequently fails to apply edits. Two likely causes and fixes? 7. A model tops coding benchmarks but disappoints on your repo. What happened?

System design 8. Design model selection + routing for a Cursor-style IDE (autocomplete, chat, agent edits, private-repo option).

Startup / product 9. Argue cost-per-accepted-change (not per token) as the right coding-product metric, and how routing improves it.

14. Takeaways

1. Optimize for diffs that apply and pass tests on your code, not "looks right."
2. SWE-bench/LiveCodeBench (pass@1) > HumanEval; verify on your repo.
3. Edit quality + format-following drive apply-patch reliability.
4. Agentic coding needs reliable tool calling.
5. Route autocomplete (fast/small) vs hard edits (strong/reasoning); open coders for private code.
6. Decide on apply-rate / test-pass-rate / cost-per-accepted-change.

15. Artifact Checklist

• Repo-based coding eval (apply-rate, test-pass-rate, diff size).
• 2–3 model comparison table.
• Routing recommendation per coding sub-task.
• (Extension) agentic task with tool-call metrics.
• Selection memo (Phase 3.07).
• Notes: the "looks right ≠ passes tests" + routing lessons.

Next: 04 — Reasoning Models