Autocomplete Models

Phase 11 · Document 05 · AI Coding Platforms Prev: 04 — Inline Edit and Apply-Patch · Up: Phase 11 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Autocomplete (ghost text as you type — Copilot/Cursor Tab) is the most-used, most-latency-brutal feature in a coding platform, and it's architecturally different from chat/edit/agent: it must respond in tens to ~200 ms, on every keystroke, which forces a small, fast, FIM-trained model and a serving path tuned for speed — not the big reasoning model. Getting it right is a UX make-or-break (laggy autocomplete is worse than none) and a cost/latency engineering problem (Phase 7). It's the clearest example of why coding platforms are multi-model (00, 06): you cannot serve autocomplete with the same model or path as an agent.

2. Core Concept

Plain-English primer: complete at the cursor, instantly

Autocomplete predicts the code that should appear at the cursor, given what's before and after it. Unlike chat, the model isn't answering a question — it's filling a hole in the middle of a file. Two defining requirements:

1. Fill-in-the-Middle (FIM) — the model must condition on both the prefix (code before cursor) and the suffix (code after), not just left context, because good completions depend on what comes next (the function's return, the closing brace, the next call).
2. Extreme latency — it fires constantly as you type, so it must return in well under a second (ideally ~tens of ms) or it's useless (the user has already typed past it).

Fill-in-the-Middle (FIM)

Plain LLMs are trained left-to-right (predict next token from the left). FIM trains/prompts the model with a reordered format so it can generate the middle given prefix and suffix:

<|fim_prefix|>def total(items):
    s = 0
    for it in items:
<|fim_suffix|>
    return s
<|fim_middle|>          ← the model generates here, aware of BOTH sides

The model sees the suffix (return s) and completes the loop body coherently. FIM dramatically beats left-only completion for code. Models trained with FIM: StarCoder/StarCoder2, CodeLlama, DeepSeek-Coder, Qwen2.5-Coder, Codestral — using FIM-aware completion is essential (a chat model without FIM makes a poor autocomplete engine).

Why a small, fast model (and often local)

The latency budget forces model choice:

• Small models (1–7B), often quantized, hit the budget; a frontier model's per-token latency + network round-trip blows it (Phase 6).
• Short outputs — autocomplete emits a line or a few, not paragraphs; cap max_tokens low.
• Local/edge serving is attractive — running the autocomplete model on the dev's machine (or a nearby edge) removes network latency and keeps code private (Phase 6, 07). Many tools run a small local FIM model for Tab and reserve cloud models for chat/agent.
• Speculative decoding/MTP (Phase 6.07) helps when generation is the bottleneck.

This is exactly the cost-quality-latency trade (Phase 5.09) with latency weighted to the extreme.

The client-side speed tricks (just as important as the model)

Hitting the budget is as much engineering as model size (01):

• Debounce — don't fire on every keystroke; wait a few ms of typing pause.
• Cancel in-flight requests — on a new keystroke, abort the previous completion (cancellation tokens) so you don't pay for or show stale suggestions.
• Caching — cache completions for identical prefix/suffix contexts; prefix caching on the server (Phase 7.05).
• Speculative/eager fetching — prefetch likely completions.
• Trim context — autocomplete uses the local window (current file region ± a little retrieved context), not the whole codebase, to stay fast.

Acceptance rate: the metric that matters

Autocomplete quality is measured by acceptance rate — the fraction of suggestions the developer keeps (and, more strictly, retains after edits). High acceptance = genuinely helpful; low = noise/distraction. It's the autocomplete analog of apply-rate (04) and the north-star you tune model + context + latency against (08). You also watch latency (p50/p95) — beyond the budget, acceptance craters because suggestions arrive too late.

Where it sits in the platform

Autocomplete is its own tier (06): a dedicated small FIM model + a low-latency serving path + aggressive client-side optimization — separate from chat (reasoning model), edit (instruction + apply model [04]), and agent (tool-loop, Phase 10.06). It's wired through the editor's inline-completion API (01).

3. Mental Model

   autocomplete = fill code AT THE CURSOR, instantly, on every keystroke
   ① FIM: condition on PREFIX + SUFFIX (not left-only) → coherent middle (StarCoder/DeepSeek-Coder/Qwen-Coder/Codestral)
   ② LATENCY ~tens–200ms → SMALL fast model (1–7B, quantized), short max_tokens, often LOCAL/edge [Phase 6]
        (+ speculative decoding [6.07]) — cost-quality-latency with LATENCY maxed [5.09]
   ③ CLIENT tricks (≈ as important as the model): DEBOUNCE · CANCEL in-flight · CACHE · prefetch · trim to LOCAL context [01]

   METRIC: ACCEPTANCE RATE (suggestions kept/retained) + p50/p95 LATENCY → past the budget, acceptance craters
   it's a SEPARATE TIER from chat/edit/agent [06]; wired via the inline-completion API [01]

Mnemonic: autocomplete = instant FIM completion at the cursor — small fast (often local) FIM model + debounce/cancel/cache. Acceptance rate is the metric; it's its own latency tier, not the agent model.

4. Hitchhiker's Guide

What to look for first: is it a FIM-trained model within the latency budget, and are debounce + cancel in place? Those three make autocomplete usable. Then acceptance rate as the quality metric.

What to ignore at first: using the biggest model for completions, and elaborate whole-repo context for autocomplete. Start with a small FIM model + local-window context + debounce/cancel.

What misleads beginners:

• Using a chat/frontier model for autocomplete. Too slow (and not FIM) — laggy ghost text the user has typed past (Phase 5.09).
• Left-only context. Without the suffix (FIM), completions clash with what follows.
• No debounce/cancel. Firing every keystroke = wasted cost + stale suggestions (01).
• Stuffing whole-codebase context. Kills the latency budget; autocomplete uses the local window (+ light retrieval).
• Measuring "looks good" not acceptance. The metric is acceptance/retention rate, plus p95 latency (08).

How experts reason: they pick a small FIM model (local/edge where possible for latency + privacy), cap output short, apply debounce + cancel + caching, trim to local context (+ a little retrieval), and tune against acceptance rate within a strict p95 latency budget (08). They treat autocomplete as a distinct tier/serving path, not a variant of chat.

What matters in production: p50/p95 latency (the budget), acceptance/retention rate, cost (it fires constantly — cheap per call matters), and privacy (local FIM keeps code on device, 07).

How to debug/verify: measure p95 latency (over budget → too-big model / network / no cancel); measure acceptance (low → wrong model/context/FIM or latency too high); confirm debounce/cancel are firing.

Questions to ask: is the model FIM-trained? what's p95 latency vs budget? local or cloud (latency/privacy)? debounce + cancel + cache? what context (local vs repo)? acceptance rate?

What silently gets expensive/unreliable: a too-big/non-FIM model (laggy, low acceptance), no cancellation (cost + stale), whole-repo context (latency), and ignoring acceptance (shipping noisy suggestions).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Autocomplete is a distinct latency tier (~tens–200 ms, every keystroke) — it needs a small, fast, FIM-trained model, not the chat/agent model (00, 06).
• FIM conditions on prefix AND suffix — essential for coherent completions; use FIM models (StarCoder2, DeepSeek-Coder, Qwen2.5-Coder, Codestral).
• Small (1–7B), quantized, short-output, often local/edge models hit the budget and keep code private (Phase 6, 07).
• Client-side speed tricks (debounce, cancel in-flight, caching, prefetch, local-context-only) are as important as the model (01).
• Acceptance/retention rate is the quality metric; p50/p95 latency is the constraint — past the budget, acceptance craters (08).
• Autocomplete uses local context (current file region + light retrieval), not the whole codebase — for speed.
• Speculative decoding/MTP can help when generation is the bottleneck (Phase 6.07).
• It's the clearest case for multi-model architecture — you literally cannot use one model for autocomplete and agent (06).

9. Observations from Real Systems

• GitHub Copilot began as FIM autocomplete (Codex/FIM) and remains latency-tuned; Cursor Tab uses a fast custom completion model.
• Local/edge FIM models (StarCoder2, DeepSeek-Coder, Qwen2.5-Coder, Codestral via Ollama/llama.cpp) power privacy-first/offline autocomplete (Phase 6.04).
• Debounce + cancellation are universal — every good autocomplete cancels stale requests on keystroke (01).
• Acceptance rate is the headline autocomplete metric teams optimize (Copilot has published acceptance/retention studies) (08).
• Speculative decoding appears in fast completion/apply paths to shave decode latency (Phase 6.07, 04).

10. Common Misconceptions

11. Engineering Decision Framework

BUILD AUTOCOMPLETE (its own tier):
 1. MODEL: small (1–7B), FIM-trained (StarCoder2/DeepSeek-Coder/Qwen-Coder/Codestral), quantized; LOCAL/edge if possible [6/07].
 2. FORMAT: FIM (send PREFIX + SUFFIX); cap max_tokens short (a line / few lines).
 3. CONTEXT: LOCAL window (current file region) + light retrieval — NOT whole codebase (latency).
 4. CLIENT: DEBOUNCE; CANCEL in-flight on keystroke; CACHE identical contexts; prefetch [01].
 5. SERVING: low-latency path; prefix caching [7.05]; speculative decoding if decode-bound [6.07].
 6. MEASURE: ACCEPTANCE/retention rate within a strict p95 LATENCY budget [08]; tune model+context+latency.

12. Hands-On Lab

Goal

Build a FIM autocomplete with a small/local model, add debounce + cancel + cache, and measure acceptance rate within a latency budget — feeling why autocomplete is its own tier.

Prerequisites

• A local FIM model via Ollama/llama.cpp (e.g., a Qwen2.5-Coder/StarCoder2/DeepSeek-Coder variant, Phase 6.04); a code file; (optional) the VS Code extension from 01.

Steps

1. FIM completion: at a cursor position, build the FIM prompt (prefix + suffix) and request a short completion from the local model; verify it completes coherently using the suffix (e.g., respects a later return).
2. Left-only contrast: request a completion with prefix only (no suffix); compare quality — FIM should be clearly better.
3. Latency: measure p50/p95 completion latency on your machine; compare a small local model vs a frontier API model — see the budget difference (Phase 5.09).
4. Client tricks: add debounce (fire after a typing pause) and cancel (abort the previous request on a new keystroke); add a cache for identical prefix/suffix; observe fewer calls + no stale suggestions (01).
5. Acceptance proxy: run a set of holes (mask a line in real code) and measure how often the completion matches/would be accepted (exact or close) — your acceptance proxy (08).
6. Context trim: show that using the local window (not the whole file/repo) keeps latency in budget with little quality loss.

Expected output

A working FIM autocomplete (local model) demonstrating FIM > left-only, a latency comparison (local vs API) against a budget, working debounce/cancel/cache, and an acceptance-proxy measurement.

Debugging tips

• Incoherent completions clashing with later code → not using FIM (suffix); fix the FIM prompt/model.
• Over budget → model too big / network / no cancel; use a smaller local model + cancellation.

Extension task

Add speculative decoding if your runtime supports it (Phase 6.07) and measure the latency gain; wire it into the VS Code inline-completion provider (01).

Production extension

Serve the FIM model on a low-latency/local path with prefix caching (Phase 7.05); track acceptance/retention + p95 in eval (08); keep code local for privacy (07).

What to measure

FIM vs left-only quality, p50/p95 latency (local vs API) vs budget, debounce/cancel effect, acceptance proxy, context-trim effect.

Deliverables

• A FIM autocomplete with a small/local model.
• An FIM vs left-only and a local vs API latency comparison.
• Debounce/cancel/cache + an acceptance-proxy measurement.

13. Verification Questions

Basic

1. What is FIM, and why is it needed for autocomplete?
2. Why does autocomplete need a different model than chat/agent?
3. What is acceptance rate, and why is latency tied to it?

Applied 4. Why run the autocomplete model locally/edge when possible? 5. Which client-side tricks keep autocomplete in its latency budget?

Debugging 6. Completions clash with the code after the cursor. Cause and fix. 7. Autocomplete feels laggy and acceptance is low. Two causes.

System design 8. Design the autocomplete tier: model, FIM, context, serving path, client tricks, metrics.

Startup / product 9. Why is autocomplete the clearest justification for a multi-model coding-platform architecture?

14. Takeaways

1. Autocomplete is its own latency tier — a small, fast, FIM-trained model, not the chat/agent model.
2. FIM (prefix + suffix) is essential for coherent completions.
3. Small/quantized/short-output, often local/edge models hit the ~tens–200 ms budget and keep code private.
4. Client tricks (debounce, cancel, cache) are as important as the model (01).
5. Acceptance/retention rate within a strict p95 latency budget is the metric — tune model + context + latency to it (08).

15. Artifact Checklist

• A FIM autocomplete (small/local model, prefix+suffix).
• An FIM vs left-only quality comparison.
• A local vs API latency comparison vs a budget.
• Debounce + cancel + cache in the client.
• An acceptance-proxy measurement.

Up: Phase 11 Index · Next: 06 — Planning vs Execution Models