Codebase Indexing

Phase 11 · Document 02 · AI Coding Platforms Prev: 01 — VS Code Extension Architecture · Up: Phase 11 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

A real codebase is millions of tokens and the context window holds a sliver, so a coding platform must retrieve the relevant code for each request — and codebase indexing is what makes that retrieval possible and fast. It's the context-selection engine behind "chat with your repo," agent context-gathering, and relevant-snippet injection for edits (00) — i.e. RAG specialized for code (Phase 9). Get indexing right and the assistant "knows your codebase"; get it wrong and it suggests code that ignores your actual APIs, conventions, and types. Indexing also forces a privacy decision (does your proprietary code get embedded/sent to a cloud?) that's central to enterprise adoption (07).

2. Core Concept

Plain-English primer: RAG over code

Indexing is the ingestion pipeline of Phase 9 applied to a repo: chunk the code, embed the chunks, store vectors + metadata, and keep it fresh — so at query time you embed the request and retrieve the most relevant code to put in the prompt (Phase 9.07).

repo → chunk code [code-aware, see below] → embed chunks [Phase 9.03] → store (vectors + metadata: path/symbol/lines)
     → incremental updates on file change                                  → retrieve relevant code per query

But code isn't prose, so naive RAG underperforms — code indexing has its own twists:

Code-aware chunking (don't split mid-function)

Fixed-size chunking shreds code (Phase 9.02). Code has natural units — functions, classes, methods — so chunk by AST/structure (03): one chunk per function/class, keeping it whole and tagging it with its symbols, file path, and line range (for citations/jump-to). Enrich each chunk with context (file path, enclosing class, imports) so an isolated function is still understandable — the code analog of metadata enrichment (Phase 9.02).

Embeddings for code (and their limits)

Use a code-aware embedding model (trained on code: Voyage-code, OpenAI/Cohere code-capable, or open code embedders) so "function that validates a JWT" finds the right code semantically (Phase 9.03). But embeddings alone miss code's defining feature: exact symbols and structure. "Where is parse_header called?" is an exact-match/graph question that semantic search answers poorly. So code retrieval is fundamentally hybrid: embeddings (semantic) + keyword/symbol search + the symbol/call graph (03, Phase 9.05). This is the reason code indexing ≠ generic RAG.

Incremental indexing (the operational core)

Codebases change constantly (every save, branch switch, pull). Re-embedding the whole repo each time is infeasible, so indexing must be incremental: watch the filesystem / git, detect changed files (hash/mtime), and re-chunk + re-embed only what changed (and delete removed) — the Phase 9.01 incremental pattern, but continuous and low-latency because the dev is editing live. A stale index suggests against code that no longer exists.

Where the index lives: local vs remote (privacy)

A defining architectural + privacy choice:

• Local indexing — embeddings computed and stored on the developer's machine (or in their environment). Code never leaves; works offline; the privacy-safe default for proprietary code. Limited by local compute/embedding model.
• Remote/cloud indexing — code (or its embeddings) is sent to a service to index. More powerful (bigger models, server compute, team-shared index), but your source code goes to a third party — a real governance issue. Some products send only embeddings (and obfuscate/anonymize) rather than raw code to mitigate, but embeddings can leak information.

This maps directly to Phase 5.02 local-vs-cloud and Phase 8.09 data residency: enterprises often mandate local indexing or self-hosted everything.

Retrieval at query time

Given the index, retrieval is RAG (Phase 9): embed the query (or the current cursor context), do hybrid retrieval (semantic + symbol/keyword), rerank (Phase 9.06), and pack the top chunks alongside the current file/diagnostics within the window (Phase 9.07). For agents, retrieval is also agentic — the model greps/reads on demand (Phase 10.06), sometimes instead of a maintained index. Many platforms combine a maintained index (fast "where is X?") with agentic grep/read (precise on-demand).

3. Mental Model

   repos are millions of tokens; window is tiny → INDEX = context-selection engine (RAG for code [Phase 9])
   PIPELINE: code-aware CHUNK (by function/class via AST [03], +path/symbols/lines) → EMBED (code model [9.03])
             → STORE (vectors + metadata) → INCREMENTAL re-index on change (continuous, low-latency [9.01])
   QUERY: embed request → HYBRID retrieve (semantic + symbol/keyword [9.05/03]) → rerank [9.06] → pack [9.07]

   ★ code ≠ prose: embeddings MISS exact symbols/structure → retrieval is HYBRID (embeddings + symbol/AST graph [03])
   LOCAL index (code stays on device, privacy-safe) ◁ vs ▷ REMOTE/cloud (powerful, but code leaves → governance [07/8.09])
   agents also retrieve AGENTICALLY (grep/read on demand [10.06]) — often combined with a maintained index

Mnemonic: codebase indexing = RAG for code: chunk by structure, embed with a code model, store + incrementally re-index, retrieve hybrid (semantic + symbol). Code needs structure, not just embeddings; and local-vs-remote indexing is a privacy decision.

4. Hitchhiker's Guide

What to look for first: is retrieval hybrid (semantic + symbol/keyword, 03) and is indexing incremental? Those two determine whether the assistant actually knows your live codebase. Then the local-vs-remote privacy posture.

What to ignore at first: a giant remote index for a small repo — agentic grep/read (Phase 10.06) is often enough until the repo is large.

What misleads beginners:

• Treating code like prose. Fixed-size chunking + embeddings-only retrieval misses exact symbols — chunk by AST, retrieve hybrid (03).
• Static index. Without incremental re-indexing, suggestions reference deleted/old code (Phase 9.01).
• Ignoring privacy. Cloud-indexing proprietary code is a governance decision — many enterprises forbid it (07/Phase 8.09).
• Index-only thinking. Combine the maintained index with agentic grep/read for precision (Phase 10.06).
• No code-aware embeddings. Generic embedders underperform on code; use a code-capable model (Phase 9.03).

How experts reason: they index with AST-aware chunking + code embeddings, retrieve hybrid (semantic + symbol/graph), rerank + pack (Phase 9.06/9.07), keep the index incrementally fresh, choose local vs remote by privacy, and blend with agentic grep/read. They measure retrieval relevance (does the right code get into context?) as the upstream driver of suggestion quality.

What matters in production: retrieval relevance (recall of the right symbols/files), index freshness/latency (keeps up with edits), privacy posture (local vs remote), and indexing cost (embedding a big monorepo isn't free).

How to debug/verify: when suggestions ignore your APIs/conventions, check what was retrieved — was the relevant function/type in context? If not, fix chunking (AST), add symbol/keyword retrieval, or refresh the index. Measure recall on labeled "which file/symbol answers this?" queries (Phase 9.09).

Questions to ask: is chunking code-aware (AST)? is retrieval hybrid (semantic + symbol)? incremental re-indexing? local or remote (privacy)? blended with agentic grep/read? cost of indexing the monorepo?

What silently gets expensive/unreliable: prose-style chunking (shredded code), embeddings-only retrieval (missed symbols), stale index (ghost code), cloud-indexing private code (governance breach), and re-embedding the whole repo (cost).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Codebase indexing is RAG for code (Phase 9): chunk → embed → store → incrementally re-index → retrieve.
• Code ≠ prose — embeddings alone miss exact symbols/structure; retrieval must be hybrid (semantic + symbol/keyword + graph) (03, Phase 9.05).
• Chunk code-aware (by function/class via AST), tagged with path/symbols/lines; enrich isolated chunks (Phase 9.02/03).
• Use code-aware embedding models, not generic prose embedders (Phase 9.03).
• Indexing must be incremental and continuous — re-embed only changed files; stale indexes suggest deleted code (Phase 9.01).
• Local vs remote indexing is a privacy decision — local keeps code on device; remote is powerful but sends code/embeddings to a third party (07/Phase 8.09).
• Retrieval relevance drives suggestion quality — bad retrieval = code that ignores your APIs/conventions.
• Maintained index + agentic grep/read are often combined (Phase 10.06).

9. Observations from Real Systems

• Cursor indexes the codebase (embeddings) with a local/remote option and explicitly addresses privacy (e.g., sending embeddings, not raw code, where possible) — the canonical example.
• Sourcegraph combines structural (symbol) + semantic code search — the hybrid principle at scale.
• Claude Code / agent-first tools lean on agentic grep/read (Phase 10.06) rather than (or alongside) a maintained embedding index.
• The recurring quality bug — "it suggests code using an API we don't have" — is a retrieval/index failure (wrong or stale context), not a model failure.
• Enterprises mandate local/self-hosted indexing for proprietary code — a frequent procurement gate (Phase 8.09).

10. Common Misconceptions

11. Engineering Decision Framework

INDEX A CODEBASE:
 1. SMALL repo? → agentic grep/read may suffice (no index) [10.06]. LARGE repo → build an index.
 2. CHUNK code-aware: by function/class via AST [03]; tag path/symbols/lines; enrich with context [9.02].
 3. EMBED with a CODE-aware model [9.03]; STORE vectors + metadata [9.04].
 4. RETRIEVE HYBRID: semantic + symbol/keyword [9.05/03] → rerank [9.06] → pack with current file/diagnostics [9.07].
 5. INCREMENTAL: watch fs/git; re-embed only changed files; delete removed (continuous, low-latency) [9.01].
 6. PRIVACY: LOCAL index for proprietary code (default); REMOTE only if governance allows; send embeddings-not-code if cloud [07/8.09].
 7. MEASURE retrieval relevance (recall of right symbols/files) [9.09].

12. Hands-On Lab

Goal

Build a code index with AST-aware chunking + embeddings, add symbol search, and show hybrid retrieval beats embeddings-only on code queries.

Prerequisites

• A Python repo; pip install openai chromadb; the AST chunker from 00/03; ~10 code queries (incl. exact-symbol ones like "where is parse_header defined/used?").

Steps

1. Code-aware chunk: parse each file's AST and emit one chunk per function/class, tagged with path, symbol, start/end_line (03).
2. Embed + store: embed chunks with a code-capable model (Phase 9.03); store vectors + metadata in Chroma.
3. Semantic retrieval: for each query, retrieve top-k by embedding; record whether the right symbol/file is found (recall) — note it misses exact-symbol queries.
4. Add symbol search: build a simple symbol→chunk map (from the AST tags) for exact-name lookup; fuse symbol hits with semantic hits (RRF, Phase 9.05).
5. Compare: measure recall for embeddings-only vs hybrid — hybrid should win, especially on exact-symbol queries (Phase 9.09).
6. Incremental: edit one file, re-index only it (hash check), and confirm retrieval reflects the change (Phase 9.01).

Expected output

A code index with AST chunks + metadata, a recall comparison showing hybrid > embeddings-only (esp. exact symbols), and a working incremental re-index — the context-selection engine for 00.

Debugging tips

• Recall low on "where is X used?" → that's a symbol/graph query; add symbol search (03).
• Chunks shredded → not chunking by AST; fix the parser.

Extension task

Add a call/import graph for "what calls this?" retrieval (03); add reranking (Phase 9.06).

Production extension

Wire incremental indexing to a file watcher, choose local vs remote per privacy (07/Phase 8.09), blend with agentic grep/read (Phase 10.06), and track retrieval relevance.

What to measure

Retrieval recall (right symbol/file) for semantic vs hybrid; exact-symbol query performance; incremental re-index latency; index size/cost.

Deliverables

• A code index (AST chunks + code embeddings + metadata).
• A hybrid vs embeddings-only recall comparison.
• An incremental re-index demonstration + a local/remote privacy note.

13. Verification Questions

Basic

1. Why does a coding platform need an index at all?
2. Why chunk code by AST instead of fixed size?
3. Why is code retrieval hybrid (not embeddings-only)?

Applied 4. What metadata should each code chunk carry, and why? 5. Local vs remote indexing — what drives the choice?

Debugging 6. The assistant suggests code using a non-existent API. What index/retrieval issue? 7. "Where is X used?" returns junk. What retrieval capability is missing?

System design 8. Design codebase indexing: chunking, embeddings, hybrid retrieval, incremental updates, privacy posture.

Startup / product 9. Why is local indexing often a procurement requirement, and how does that constrain your architecture?

14. Takeaways

1. Codebase indexing is RAG for code — the context-selection engine behind the platform (Phase 9).
2. Chunk code-aware (by AST), embed with a code model, tag with path/symbols/lines.
3. Code retrieval is hybrid — embeddings + symbol/keyword + graph; embeddings alone miss exact symbols (03).
4. Index incrementally and continuously — stale indexes suggest deleted code.
5. Local vs remote indexing is a privacy decision; blend the index with agentic grep/read (Phase 10.06).

15. Artifact Checklist

• A code index (AST chunks + code embeddings + metadata).
• A hybrid vs embeddings-only retrieval-recall comparison.
• An incremental re-index (changed-file-only) demo.
• A local-vs-remote privacy decision note.
• (Optional) a symbol/call-graph retrieval addition (03).

Up: Phase 11 Index · Next: 03 — Symbol Search and AST