« Phase 05 README · Warmup · Hitchhiker's · Deep Dive · Core Contributor · Staff Notes

Phase 05 — Principal Deep Dive: Retrieval Foundations

The lab fits a whole retriever in one process against twelve documents. This document is about what the same pipeline becomes when it is a shared platform serving many tenants, millions of chunks, and a latency SLO — and where it breaks. The algorithms do not change; the system around them is the whole job.

The build/serve split is the primary architectural boundary

HybridRetriever hides a decision every production system makes explicit: indexing is an offline, throughput-bound batch pipeline; retrieval is an online, latency-bound service. In the lab, add() stages chunks and _ensure_built() rebuilds both indexes lazily on the next query. That laziness is a toy; at scale it is a career-ending anti-pattern, because rebuilding an HNSW graph or a BM25 posting list on the request path adds seconds of tail latency and couples write load to read latency.

The real topology is two services with a durable artifact between them:

• Index build: documents → chunk → embed (batched, GPU) → upsert into vector store + BM25 postings. Throughput-bound, restartable, idempotent on chunk id, versioned. Runs on ingest and on backfills.
• Query serve: query → embed(query) + tokenize → dense ANN top-n ∥ BM25 top-n → RRF → rerank top-k. Latency-bound, stateless, horizontally scalable behind a load balancer.

The chunk id is the contract across the boundary. chunk_text emitting a stable doc_id#idx is what lets the build pipeline upsert (delete-then-insert keyed by id) instead of append — without stable ids you cannot update a document without duplicating it.

The performance envelope: recall, latency, cost, memory — pick your point

There is no "fast and accurate" setting; there is a surface you choose a point on.

Recall vs latency (the ANN knob). Exact kNN is O(N·dim) — the lab's DenseIndex. At 10M × 768-dim floats that is ~7.7 GFLOPs per query, tens of ms per shard single-threaded, and it does not fit a tight SLO. HNSW turns it into ≈ O(log N) graph hops at the cost of a few points of recall, tuned by efSearch: low efSearch is fast and lossy, high is slow and near-exact. The principal-level statement is quantitative: "at efSearch=64 we hold recall@10 ≥ 0.95 with p99 under 15 ms/shard; pushing to 0.99 costs us 2.5× latency, which blows the budget, so we buy the last 4 points of recall with the reranker instead."

Memory footprint. 10M × 768-dim × 4 bytes = ~30 GB of raw vectors before the HNSW graph (add M neighbor ids per node, typically another 20–40%). This is why PQ exists: compress to tens of bytes per vector and the index is RAM-resident on one box. Memory, not compute, is usually the first wall you hit, and it drives the sharding decision.

Cost. Two cost centers. Embedding is a per-token model call on every chunk at build time and every query at serve time — cheap per call, brutal across a 50M-chunk corpus and a re-embed. Serving is RAM (vectors resident) plus the reranker, which is a cross-encoder forward pass per candidate: reranking 100 candidates is 100 model calls, so it is the dominant serve-time cost and you cap the shortlist deliberately.

Latency budget. A concrete decomposition for a 300 ms retrieval SLO: query embed ~20 ms, dense ANN ~15 ms, BM25 ~10 ms (parallel with dense), RRF ~1 ms (it is arithmetic over ids), rerank 50 candidates ~80 ms, network/overhead ~40 ms. The reranker is a third of the budget, which is why "is reranking worth it here" is a real decision, not a default.

Sharding, and why fusion is naturally shard-friendly

Past one machine's memory you shard: partition chunks across N nodes, each holds a sub-index, fan out the query, merge. Dense ANN merges by score (top-k from each shard, re-sort). BM25 needs care — IDF is a corpus-global statistic, so naively sharded BM25 computes IDF per shard and mis-weights rare terms; production systems either replicate global df or accept the approximation. RRF is the pleasant part: it fuses by rank, so a scatter-gather that returns ranked id lists per shard fuses correctly without reconciling score scales across shards. The lab's rrf being order-independent and score-free is exactly the property that survives sharding.

Failure modes and blast radius

• Stale index. A document changed but its old embedding lingers because the pipeline appended instead of upserting. Blast radius: wrong-but-confident answers citing a real source, the hardest class to detect because nothing errors. Mitigation: delete-then-insert keyed by chunk id, and a freshness SLA measured as index-lag, not "we reindex nightly."
• Embedding-model drift. Swapping the embedder (or a silent provider-side version bump) invalidates every stored vector — query vectors from model B are compared against document vectors from model A, and cosine becomes noise. Recall craters globally and silently. This is the single largest-blast-radius failure in retrieval: mitigate with a versioned index (index@modelv3), a full re-embed behind a dual-read migration, and pinning the embedder version as a hard dependency.
• Chunk-boundary bugs. An off-by-one in chunk_text's step = size − overlap, or dropping the overlap, splits a fact across two chunks so no single chunk answers the question. Recall looks fine on aggregate metrics and fails precisely on the boundary-straddling queries. Caught only by the invariant tests (overlap seam, full coverage) and by evaluating on hard cases.
• ANN recall cliff. efSearch/nprobe set too low silently drops the true neighbor. The index returns plausible results, so nothing looks broken; only recall@k against a labeled set reveals it.

Cross-cutting concerns the lab omits and production cannot

• Access-control filtering at retrieval time. A retriever that ignores who is asking will surface a chunk the caller may not read, and the model summarizes confidential data into the answer. A shared index across tenants is the #1 multi-tenant leak channel. Enforce it architecturally: per-tenant namespaces/collections, or a mandatory tenant_id filter injected server-side on every query — never a prompt instruction, never client-supplied. The subtle engineering question is pre-filter vs post-filter: pre-filtering (restrict the ANN search to the allowed set) preserves recall but is harder to index; post-filtering (ANN then drop unauthorized hits) is trivial but can empty your top-k when the allowed set is sparse. Principals pick per-query based on filter selectivity.
• Freshness vs cost. Real-time upsert on every document change is expensive; nightly batch is cheap but stale. The answer is per-corpus: an incident-runbook corpus needs minutes of lag; an archived-policy corpus tolerates a day.
• Latency budget ownership. Retrieval is one hop in a larger agent turn; its budget is negotiated against generation, not maximized in isolation.

"Looks wrong but intentional" decisions

• Brute-force DenseIndex. Not naïveté — it is the exact oracle the ANN indexes approximate, and the reference you measure ANN recall against. Ship ANN, keep the exact scan as the recall ground truth in CI.
• chunk_overlap=0 as the retriever default. Overlap costs storage and duplicate retrieval; it is a per-corpus lever you raise for boundary-sensitive corpora, not a universal on.
• A separate reranker instead of a bigger retriever. Buying the last few points of recall with efSearch costs latency on every query; buying it with a reranker on a capped shortlist is cheaper and more precise. Retrieve wide and cheap, rerank narrow and precise is an architecture, not a preference.
• RRF instead of a tuned score blend. A learned weighted blend can beat RRF on a fixed corpus, but it needs retuning whenever the corpus statistics move; RRF is parameter-light and never drifts. Principals trade a little peak quality for zero maintenance.

The capacity math, stated once

For a corpus of C chunks at d dimensions in f-byte floats: raw vector memory ≈ C·d·f, HNSW overhead ≈ C·M·(id size), embed build cost ≈ C · (per-chunk embed latency), and a re-embed on model swap costs the entire build again. Serve QPS is bounded by (shard count · per-shard throughput) where per-shard throughput is dominated by the reranker's per-candidate model calls, not the ANN lookup. Write those four quantities on the whiteboard before you argue about which vector database to buy — the database choice rarely changes them.