« Track Overview · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes
Phase 06 — The Knowledge Foundation: Vector Topology, Hybrid Retrieval & Grounding
Answers these JD lines: "Architect the platform's knowledge foundation, including vector store selection and topology, hybrid retrieval (BM25, dense, graph), embeddings strategy, … context engineering, and grounding patterns" · "Deep experience in retrieval and knowledge architecture, including vector databases (pgvector, Azure AI Search, Pinecone, Weaviate, Qdrant, or equivalent), hybrid retrieval, reranking, embeddings model selection … and context engineering at scale."
Why this phase exists
Retrieval is where a bank's AI platform touches the bank's information, and it is therefore where two things that look like quality problems are actually security problems.
The first is authorization. A vector index returns the nearest chunk, and "nearest" has no opinion about who owns it. A shared index with a post-hoc filter is the single most common multi-tenant AI data breach, and it produces a 200 OK with a plausible answer — no error, no alert, no detection. Retrieval must be authorized, not merely relevant.
The second is grounding. An answer that cites nothing is an answer you cannot defend to an auditor, a customer, or a model-risk reviewer. Citations are not a UX nicety in a regulated platform; they are the evidence that the answer came from the bank's own record rather than from the model's priors.
The engineering content underneath those two constraints is where most of the measurable quality lives: chunking (the most under-invested lever in RAG), the embedding-model choice and the migration it implies, hybrid retrieval because exact identifiers are BM25's strength and embeddings' weakness, reranking because bi-encoders are cheap and imprecise, and the topology decision that determines whether isolation is structural or hopeful.
Concept map
- Chunking: fixed-size vs structure-aware; overlap as an index-size/recall trade; why a chunk boundary in the middle of a table destroys the table.
- Embeddings strategy: model selection, dimensionality, normalization, and the re-embedding migration — the operational event nobody plans for and everybody eventually has.
- Vector stores and topology: pgvector · Azure AI Search · Pinecone · Weaviate · Qdrant; silo / pool / bridge per tenant; HNSW vs IVF-PQ; the filtering cliff when a selective filter is applied after ANN search.
- Lexical retrieval: BM25 derived from TF-IDF, with \( k_1 \) saturation and \( b \) length normalization; why product codes and payment references need it.
- Hybrid fusion: Reciprocal Rank Fusion, \( \sum 1/(k+\mathrm{rank}_i) \) with \( k=60 \); score-free merging and why calibration is a trap.
- Reranking: cross-encoder over the top-k; the cost/accuracy trade and where it sits in the latency budget (it is the first thing you shed).
- Authorized retrieval: entitlement as a pre-filter or a namespace, never a post-filter; information barriers as retrieval constraints.
- Grounding: citation spans, faithfulness checks, freshness contracts, and what "the answer is not in the corpus" must look like.
- Context engineering: assembling the final prompt — instructions, tools, memory, retrieved spans — under a token budget, ordered for prefix caching (Phase 04).
The lab
| Lab | You build | Proves you understand |
|---|---|---|
| 01 — The Authorized Hybrid Retriever | a structure-aware chunker; a deterministic hashing embedder and a cosine index; BM25 from first principles; RRF fusion; a cross-encoder-shaped reranker; per-tenant namespaces with entitlement applied before search; a grounding checker that maps every claim to a citation span or fails; a freshness/provenance record on every chunk; and a context assembler that fills a token budget in cache-friendly order | that retrieval quality is engineering (chunking, fusion, reranking) and retrieval safety is architecture (namespaces, pre-filtering, citations) — and that the two are not in tension |
Test contract, per LAB-STANDARD: a cross-tenant query returns nothing, not a filtered list; RRF is order-independent; an answer with an unsupported claim fails the grounding check; and the assembler never exceeds its budget. 67 tests, all green.
Deliverables checklist
-
Lab 01 green under
LAB_MODULE=solution pytestand under your ownlab.py. - You can explain why a shared index with a post-hoc filter is a breach with no detection.
- You can derive BM25's two parameters and say what each controls.
- You can compute an RRF score and explain why score-free fusion avoids calibration.
- You can describe the filtering cliff and three ways to avoid it.
- You can plan a re-embedding migration without downtime.
- You can state a freshness contract and say where it is enforced.
- You can place reranking in a latency budget and say what you shed first.
Key takeaways
- Retrieval must be authorized, not merely relevant. Filter before you search.
- Chunking is the highest-leverage quality lever and the least glamorous.
- BM25 and embeddings fail on opposite inputs — that is why hybrid wins, not because more is better.
- RRF needs no calibration, which is why it survives model changes that score-fusion does not.
- Citations are evidence, and an answer that cannot cite is an answer you cannot defend.
- Re-embedding is a migration, not a config change. Plan it before you need it.