« Phase 06 · Warmup · Track Overview
Lab 01 — The Authorized Hybrid Retriever
The problem
Wholesale, Retail and Group Compliance all use the same "ask your documents" agent, backed by one knowledge foundation. A Legal user asks a question, the retriever returns the three most similar chunks, and one of them is a Wholesale payment record — because similarity has no opinion about ownership.
That is the failure this lab is built around, and it is the worst kind: it produces a 200 OK with a plausible answer. No error, no alert, no detection. You find out months later.
So the lab builds two things at once. The quality half — structure-aware chunking, BM25, dense retrieval, RRF fusion, reranking — is ordinary information retrieval done properly. The safety half — namespaces, entitlement as a pre-filter, citations, freshness contracts — is what makes it a bank's knowledge foundation rather than a demo.
What you build
| # | Component | What it does |
|---|---|---|
| 1 | Document, Chunk, chunk_document | structure-aware chunking (sections first, size second) with provenance and a citable span on every chunk |
| 2 | hash_embed, cosine | signed feature hashing, L2-normalized — deterministic so retrieval behaviour is testable |
| 3 | BM25Index | Okapi BM25 from first principles: k1 saturation, b length normalization, clamped IDF |
| 4 | VectorIndex | a namespaced dense index — the topology decision, made in the data structure |
| 5 | reciprocal_rank_fusion | score-free merging, k=60 |
| 6 | rerank, lexical_overlap_reranker | second stage over the top-k only, with a relevance floor |
| 7 | AuthorizedRetriever | three independent enforcement points: namespace, entitlement pre-filter, freshness contract |
| 8 | check_grounding | every claim maps to a retrieved span or the report names it |
| 9 | assemble_context | fills a token budget in cache-friendly order, drops lowest-ranked first, and reports what went |
Key concepts
| Concept | Where | Why it matters |
|---|---|---|
| Structure before size | chunk_document | a fixed-size splitter cuts through a policy clause and produces a retrievable, unusable chunk |
| Namespace ≠ filter | namespace_of, VectorIndex | a cross-tenant chunk is never retrieved, not filtered out |
| Two mechanisms, deliberately | namespace_of vs visible | a classification bug leaks within a tenant; a tenant-filter bug leaks across customers |
| Pre-filter, not post-filter | AuthorizedRetriever.retrieve | filtering after ranking leaks existence through result-set size and is one refactor from leaking content |
| Signed hashing | hash_embed | without random signs, collisions always add constructively and nothing is ever dissimilar |
| Clamped IDF | BM25Index._idf | a term in every document must contribute 0, never a negative score |
| Score-free fusion | reciprocal_rank_fusion | BM25 and cosine live on incomparable scales; normalizing is a calibration that breaks |
| Relevance floor | rerank(min_score=…) | first-stage retrieval always returns something; without a floor, "nothing relevant" looks like "least irrelevant" |
| Freshness is a contract | RetrievalPolicy.max_stale_ticks | staleness is a property you promise, not one you hope for |
| Rerank is sheddable | enable_rerank | quality degrades, the answer survives — that is what makes retrieval a degradable dependency |
| Measure the assembled text | assemble_context | headers and separators are real tokens; budgeting the pieces over-fills every time |
| Dropping is reported | dropped_chunks | "we answered without the third source" is a fact grounding and audit both need |
Files
| File | Role |
|---|---|
| lab.py | your implementation |
| solution.py | reference; python solution.py runs a nine-part worked session |
| test_lab.py | 67 tests |
| requirements.txt | pytest |
Run
pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v
python solution.py
Success criteria
-
All 67 tests green against your
lab.py. - A wholesale principal asking a retail question gets no retail chunks, and the retail principal gets them.
-
visible()is a conjunction — failing any of tenant, classification or barrier hides the chunk. -
excluded_by_entitlementis non-zero when a clearance is too low: the exclusion is counted, which is how you prove the pre-filter ran. -
BM25 saturates: going 9 → 10 occurrences adds less than 1 → 2 (test with
b=0to isolate it). - BM25 IDF never goes negative for a term present in every chunk.
- RRF ranks a consistently-3rd chunk above a 1st-and-10th one.
- Feature hashing produces at least one non-positive similarity across 40 unrelated pairs — proof the signs are doing their job.
- A query matching nothing returns an empty result, not the least-irrelevant chunk.
- The assembled context never exceeds its budget, and an impossible budget raises rather than truncating the question.
How this maps to the real stack
| This lab | The real thing | What we simplified |
|---|---|---|
VectorIndex | pgvector · Azure AI Search · Pinecone · Weaviate · Qdrant | exact search over a list; real stores use HNSW or IVF-PQ and trade recall for speed |
| Namespaces | Pinecone namespaces, Qdrant collections, per-tenant pgvector schemas, Azure AI Search index-per-tenant | the decision is identical; the storage is not |
BM25Index | Elasticsearch/OpenSearch, Azure AI Search's keyword mode, rank_bm25 | ours has no stemming, stop-word list, or field boosting |
hash_embed | a real embedding model (and the re-embedding migration when you change it) | ours captures lexical overlap, not meaning |
| RRF | Azure AI Search hybrid ranking, Weaviate's fusion, LangChain's EnsembleRetriever | identical formula, same k=60 default |
rerank | a cross-encoder (Cohere Rerank, BGE-reranker, Azure semantic ranker) | ours is deterministic; a real one is a model call in the latency budget |
check_grounding | RAGAS faithfulness, Bedrock contextual grounding checks, an LLM-as-judge | ours is token overlap; a real one is an entailment model |
assemble_context | prompt assembly in the agent kernel, ordered for provider prefix caching | no truncation of individual chunks, no summarization fallback |
Honest limits. No ANN index, so no filtering cliff to observe — the phenomenon where a selective post-filter collapses recall is precisely what namespaces avoid, and you cannot see it without approximate search. No re-embedding migration. No stemming or lemmatization. No multi-vector or late-interaction retrieval. And the grounding check is lexical, so a correct paraphrase can fail it — which is exactly the precision/recall trade the WARMUP asks you to tune deliberately.
Extensions
- Build an HNSW index and then watch the filtering cliff: apply a 1-in-500 metadata filter after ANN search and measure recall@10 collapse. Then fix it with a namespace and compare.
- Re-embedding migration. Change the embedding dimension, and migrate a populated index with no downtime: dual-write, backfill, shadow-read, compare, cut over. Time it.
- Chunking bake-off. Fixed-size vs structure-aware vs structure-aware-with-overlap, measured as recall@k on a golden set. The result is usually decisive and surprises people.
- Graph expansion. Take Phase 07's graph and expand the retrieved set along ownership edges before reranking.
- A real reranker interface. Swap
lexical_overlap_rerankerfor a callable that batches pairs, and put it in the Phase 00 latency budget. Measure what shedding it costs in recall. - Query rewriting. Add a step that expands "why is it held" into the vocabulary the corpus uses. Then measure whether it helps BM25, dense, or both — the answer is instructive.
Interview / resume bullets
- "Made retrieval authorized by construction: tenant isolation enforced by the index namespace rather than a filter, with classification and information-barrier checks applied before ranking — so an unentitled chunk is never a candidate, not merely never returned."
- "Implemented hybrid retrieval with BM25, dense search and reciprocal rank fusion, chosen score-free so that changing the embedding model does not require re-calibrating the merge."
- "Added a relevance floor to reranking, which turned 'the retriever always returns something' into an explicit 'nothing relevant' — and stopped the grounding check being asked to support claims against noise."
- "Built a grounding check that maps every claim to a retrieved span and fails the answer otherwise, so an unsupported statement is caught before it reaches a customer or an auditor."
- "Ordered prompt assembly stable-content-first and measured the cacheable fraction, making the prefix-cache discount a number the platform reports rather than an accident."