« Phase 06 · Warmup · Track Overview

Hitchhiker's Guide — The Knowledge Foundation

The 30-second mental model

Retrieval has a quality half and a safety half, and they are different disciplines.

Quality: chunk on structure, retrieve lexically and densely because they fail on opposite inputs, fuse with RRF because scores are incomparable, rerank the top-k with a cross-encoder.

Safety: retrieval must be authorized, not merely relevant. Tenant isolation is the namespace; classification and barriers are a pre-filter; every claim cites a span.

And the one sentence that justifies the whole phase: a shared index with a post-hoc filter fails with a 200 OK. It is the only failure in the platform with no runtime detection.

The formulas

The numbers

One-liners

• Recall@k caps everything. Measure it before touching the generator.
• Structure before size. A fixed-size splitter cuts a policy clause in half.
• overlap < max_tokens, or chunks stop advancing.
• Provenance at ingestion, including doc_version — you cannot reconstruct a span later.
• The sign in feature hashing is load-bearing. Without it nothing is ever dissimilar.
• Changing the embedding model is a migration: dual-write → backfill → shadow-read → cut over → retain.
• Clamp IDF at 0, or common domain terms invert your ranking.
• BM25 for identifiers, dense for intent. Complementary, not redundant.
• RRF reads positions, never scores. No calibration to break.
• A cross-encoder cannot precompute — top-k only, and it is the first thing you shed.
• A relevance floor makes "nothing relevant" expressible. Without one it is not.
• Namespace for tenants, pre-filter for classification. Different blast radius, different mechanism.
• An information barrier is a retrieval constraint, not a policy document.
• Stable content first in the prompt. The first changed byte kills the prefix-cache discount.

Vocabulary

Chunk · the retrievable unit. Overlap · re-included tail of the previous chunk. Provenance · doc id, version, section, span. Bi-encoder · embeds query and document separately (fast, precomputable). Cross-encoder · scores the pair jointly (accurate, per-pair). ANN · approximate nearest neighbour. HNSW / IVF-PQ · the two ANN families. Filtering cliff · recall collapse when a selective filter is applied after ANN search. Silo / pool / bridge · per-tenant, shared, or hybrid index topology. Namespace · the partition that makes isolation structural. Information barrier · enforced separation between businesses (advisory vs trading). MNPI · material non-public information. Grounding · every claim maps to a retrieved span. Freshness contract · the staleness you promise.

War stories

The answer from the wrong desk. One index, tenant applied as a post-filter. A refactor moved the filter one function up the call stack. Hit rate unchanged, no errors, and a Legal user received a Payments incident report inside a summary. Found by a person, months later.

The empty result set that wasn't a bug. ANN returned 100 neighbours across 10 million chunks, the tenant filter kept 3, and the two other filters kept none. The team spent a week on "the embedding model is bad." It was the filtering cliff.

The half table. Fixed-size chunking split a fee schedule between the header row and the rows. The agent read amounts with no idea which column they were in, and answered confidently.

The negative IDF. "Account" appeared in 80% of the corpus. Unclamped IDF made it negative, so documents containing the user's own search term ranked lower. Nobody noticed for a quarter because the results were still plausible.

The magic weight. 0.6 * bm25_normalized + 0.4 * cosine. Nobody could say where 0.6 came from. Changing the embedding model degraded retrieval and the weight was re-tuned by hand, twice.

The confidently superseded policy. No freshness contract. An agent cited clause 4.2 of a policy that had been replaced six weeks earlier — correctly cited, correctly retrieved, completely wrong.

"I don't know" was unreachable. No relevance floor. Asked about a topic entirely absent from the corpus, the retriever returned its nearest neighbours and the model dutifully answered from them.

Beginner mistakes

1. Fixed-size chunking.
2. Overlap ≥ chunk size (infinite loop).
3. No doc_version on the chunk.
4. Unsigned feature hashing.
5. Not realising a new embedding model invalidates the index.
6. Unclamped IDF.
7. Weighted score fusion with a hand-tuned constant.
8. Reranking the whole index.
9. No relevance floor.
10. One index, tenant as a filter.
11. Filtering after ranking.
12. Treating an information barrier as a document.
13. No freshness contract.
14. Volatile content at the top of the prompt.
15. Budgeting the prompt's pieces rather than the assembled text.
16. Dropping retrieved chunks silently.

What "good" sounds like

"Tenant isolation is the namespace — a cross-tenant chunk is never a candidate, not filtered out — and classification and barrier checks are a pre-filter inside it, before ranking. Two mechanisms because the blast radius differs: a tenant bug crosses a customer boundary, a classification bug doesn't leave the tenant. Quality is structure-aware chunking with provenance and a document version, hybrid BM25 plus dense because identifiers and intent fail on opposite retrievers, RRF to merge so I never re-calibrate when the embedding model changes, and a cross-encoder over the top-50 with a relevance floor so 'nothing relevant' is expressible. Then grounding — every claim cites a span or the answer fails — and a freshness contract, because a correctly-cited superseded policy is the worst kind of wrong. And I'd measure recall@k on a golden set before touching the generator, because recall caps everything downstream."