« Phase 06 · Warmup · Track Overview

Deep Dive — Mechanism & Internals

Table of Contents

• 1. The chunker's two-level split
• 2. Character offsets, and the repeated-word trap
• 3. Signed feature hashing, in detail
• 4. BM25's three guards
• 5. RRF as a rank-only reduction
• 6. The retrieve pipeline's ordering
• 7. The assembler's fixed-point loop
• 8. A traced retrieval
• 9. Invariants, complexity, determinism

1. The chunker's two-level split

sections = _split_sections(document.text)          # structure
for name, body, offset in sections:
    for piece, start, end in _split_by_size(body, max_tokens, overlap_tokens):
        yield Chunk(..., section=name, start_char=offset + start, ...)

Two levels, in this order, and the order is the design. Reversing it — size first, then trying to attribute sections — cannot work, because a size-first split has already destroyed the boundary information you would need.

_split_sections returns triples of (name, text, char_offset) rather than pairs. The offset is what lets the inner splitter work in section-local coordinates while the chunk records document-global ones. Without it every citation span would be relative to a section and useless for pointing a human at a document.

The no-headings case returns [("body", stripped, 0)] rather than []. Returning empty would make an unstructured document unretrievable, which is a silent data-loss bug — the document is ingested, reports success, and never appears in a result.

Text before the first heading gets its own "body" section. Dropping it is the other silent-loss variant, and it is common because preambles look like boilerplate right up until the one that contains the definition you need.

2. Character offsets, and the repeated-word trap

_char_offset looks over-engineered:

offset = 0
for i in range(word_index):
    offset = text.index(words[i], offset) + len(words[i])
return text.index(words[word_index], offset)

The naive version is text.index(words[word_index]). It is wrong whenever a word repeats — which in a policy document is every word. Searching for the 40th word "payment" from position 0 finds the first "payment", and the citation span points at the wrong clause.

Scanning forward with a running cursor makes each lookup resolve to the correct occurrence. It is \( O(n) \) in the word index, so building all offsets for a section is \( O(n^2) \) in the worst case — acceptable at chunk scale (tens to hundreds of words), and the correct trade against a citation that quietly points somewhere else.

A production implementation tokenizes once with spans, avoiding the re-scan entirely. The lab keeps the naive-but-correct version because the bug it avoids is the teaching point.

3. Signed feature hashing, in detail

digest = blake2b(token, digest_size=8)
index  = int.from_bytes(digest[:4], "big") % dimensions
sign   = +1 if digest[4] % 2 == 0 else -1
vector[index] += sign

One digest, two independent uses: bytes 0–3 pick the dimension, byte 4 picks the sign. They must be independent, or the sign correlates with the bucket and the cancellation property fails.

Why cancellation matters, precisely. Let \( h \) map tokens to dimensions and \( s \) to \( \pm 1 \). The hashed dot product between documents \( x \) and \( y \) is

$$\langle \phi(x), \phi(y)\rangle = \sum_{i,j} x_i y_j , s(i)s(j),[h(i)=h(j)]$$

For \( i = j \) the term is \( x_i y_i \) — the true contribution. For \( i \neq j \) with a collision, \( s(i)s(j) \) is \( +1 \) or \( -1 \) with equal probability, so the expectation of the cross terms is zero. The hashed dot product is an unbiased estimator of the true one.

Drop the signs and every cross term is \( +x_i y_j \), strictly positive: similarity is systematically inflated, and it inflates more for longer documents (more tokens, more collisions). The observable symptom is that unrelated documents never score at or below zero — which is exactly what the lab's test_signed_hashing_allows_negative_similarity checks across 40 pairs.

blake2b rather than hash(): Python salts string hashing per process, so an index built in one process would not be searchable from another. Same reasoning as the affinity ring in Phase 01.

The all-zero case (empty text) returns the zero vector rather than dividing by its zero norm. Cosine against it is 0 — a miss, which is correct.

4. BM25's three guards

def _idf(self, ns, term):
    n = len(self._by_namespace.get(ns, ()))
    if n == 0: return 0.0                                        # (1)
    df = self._df[ns].get(term, 0)
    return max(0.0, math.log((n - df + 0.5) / (df + 0.5) + 1.0)) # (2) (3)

1. Empty namespace returns 0, not a division by zero. Reachable whenever a tenant has no documents yet, which is every tenant on day one.
2. The +1 inside the log shifts the argument above 1 for all \( df \le N \), so the logarithm is non-negative in the common case. Without it, \( df > N/2 \) gives a ratio below 1 and a negative log.
3. max(0, …) is belt and braces for the edge where smoothing still produces a negative.

Guards 2 and 3 exist for the same failure and it is worth being explicit about it: with a negative IDF, a document containing the user's search term scores lower than one that does not. In a bank corpus where "payment" or "account" appears in most documents, that inverts ranking for exactly the terms users type. It is a real bug, it ships, and it is invisible because the results are still plausible.

The scoring loop skips terms with f == 0 before computing anything:

frequency = tf.get(term, 0)
if frequency == 0: continue

Not an optimization — a correctness guard. With f = 0 the numerator is 0 and the term contributes nothing anyway, but computing IDF for a term absent from the document is wasted work proportional to query length × corpus size.

Zero-scoring chunks are dropped before sorting, so a query with no matching terms returns [] rather than the whole corpus at score 0. That is what makes "BM25 found nothing" a distinguishable state.

5. RRF as a rank-only reduction

for ranking in rankings:
    for rank, scored in enumerate(ranking, start=1):
        fused[key] = fused.get(key, 0.0) + 1.0 / (k + rank)

The input Scored.score is never read. That is the whole mechanism: the reduction consumes positions and discards magnitudes, which is what makes it immune to the calibration problem.

enumerate(..., start=1) matters. Zero-based ranks would make the top result contribute \( 1/k \) and the second \( 1/(k+1) \) — a smaller gap, and inconsistent with every published formulation, so a comparison against a reference implementation would silently differ.

The identity of a chunk across rankings is chunk_id. Two rankings referring to the same chunk must produce the same key or fusion degenerates into concatenation — which is why chunk ids are deterministic ({doc_id}::{n}) rather than generated.

The agreement property, arithmetically. For \( k = 60 \):

The gap between the 2nd and 3rd rows is the behaviour you are buying. It is small — about 3% — and it is systematic, which is what matters over a result set.

k controls how flat the curve is. As \( k \to \infty \) every rank contributes \( 1/k \) and fusion becomes a vote count. As \( k \to 0 \) the top rank dominates and fusion approaches "whichever retriever ranked it first." 60 sits far enough along that ranks beyond ~20 barely differ, which matches how far down a candidate list anyone actually looks.

6. The retrieve pipeline's ordering

search both indexes (namespace-scoped)
  → count `considered`
    → PRE-FILTER (visibility, freshness)
      → RRF
        → rerank
          → limit

Every arrow is a decision.

Search is namespace-scoped, so considered counts only in-namespace candidates. That is deliberate: the metric answers "how many candidates did we evaluate", and a cross-tenant chunk was never evaluated. Counting it would imply the tenant boundary was a filter.

The pre-filter runs on each ranking separately, before fusion. Three reasons, and only the first is obvious:

1. an ineligible chunk cannot occupy a slot in the fused result;
2. it cannot influence other chunks' fused scores by shifting their ranks;
3. the exclusion counters attribute correctly — excluded_by_entitlement counts what the entitlement check removed, not what survived a later stage.

Filtering after fusion passes the "no cross-tenant results" test and still leaks: result-set size and ordering both vary with what was filtered, which is an oracle for the existence of content.

considered is computed before the filter, the exclusions after. So considered, excluded_by_entitlement and excluded_by_freshness together describe the funnel, and a rising exclusion rate is an operational signal — a freshness-exclusion spike means an ingestion pipeline has stalled.

Empty rankings are dropped before fusion (rankings = [r for r in (lexical, dense) if r]). Passing an empty ranking to RRF is harmless but makes "one retriever found nothing" invisible; the filter keeps that state observable.

Rerank is last and optional. Disabling it appends "rerank" to degraded and returns the fused order — an answer of lower quality, not an error. That is precisely what makes retrieval a degradable dependency in the Phase 00 sense, and the lab's structure is what lets you prove it.

7. The assembler's fixed-point loop

included = list(retrieved)
while True:
    text, stable = render(included)
    if estimate_tokens(text) <= budget_tokens: break
    if not included: raise ValueError(...)
    dropped.insert(0, included.pop().chunk_id)

The obvious implementation sums the segments' token counts and subtracts. It is wrong, and the lab found out by testing it: the rendered form adds [segment] headers and "\n\n" separators, so a budget computed from the pieces over-fills by exactly the amount nobody accounts for. The first version over-ran a 120-token budget by 6.

Rendering and re-measuring is \( O(n^2) \) in the number of chunks — n renders, each \( O(\text{text}) \). At 5–20 chunks that is microseconds, and it is exact, which the incremental version is not. Choosing correctness over an irrelevant asymptotic is the right call here, and knowing that you chose it is the point.

Two details:

• included.pop() drops the last element, which is the lowest-ranked because the retrieved sequence arrives in rank order. dropped.insert(0, …) then rebuilds the dropped list in descending-rank order, so it reads as "we dropped these, best-first".
• The raise happens only when included is empty, i.e. the fixed segments alone exceed the budget. That is the case where truncating the question would be the only remaining option, and the error message says so explicitly.

stable_prefix_tokens counts the rendered instruction, tool-schema and policy blocks — headers included — because that is what the provider actually sees and caches. Counting the raw values would under-report the cacheable fraction.

8. A traced retrieval

Query: "PMT-771 hold reason and release rules". Principal: tenant wholesale, cleared to confidential, no barriers. Corpus: 4 documents → 7 chunks across 2 namespaces.

The exclusion count of 2 for one chunk is worth understanding: it is excluded from the lexical ranking and from the dense ranking, and each removal is counted. That is intentional — the counter measures filter actions, not distinct chunks — and it is documented rather than "corrected", because a per-ranking count is what tells you which retriever was surfacing ineligible content.

Then grounding over those 3 chunks with 3 claims: two match a chunk above the 0.6 overlap threshold and receive its citation; the invented one ("the customer has been notified by email") matches nothing and is reported unsupported. Coverage 67%, is_grounded False.

Then assembly into a 260-token budget: 3 chunks fit, stable_prefix_tokens = 47, cacheable fraction 21%. Raising the budget does not raise the cacheable fraction — it lowers it, because the extra tokens are all volatile retrieved content. That is a real and slightly counter-intuitive property: more retrieval reduces your prefix-cache discount, which is a cost consideration that belongs in the retrieval-depth decision.

9. Invariants, complexity, determinism

Invariants (each tested):

1. A chunk from another tenant is never in a result — and considered never counts it.
2. visible() is a conjunction: failing any of tenant / classification / barrier hides the chunk.
3. excluded_by_entitlement > 0 whenever a clearance is insufficient — the filter is observable.
4. BM25 IDF is never negative.
5. BM25 saturates: the 9→10 increment is smaller than the 1→2 increment.
6. RRF ranks a consistently-3rd chunk above a 1st-and-10th one.
7. RRF is order-independent across rankings.
8. Feature hashing produces at least one non-positive similarity across 40 unrelated pairs.
9. A query matching nothing returns an empty result, not the least-irrelevant chunk.
10. The assembled context never exceeds its budget; an impossible budget raises.
11. included ∩ dropped = ∅ and their union is the input.
12. Identical inputs produce identical outputs, everywhere.

Complexity:

The two that do not survive scale are the searches: both are linear in namespace size. That is correct for a lab and it is precisely where an ANN index goes in production — and where the filtering cliff (WARMUP §7.2) appears, which the exact version cannot exhibit. Knowing that the lab cannot show you the cliff is part of reading it honestly.

Determinism. No clock (freshness uses an injected now_tick), no RNG, no uuid4, no hash(). Every sort has an explicit tie-break on chunk_id. Chunk ids are derived from doc_id and a counter. VectorIndex.namespaces() returns sorted output. The result is that two runs — or two machines — produce byte-identical retrieval, which is what makes the grounding and assembly tests equality assertions rather than approximations.