« Phase 31 README · Warmup · Hitchhiker's · Principal Deep Dive · Core Contributor · Staff Notes

Phase 31 — Deep Dive: Cohere

The load-bearing idea in this phase is a single asymmetry in the transformer: when do the query and the document get to attend to each other, and when do they not. Everything else — why Rerank is accurate but expensive, why Embed is cheap but coarse, why the two-stage pipeline is not a hack but the only shape that satisfies both constraints — falls out of that one distinction. Get it at the mechanism level and the rest of the phase is bookkeeping.

Bi-encoder: independence is the whole trick, and the whole limitation

A bi-encoder (Cohere's Embed) runs the query through the network and each document through the network separately, producing one vector each, then compares those vectors with a cheap similarity — cosine. Formally, for query q and document d, the score is cos(f(q), f(d)) where f is the same encoder applied independently. The two never share a forward pass.

That independence is exactly what makes it scale. Document vectors f(d) depend only on d, so you compute them once, offline, and store them in an index. At query time you pay one forward pass for f(q) plus a nearest-neighbor lookup — O(1) model inferences regardless of corpus size (the ANN structure handles the N comparisons in sublinear time). A million-document corpus costs the same per query as a thousand-document one, up to the ANN's log factor.

The same independence is the ceiling. Because f(q) is computed before f(d) is ever seen, the score cos(f(q), f(d)) is a comparison of two summaries that were each written without knowledge of the other. Word order inside the document, the proximity of two query terms, whether "reset the VPN password" appears as a phrase or as three words scattered across unrelated sentences — all of that has to be compressed into a fixed vector before the query is known. A bag-of-words-flavored representation cannot un-blur that after the fact. This is why the Lab 03 miniature encodes documents as hashed token counts and queries as hashed content-term presence: term frequency is document signal, a query term "votes once," and the two land in a shared hash space so cross-type cosine is meaningful. It is a deliberately coarse f, and it reproduces the coarse failure faithfully — a term-stuffed page that repeats every query word scores high because repetition inflates counts, exactly the real bi-encoder pathology.

input_type is the same mechanism made visible. f is not one function but a family: the search_document encoding and the search_query encoding are different projections trained so a query lands near the documents that answer it, not the documents that merely resemble it. Using search_document for both collapses the family to one projection — nothing errors, because the vectors are still valid points in the space; the geometry just silently loses the asymmetry the model was trained to exploit. In the miniature, that is the counts-vs-presence split; in the real model it is two learned heads.

Cross-encoder: joint attention is the accuracy, and the cost

A cross-encoder (Cohere's Rerank) does the opposite. It concatenates the query and one document into a single input, [q ; d], and runs that pair through the transformer together, so self-attention spans across the pair — every query token can attend to every document token and vice versa in every layer. The output is a single scalar g(q, d): a calibrated relevance score, not a vector. There is no shared space, no cosine; the model reads the query and the document at the same time and answers one question — how relevant is this document to this query.

That joint pass is why it sees what the bi-encoder structurally cannot. "Order, phrasing, proximity" are not lost, because the document representation is now conditioned on the query: the attention pattern over the document is built in the presence of the query tokens. The distinction between "the page that answers the question in one sentence" and "the page that scatters the same five words across three unrelated sentences" is visible in the attention, so it is visible in the score.

The cost is dual to the bi-encoder's benefit. Because g(q, d) depends on both arguments, you cannot precompute anything — there is no g(d) to cache, because the score does not factor. Every (query, document) pair is a fresh forward pass. Scoring M candidates is O(M) transformer inferences per query, each far heavier than a cosine. Run that over a million-document corpus and you have a million forward passes per query; it is a non-starter. So you never do. You run the cross-encoder over the shortlist the bi-encoder already produced. Lab 01's relevance score (coverage + phrase + proximity) is the miniature: coverage is the part a bi-encoder could roughly approximate, and the phrase and proximity terms are precisely the interaction signal a bi-encoder's independent vectors cannot represent.

Why two stages is forced, not chosen

Put the two complexities side by side and the architecture is not a design preference; it is the only assignment that respects both constraints:

Stage 1 (bi-encoder, whole corpus): O(1) inferences per query, coarse. Its job is recall — do not lose the right document. If the answer chunk is not in the top ~50–100, no later stage can recover it. The metric is recall@k; the failure mode is a relevant doc that never makes the shortlist.
Stage 2 (cross-encoder, shortlist only): O(k) inferences per query with small k, precise. Its job is precision at the top — reorder the shortlist so the genuinely relevant rise. It fixes the two failures stage 1 cannot see: the relevant-but-not-top document and the term-stuffed decoy on top. The metric is rerank lift (recall/MRR of retrieve-only vs retrieve→rerank).

Run cross-encoder alone: correct, unaffordable. Run bi-encoder alone: affordable, and it will rank the term-stuffed policy page above the procedure that actually answers the question — the exact bug in Lab 01's planted corpus, where a document at first-stage rank #4 is promoted to reranked #1. Two stages is the composition that is both affordable and precise, and each stage's top_n / candidate-count is the knob trading latency for recall at the top.

Citation spans: alignment, not generation

Grounded generation adds a third mechanism that is easy to mistake for the first two but is distinct. A citation is {start, end, text, document_ids}: start/end are character offsets into the generated answer, text is exactly answer[start:end], and document_ids names the source(s) grounding that span. The offsets index the answer, not the document — this is the part people get backwards.

The real Command models emit these spans out of the decoder: they were fine-tuned so the citation markers are aligned to the evidence as part of generation, using learned entailment. Lab 02 inverts that into an inspectable pipeline that preserves the identical contract and invariants: an injected generator proposes claims (some grounded, one hallucinated); a support check computes, per claim, the fraction of its content terms attested by each document and locates the minimal source span attesting them; supported claims are assembled into the answer while tracking exact offsets; unsupported claims are dropped with an audit trail; and grounding_score = supported / generated quantifies faithfulness. Term overlap stands in for learned entailment, but the claim → span → source chain is byte-for-byte the same.

The critical property is that this chain is independently verifiable, which is what Lab 02's verify_citations auditor checks. It re-derives, from scratch, three things a lying citation gets wrong: does answer[start:end] actually equal the claimed text (tampered offsets); does each named document actually exist (phantom source); and does the named real document actually support the span (misattribution — the subtle one a "does it have citations?" check misses entirely). Verification is possible only because the citation carries both answer offsets and document ids; a mapping with only one half would be unauditable.

The worked trace

Query: "how do I reset the vpn password". Corpus has doc A (a policy index page repeating "vpn," "password," "reset" across headings) and doc B (the actual procedure, one paragraph).

Embed, offline. f_doc(A), f_doc(B) computed once, indexed. A's repetition inflates its term counts.
Embed query. f_query(q) computed with the query projection.
Stage 1. cos(f_query(q), f_doc(A)) > cos(f_query(q), f_doc(B)) — A ranks #1, B ranks #4. The bi-encoder cannot see that B's words form the answering phrase; it only sees that A shares more term mass. Both survive into the top ~50 shortlist. Recall is intact; precision is wrong.
Stage 2. g(q, A) and g(q, B) are computed by joint passes. The cross-encoder sees that B's tokens form the procedure in the presence of the query tokens and A's do not. g(q, B) > g(q, A). B is promoted to #1. top_n=3 keeps B, A, and one more.
Generate. Command receives B (and the shortlist) as documents, produces the answer, and emits a citation {start, end, text, [B]} for the grounded span. The scripted hallucination returns uncited.
Enforce. The system verifies the citation (offsets honest, B supports the span), keeps the grounded claim, drops the uncited one, logs the citation, and reports grounding_score.

Naive single-stage vector RAG stops at step 3, hands A to the model, and — because A is the policy index, not the procedure — the model either answers wrong or hallucinates a plausible procedure with no source to show for it. The mechanism failure is precisely locatable: independence in stage 1 lost the interaction signal, and without a cross-encoder to restore it and a citation contract to check the output, the pipeline had no place to catch the error. That is why the shape is two-stage-retrieve-then-ground, and why each stage exists.