« Phase 31 README · Warmup · Hitchhiker's · Deep Dive · Core Contributor · Staff Notes

Phase 31 — Principal Deep Dive: Cohere

The mechanism deep-dive answers why two stages exist. This one answers the questions a principal actually gets asked in a design review: how much does each stage cost in latency and dollars, where does it physically sit, how does it fail, and what is the blast radius when it does. The Cohere pipeline — Embed → first-stage ANN → Rerank → Command-with-documents → citation enforcement — is a good system to reason about precisely because every stage has a different cost shape and a different failure mode.

The latency and cost envelope, stage by stage

First-stage retrieval is the cheap stage by construction. Document embeddings are computed once offline and live in an ANN index; a query costs one embed call plus a sublinear index lookup. On the cost side, embedding is priced per token and the corpus is embedded once (plus deltas), so the recurring cost is the query embed — negligible. The index itself is the cost you underestimate: float embeddings at 1024 dims are 4KB per vector, so ten million documents is 40GB of vectors before ANN overhead, and that has to live in RAM to be fast. This is where int8 and binary stop being a curiosity and become a capacity decision (below).

Rerank is the stage that changes your latency budget. It adds a network hop and, more importantly, O(candidates) cross-encoder inferences that cannot be cached — the score depends on the query, so there is no reuse across queries. Sending 100 candidates is roughly 100× the compute of sending 10 (cross-encoder passes dominate; batching amortizes overhead but not the passes themselves). The design lever is the candidate count: too few and you have re-imposed the first-stage recall ceiling you added Rerank to escape (if the right doc was at first-stage rank 60 and you only rerank the top 40, Rerank never sees it); too many and latency and cost climb linearly for diminishing precision gains. The defensible default is "rerank the smallest candidate set whose recall@candidates is measured to contain the answer" — typically 50–100 in, top_n 3–10 out — and the word measured is the whole point. Batch the candidate documents in one Rerank call, not one call per document; the API takes a list precisely so the server can batch the forward passes.

Generation is priced per input and output token, and with grounded RAG the input is dominated by the documents you pass. This is the second reason top_n matters: every reranked document you forward is context tokens Command pays for on every call. Reranking to top_n=5 instead of dumping 50 candidates into the prompt is simultaneously a quality decision (precision at the top) and a cost decision (5× fewer document tokens per generation). The two goals align here, which is rare and worth stating explicitly in a review.

Napkin math for a design review: first stage ~single-digit ms plus index lookup; Rerank adds one RTT plus tens of ms for a ~100-candidate batch; generation is the long pole at hundreds of ms to seconds depending on output length. Rerank is a small, controllable addition to a budget the LLM already dominates — which is exactly why "the reranker is too slow" is almost always a candidate-count misconfiguration, not an architectural problem.

Where each stage physically sits

The ordering is not negotiable and the reasons are architectural. Rerank goes after the ANN and before the LLM: after ANN because a cross-encoder over the whole corpus is unaffordable (it must operate on a shortlist), and before the LLM because its entire purpose is to decide which documents earn the expensive, token-metered context slot. Putting Rerank before first-stage retrieval is a category error — there is no shortlist yet. Putting it after generation is pointless — the documents are already spent. The pipeline is a funnel: wide-and-cheap narrows to narrow-and-precise narrows to grounded-and-cited, and each stage's output is the next stage's input at a smaller cardinality. Lab 03's TwoStageRag.answer (retrieve first_k → rerank to rerank_n → grounded-generate) is this funnel, and it deliberately carries both stages' document ids in the response — because in production the only way to debug "why did it cite the wrong doc" is to know what each stage saw and kept. Observability is designed into the data shape, not bolted on.

Embedding format is a capacity decision, not a default

embedding_types looks like a knob you can ignore; at scale it is the difference between an index that fits in RAM and one that does not. float (32-bit) is the fidelity baseline and the storage worst case. int8 is ~4× smaller with cosine similarity almost identical — for large indexes this is close to a free lunch and should be the default you reach for, not the exotic option. binary is ~32× smaller but similarity degrades to a Hamming-style bit overlap that ranks easy cases correctly and costs recall on hard ones — a real dial with a real price you measure on your own eval set before committing an index format you cannot cheaply change. Matryoshka (truncatable dimensions) adds a second axis: store a short prefix for cheap shortlisting and the full vector for precise rescoring, but only because the model was trained to front-load information into the prefix. Lab 03 makes the honesty of these tradeoffs concrete — binary keeps the easy top-1; aggressive Matryoshka truncation (16 of 64 dims) loses it, because a hashing embedder is not trained to front-load the way a real Matryoshka model is. The principal-level statement is: "we A/B'd int8 vs binary and binary dropped recall@10 by N points on our set, so we shipped int8" — a number, not a preference.

Grounded generation as a hallucination control, and its blast radius

Grounded generation with citations is best understood as a control, and controls are defined by what they catch and what they do not. What it catches: an answer whose claims are not traceable to a retrieved document — the citation contract makes "unsupported" a checkable property, and the enforcement layer (drop uncited claims, log the citation, compute grounding_score) turns that property into an action. What it does not catch, and the blast radius if you assume it does: citations attribute, they do not verify truth. A retrieved document can itself be wrong; a poisoned or prompt-injected document can carry an instruction the retrieval agent tries to follow; and — the sharpest edge — a model trained to cite can still miscite, naming a real document that does not actually support the span. That last one is why the citation enforcement is the deployment's job and not the model's promise: Cohere returns citations, it does not refuse to emit an uncited sentence, so the last mile of the trust guarantee is code you write (Lab 02's verify_citations). The blast radius of skipping it is a fluent, confidently-cited-looking paragraph with one ungrounded claim wedged in, shipped to a regulated customer — which is the exact failure the whole pipeline exists to prevent.

Cross-cutting concerns

Multi-tenancy and access governance. In a real North deployment, retrieval must respect the customer's existing permissions: a user must not be able to ground an answer in a document they are not allowed to read. That means the access filter belongs at the first stage — you filter the candidate set by the caller's entitlements before reranking and generation, not after, or you leak the existence and content of restricted documents through citations. This is the authorization/isolation discipline from Phase 13 applied to RAG, and it is a place where "looks wrong but is intentional" bites: the filter must run early even though it makes the shortlist per-user and less cacheable, because the alternative is a compliance breach.

Cost observability. The two metered stages are Rerank (per candidate) and generation (per document token in, per token out). Both scale with top_n/candidate count, so that single number is your primary cost and latency dial and your primary quality dial. Instrument it: log candidate count, top_n, per-stage latency, and the per-stage document ids on every request. When someone asks "why did last month's bill jump," the answer is almost always "someone raised the candidate count," and you can only prove it if you logged it.

The deployment boundary as an architectural constraint. Cohere's SaaS / VPC / on-prem / air-gapped spectrum is not a sales page item; it is a constraint that changes the architecture. In air-gapped mode there is no api.cohere.com hop — models run inside the boundary, which changes your latency (no internet RTT) and your capacity planning (you own the GPUs). The principal skill is placing the retrieval and generation stages relative to that boundary correctly: filter and retrieve inside the boundary, and never design a stage that requires an outbound call the customer's compliance forbids. The pipeline shape is identical across deployment modes; where it runs is the variable, and picking the mode that clears the customer's data-residency requirement is the design decision that gates the entire deal.