« Phase 31 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Staff Notes

Phase 31 — Core Contributor Notes: Cohere

This is the view from inside the real API surface — the design decisions a committer on the Cohere SDK or someone who has read a lot of production Cohere integration code would flag, and the places our stdlib miniature deliberately simplifies. The goal is to know the actual shapes cold, and to know why they are shaped that way, because the "why" is where the non-obvious decisions live. Where an exact field name or value could have drifted between versions, I describe the pattern rather than assert a specific string.

Three endpoints, three cost models, one SDK

Cohere exposes the phase's three mechanisms as three separate endpoints — /embed, /rerank, /chat — and the separation is deliberate, not incidental. They have different cost models (embed is per-token and cacheable offline; rerank is per-candidate and not cacheable; chat is per-token in and out), different scaling shapes, and different adoption paths. Selling Rerank as a standalone API — not bundled into a "search product" — is the decision that made it Cohere's most-adopted surface: it drops into an existing Elasticsearch, pgvector, or even a competitor's embedding stack with one call, because it takes raw strings, not Cohere embeddings. That is an API-design choice with a business consequence, and it is the kind of thing a maintainer notices: the endpoint's input contract (a query and a list of document strings) was kept deliberately stack-agnostic so it could be adopted without buying the rest of the platform.

The Python SDK went through a visible v1 → v2 evolution, and the seams still show. The v2 client (ClientV2, constructed reading an API key from the environment) unified on a messages list of role-tagged turns — system, user, assistant, tool — matching the industry-standard chat shape. v1 exposed a different, Cohere-specific shape: a preamble for the system instruction and a chat_history plus a separate message string for the current turn. If you read older integration code and see preamble and chat_history, that is v1; the migration to messages was Cohere aligning to the shape the rest of the ecosystem converged on, and it is the single biggest source of "the docs don't match this code" confusion. Know both; write new code against v2.

input_type on Embed: why a required argument exists

The single most instructive API decision in the phase is that Embed requires input_type on v3+ — search_document, search_query, classification, or clustering. Most APIs would have made this optional with a default. Cohere made it required, and the reason is a hard-won lesson encoded into the type system: the failure it prevents is silent. If input_type defaulted to, say, search_document, every engineer who embedded queries without thinking would get a working call and quietly degraded retrieval — no error, just worse recall for months (the exact war story in the Warmup and Hitchhiker's Guide). By making the argument required, the API forces the developer to state which side of the asymmetry they are on, converting a silent quality bug into a decision you cannot skip. That is a maintainer choosing friction now over silent failure later, and it is worth being able to explain in an interview: the required argument is not bureaucracy, it is a guardrail against the most common way people misuse asymmetric embeddings.

The asymmetry itself is two trained projections (query-side and document-side) sharing one output space, so cross-type cosine is meaningful. Our Lab 03 miniature reproduces the contract — document vectors are hashed token counts, query vectors are hashed content-term presence, sharing one hash space — but not the mechanism: there is no learned projection, just a hand-designed asymmetry that makes the counts-vs-presence distinction stand in for what training would learn. That is the honest simplification: same interface, same failure-if-you-swap-them behavior, different internals.

Embed response shape and compression types

Embed's response nests the vectors under the requested embedding_types: you ask for ["float"] or ["int8"] or ["binary"] (and can request several at once), and the response carries each under its own key, so you read .embeddings.float or .embeddings.int8 rather than a single flat array. This shape exists because a single call can return multiple representations of the same text — the Matryoshka/multi-precision pattern where you might store a compact form and a precise form together. The compression types map to real storage math: int8/uint8 is ~4× smaller with cosine barely moving; binary/ubinary is ~32× smaller with similarity degrading to a bit-overlap comparison. Embed v4 added multimodal inputs and Matryoshka (selectable output dimensions via prefix-truncation), whereas the v3 models (embed-english-v3.0, embed-multilingual-v3.0) are fixed-dimension. The maintainer-level point: the response is shaped around the assumption that format is a first-class decision, which is why the types are explicit keys and not an afterthought.

Rerank v2/v3: top_n, return_documents, and index preservation

Rerank's contract has three sharp edges a committer knows. First, top_n controls how many results come back, and the results carry each document's original index into the list you passed — because the whole point is to reorder your candidates and let you rejoin your metadata; the API returns indices, not (by default) the document text, to keep the response small. Second, return_documents is the flag that decides whether the response echoes the document content back or just the index and score — off by default because you already have the documents; you sent them. Third, long-document handling: historically a max_chunks_per_doc-style mechanism chunked overlong documents and scored each by its best chunk, because a cross-encoder has a finite input window and a document longer than that window has to be split. Lab 01 mirrors all three — the index-preserving top_n contract, descending calibrated scores, and best-chunk long-document handling — which is why the lab's output object carries the original index alongside the relevance score. The non-obvious decision here is return_documents=False as the default: it optimizes for the common case (you have the docs, you want the ranking) and it is a small thing that signals whoever designed it was thinking about payload size at scale.

Chat with documents: the citation contract

The documents parameter is what turns /chat from a chat API into a RAG API. You pass a list of document objects — each an id plus content (in v2 the content sits under a data/text-style structure) plus arbitrary metadata — and the model generates grounded in them and returns a citations array on the response message. Each citation carries character offsets into the generated answer (start/end), the exact text substring, and the source document(s) — v2 nests these under a sources list of typed source objects ({type: "document", document: {id}}), where v1 exposed a flatter document_ids. If you see document_ids in older code and sources in newer, that is the same v1 → v2 evolution: the flat id list became a structured source list to accommodate multiple source types (documents, tool results) grounding a single span.

Two things a maintainer flags. First, the offsets index the answer, not the document — this trips up nearly everyone the first time, and it is the design that makes streaming citations possible: as the answer streams, the client can attach a source marker to a span the instant that span is emitted, because the offset is into the text being produced. Second, and this is the one to say out loud in an interview: Cohere returns citations but does not refuse to emit an uncited sentence. The API's contract is "here is what I generated and here is what grounds the parts that are grounded" — it is not "everything I generated is grounded." Enforcement (drop or flag uncited claims, verify each citation, compute a faithfulness score) is left to the caller by design, because the right policy is application-specific — a consumer toy tolerates uncited text, a bank does not. Lab 02 is exactly this missing last mile: verify_citations re-checks that answer[start:end] matches, that named documents exist, and that they actually support the span, catching the misattribution the model can still produce.

The Command family and what it means to be "trained for this"

Command is the generation line — Command R (the RAG/citation/tool workhorse, 128k context), Command R+ (stronger reasoning and multi-step tool use), Command R7B (small/edge), and the Command A flagship direction (long context — 256k-class — tuned for agentic/enterprise workloads). The committer-relevant fact is what "trained for RAG" changes at the API level: these models were fine-tuned to consume a documents list and emit grounded spans and structured tool_calls as part of decoding, so the citation offsets and tool-call JSON come out of the model aligned to the evidence, not reconstructed by a fragile post-hoc parser. That is why documents is a distinct parameter and not just "concatenate these into the prompt yourself": passing documents through the trained seam gets you the citation machinery; pasting them into the prompt string gets you influence without attribution.

What our miniature deliberately does not do

The honest ledger: our labs replace every learned component with a deterministic, offline stand-in — cosine over hashed vectors for Embed, a coverage+phrase+proximity score for Rerank's cross-encoder, and term-overlap entailment for grounded generation — so the mechanism is inspectable and the tests are deterministic. What we keep faithfully is every contract and invariant: the input_type asymmetry and its silent-degradation failure, the index-preserving top_n rerank result, the {start, end, text, document_ids} citation shape with answer-relative offsets, and the caller-owned enforcement layer. What we drop is the learning — real relevance and real entailment are trained, not hand-coded. Knowing exactly which half is real and which is stand-in is the difference between someone who ran the lab and someone who understands what the lab is a model of.