« Phase 31 README · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 31 Warmup — Cohere Platform: Command, Embed, Rerank & Grounded Generation (North)

Who this is for: someone who has built agent infrastructure from scratch (Phases 00–17), studied the runtime frameworks (18–24), and now needs to speak fluently about Cohere — not as "another LLM API," but as the enterprise-RAG company whose whole product line is shaped around retrieval-grounded, citable, privately deployable AI. By the end you will be able to explain, from first principles, how Cohere's Command models generate grounded answers with inline citations, how Embed's input_type asymmetry and compressed embeddings work, why Rerank is a cross-encoder and where it sits in a pipeline, how the two-stage retrieve→rerank→generate RAG stack composes end to end, what North actually is, and how Cohere stacks up against OpenAI, Anthropic, and open-weight models for a regulated enterprise — with a real decision framework, not a feature list. No Cohere account, no SDK, no network — the labs are all mechanism.

What Cohere actually is: the enterprise-RAG thesis
Security-first positioning and private deployment
The Command model family
The Chat API: messages, documents, tools, streaming
Grounded generation with inline citations, in depth
Embed: input_type, multilingual, and compressed embeddings
Rerank: cross-encoder vs bi-encoder
The two-stage RAG pipeline end to end
North: the secure enterprise AI workspace
Agentic workflows on Cohere: tool use and multi-step
Evaluating grounded RAG: faithfulness, citation accuracy, relevance
Enterprise security, compliance, and auditability
Cohere vs OpenAI, Anthropic, and open models for enterprise RAG
Common misconceptions
Lab walkthrough
Success criteria
Interview Q&A
References

1. What Cohere actually is: the enterprise-RAG thesis

Most LLM companies sell intelligence and let you figure out deployment. Cohere sells attributable, deployable intelligence and lets the frontier-IQ race happen without it. That is not a hedge — it is a deliberate market position, and every product decision follows from it.

Cohere is a foundation-model company (founded 2019 by ex-Google-Brain researchers, including a co-author of the original Transformer paper) whose product line is unusually coherent around retrieval-augmented generation for the enterprise:

Command — text-generation models tuned for RAG, tool use, and grounded answers.
Embed — embedding models for the retrieval half of RAG, with query/document asymmetry and aggressive compression options.
Rerank — a cross-encoder that re-scores retrieval results, sold as a standalone API that drops into anyone's search stack.
North — a secure AI workspace that packages models + retrieval + agents + connectors to run inside a customer's own boundary.

The through-line: an enterprise's blocker is rarely "the model isn't smart enough." It is "I can't put my data in someone else's cloud," "I can't ship an answer I can't trace to a source," and "my search returns the policy index instead of the procedure." Cohere's four products map one-to-one onto those three blockers. When you interview for a forward-deployed role, the thing being tested is whether you understand that — that Cohere's value proposition is the enterprise deployment envelope around competent models, not a leaderboard score.

A forward deployed engineer (FDE) is the person who operationalizes that thesis inside a specific customer: translate an ambiguous business ask into a grounded, evaluable, privately-deployed workflow built from Command + Embed + Rerank, hosted in North or the customer's VPC. This warmup is the mechanism knowledge that role assumes.

2. Security-first positioning and private deployment

The single most important non-model fact about Cohere: it is built to run where the customer's data already lives, and it does not train on customer data by default. Concretely, Cohere supports a spectrum of deployment modes:

SaaS API — the hosted api.cohere.com, the fastest path, data processed in Cohere's environment under its enterprise terms.
Cloud-private / VPC — Cohere models available through the major clouds' managed catalogs (Amazon Bedrock and SageMaker, Azure AI Foundry, Google Vertex AI, Oracle OCI), so inference runs inside your cloud account and network boundary, under that cloud's IAM/VPC/compliance envelope. This is the same federation you saw in Phase 24: Cohere's Command/Embed/Rerank are first-class models in Bedrock's catalog.
On-premises / private deployment — Cohere will deploy its models into a customer's own infrastructure (including via containerized deployments), for organizations that cannot use any public cloud.
Air-gapped — the extreme: a fully disconnected environment for defense/intelligence/highly regulated workloads.

Paired with a no-training-on-your-data-by-default posture and enterprise controls (SOC 2, data residency, RBAC/SSO in North), this is the whole sales motion: you get frontier-adjacent model quality without your proprietary data ever leaving a boundary you control. For a bank, a hospital, or a defense contractor, that is not a nice-to-have — it is the gate that most model vendors simply cannot clear. Whenever a scenario says "regulated," "on-prem," "can't use the public API," or "data can't leave," that is Cohere's home turf, and naming the deployment mode that fits is the senior move.

3. The Command model family

Command is Cohere's text-generation line, and its defining trait is that it is trained for retrieval-grounded generation and tool use, not just open-ended chat. The lineage worth knowing:

Command / Command Light (the original generation) — general instruction-following text models; now legacy.
Command R (R = "retrieval") — the model that defined the family: optimized for RAG, grounded generation with citations, tool use, and multilingual work, with a 128k-token context window, at a price point meant for high-volume production RAG rather than frontier reasoning. This is the workhorse.
Command R+ — the larger sibling: stronger reasoning and multi-step tool use, same RAG/citation/multilingual orientation, 128k context — for workloads where R's quality isn't quite enough.
Command R7B — a small (7B) model for latency-sensitive, cost-sensitive, or on-device/edge deployments, keeping the RAG/tool orientation in a much smaller package.
Command A (and Command A Vision / reasoning variants) — the current flagship direction: a 256k-token context window, tuned specifically for agentic and enterprise workloads (tool use, multi-step, efficiency on modest hardware), positioned as Cohere's most capable generation model. Treat exact model names and numbers as drifting — new variants ship regularly — but the shape is stable: a RAG/agent-optimized family with long context, offered across the deployment spectrum in §2.

When to pick which, as you'd say it to a customer: Command R for the bulk of grounded-RAG traffic (best cost/quality for "answer from these documents, with citations"); Command R+ or Command A when the task needs deeper reasoning or longer multi-step tool chains; Command R7B when latency, cost, or edge/on-device constraints dominate and the task is narrow. The differentiator versus a general chat model isn't a benchmark — it's that these models were fine-tuned to consume a documents list and emit grounded, cited output and structured tool calls, which is exactly the behavior §4 and §5 build on.

4. The Chat API: messages, documents, tools, streaming

Cohere's Chat API (co.chat / the /v2/chat endpoint) is the generation front door, and it has four moving parts an FDE must know cold.

1. messages — the conversation, a list of role-tagged turns (system, user, assistant, tool). Standard multi-turn chat shape; the system message carries instructions and persona. (Command R also historically exposed a preamble for the system instruction and a chat_history + message split in the v1 API; v2 unifies on messages.)

2. documents — the parameter that makes this a RAG API and not a plain chat API. You pass a list of retrieved documents (each an object with content plus an id and arbitrary metadata), and the model generates its answer grounded in them, returning citations that map answer spans back to those documents (§5). This is fundamentally different from concatenating documents into the prompt string yourself: the model was trained to treat documents as a distinct, citable evidence set, and Cohere's server-side retrieval-aware machinery produces the citation offsets. You can also hand Cohere connectors / a retrieval step so it fetches documents itself, but the common FDE pattern is you retrieve (Embed + Rerank, §7–8) and pass the top documents in.

import cohere
co = cohere.ClientV2()  # reads CO_API_KEY

resp = co.chat(
    model="command-r-08-2024",
    messages=[{"role": "user", "content": "What's our enterprise refund window?"}],
    documents=[
        {"id": "doc_refund", "data": {"text": "Enterprise customers may request a refund within 45 days of purchase."}},
        {"id": "doc_sla",    "data": {"text": "Severity-one incidents get a one-hour response."}},
    ],
)
print(resp.message.content[0].text)   # the grounded answer
print(resp.message.citations)         # [{start, end, text, sources:[{document:{id:...}}]}...]

3. tools — function-calling / tool use (§10): you declare tools with JSON-schema parameters, the model returns structured tool_calls, you execute them and feed results back as tool messages. Command R/R+/A are specifically tuned for multi-step tool use.

4. Streaming — co.chat_stream(...) emits typed events (content deltas, tool-call deltas, and — the Cohere-specific part — citation events that arrive alongside the text as it streams). Streaming citations is why the start/end offsets matter: the client can attach a source marker to a span the moment that span is emitted.

The response object carries message.content (the answer), message.citations (§5), message.tool_calls (if any), a finish_reason, and usage (token accounting) — the shape Lab 02 and Lab 03 mirror with Citation/GroundedReply/RagResponse.

5. Grounded generation with inline citations, in depth

This is Cohere's signature capability, the one to be able to explain at mechanism depth, because it is the answer to "how does the enterprise trust the output."

The problem it solves. Naive RAG — retrieve chunks, paste them into the prompt, ask the model to answer — produces an answer that is influenced by the documents but carries no record of which document supports which claim, and no guarantee that every claim is supported at all. The model can fluently blend a real fact from doc A, a real fact from doc B, and a plausible hallucination that came from neither, into one confident paragraph. For a consumer toy that's tolerable; for an enterprise assistant answering from contracts or medical policy it is disqualifying, because the operational question is always "show me the source," and naive RAG has no source to show.

What Cohere returns. When you pass documents, the Chat response includes a citations array. Each citation is, in essence:

{ "start": 12, "end": 58, "text": "may request a refund within 45 days",
  "sources": [{ "type": "document", "document": { "id": "doc_refund" } }] }

start/end are character offsets into the generated answer; text is that exact substring; and sources (v1 called it document_ids) names the document(s) grounding that span. So for any sentence the assistant produced, a reviewer — or an automated guardrail, or an auditor a year later — can jump to the precise passage that backs it. Lab 02 builds this exact contract: Citation(start, end, text, document_ids) with offsets into the answer and ids into your docs.

How it differs from "stuff context in the prompt," precisely:

Attribution is structured, not inferred. You don't parse the answer hoping to guess which chunk it came from; the model emits the mapping. The Command models were fine-tuned to produce grounded spans and citation markers as part of RAG training — the citations come out of the decoder, aligned to the evidence, not bolted on afterward.
The claim→span→source chain is checkable. Because a citation carries answer offsets and document ids, you can independently verify it: does answer[start:end] really appear, and does the named document really support it? Lab 02's verify_citations is exactly this auditor, and it catches three lies — tampered offsets, a citation naming a document that doesn't exist, and a citation naming a real document that doesn't actually support the span.
Grounding enforcement is a deployment decision. Cohere returns citations; it does not refuse to emit an uncited sentence. So the production system must decide what to do with a claim that came back uncited (i.e., the model asserted something no document grounds). Lab 02 hard-codes the strict enterprise choice — drop it, log it, compute a grounding score — because "a model trained to cite can still miscite," and the whole point of choosing Cohere was to not ship unattributable claims.

The mechanism, in the miniature. Lab 02 inverts the trained-decoder approach into an inspectable one: an injected generator proposes claims (some grounded, one hallucinated); a support check computes, per claim, the fraction of its content terms attested by each document and locates the minimal source span attesting them; supported claims are assembled into the answer with exact offsets; unsupported claims are dropped with an audit trail; and a faithfulness / grounding score = supported / generated quantifies how much the generator could be trusted. The trained model does this with learned entailment instead of term overlap, but the contract and the invariants are identical — which is precisely why the lab is a faithful teaching model even though it uses no ML.

Why "auditability" is the real product. An enterprise doesn't adopt grounded generation because it's elegant; it adopts it because when regulators, or a customer's lawyer, or an internal risk team asks "on what basis did your AI tell this customer they qualified for a refund," the answer is a citation to a specific clause of a specific document, retrievable from the log — not "the model said so." Citations turn an LLM answer from an opinion into an attributable assertion, and that is the difference between a pilot and a production deployment.

6. Embed: input_type, multilingual, and compressed embeddings

Embed is the retrieval half of RAG. Cohere's embedding models (embed-english-v3.0, embed-multilingual-v3.0, and the newer Embed v4 with multimodal and Matryoshka support) are trained bi-encoders, and three of their properties are FDE-interview material.

1. input_type — asymmetric embeddings. This is the parameter people forget and then can't explain why their retrieval is mediocre. Every Embed v3+ call requires an input_type:

search_document — for the texts you index (your corpus).
search_query — for the queries you search with.
classification — for text you'll feed a classifier.
clustering — for text you'll cluster.

search_document and search_query produce genuinely different vectors for the same text, and that asymmetry is by design: a query ("refund window enterprise") and a document (a paragraph of policy prose) come from different distributions — different length, different redundancy, different keyword density — and the model is trained to project each into a shared space where a query lands near the documents that answer it, not merely the documents that resemble it. Use search_document for both sides and nothing errors; retrieval quality just quietly sags because you've thrown away the asymmetry the model was trained to exploit. Lab 03 makes this mechanical: document vectors are hashed token counts (term frequency is document signal), query vectors are hashed content-term presence (a query term votes once), sharing one hash space so cross-type cosine is meaningful — different encoders, one geometry, exactly the real model's design intent.

2. Multilingual. embed-multilingual-v3.0 (and Embed v4) embed 100+ languages into a shared space, so a French query retrieves an English document about the same thing. For a multinational enterprise this is a first-class requirement, and it's a genuine Cohere strength.

3. Compressed embeddings. Embed v3+ can return the embedding in several embedding_types:

float — 32-bit floats, the default, highest fidelity, largest index.
int8 / uint8 — 8-bit integers, 4× smaller, with cosine similarity almost identical to float — usually a free lunch for large indexes. Lab 03 quantizes by scaling to ±127.
binary / ubinary — 1 bit per dimension, ~32× smaller; similarity degrades to a Hamming-style "how many bits do we share," which still ranks easy cases correctly but costs recall on hard ones. Lab 03 shows exactly this: binary keeps the easy top-1, and you'd measure the recall cost on your corpus before committing.
Matryoshka / shortened dimensions (Embed v4) — the model is trained so that prefixes of the vector are themselves usable embeddings, so you can store a short vector for cheap shortlisting and the full vector for precise rescoring. Lab 03's truncate_matryoshka mirrors the usage pattern (truncate, renormalize, retrieve) and honestly shows the tradeoff: 48 of 64 dims keeps the easy top-1, 16 dims loses it — because a hashing embedder isn't trained to front-load information the way a real Matryoshka model is, which is the stated limit of the miniature.

Embedding dimensions for the v3 models are model-fixed (e.g. 1024 for the large English/ multilingual v3); Embed v4 exposes selectable output dimensions via the Matryoshka training. The practical lesson: the embedding format (input_type, embedding_types, dimension) is a deliberate index-design decision with real storage/latency/recall consequences, not a default to ignore.

7. Rerank: cross-encoder vs bi-encoder

Rerank is Cohere's most-adopted API, including by shops that use no other Cohere product, and the reason is architectural. First-stage retrieval — dense embeddings, BM25, or the hybrid you built in Phase 05 — is a bi-encoder: the query and every document are embedded separately. That is what makes it scalable (document vectors precompute into an index; query time is one embed + a nearest-neighbor lookup), and it is exactly what makes it coarse: because the query and document vectors are built independently, the scoring dot product never lets the query "see" the document. Word order, phrasing, and term proximity — everything about how the query and this specific document interact — is invisible.

A cross-encoder — Rerank — instead concatenates the query and one document into a single input and scores them jointly, with a transformer attending across the pair. It can tell "reset the vpn password on the corporate gateway" (the actual procedure, phrased that way) from a policy page that merely contains those five words scattered across three sentences — a distinction a bi-encoder's independent vectors structurally cannot represent. The cost is that scoring is O(candidates) model inferences per query (you can't precompute, because the score depends on the query), so you never run a cross-encoder over the whole corpus. You run it over the shortlist the cheap bi-encoder already produced.

Hence the pattern, and Cohere's whole pitch for the product: retrieve wide and cheap (bi-encoder over the corpus, top ~50–100), then rerank narrow and precise (cross-encoder over the shortlist, keep top ~3–10). Rerank (co.rerank) takes a query, a list of documents, and top_n, and returns results carrying each document's original index (so you rejoin your metadata), a calibrated relevance_score, and the reordering. It handles long documents by chunking (historically max_chunks_per_doc) and scoring a document by its best chunk. The latency/quality tradeoff is the only real knob: a larger candidate set and larger top_n mean more cross-encoder inferences and higher latency, for higher recall of the truly-relevant at the top. Lab 01 builds all of this — the coverage+phrase+proximity relevance score (the phrase and proximity terms are precisely the interaction a bi-encoder loses), the index-preserving top_n contract, chunked long-doc handling, and a planted corpus where a document buried at first-stage rank #4 is promoted to reranked #1.

Cross-reference: this is the reranking stage Phase 05's hybrid retriever ends on, productized — Phase 05 taught the bi-encoder + BM25 + RRF first stage; this phase's Rerank is the cross-encoder that comes after.

8. The two-stage RAG pipeline end to end

Put §5–7 together and you have the reference architecture an FDE actually deploys:

                 ┌── Embed(search_document) ──> vector index  (offline, once)
  corpus  ───────┤
                 └── (optional BM25 index)                    (Phase 05 hybrid)

  query  ──> Embed(search_query) ──> first-stage retrieval (cosine / hybrid), top ~50–100
                                            │
                                            ▼
                                 Rerank (cross-encoder), keep top ~3–10
                                            │
                                            ▼
                          Chat(model=Command, documents=top docs)
                                            │
                                            ▼
                          grounded answer + citations  ──> enforce grounding, log, return

Each stage has a distinct job and a distinct failure mode. First stage maximizes recall — its job is to not lose the right document; if the answer chunk isn't in the top-100, no downstream stage can recover it (recall@k is the metric, Phase 05). Rerank maximizes precision at the top — it fixes the "relevant-but-not-top" and "term-stuffed-decoy-on-top" failures the bi-encoder can't see. Generation maximizes faithfulness — it must answer only from what the reranked documents support, and cite it. Lab 03 composes exactly this (TwoStageRag.answer: retrieve first_k → rerank to rerank_n → grounded-generate) and carries both stages' document ids in the response — because when someone asks "why did it cite the wrong doc," the only way to debug is knowing what each stage saw and kept. The planted example is the whole argument in miniature: first-stage cosine ranks the term-stuffed policy page #1 (bag-of-words loves repetition), rerank promotes the actual how-to, and grounded generation cites it, dropping the scripted hallucination.

9. North: the secure enterprise AI workspace

North is Cohere's application-layer product: a secure AI workspace platform that packages the models (Command), retrieval (Embed + Rerank), and agents into an environment that runs inside the customer's security boundary. The mental model: where OpenAI has ChatGPT Enterprise and the like, North is Cohere's equivalent — but the defining property is deployment location and data control, not features.

What North gives an enterprise:

A workspace / assistant surface — employees chat with an assistant that has access to their workplace knowledge and tools, with the grounded-citation behavior of §5 so answers are traceable.
Connectors to workplace tools — it connects agents to the systems work actually happens in (document stores, search, internal apps, SaaS tools), so the assistant can retrieve from and act on real enterprise data.
Agents with tool access — not just Q&A: North agents can take multi-step actions through those connectors (§10), inside a permission model.
Private / VPC / on-prem / air-gapped deployment — the §2 spectrum, applied to the whole application: North can run so that customer data never leaves the customer's control, and Cohere does not train on it.
Enterprise governance — RBAC, SSO, access controls, and audit trails, so which employee can ask what, of which data, is itself governed.

For the FDE, North is the thing you deploy: a customer licenses North, and your job is to stand it up in their environment, wire the connectors to their data, configure the retrieval + grounding so answers are precise and cited, set up the agents and permissions, and put an evaluation harness (§11) around it. Every mechanism in this phase — Rerank precision, Embed input_type, grounded citations, tool use — is a dial you turn inside a North deployment. The one-sentence version for an interview: "North is Cohere's secure AI workspace — models plus retrieval plus agents plus connectors, deployable inside the customer's own boundary, with the data never leaving their control."

10. Agentic workflows on Cohere: tool use and multi-step

Grounded Q&A is the floor; enterprises quickly want the assistant to do things, which means tool use and multi-step agentic loops — exactly what Phase 01 built from scratch and Phase 07 orchestrated.

Cohere supports this through the Chat API's tools parameter. You declare tools with JSON-schema parameters; the model (Command R/R+/A, tuned for this) returns structured tool_calls; your code executes them and feeds results back as tool messages; the model continues, possibly calling more tools, until it produces a final grounded answer. This is the same reason→act→observe loop from Phase 01, with Cohere's model as the policy and its tool-calling fine-tuning as the reason the structured calls are reliable. Two Cohere-specific notes:

Grounding composes with tools. A tool that returns documents (a search tool, a database query) feeds the same documents/citation machinery — so an agent that retrieves mid-loop can still produce cited answers, which is the enterprise requirement that separates a real deployment from a demo.
Multi-step is what Command A is tuned for. The flagship direction explicitly targets agentic workloads — longer tool chains, more reliable structured output, efficient enough to run the loop at enterprise volume.

In North, this shows up as agents with connector-backed tools operating under a permission model: the agent can search the doc store, file a ticket, or update a record — but what it may do is governed (which ties directly to the authorization discipline of Phase 13). The FDE framing: translate "the assistant should also file the exception ticket" into a tool definition, an execution path, a permission scope, and an evaluation that checks it filed the right ticket — i.e., an ambiguous business problem into a grounded, tool-using, evaluable workflow.

11. Evaluating grounded RAG: faithfulness, citation accuracy, relevance

A forward-deployed system you can't measure is a liability, and grounded RAG has specific metrics beyond generic answer quality. This section ties directly to Phase 11's eval harness.

Evaluate the two halves separately, because they fail separately:

Retrieval quality (stages 1–2):

Recall@k — did the relevant document make the first-stage top-k? If not, nothing downstream can save the answer; this is the metric first-stage retrieval is tuned on.
Rerank lift — recall/MRR of retrieve-only vs retrieve→rerank on a labeled set. This is how you justify the Rerank latency cost with a number (Lab 01's planted promotion is the single-example version).

Generation quality (stage 3):

Faithfulness / groundedness — is every claim in the answer actually supported by the retrieved documents? This is the metric that catches hallucination, and it's exactly Lab 02's grounding_score = supported / generated. In production you compute it with an NLI model or an LLM judge for entailment; the lab computes it lexically to keep the mechanism visible.
Citation accuracy — of the citations emitted, how many correctly attribute their span to a document that genuinely supports it (precision), and of the claims that could be cited, how many got a citation (recall)? Lab 02's verify_citations is the precision check made executable — it catches a citation that names a real document which doesn't actually support the span, which is the subtle failure a naive "does it have citations" check misses.
Answer relevance — does the answer actually address the query (as opposed to being faithful but off-topic)? Typically an LLM-judge dimension.

The production pattern (same layering as Phase 11 and Phase 24 §9): cheap automatic metrics (recall@k, lexical faithfulness) as a regression gate in CI on every config change; an LLM judge for faithfulness/relevance during iteration; human review as the final gate before a customer-facing change. The FDE-specific point: you wire this harness around the North deployment before you tune anything, because "we improved retrieval" is only credible with a faithfulness-and-recall number that moved, and a regulated customer will require the eval evidence.

12. Enterprise security, compliance, and auditability

Pulling the security story together, because it's the spine of the whole phase:

Deployment boundary (§2) — SaaS, VPC/cloud-private (Bedrock/SageMaker/Azure/Vertex/OCI), on-prem, air-gapped. The FDE picks the mode that clears the customer's data-residency and network-isolation requirements. This is the architectural trust boundary, the same discipline as Phase 00 and Phase 24's VPC/PrivateLink story, applied to a model vendor.
No training on customer data by default — the contractual/technical guarantee that your proprietary corpus and prompts aren't absorbed into a shared model. For many enterprises this single clause decides the vendor.
Auditability via citations (§5) — grounded generation isn't only a quality feature; it's a compliance feature. Every answer's claim→source mapping, logged, is the audit trail that lets a regulated org defend an AI-assisted decision after the fact. This is why the enforcement layer (drop uncited claims, store the citation with the answer) is non-negotiable in a real deployment.
Access governance in North — RBAC/SSO controls who may ask what of which data, so the retrieval itself respects the customer's existing permissions (a user can't ground an answer in a document they're not allowed to read). This is the authorization/isolation discipline of Phase 13, applied to RAG.
Guardrails and injection defense — a retrieval-augmented agent that ingests documents and calls tools has the prompt-injection surface of Phase 10: a poisoned document can try to hijack the agent. Grounding + citation enforcement helps (an injected instruction isn't a grounded claim), but it's not a complete defense, and an FDE should say so.

The synthesis: Cohere's security posture isn't one feature, it's the composition of where the model runs, what it does with your data, whether its answers are traceable, and who's allowed to ask — and an FDE's job is to make all four true for a specific customer.

13. Cohere vs OpenAI, Anthropic, and open models for enterprise RAG

The platform-selection question, answered by axis rather than by favorite — the Staff/Principal move.

Cohere optimizes for enterprise RAG + deployment control: models tuned for grounded, cited generation; a best-in-class Rerank; strong multilingual embeddings; and the full deployment spectrum (VPC/on-prem/air-gapped) with no-training-on-your-data. You pick Cohere when the customer's blockers are data can't leave the boundary, answers must be auditable/cited, and retrieval precision matters — and when frontier reasoning IQ is not the deciding factor. Rerank specifically is worth adopting even if you use another vendor's generation model.

OpenAI optimizes for frontier capability and ecosystem breadth: typically the newest reasoning/multimodal capabilities first, the largest tooling/ecosystem, the Assistants/Responses APIs and built-in retrieval. You pick it when the product's edge is raw capability the day it ships, and when a fully-managed public API is acceptable. Its enterprise deployment story exists (Azure OpenAI gives you the VPC/compliance envelope) but on-prem/air-gapped is not its native posture the way it is Cohere's.

Anthropic (Claude) optimizes for strong reasoning + a safety/steerability posture and long context, with a clean tool-use and (via Bedrock/Vertex) enterprise-deployment story. You pick Claude when reasoning quality and controllability are central and the Bedrock/Vertex envelope satisfies compliance. Its RAG is excellent but it does not sell a dedicated Rerank/Embed retrieval stack the way Cohere does, nor a North-style on-prem workspace.

Open-weight models (Llama, Mistral, Qwen, etc.) optimize for maximum control and no per-token vendor cost: you run the weights yourself, anywhere, air-gapped if you like, and pay only for compute. You pick them when total data control and cost-at-scale dominate and you have the MLOps capacity to serve, secure, and evaluate models yourself — but you own the entire burden of grounding, citations, retrieval quality, and evaluation that Cohere ships as product. (A common real pattern: open-weight generation with Cohere Rerank + Embed, because the retrieval stack is the hard part to reproduce.)

The decision framework, said aloud: regulated, data-can't-leave, needs cited/auditable answers, retrieval precision matters → Cohere (often North). Product edge is frontier capability, public-API-acceptable → OpenAI. Reasoning/steerability central, Bedrock/Vertex envelope fine → Anthropic. Total control / air-gap / cost-at-scale with MLOps capacity → open weights, very possibly with Cohere's retrieval APIs on top. Naming the axis you're optimizing for is the answer; declaring a universal winner is the junior tell.

14. Common misconceptions

"Cohere is just another chatbot API." It's an enterprise-RAG platform; the models are tuned for grounded, cited generation and tool use, and the products (Embed, Rerank, North, private deployment) exist to solve enterprise retrieval and deployment, not to win a chat leaderboard.
"Grounded generation is just RAG." Naive RAG stuffs context into a prompt; Cohere's grounded generation returns a structured citation mapping (answer span → source document) the model was trained to emit — attribution you can verify, not a prompt you hope worked.
"Citations mean the answer is guaranteed correct/grounded." No — Cohere returns citations but does not refuse uncited sentences; a model trained to cite can still miscite. Enforcement (drop/flag uncited claims, verify each citation) is the deployment's job (Lab 02).
"input_type is optional / cosmetic." It changes the embedding. Index with search_document, query with search_query; using the wrong one silently degrades retrieval and never errors.
"Rerank is just a second, better embedding search." It's a cross-encoder — it scores the query and document jointly, seeing interactions (order, proximity) a bi-encoder embedding physically cannot, which is why it runs only on a shortlist, not the corpus.
"Compressed embeddings are lossy junk." int8 is near-free (≈4× smaller, cosine barely moves); binary (~32×) and aggressive Matryoshka truncation cost recall you should measure — they're a deliberate storage/recall dial, not damage.
"North is just ChatGPT Enterprise for Cohere." The defining property is where it runs and who controls the data — private/VPC/on-prem/air-gapped with no training on customer data — not the chat surface.
"Cohere competes with OpenAI on model IQ." It competes on the enterprise envelope (grounding, retrieval precision, deployment control), deliberately not on frontier-benchmark supremacy — conflating the two misreads the whole company.

15. Lab walkthrough

Build the three miniatures; each isolates one Cohere capability and injects the model (and, in Lab 03, the embedder/generator) as a pure function so everything stays deterministic and offline.

Lab 01 — Rerank. Implement the cross-encoder relevance score (coverage + phrase + proximity — the phrase/proximity terms are the interaction a bi-encoder loses), Cohere's rerank contract (top_n, preserved original indices, descending calibrated scores), and max_chunks_per_doc-style long-document chunking, then prove the two-stage win: a document first-stage-ranked #4 is promoted to reranked #1. 27 tests.
Lab 02 — Grounded Generation & Citations. Implement per-claim support checking with minimal source spans, assemble an answer with Cohere-shaped citations ({start, end, text, document_ids} into the answer), enforce grounding (drop the injected hallucination, compute the grounding score), and build the independent verify_citations auditor that catches tampered offsets and misattributed sources. 28 tests.
Lab 03 — Embed & Two-Stage RAG. Implement the asymmetric embedder (search_query ≠ search_document), int8/binary/Matryoshka compression with measured tradeoffs, a cosine index, the compact rerank + grounded-generation stages, and the TwoStageRag pipeline tying them together — proving two-stage beats retrieve-only on a planted corpus. 27 tests.

Run each with LAB_MODULE=solution pytest test_lab.py -v first (green reference), then fill your lab.py to match, then read solution.py's main() output.

16. Success criteria

You can explain, in one sentence each, what Command, Embed, Rerank, and North are.
You can describe Cohere's citation shape ({start, end, text, document_ids}, offsets into the answer) and why enforcement/verification is the deployment's job, not the model's guarantee.
You can explain why search_query and search_document embeddings differ and what breaks (silently) if you use the wrong one.
You can explain bi-encoder vs cross-encoder and why Rerank runs on a shortlist, not the corpus.
You can draw the two-stage retrieve→rerank→grounded-generate pipeline and name each stage's metric and failure mode.
You can name Cohere's deployment modes and the compliance blocker each one clears.
You can compare Cohere to OpenAI, Anthropic, and open-weights by axis and pick per constraint.
You can name the specific grounded-RAG eval metrics (recall@k, faithfulness, citation accuracy, answer relevance) and where each fits.
All three labs pass under both lab and solution (82 tests total).

17. Interview Q&A

Q: What actually differentiates Cohere from OpenAI or Anthropic? A: Not model IQ — the enterprise envelope. Cohere optimizes for retrieval-grounded, citable generation; a best-in-class Rerank; strong multilingual Embed; and a full deployment spectrum (SaaS/VPC/on-prem/air-gapped) with no training on customer data. The thesis is that an enterprise's blocker is rarely intelligence — it's data residency, auditability, and retrieval precision — and every Cohere product maps onto one of those.

Q: Explain grounded generation with citations and how it differs from stuffing context into a prompt. A: You pass documents to the Chat API and the model returns a citations array — each citation has character offsets into the answer (start/end/text) plus the document_ids grounding that span. Unlike pasting chunks into the prompt, the attribution is structured and checkable: you can verify that answer[start:end] exists and that the named document actually supports it, and you can enforce grounding by dropping any claim that came back uncited. The Command models were fine-tuned to emit these grounded spans, so the mapping comes out of the decoder, not a fragile post-hoc parse.

Q: Do citations guarantee the answer is faithful? A: No. Cohere returns citations but doesn't refuse to emit an uncited sentence, and a model trained to cite can still miscite. So the production system owns enforcement: verify each citation (offsets honest, named document truly supports the span), drop or flag uncited claims, and track a faithfulness/grounding score. That enforcement layer is exactly what you build in Lab 02, including an auditor that catches a citation naming a real document that doesn't actually support its span.

Q: What is input_type and why does it matter? A: Embed v3+ requires it: search_document for the corpus you index, search_query for queries, plus classification/clustering. Query and document embeddings are asymmetric on purpose — a short keyword query and a prose paragraph come from different distributions, and the model projects each so a query lands near the documents that answer it. Use search_document for both and nothing errors; retrieval just silently gets worse because you discarded the asymmetry.

Q: Bi-encoder vs cross-encoder — why does Rerank exist? A: First-stage retrieval is a bi-encoder: query and documents embedded separately, so vectors precompute and index — fast, but the scoring dot product never lets the query see the document, so word order and proximity are invisible. Rerank is a cross-encoder: it scores the (query, document) pair jointly, capturing those interactions — far more accurate, but O(candidates) inferences, so you run it only on the shortlist the bi-encoder produced. Retrieve wide and cheap, rerank narrow and precise.

Q: Walk me through a full two-stage RAG pipeline. A: Offline, embed the corpus with search_document into a vector index. Per query: embed with search_query, retrieve the top ~50–100 by cosine/hybrid (first stage, tuned for recall), Rerank those and keep the top ~3–10 (second stage, tuned for precision at the top), pass those documents to Command via the Chat API's documents parameter (grounded generation, tuned for faithfulness), and return the answer with citations after enforcing grounding. Each stage has its own metric — recall@k, rerank lift, faithfulness/citation accuracy — and you log every stage's candidates for debuggability.

Q: What is North? A: Cohere's secure AI workspace: models (Command) + retrieval (Embed + Rerank) + agents + connectors to workplace tools, deployable inside the customer's own boundary (private/VPC/on-prem/air-gapped) with data never leaving their control and RBAC/SSO governance. It's the application an FDE actually deploys; every mechanism in this phase is a dial inside a North deployment.

Q: A customer says "we can't put our data in anyone's cloud, and every AI answer has to be traceable to a source document." Which Cohere features answer that? A: Two separate things. "Can't leave our cloud" → private/on-prem (or air-gapped) deployment of Cohere models, or VPC deployment via Bedrock/Azure/Vertex, with no training on their data — North stood up inside their boundary. "Traceable to a source" → grounded generation with citations through the Chat documents API, plus an enforcement layer that drops uncited claims and logs the citation with every answer, so there's an audit trail. I'd wire retrieval as Embed(search_document/ search_query) → Rerank → Command, and put a faithfulness/citation-accuracy eval around it.

Q: How do you evaluate a grounded RAG system? A: Separate the halves. Retrieval: recall@k for the first stage, and rerank lift (recall/MRR of retrieve-only vs retrieve→rerank) to justify the reranker. Generation: faithfulness/groundedness (is every claim supported — Lab 02's grounding score), citation accuracy (do emitted citations correctly attribute — precision — and do supportable claims get cited — recall), and answer relevance (does it address the query). Cheap automatic metrics gate CI on every config change; an LLM judge covers faithfulness/relevance during iteration; humans are the final gate — and a regulated customer will require the evidence.

Q: As an FDE, how do you turn "make our support smarter with AI" into a concrete build? A: Pin down the ambiguity into a workflow: what data grounds answers (which doc stores → connectors), what "smarter" means (answer procedures, not recite the index → retrieval precision, so Rerank), what trust requires (cited, auditable answers → grounded generation + enforcement), where it can run (compliance → deployment mode/North), and what "done" means (a faithfulness/recall eval on a labeled regression set). Then build Embed→Rerank→Command with citation enforcement inside North, add tool use if it must act (file a ticket), and stand up the eval harness before tuning. The skill is translating the vague ask into stages, each with a metric.

Q: When would you NOT recommend Cohere? A: When the product's edge is frontier reasoning IQ the day it ships (lean OpenAI/Anthropic), or when total control and cost-at-scale dominate and the team has the MLOps depth to serve open weights themselves. Even then I'd often keep Cohere Rerank (and Embed) on top of an open-weight generator, because a precise, multilingual retrieval stack is the hard part to reproduce — which is exactly why Rerank is Cohere's most-adopted API.

Q: What's the honest limitation of Cohere's pitch? A: It's deliberately not the frontier-IQ leader, so for tasks bottlenecked on hard reasoning it can trail. And citations solve attribution, not truth — a document can be wrong, or an injected/poisoned document can try to hijack a retrieval agent; grounding helps but isn't a complete injection defense. The value is real but scoped: attributable, deployable, precise retrieval-grounded AI — not "the smartest model in every benchmark."

18. References

Cohere — Documentation home (models, Chat, Embed, Rerank, RAG, tool use). https://docs.cohere.com/
Cohere — Command models overview (Command R, R+, R7B, Command A). https://docs.cohere.com/docs/command-r
Cohere — Chat API and RAG with the documents parameter (retrieval-augmented generation). https://docs.cohere.com/docs/retrieval-augmented-generation-rag
Cohere — Citations / grounded generation in Chat. https://docs.cohere.com/docs/documents-and-citations
Cohere — Embed API and input_type (search_document / search_query / classification / clustering). https://docs.cohere.com/docs/embeddings
Cohere — Embed v3 / v4, multilingual, and compressed embedding types (int8 / binary / Matryoshka). https://docs.cohere.com/docs/embed-models
Cohere — Rerank API (cross-encoder reranking, top_n, long-document chunking). https://docs.cohere.com/docs/rerank-overview
Cohere — Tool use / function calling with Command. https://docs.cohere.com/docs/tool-use
Cohere — North (secure enterprise AI workspace). https://cohere.com/north
Cohere — Private / cloud deployment options (VPC, on-prem, air-gapped; Bedrock, SageMaker, Azure AI Foundry, Vertex AI, OCI). https://docs.cohere.com/docs/cohere-on-cloud
Cohere — Security and trust (no training on customer data, compliance). https://cohere.com/security
Amazon Bedrock — Cohere models in the Bedrock catalog (the §2 VPC/federation contrast; see also Phase 24). https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html
Nogueira & Cho — "Passage Re-ranking with BERT" (the cross-encoder reranking idea behind §7). https://arxiv.org/abs/1901.04085
Kusupati et al. — "Matryoshka Representation Learning" (the truncatable-embedding idea in §6). https://arxiv.org/abs/2205.13147
This track — Phase 05 Retrieval Foundations (the bi-encoder + BM25 + RRF first stage Rerank sits after) and Phase 11 Agent Evaluation (the eval harness §11 wires around a grounded-RAG deployment).

Agentic AI Engineer — Agent Infrastructure, Platforms & Production Systems