Selecting RAG Models

Phase 5 · Document 05 · Model Selection Prev: 04 — Reasoning Models · Next: 06 — Agent Models

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

RAG (retrieval-augmented generation) is the most common enterprise LLM pattern — answer questions from your documents. The selection twist: RAG needs three model roles, not one — an embedding model (retrieval), an optional reranker (precision), and a generator (the answer) — and beginners over-focus on the generator while retrieval quality actually dominates results. Pick the wrong embedding model and the best generator still answers from garbage context. This doc applies the framework to all three roles and teaches the RAG-specific quality axes (faithfulness, grounding, citations), feeding the full RAG build in Phase 9.

2. Core Concept

Plain-English primer (the three RAG roles + quality words)

• RAG — instead of hoping the model "knows," you retrieve relevant text from your documents and paste it into the prompt, then the model answers from it (Phase 9).
• Embedding model — turns text into a vector (list of numbers) so you can find passages similar to a query via cosine similarity. The retrieval engine. (Phase 2.01)
• Reranker — given the query and the top-N retrieved passages, scores how relevant each is and reorders them, so the best context goes to the generator. Boosts precision.
• Generator — the LLM that reads the retrieved context + question and writes the answer (with citations).
• Faithfulness / grounding — the answer is supported by the retrieved sources (not invented). The key RAG quality metric (Phase 1.07).
• Citations — the answer points to which source/passage backs each claim — essential for trust/audit.

Three roles, three selections

Retrieval quality usually dominates answer quality. A modest generator with great retrieval beats a frontier generator fed irrelevant chunks. Invest selection effort in the embedder/reranker first.

What to optimize the generator for (RAG-specific)

1. Instruction-following / groundedness — will it actually use the provided context and not fall back on parametric memory or invent facts?
2. Faithfulness — answers supported by sources; low hallucination when the context lacks the answer (ideally says "I don't know").
3. Citation-following — reliably attributes claims to sources when asked.
4. Long-context coherence — reasons across many retrieved chunks without "lost in the middle" (Phase 2.01) — but remember retrieval beats stuffing.
5. Structured output — emit citation metadata/JSON reliably (07).

Selecting the embedding model (the underrated decision)

Optimize on retrieval quality on your corpus (not a generic MTEB rank), dimension (bigger = better recall but more storage/latency in the vector DB), max input length (fits your chunk size), domain/language fit (code vs prose vs multilingual), and cost (you embed every doc + every query). Open options (BGE, E5, nomic, GTE) enable self-hosting; commercial options (OpenAI, Cohere, Voyage) are strong and zero-ops. Always evaluate retrieval on your data (Phase 1.04, Phase 9.03).

3. Mental Model

RAG = THREE models, not one:
  query ──[EMBEDDING model]──► find similar passages ──[RERANKER]──► best context ──[GENERATOR]──► grounded answer + citations

RETRIEVAL QUALITY usually DOMINATES → invest in embedder/reranker first (a great generator can't fix bad context).

GENERATOR axes (RAG): groundedness · faithfulness (say "I don't know") · citations · long-context coherence · structured output
EMBEDDING axes: retrieval quality on MY corpus · dimension · max input · domain/language · cost   (SAME model for q & docs!)
DECIDE on a RAG eval (faithfulness, context precision/recall, citation accuracy) — Phase 9/13, not generic ranks.

4. Hitchhiker's Guide

What to select first: the embedding model, measured by retrieval quality on your corpus — it gates everything downstream.

What to ignore at first: generic embedding leaderboards (MTEB) and generator "smartness" rankings — measure on your data and task.

What misleads beginners:

• Tuning the generator while retrieval feeds it junk.
• Using different embedding models for queries vs documents (breaks similarity).
• Stuffing huge context instead of retrieving well ("lost in the middle" + cost).
• Picking a generator that ignores context and answers from memory.

How experts reason: fix retrieval first (embedder + maybe reranker), evaluate with RAG metrics (faithfulness, context precision/recall), then choose the cheapest generator that stays grounded and cites well on your eval. They route: cheap grounded generator for most, stronger for hard synthesis.

What matters in production: faithfulness/grounding, citation accuracy, retrieval precision/recall, latency (embedding + rerank + generate), and cost per answered question.

How to verify: build a small RAG eval (questions with known-good source passages) and measure retrieval recall, faithfulness, and citation accuracy across embedder/reranker/generator choices (Phase 9).

Questions to ask: Does the embedder fit my domain/language/chunk size, and is it the same for q+docs? Does a reranker improve precision enough to justify latency? Does the generator stay grounded and cite reliably on my data?

What silently gets expensive/unreliable: generator hallucination when context is thin; mismatched embedders; over-long stuffed context (cost + recall loss); skipping reranking on noisy corpora.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• RAG needs three model roles: embedding (retrieval), reranker (precision, optional), generator (answer).
• Retrieval quality usually dominates — a great generator can't fix bad context.
• Embed queries and documents with the SAME model (Phase 1.04).
• Faithfulness/grounding and citations are the decisive generator metrics, not raw smartness.
• Retrieve well; don't just stuff long context — "lost in the middle" + cost (Phase 2.01).
• Evaluate retrieval and answers on YOUR corpus (Ragas-style metrics), not generic leaderboards.
• Embedding dimension trades recall vs storage/latency.
• A modest grounded generator often beats a frontier model fed poor context — and is cheaper.

9. Observations from Real Systems

• Enterprise doc assistants spend most engineering on retrieval (chunking, embedder, reranker) because that's where quality lives (Phase 9).
• Cohere/Voyage/OpenAI embeddings + a reranker are common commercial stacks; BGE/E5/nomic are common open/self-hosted embedders (01).
• Ragas-style faithfulness gates are used to catch RAG hallucination before shipping (Phase 13).
• Citation enforcement (the generator must cite, validated) is how trustworthy RAG products reduce hallucination.
• Generator over-investment (frontier model, weak retrieval) is the classic RAG anti-pattern that an eval exposes.

10. Common Misconceptions

11. Engineering Decision Framework

Select RAG models role-by-role (fix retrieval FIRST):
  1. EMBEDDING: shortlist on MTEB + domain/language; evaluate RETRIEVAL on YOUR corpus
     (context precision/recall); pick by recall + dimension/latency/cost. SAME model q+docs.
  2. RERANKER: add if first-stage retrieval is noisy; keep only if precision gain > latency/cost.
  3. GENERATOR: among models that pass faithfulness + citation on YOUR RAG eval, pick the CHEAPEST.
  4. ROUTE generator: cheap grounded model for most; stronger for hard synthesis.
  5. Eval (Ragas-style) gate + memo (3.07).

Optimize order:  chunking/retrieval ≫ reranking > generator choice.

12. Hands-On Lab

Goal

Show that retrieval choice dominates: hold the generator fixed and vary the embedding model; then hold retrieval fixed and vary the generator — measure faithfulness + retrieval recall on your data.

Prerequisites

• A small doc corpus + 15–20 questions with known supporting passages; two embedding models; two generators; a vector store (or in-memory cosine from Phase 2.01 lab).

Steps

1. Build a RAG eval set: questions + the source passage(s) that should be retrieved + a known-good answer.
2. Vary embedder (generator fixed): for each embedder, retrieve top-k and measure retrieval recall (did the right passage come back?) and downstream faithfulness.
3. Vary generator (retrieval fixed, good context): measure faithfulness, citation accuracy, cost, latency.
4. Compare deltas: quantify how much each axis moves answer quality (retrieval should dominate).
5. Decide: best embedder, whether to add a reranker, cheapest grounded generator; memo it.

# retrieval recall sketch (in-memory cosine)
def recall_at_k(eval_set, embed, k=5):
    hits = 0
    for q in eval_set:
        ranked = retrieve(embed(q["question"]), k)        # top-k passage ids
        hits += int(any(p in ranked for p in q["gold_passages"]))
    return hits / len(eval_set)
# faithfulness via judge or programmatic check on the generated answer

Expected output

• Recall and faithfulness tables showing the embedder choice swings results more than the generator choice — the core lesson.

Debugging tips

• Recall low across embedders? Fix chunking first (Phase 9) — selection can't save bad chunks.
• Faithfulness low with good context? The generator ignores context — pick a more grounded one or enforce citations.

Extension task

Insert a reranker between retrieval and generation; measure the precision/faithfulness gain vs added latency.

Production extension

Wire the RAG eval into CI (faithfulness + recall thresholds) so embedder/generator swaps can't silently regress (Phase 13).

What to measure

Retrieval recall per embedder; faithfulness/citation accuracy per generator; latency + cost per answer; reranker gain (extension).

Deliverables

• A RAG eval set + harness.
• Embedder-varied and generator-varied result tables.
• A 3-role selection (embedder, reranker?, generator) + memo.

13. Verification Questions

Basic

1. What three model roles does RAG need, and what does each do?
2. Why must queries and documents use the same embedding model?
3. What is faithfulness, and why is it the key RAG metric?

Applied 4. Retrieval recall is poor. Where do you intervene before changing the generator? 5. How do you decide whether a reranker is worth adding?

Debugging 6. Great generator, but answers are wrong/irrelevant. What's the likely cause? 7. The generator invents facts not in the context. Two fixes?

System design 8. Design model selection for a multilingual, private document assistant (all three roles + data policy).

Startup / product 9. Explain why investing in retrieval (embedder/reranker) over a fancier generator improves both quality and unit economics.

14. Takeaways

1. RAG selection is three roles: embedding, reranker (optional), generator.
2. Retrieval quality dominates — fix the embedder/reranker before the generator.
3. Same embedding model for queries and documents.
4. Choose the generator for faithfulness + citations, then cheapest that passes.
5. Retrieve well; don't stuff context.
6. Decide on a RAG eval (faithfulness, recall, citation accuracy), not leaderboards.

15. Artifact Checklist

• RAG eval set (questions + gold passages + answers).
• Embedder-varied retrieval-recall table.
• Generator-varied faithfulness/citation table.
• 3-role selection (embedder, reranker?, generator) + memo.
• (Extension) reranker gain measurement.
• Notes: "retrieval dominates" + same-embedder rule.

Next: 06 — Agent Models