RAG — Overview and the Pipeline

Phase 9 · Document 00 · RAG Prev: Phase 8 — LLM Gateways · Up: Phase 9 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

RAG (Retrieval-Augmented Generation) is the most common production pattern for giving an LLM knowledge it doesn't have — your private docs, fresh data, anything bigger than the context window — by retrieving the relevant pieces at query time and putting them in the prompt (what-happens §3.4). It's how you build a knowledge-base assistant, a support bot grounded in product docs, a legal/research tool over private datasets, or a codebase assistant — with citations and without retraining the model. Nearly every serious LLM application eventually needs RAG, and the single most important lesson of this phase is counterintuitive: RAG quality is dominated by retrieval, not by the generator model (Phase 5.05). A frontier model fed the wrong chunks gives a confident wrong answer; a cheap model fed the right chunks is correct. This overview is the map of Phase 9; the dedicated docs go deep on each stage.

2. Core Concept

Plain-English primer: retrieve, then generate

The model is stateless and bounded by its context window (what-happens §0); it only "knows" its training data plus whatever you put in the prompt. RAG answers: how do I give it the right information at the right moment? Instead of stuffing everything into context (too big, expensive, and "lost in the middle"), you keep a corpus outside the model and fetch only the relevant slice per query, then generate grounded in it (what-happens §3.4).

It splits into two pipelines:

INGESTION (offline, once per doc change):
  documents → LOAD/PARSE [01] → CHUNK [02] → EMBED [03] → STORE in vector DB (+metadata) [04]

QUERY (online, per request):
  query → (rewrite) → RETRIEVE (dense + sparse) [03,05] → RERANK [06] → PACK context [07]
        → GENERATE grounded answer with CITATIONS [08]  → (evaluate [09])

The two-pipeline split (why it matters)

• Ingestion is offline ETL: turn messy sources into clean, chunked, embedded, metadata-tagged records in a vector store. You re-run it when documents change (01/10).
• Query is the online hot path: embed the question, find candidate chunks, rerank for precision, pack the best into the prompt within budget, and generate a grounded, cited answer.

The central principle: retrieval dominates quality

If retrieval returns the wrong chunks, no model can save you — it will either hallucinate or honestly say it doesn't know. So most RAG effort goes into getting the right chunks to the top: good chunking (02), the right embedding model (03), hybrid search (semantic + keyword) for recall (05), and reranking for precision (06). The generator's job is then comparatively easy: answer only from the provided context and cite it (08).

RAG vs stuffing vs fine-tuning vs long-context

A decision you must be able to make (Phase 5.05, Phase 13.00):

• Stuffing (paste it all): fine for a small, fixed corpus that fits the window; fails as it grows (cost, window, lost-in-the-middle).
• Long-context (huge windows): helps, but is expensive per call and still degrades on a large/changing corpus; doesn't give attribution. Often paired with retrieval.
• Fine-tuning: teaches behavior/format/style, not fresh facts reliably; expensive to keep current. Use RAG for knowledge, fine-tuning for behavior.
• RAG: best for large, changing, private knowledge with citations and freshness — the default for "the model needs to know our stuff."

3. Mental Model

   the model knows: training data + what's in the prompt. RAG = put the RIGHT slice in the prompt.

   INGESTION (offline)            QUERY (online, per request)
   docs                            question
    → load/parse  [01]              → (rewrite/HyDE [07])
    → chunk       [02]              → RETRIEVE: dense(embed) + sparse(BM25) [03,05]
    → embed       [03]              → RERANK (cross-encoder, precision) [06]
    → store + metadata [04]         → PACK top chunks into the window (order matters) [07]
                                    → GENERATE grounded + CITED answer [08]
                                    → EVALUATE faithfulness/relevancy + retrieval metrics [09]

   ★ RETRIEVAL DOMINATES QUALITY: wrong chunks → no model can save you.
   RAG (large/changing/private + citations)  vs  stuffing (small/fixed)  vs  fine-tune (behavior, not facts)

Mnemonic: ingest (load→chunk→embed→store) offline; query (retrieve→rerank→pack→generate→cite) online. Fix retrieval first — it dominates.

4. Hitchhiker's Guide

What to look for first: is this a knowledge problem (use RAG) or a behavior problem (consider fine-tuning)? Then build the simplest pipeline (chunk → embed → store → retrieve → generate-with-citations) and measure retrieval before optimizing anything.

What to ignore at first: exotic chunking, fancy query rewriting, and multi-vector tricks. Get a baseline working and instrument it; add complexity where the eval shows a gap.

What misleads beginners:

• Blaming the model for RAG failures. Most "hallucinations despite RAG" are retrieval failures — the right chunk wasn't retrieved (09).
• Pure semantic search. Embeddings miss exact terms, IDs, names — hybrid (add BM25) is usually better (05).
• Skipping reranking. Top-k vector hits are noisy; a cheap cross-encoder reranker is high ROI (06).
• No evaluation. Without separating retrieval vs generation metrics, you tune blind (09).
• Ignoring freshness/ACL. Stale docs give wrong answers; missing ACL leaks data (10, Phase 8.09).

How experts reason: they treat RAG as a retrieval problem with a generation step on top: maximize recall (hybrid + good chunking), then precision (reranking), then pack the window thoughtfully (order, dedup, budget), and finally make the generator answer only from context with citations. They eval retrieval and generation separately and build a feedback loop (10).

What matters in production: retrieval quality (recall@k/precision), faithfulness (no hallucination), citations/grounding, freshness (re-ingestion), ACL-aware retrieval, latency, and cost (10).

How to debug/verify: when an answer is wrong, first check was the right chunk retrieved? If no → fix retrieval (chunking/embeddings/hybrid/rerank). If yes but the answer is wrong → fix generation (prompt/grounding) (09). This retrieval-vs-generation split is the RAG debugging move.

Questions to ask: Is the corpus large/changing/private (RAG) or small/fixed (stuff it)? What's my retrieval recall? Hybrid + rerank? Citations enforced? Freshness + ACL handled?

What silently gets expensive/unreliable: poor retrieval (confident wrong answers), no reranking (noisy context), stuffing too many chunks (cost + lost-in-the-middle), stale corpus, and unfiltered ACL (data leak).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• RAG = retrieve relevant chunks at query time + inject into the prompt + generate (what-happens §3.4).
• Two pipelines: offline ingestion (load→chunk→embed→store) and online query (retrieve→rerank→pack→generate→cite).
• Retrieval dominates quality — wrong chunks ⇒ no model can save you (Phase 5.05).
• Hybrid search (semantic + BM25) usually beats pure semantic for recall (05).
• Reranking (cross-encoder) is high-ROI precision on the top-k (06).
• RAG is for knowledge; fine-tuning is for behavior — don't fine-tune to teach fresh facts (Phase 13).
• Evaluate retrieval and generation separately — it's the key debugging split (09).
• Production needs freshness + ACL-aware retrieval (10, Phase 8.09).

9. Observations from Real Systems

• Most "enterprise GPT" products are RAG over internal docs (Confluence/Notion/SharePoint/PDFs) with citations (10).
• Coding assistants RAG over the repo (symbol/AST + embeddings) to ground edits (Phase 11).
• Agentic/tool-based retrieval (grep/read on demand) is RAG without a vector DB — same principle (what-happens §3.4).
• Frameworks (LlamaIndex, LangChain) provide the pipeline; Ragas provides eval — but the quality comes from your retrieval tuning, not the framework.
• The recurring production failure is a retrieval miss masquerading as a model hallucination — caught only by separating retrieval vs generation eval (09).

10. Common Misconceptions

11. Engineering Decision Framework

KNOWLEDGE PROBLEM?
  small + fixed corpus that fits the window → STUFF it (no RAG needed)
  behavior/format/style (not facts)         → consider FINE-TUNE [Phase 13]
  large / changing / private + need citations → RAG (this phase)

BUILD RAG (then optimize where eval shows a gap):
 1. INGEST: load/parse [01] → chunk [02] → embed [03] → store + metadata(ACL/freshness) [04]
 2. RETRIEVE: dense + sparse (hybrid) for RECALL [05]
 3. RERANK: cross-encoder for PRECISION [06]
 4. PACK: fit top chunks in budget; order for lost-in-the-middle; dedup [07]
 5. GENERATE: answer ONLY from context, with CITATIONS [08]
 6. EVALUATE: retrieval (recall@k) AND generation (faithfulness) separately [09]
 7. PRODUCTIONIZE: ACL-aware retrieval, freshness/re-ingestion, feedback loop [10]

12. Hands-On Lab

Goal

Build a minimal end-to-end RAG pipeline, then prove the central lesson by showing a wrong answer is a retrieval failure, not a model one.

Prerequisites

• pip install openai chromadb; an embedding+chat API (or local, Phase 6.04); a folder of ~10 markdown/text docs.

Steps

1. Ingest: load docs → chunk (~300 tokens, overlap) → embed → store in Chroma with source metadata (01–04).
2. Query: embed the question → retrieve top-k → format chunks with [SOURCE n] → generate with a grounding system prompt ("answer only from sources, cite [SOURCE n], else say you don't know") (08).
3. Test 10 Q&A: ask 10 questions you know the answers to; record answer + cited sources.
4. The key diagnostic: for any wrong/"don't know" answer, inspect the retrieved chunks. Was the correct chunk retrieved? If no → it's a retrieval failure (try bigger/overlapping chunks, a better embedding model, or add keyword search [05]). If yes but the answer is wrong → it's a generation failure (strengthen the grounding prompt). Log which bucket each failure falls in.
5. Quick win: add a tiny rerank (or just retrieve more then keep the best) and re-measure (06).

Expected output

A working RAG Q&A pipeline with citations, plus a failure analysis splitting errors into retrieval vs generation — demonstrating that retrieval dominates.

Debugging tips

• "Don't know" despite the info existing → retrieval miss (chunking/embeddings/hybrid).
• Confident wrong answer → either retrieval miss or weak grounding; check the chunks first.

Extension task

Add hybrid search (BM25 + vector with RRF) and show recall improves on exact-term queries (names/IDs) (05).

Production extension

Add source/acl/updated_at metadata and filter retrieval by ACL + freshness; wire a thumbs-up/down feedback loop (10, Phase 8.09).

What to measure

Per-question: was the right chunk retrieved? answer correct? cited? Failure bucket (retrieval vs generation).

Deliverables

• A working RAG pipeline with citations.
• A 10-question test with answers + sources.
• A retrieval-vs-generation failure analysis + one improvement (hybrid or rerank) with before/after.

13. Verification Questions

Basic

1. What problem does RAG solve, and how (one sentence)?
2. Name the stages of the ingestion and query pipelines.
3. Why does "retrieval dominates quality"?

Applied 4. RAG vs stuffing vs fine-tuning vs long-context — when do you choose each? 5. A user gets a hallucinated answer despite the info existing. What's your first diagnostic step?

Debugging 6. Exact-term queries (an error code, a product name) fail. What retrieval fix? 7. The right chunk is retrieved but the answer is still wrong. Where's the bug?

System design 8. Design a RAG pipeline for an internal docs assistant with citations, ACL, and freshness.

Startup / product 9. Why is RAG (not fine-tuning) the default for a product that must answer from a customer's changing private data?

14. Takeaways

1. RAG = retrieve the right slice at query time + generate grounded, cited answers — for large/changing/private knowledge.
2. Two pipelines: offline ingestion (load→chunk→embed→store), online query (retrieve→rerank→pack→generate→cite).
3. Retrieval dominates quality — fix chunks/embeddings/hybrid/rerank before blaming the model.
4. RAG is for knowledge; fine-tuning is for behavior.
5. Evaluate retrieval and generation separately, and productionize with freshness + ACL + feedback.

15. Artifact Checklist

• A working end-to-end RAG pipeline with citations.
• A 10-question test (answers + sources).
• A retrieval-vs-generation failure analysis.
• One retrieval improvement (hybrid or rerank) with before/after.
• A RAG-vs-alternatives decision note for your use case.

Up: Phase 9 Index · Next: 01 — Document Ingestion