Phase 9 — RAG (Retrieval-Augmented Generation)
How to give an LLM knowledge it doesn't have — your private, changing, large corpus — by retrieving the right pieces at query time and generating grounded, cited answers. The full pipeline: ingestion → chunking → embeddings → vector DB → hybrid search → reranking → context packing → citations/grounding → evaluation → production.
Why this phase matters
The model only knows its training data plus what's in the prompt (Phase 1, what-happens §0). RAG is the dominant production pattern for "the model needs to know our stuff" — with citations and freshness, without retraining. The one lesson that governs the whole phase: retrieval dominates quality — a frontier model fed the wrong chunks is confidently wrong; a cheap model fed the right chunks is correct (Phase 5.05). So most of the work is getting the right chunks to the top, then making the generator answer only from them.
Documents
| # | Document | What you'll be able to do |
|---|---|---|
| 00 | RAG Overview | See the two-pipeline model; choose RAG vs stuffing vs fine-tune; debug retrieval-vs-generation |
| 01 | Document Ingestion | Parse messy sources, attach ACL/freshness metadata, ingest incrementally |
| 02 | Chunking | Split for precision+recall; structure-aware; parent–child; the cheapest big win |
| 03 | Embeddings | Pick + use an embedder right (metric, asymmetric prefixes, MTEB, re-embed) |
| 04 | Vector Databases | ANN (HNSW/IVF), recall/latency tuning, metadata-filtered (ACL) search |
| 05 | Hybrid Search | Combine dense + BM25 via RRF; fix exact-term recall |
| 06 | Reranking | Retrieve wide, rerank to a precise few (cross-encoder); the precision stage |
| 07 | Context Packing | Budget, order (lost-in-the-middle), dedup, query rewriting/HyDE, compression |
| 08 | Citations and Grounding | Answer only from sources, cite + verify, refuse — kill hallucination |
| 09 | RAG Evaluation | Measure retrieval vs generation separately; the metric quadrant; eval gate |
| 10 | Production RAG | ACL-aware retrieval, freshness, feedback loop — the internal-docs-assistant capstone |
How to work through it
Read 00 (the map) first — it frames the two pipelines and the "retrieval dominates" principle. Then follow the pipeline in order: 01–04 build the index (ingest → chunk → embed → store), 05–07 are the retrieval-quality core (hybrid recall → rerank precision → pack), 08 makes generation trustworthy (grounding + citations), 09 is the measurement discipline that ties it all together (and the daily debugging tool: retrieval miss or generation failure?), and 10 is the production capstone. Every doc opens with a from-zero plain-English primer and ends with a measured lab; together they build the internal-docs assistant. RAG composes with serving (Phase 7), gateways (Phase 8), eval (Phase 12), and security (Phase 14) — cross-links are explicit.
Phase 9 artifacts
- A working end-to-end RAG pipeline with citations + a retrieval-vs-generation failure analysis (00).
- An ingestion pipeline (parsing + metadata/ACL + incremental/dedup) (01).
- A chunking recall@k comparison + enrichment (02); an embedder recall@k A/B + prefix lesson (03).
- A recall@k-vs-latency curve + ACL-filtered search on a real vector DB (04).
- A dense vs sparse vs hybrid recall comparison (05) + a rerank precision@k uplift (06).
- A packing-variant comparison (k/order/dedup/HyDE) (07) + a grounded, cited, verified generator (08).
- A golden set + quadrant eval + CI regression gate (09).
- The flagship internal-docs assistant: ACL + freshness + citations + eval + feedback (10).