Cheatsheet 14 — RAG Design
The RAG pipeline and its design knobs. Full: Phase 9. Labs: 03, 09.
The pipeline
INGEST: load → clean → CHUNK → EMBED → store (vector DB) [+ keyword index]
QUERY: embed query → RETRIEVE (vector [+ BM25]) → RERANK → PACK context → GENERATE (grounded) → CITE
Design knobs & defaults
| Stage | Knob | Default / guidance |
|---|---|---|
| Chunking | size / overlap | 200–500 tokens, ~10–20% overlap; respect structure (headings) |
| Embedding | model | A strong embedding model; match query+doc model |
| Vector DB | ANN index | HNSW (most cases); per-tenant namespaces for multi-tenant (14.04) |
| Retrieval | top-k | 10–30 candidates before rerank |
| Hybrid | dense + BM25 + RRF | Add keyword search; fuse with Reciprocal Rank Fusion |
| Rerank | cross-encoder | Rerank to top 3–8 (precision boost) |
| Packing | order / count | Put best chunks at start/end (lost-in-the-middle, 9.07) |
| Generation | grounding | "Answer only from context; cite; say 'I don't know'" |
RAG vs fine-tuning (the cardinal rule)
- RAG → knowledge (facts, current/proprietary data, citations).
- Fine-tuning → behavior (format/style/skill).
- Need facts? RAG. Need a behavior? Fine-tune. (Ladder: prompt → few-shot → RAG → fine-tune, Phase 13.00.)
Evaluate retrieval and generation SEPARATELY
| Half | Metrics |
|---|---|
| Retrieval | recall@k, precision@k, MRR, hit rate |
| Generation | faithfulness/groundedness, answer relevance, citation accuracy |
"Bad answer" = first ask did retrieval find the right chunks? (retrieval) vs did the model use them? (generation) (Phase 9.09).
Common failure modes 🚩
| Symptom | Likely cause |
|---|---|
| "I don't know" but doc exists | Retrieval miss (chunking/embedding/query mismatch) |
| Hallucinated answer | Weak grounding prompt; missing chunks; model ignoring context |
| Right chunks, wrong answer | Generation/packing (lost-in-the-middle) |
| Cross-tenant leak | Missing per-tenant retrieval filter (14.04) |
| Stale answers | Ingestion freshness/re-index gap |
Production extras
- ACLs at retrieval (user only sees permitted docs), freshness/re-indexing, feedback loop (Phase 9.10).
- HyDE / query rewriting for hard queries; prefix caching for shared context (7.05).
Next: 15 — Tool Calling · Full: Phase 9