Interview Prep 05 — RAG Engineer

Specialist in retrieval quality, grounding, and RAG evaluation. Phases: 9, 12.03. Cheatsheet: 14.

1. Core concepts

  • The pipeline: ingest → chunk → embed → store → retrieve → rerank → pack → generate → cite.
  • Embeddings + ANN (HNSW/IVF); hybrid (dense + BM25 + RRF).
  • Chunking (size/overlap/structure); lost-in-the-middle / context packing.
  • Reranking (cross-encoder): retrieve-wide-then-rerank-narrow.
  • Grounding, citations, faithfulness; "say I don't know."
  • Retrieval eval vs generation eval (the quadrant).
  • RAG vs fine-tuning (knowledge vs behavior).

2. System design questions

  1. Design RAG over 10M docs, multi-tenant, freshness-sensitive (template 03).
  2. Design RAG eval that tells you whether retrieval or generation is failing.
  3. Reduce hallucination in a high-trust domain. (Grounding, citations, faithfulness gate, human approval.)

RAG diagram.

3. Debugging questions

  • "I don't know" but doc exists → retrieval miss (chunking/embedding/hybrid/top-k) (exercise RA1).
  • Right chunks, wrong answer → generation/packing (lost-in-the-middle).
  • Cross-tenant leak → missing retrieval filter (Phase 14.04).
  • Stale answers → ingestion freshness / re-index gap.

4. Coding exercises

  • Build ingestion + retrieval + grounded generation with citations (labs/lab-03).
  • Implement hybrid search (dense + BM25 + RRF) (labs/lab-09).
  • Implement a retrieval eval (recall@k, MRR) and a faithfulness check.

5. Architecture exercises

  • Draw the RAG architecture incl. the per-tenant filter.
  • Design re-indexing/freshness and ACL-at-retrieval.

6. Model selection questions

  • Choose an embedding model (and why query+doc must match).
  • Choose a generation model for grounded answers (faithfulness, cost, latency).

7. Production incident questions

  • Recall dropped after a chunking change → eval gate caught (or should have); roll back.
  • Hallucination spike → grounding prompt regressed; faithfulness eval; tighten.
  • Cross-tenant doc surfaced → disable, fix filter, audit, leak test.

8. Portfolio project

A production-grade RAG system: hybrid search + rerank + citations + per-tenant isolation; a retrieval+generation eval quadrant with numbers (recall@k, MRR, faithfulness, citation accuracy); cost per query; failure analysis. README with the metrics.

9. 30-minute drill

  • 5 min: the RAG pipeline + retrieval-vs-generation eval.
  • 10 min: design RAG for a corpus (whiteboard).
  • 10 min: debug "I-don't-know" + "right chunks wrong answer."
  • 5 min: chunking + reranking + hybrid tradeoffs.

10. 2-hour take-home

Build RAG over provided docs with hybrid search + rerank + grounded citations; evaluate retrieval (recall@k) and generation (faithfulness) on 10–15 questions; writeup of failure modes, chunking choice, and how you'd isolate tenants + keep the index fresh.

11. Senior-level expectations

  • Junior: basic embed-retrieve-generate.
  • Mid: hybrid + rerank + citations + a basic eval.
  • Senior: splits retrieval vs generation eval, diagnoses which half fails, handles isolation/freshness/ACLs, and tunes chunking/packing with data.
  • Principal: sets RAG architecture + eval standards; designs the retrieval layer as a reusable, secure, measured platform.

Next: 06 — Agent Engineer · Index: interview-prep/