Production RAG

Phase 9 · Document 10 · RAG Prev: 09 — RAG Evaluation · Up: Phase 9 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

This is the capstone of Phase 9: taking the RAG pipeline you've built stage by stage (01–09) and making it a real, operable, trustworthy product. A prototype that answers your test questions is easy; a production RAG system that stays fresh as docs change, enforces ACL so users only see what they're allowed to, holds latency/cost SLOs under load, survives provider issues, and improves over time via a feedback loop is the actual job. Production RAG is where Phase 9 meets the serving/gateway/eval/security disciplines (Phases 7, 8, 12, 14) — and where most RAG projects fail, not on the demo but on freshness, permissions, drift, and the un-instrumented quality regressions that erode user trust. The flagship deliverable: an internal-docs assistant with citations, ACL-aware retrieval, freshness, and an eval harness.

2. Core Concept

Plain-English primer: two living pipelines + the ops around them

Production RAG is the prototype pipeline (00) hardened into two continuously-running systems plus the cross-cutting operations:

INGESTION PIPELINE (continuous):
  connectors (S3/Confluence/Notion/Slack/DB) → parse → chunk → embed → vector DB (+ metadata DB: ACL, freshness, version)
        ▲ incremental sync (hash/updated_at), dedup, deletes propagate   [01]

QUERY PIPELINE (per request, online):
  query → (rewrite/HyDE [07]) → ACL filter + dense∥sparse retrieve [04/05] → rerank [06]
        → pack (budget/order/markers) [07] → generate grounded+cited [08] → citation verify [08]
  → response

OPERATIONS (cross-cutting):
  ACL enforcement · freshness/re-ingestion · eval (offline gate + online) [09] · observability/cost [7.08/7.09]
  · feedback loop · gateway (routing/limits/policy) [Phase 8]

The four things that separate prod from prototype

1. ACL-aware retrieval (security). Users must only retrieve chunks they're authorized to see. The ACL captured at ingestion (01) becomes a hard metadata filter at retrieval (04), applied before ranking — fail closed. This is the same fail-closed data-policy discipline as the gateway's policy engine (Phase 8.09, Phase 14.04). A RAG system that leaks a doc across permission boundaries is a breach, not a bug.

2. Freshness (correctness over time). Docs change; a stale index gives wrong answers confidently. Production needs incremental re-ingestion (event-driven or scheduled) keyed on updated_at/hash, delete propagation (removed docs leave the index), and ideally freshness filtering/boosting at query time so recent versions win. Track index staleness as a metric (01).

3. Quality under change (eval + feedback loop). Corpora and models drift; an offline eval gate (09) blocks regressions on every change, while a feedback loop turns production signals (thumbs, escalations, citation clicks) into new golden examples and retrieval fixes. This is what makes RAG improve instead of decay.

4. Latency, cost, reliability (serving). Each query is a multi-step pipeline (rewrite + retrieve + rerank + generate) — manage its p95 latency and cost (Phase 7.08/7.09), cache where possible (embeddings, frequent queries, prompt prefixes Phase 7.05), and front the generator with a gateway for routing/fallback/budgets/policy (Phase 8).

The full reference architecture

DATA PLANE (ingestion)            CONTROL/QUERY PLANE                 OPS
sources → connectors              user+identity                       observability (latency/cost/quality) [7.08]
  → parse/chunk/embed [01-03]       → auth → ACL scope                eval: offline gate + online [09]
  → vector DB + metadata [04]       → query rewrite [07]              feedback loop (thumbs→golden set)
  (incremental, dedup, deletes)     → ACL-filtered hybrid retrieve [04/05]   freshness monitor / re-ingest [01]
                                    → rerank [06] → pack [07]          gateway: routing/limits/policy [Phase 8]
                                    → generate grounded+cited [08]     security/audit [Phase 14]
                                    → citation verify [08] → respond

It's a system-of-systems

Production RAG is where Phase 9 composes with the rest of the curriculum: serving (Phase 7) for latency/cost/observability, gateways (Phase 8) for routing/policy/metering in front of the generator and embedder, evaluation (Phase 12) for the quality discipline, and security (Phase 14) for ACL/PII/audit. Treat it as an application on that infrastructure, not a standalone script.

3. Mental Model

   PROTOTYPE → PRODUCT adds 4 things on top of the [00] pipeline:
     ① ACL-AWARE RETRIEVAL: ingestion ACL [01] → hard metadata FILTER at retrieval [04], fail-closed [8.09/14]
     ② FRESHNESS: incremental re-ingest (hash/updated_at), delete-propagation, recency filtering [01]
     ③ QUALITY-UNDER-CHANGE: offline eval GATE + online feedback loop (thumbs→golden set) [09]
     ④ LATENCY/COST/RELIABILITY: p95 budget, caching, gateway routing/fallback/policy [7.08/7.09/8]

   TWO continuous pipelines: INGESTION (data plane) + QUERY (online) + OPERATIONS (cross-cutting)
   system-of-systems: RAG (9) on top of serving (7) + gateways (8) + eval (12) + security (14)
   FLAGSHIP: internal-docs assistant — citations + ACL + freshness + eval harness + feedback

Mnemonic: prototype → product = ACL (security) + freshness (correctness over time) + eval/feedback (quality under change) + serving (latency/cost/reliability). RAG is an app on the serving + gateway + eval + security stack.

4. Hitchhiker's Guide

What to look for first: ACL-aware retrieval (does a user ever see an unauthorized chunk?) and freshness (does a changed/deleted doc update the index?). These are the two that turn a demo into something shippable — and the two most often skipped.

What to ignore at first: premature scale tuning and exotic retrieval research. Harden ACL, freshness, eval-gate, and observability on a modest pipeline before optimizing.

What misleads beginners:

• Shipping the prototype. It works on your test docs but lacks ACL, freshness, eval, and observability — the production realities (00).
• ACL as a post-filter (or afterthought). Must be a fail-closed filter applied during retrieval (04), from ingestion-captured ACL (01) — else cross-permission leaks (Phase 8.09).
• Static index. No incremental re-ingestion/delete-propagation → confidently stale/ghost answers (01).
• No eval gate / feedback loop. Quality silently decays as docs/models drift (09).
• Ignoring p95/cost. The multi-step pipeline blows latency/cost budgets under load (Phase 7.08/7.09).

How experts reason: they build RAG as two living pipelines + ops on the serving/gateway/eval/security stack: ACL enforced as a fail-closed retrieval filter, incremental freshness with delete-propagation, an offline eval gate + online feedback loop, caching + a gateway for latency/cost/policy, and full observability of quality, latency, and cost. They treat the corpus and golden set as living assets.

What matters in production: zero ACL leaks, index freshness (staleness metric), faithfulness/recall holding over time (eval gate + online sampling), p95 latency and cost/query within SLO, and a working feedback loop that improves retrieval.

How to debug/verify: use the 09 split (retrieval vs generation) for quality; verify ACL filters never return out-of-scope chunks (red-team it); confirm a changed/deleted doc reflects in retrieval; watch p95/cost dashboards; check the feedback loop produces new golden cases.

Questions to ask: is ACL a fail-closed retrieval filter from ingestion metadata? is re-ingestion incremental with delete-propagation? is there an eval gate + online feedback? what's p95/cost per query? is the generator behind a gateway (routing/fallback/policy)?

What silently gets expensive/unreliable: ACL leaks (breach), stale/ghost docs (wrong answers), quality drift (no gate/feedback), p95/cost blowups (multi-step pipeline), and an un-monitored corpus that rots.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Production RAG = two continuous pipelines (ingestion + query) + cross-cutting ops on the serving/gateway/eval/security stack.
• ACL-aware retrieval is a fail-closed metadata filter applied during retrieval (04), from ingestion-captured ACL (01) — a leak is a breach (Phase 8.09, Phase 14).
• Freshness requires incremental re-ingestion + delete-propagation (hash/updated_at) and recency handling — static indexes go stale (01).
• Quality decays without an offline eval gate + online feedback loop (09).
• Each query is multi-step (rewrite+retrieve+rerank+generate) — manage p95 latency and cost; cache; front with a gateway (Phase 7/8).
• Most RAG projects fail on freshness/ACL/drift, not the demo.
• It's a system-of-systems — compose with Phases 7/8/12/14, don't reinvent.
• Flagship deliverable: an internal-docs assistant with citations + ACL + freshness + eval harness + feedback.

9. Observations from Real Systems

• "Enterprise ChatGPT over our docs" products are production RAG: ACL-aware, citation-backed, freshness-maintained, eval-gated (00).
• The two most common production failures are ACL leaks (a user sees a doc they shouldn't) and stale answers (a deleted/updated doc still returned) — both ingestion/retrieval discipline (01/04).
• Teams that gate CI on a golden set + run online feedback keep quality stable as corpora and models change (09).
• RAG generators sit behind gateways for routing/fallback/budgets/policy — RAG is an app on Phase-8 infra (Phase 8).
• Caching (embeddings, frequent queries, prompt prefixes) is a standard latency/cost lever in production RAG (Phase 7.05).

10. Common Misconceptions

11. Engineering Decision Framework

PRODUCTIONIZE RAG:
 1. SECURITY: capture ACL at ingestion [01] → fail-closed ACL FILTER at retrieval [04]; PII/audit [8.09/14].
 2. FRESHNESS: incremental re-ingest (hash/updated_at) + delete-propagation + recency handling; staleness metric. [01]
 3. QUALITY: offline eval GATE in CI (faithfulness + recall@k) + online feedback loop (thumbs→golden set). [09]
 4. SERVING: manage p95 + cost; cache (embeddings/queries/prefix [7.05]); gateway routing/fallback/budgets/policy. [7.08/7.09/8]
 5. OBSERVABILITY: quality + latency + cost dashboards; alert on drift/staleness/ACL anomalies. [7.08]
 6. COMPOSE on Phases 7/8/12/14 — don't reinvent serving/gateway/eval/security.
 7. SHIP the flagship: internal-docs assistant (citations + ACL + freshness + eval + feedback).

12. Hands-On Lab

Goal

Build the flagship internal-docs assistant: a production-shaped RAG with ACL-aware retrieval, freshness (incremental re-ingestion), citations, and an eval gate + feedback hook.

Prerequisites

• The full pipeline from 01–09; a vector DB with metadata filtering (04); a small multi-"team" doc set with ACLs; the golden set from 09.

Steps

1. ACL-aware retrieval: tag docs with acl at ingestion (01); at query time pass the user's identity/groups and apply a fail-closed metadata filter so only authorized chunks are retrieved (04). Red-team: as user A, ask about a team-B-only doc; confirm it's never retrieved or cited (Phase 8.09).
2. Freshness: implement incremental re-ingestion (hash/updated_at); edit a doc and confirm answers update; delete a doc and confirm it disappears from retrieval (delete-propagation). Add an index-staleness metric (01).
3. Grounded + cited generation: wire the grounding prompt + citation verification (08) over the ACL-filtered, reranked, packed context (06/07).
4. Eval gate: run the golden-set eval (09) (recall@k + faithfulness); make a change (e.g., chunk size) and show the gate catches a regression.
5. Feedback loop: capture thumbs up/down per answer; route a few thumbs-down into new golden-set entries; show how that would drive a retrieval fix.
6. Observe: log per-query latency (p50/p95) and cost; note the multi-step pipeline's budget (Phase 7.08/7.09).

Expected output

A working internal-docs assistant that: enforces ACL (red-team passes — no leak), updates on doc changes/deletes, answers with verified citations, gates quality in CI, captures feedback, and reports latency/cost.

Debugging tips

• ACL leak → filter applied post-retrieval or not fail-closed; enforce during retrieval (04).
• Deleted doc still answered → delete didn't propagate to the index (01).

Extension task

Front the generator (and embedder) with your gateway (Phase 8) for routing/fallback/budgets/policy, and add prompt-prefix caching for the system prompt (Phase 7.05).

Production extension

Wire connectors (Confluence/Notion) with ACL + scheduled incremental sync; ship the eval gate to CI; stream feedback into a label store; add staleness/ACL-anomaly/quality alerts (Phase 7.08, Phase 14).

What to measure

ACL-leak rate (target 0), index staleness, recall@k + faithfulness (gate), p50/p95 latency, cost/query, feedback volume → golden-set growth.

Deliverables

• A flagship internal-docs assistant (ACL + freshness + citations + eval gate + feedback).
• A red-team ACL result (no cross-permission leak).
• A freshness demo (edit/delete propagate) + a regression-gate + latency/cost report.

13. Verification Questions

Basic

1. What four things separate production RAG from a prototype?
2. Why must ACL be a fail-closed filter applied during retrieval?
3. What makes an index stale, and how do you keep it fresh?

Applied 4. Design ACL-aware retrieval from ingestion metadata to query-time filtering. 5. How does the offline eval gate + online feedback loop keep quality from decaying?

Debugging 6. A user retrieves a document they shouldn't see. Where did the pipeline fail? 7. Answers reference a deleted document. Cause and fix.

System design 8. Design a production internal-docs assistant: ingestion connectors + ACL + freshness + query pipeline + eval + observability + gateway.

Startup / product 9. Why do most RAG products fail on freshness/ACL/drift rather than the demo, and how does this architecture prevent that?

14. Takeaways

1. Production RAG = two continuous pipelines (ingestion + query) + cross-cutting ops on the serving/gateway/eval/security stack.
2. Four prod additions: ACL-aware retrieval (fail-closed), freshness (incremental + delete-propagation), quality-under-change (eval gate + feedback), and serving (p95/cost/caching/gateway).
3. ACL leaks and stale answers are the top production failures — handle them at ingestion + retrieval.
4. Quality decays without an offline eval gate + online feedback loop.
5. It's a system-of-systems — compose with Phases 7/8/12/14; ship the internal-docs-assistant capstone.

15. Artifact Checklist

• A flagship internal-docs assistant (ACL + freshness + citations + eval + feedback).
• A red-team ACL result proving no cross-permission leak.
• A freshness demo (edit/delete propagate) + staleness metric.
• A CI eval gate (recall@k + faithfulness) catching a regression.
• A feedback loop + latency/cost report; generator behind a gateway.

Up: Phase 9 Index · Next: Phase 10 — Agents and Tools