« System Design Index · Agentic AI Engineer · Cheat Sheet · Glossary

04 — Design a Production RAG System at Scale (Hybrid, Rerank, GraphRAG)

The archetype: Citi (Agentic AI Technical Lead — "GraphRAG, LightRAG, RAPTOR, Neo4j, pgvector, advanced retrieval") and Wolters Kluwer (regulated-domain knowledge platform). The prompt: "Design retrieval for an enterprise agent over 50M+ chunks across many tenants — hybrid search, reranking, freshness, security-trimming, and knowing when to reach for GraphRAG or RAPTOR — with real numbers for recall, latency, and cost." This walkthrough explodes the retrieval component and the data/storage plane from Walkthrough 01. It builds on Phase 05 (retrieval foundations) and Phase 06 (GraphRAG/RAPTOR).

Table of Contents

1. Step 1 — Requirements & scale
2. Step 2 — The contract
3. Step 3 — Architecture
4. The ingestion pipeline
5. The query lifecycle (hybrid → rerank → pack)
6. When to reach for GraphRAG / RAPTOR / LightRAG
7. Freshness & incremental updates
8. Security: ACL / tenant trimming
9. Vector DB choice
10. Reliability
11. Evaluation
12. Cost & latency
13. Observability
14. Tradeoffs
15. Failure modes
16. Staff vs junior recap

Step 1 — Requirements & scale

• Corpus: ~50M chunks across ~40 tenants (policies, contracts, tickets, wikis, filings). Mixed freshness — some documents update hourly (tickets), some are static (regulations).
• Query load: ~300 retrieval QPS peak, embedded inside agent steps (so retrieval latency is on the agent's critical path, multiplied by steps).
• Quality: high recall@k (the answer's evidence must be in the retrieved set) and high groundedness (the answer cites it). In a regulated domain, an ungrounded answer is a liability, not just a miss.
• Latency: retrieval p95 < 300 ms end-to-end (embed + search + rerank + trim), because it runs multiple times per agent task.
• Security: hard ACL trimming — a user must never retrieve a chunk they can't read, across or within a tenant. This is the #1 leak channel.
• Freshness: new/updated documents searchable within minutes; deletions honored immediately (right-to-be-forgotten, permission revocation).

Step 2 — The contract

POST /v1/retrieve
  headers: Authorization: Bearer <OIDC>          # tenant + principal come from here
  body: { query, k: 8, filters?: {source, date_range}, mode?: "hybrid"|"graph"|"raptor" }
  -> { chunks: [{id, text, score, source, doc_id, acl_ok: true}], usage: {latency_ms, ...} }

POST /v1/ingest                                   # async; returns ingest_job_id
  body: { doc, source, acl: {readers}, tenant_id, effective_date }

The contract's non-negotiable: tenant_id and the principal's ACL come from the validated token, never the request body — retrieval filters are applied server-side from the Envelope. The mode lets the agent (or the router) pick the retrieval strategy per question type.

Step 3 — Architecture

   INGEST PATH (async, batch + stream)
   docs ─▶ [ parse ] ─▶ [ chunk ] ─▶ [ embed ] ─▶ [ index writers ]
                                                     ├─▶ VECTOR DB (dense, per-tenant namespace)
                                                     ├─▶ BM25 / lexical index (per-tenant)
                                                     ├─▶ GRAPH store (entities/relations, Neo4j)   [optional]
                                                     └─▶ RAPTOR tree (summary levels)              [optional]
                                        └─▶ ACL + metadata written alongside every chunk

   QUERY PATH (sync, on the agent's critical path)
   query ─▶ [ gateway: tenant+ACL from token ] ─▶ [ retriever ]
                                                     ├─ dense ANN  ─┐
                                                     ├─ BM25       ─┤─▶ [ RRF fuse ] ─▶ [ ACL trim ]
                                                     └─ (graph/raptor) ┘                    │
                                                                              [ cross-encoder RERANK ]
                                                                                        │
                                                                              [ pack to token budget ]
                                                                                        ▼
                                                                                 chunks → agent

Two decoupled paths: an async ingestion pipeline (heavy, batched, eventually consistent) and a sync query pipeline (fast, on the agent's hot path). They meet at the indexes, which always carry tenant_id + ACL alongside every chunk so trimming is possible.

The ingestion pipeline

1. Parse — extract text + structure from PDFs, HTML, tickets (layout matters; a bad parse poisons everything downstream).
2. Chunk — split into retrievable units. Size/overlap is a tradeoff (Phase 05): too big dilutes relevance and wastes context tokens; too small loses context. Typical ~256–512 tokens with ~10–20% overlap; prefer structure-aware splits (by heading/section) over blind fixed windows. 2b. Attach metadata + ACL — tenant_id, doc_id, source, effective_date, and the reader set (who may see this chunk). ACL travels with the chunk — you cannot trim what you didn't store.
3. Embed — a bi-encoder produces a dense vector per chunk (fast, batched). Store normalized so cosine = dot product.
4. Index — write to the dense vector index (per-tenant namespace), the BM25/lexical index, and — if this tenant uses advanced retrieval — the graph store and/or RAPTOR tree.
5. Freshness bookkeeping — record document versions so updates re-index the delta and deletions purge from every index (see Freshness).

Ingestion is where cost and quality are set: a good chunking + metadata strategy makes the query path cheap and accurate; a bad one can't be fixed by a better model.

The query lifecycle (hybrid → rerank → pack)

The pipeline every senior RAG design uses (Phase 05):

1. Gateway resolves tenant_id + ACL from the token, opens a span.
2. Dense retrieval — embed the query, ANN search the tenant's namespace for top-N (say 50). Fast, catches paraphrase/semantic matches.
3. Lexical retrieval (BM25) — top-N by k1 ≈ 1.5, b ≈ 0.75. Catches exact tokens, IDs, codes, rare terms that dense retrieval fuzzes away. Hybrid beats either alone — this is the single most important production RAG lesson.
4. RRF fusion — combine the two ranked lists with Reciprocal Rank Fusion, \\(\sum 1/(k+\text{rank})\\), k ≈ 60. RRF is score-scale-free, so you don't have to reconcile BM25 scores with cosine similarities.
5. ACL trim — drop any fused candidate the principal can't read. Do it after fusion but before rerank so you don't waste the expensive reranker on forbidden chunks. (Or push the ACL as a pre-filter into the index if the DB supports it — usually the better latency/security choice.)
6. Rerank — a cross-encoder scores the query against each surviving candidate jointly (slow but accurate), reducing top-50 → top-8. The bi-encoder retrieves; the cross-encoder reranks — the two-stage pattern that lifts precision without paying cross-encoder cost over the whole corpus.
7. Pack — assemble the top-8 into the context within the token budget, best evidence at the edges (mitigate lost-in-the-middle, Phase 04), each with a citation handle for groundedness.

Numbers to quote: hybrid + rerank typically lifts recall@10 well above dense-only on enterprise corpora; RRF k = 60; retrieve 50, rerank to 8. The cross-encoder is the latency long pole — budget for it and cap the candidate count.

When to reach for GraphRAG / RAPTOR / LightRAG

Hybrid + rerank answers local questions ("what does clause 7 of contract X say?"). It fails on global/thematic questions ("what are the recurring risk themes across all Q3 filings?") because no single chunk contains the answer — it must be synthesized across many. That's when you climb the ladder (Phase 06, the raptor-graphrag lab):

Staff vs junior: a junior reaches for GraphRAG because it's impressive. A Staff engineer says: "GraphRAG's build cost is an LLM pass over the entire corpus to extract entities/relations — that's a real bill and it goes stale on every update. I use hybrid+rerank for the 80% that are local questions, add RAPTOR where abstraction helps, and reserve GraphRAG for genuinely global/multi-hop questions on a stable corpus — or LightRAG if it also needs to stay fresh." Naming the build cost and the staleness is the signal.

Freshness & incremental updates

• Updates: re-embed and re-index only the changed chunks (delta), versioned by doc_id. Stream high-churn sources (tickets) through a near-real-time path; batch static ones.
• Deletions/revocations: must purge from every index (dense, BM25, graph, RAPTOR) and honor immediately — a revoked permission or a deleted document that's still retrievable is a security incident, not a freshness bug.
• GraphRAG staleness is a first-class problem: a full re-extraction is expensive, so use LightRAG's incremental model or scheduled partial rebuilds, and know that graph answers lag the corpus.
• Eventual consistency: the ingest path is async, so define and monitor the freshness SLA (e.g. "searchable within 5 minutes") rather than pretending it's instant.

Security: ACL / tenant trimming

The highest-stakes part of enterprise RAG — the shared vector index is the #1 multi-tenant leak channel (Phase 13):

• Per-tenant namespace — physically or logically separate index per tenant; the query is scoped to the caller's namespace from the token, default-deny.
• Row/chunk-level ACL trimming — within a tenant, filter to chunks the principal may read, enforced server-side. Prefer pre-filtering in the index (secure and often faster) over post-filtering (which risks over-fetching and leaking via counts/latency side channels).
• Never trust the request body for identity or filters — a client-supplied tenant_id is an attack.
• Test it: a CI isolation test queries as tenant/principal A and asserts zero chunks belonging to B ever surface. Untested isolation is not isolation.
• Injection surface: retrieved chunks are untrusted data — a poisoned document can carry an indirect prompt injection into the agent's context. The retrieval layer scans/marks retrieved content and the agent runtime never treats retrieved text as instructions (Phase 10).

Vector DB choice

State the tradeoff, don't cargo-cult a brand (Phase 05):

For this enterprise brief: pgvector if the corpus and QPS fit and you value ACL/transactionality next to the vectors (the regulated-domain win — your permissions live in the same DB); a managed ANN (Pinecone/Weaviate) if you're past pgvector's comfortable scale; Neo4j alongside for the graph tier. The ANN index itself is HNSW (great recall/latency, higher memory) or IVF/PQ (lower memory, tunable recall) — name the recall-vs-memory-vs-latency knob.

Reliability

• Retrieval sits inside the agent loop, so its reliability compounds into the agent's 0.95^n. A retrieval miss (evidence not in top-k) is an agent failure, not a soft one.
• Fallbacks: if the reranker times out, return the RRF-fused top-k un-reranked (degraded but usable); if the vector DB is down, BM25-only; define the ladder.
• Idempotent ingest keyed by doc_id+version so re-processing doesn't duplicate chunks.

Evaluation

Retrieval and generation are evaluated separately (Phase 11):

• Retrieval metrics: recall@k (is the needed evidence retrieved?), precision@k, MRR, nDCG — measured against a golden set of query→relevant-chunk labels. Recall@k is the one that gates answer quality: if the evidence isn't retrieved, no model can ground on it.
• Generation metrics: groundedness/faithfulness (does the answer follow only from retrieved chunks?) and citation accuracy, via LLM-as-judge (κ-validated) or programmatic checks.
• Regression gate: re-run both on every change to chunking, embeddings, k, reranker, or mode; block a deploy that drops recall@k or groundedness.
• Security as a gate: an ACL-leak in the eval set fails the build outright.

Cost & latency

• Latency budget (p95 < 300 ms): embed (~10–30 ms) + dense ANN (~10–50 ms) + BM25 (~10 ms) + RRF (~1 ms) + rerank (the long pole, ~50–150 ms for 50 candidates) + pack. Cap candidate count and use a fast cross-encoder; cache query embeddings for repeats.
• Cost drivers: embedding at ingest (one-time per chunk, batched), reranker inference per query, and — for GraphRAG — the LLM extraction pass over the corpus (the big, often-forgotten line item). Amortize ingest cost; cache retrieval results for repeated queries (semantic cache, Phase 14).
• Token cost downstream: retrieval controls how many tokens land in the prompt — good reranking means fewer, better chunks, which is both a quality and a cost win.

Observability

• Span per stage (embed, dense, lexical, fuse, trim, rerank, pack) with latency + candidate counts — you find the long pole and the recall cliff.
• Log the trimmed set size — a sudden drop can mean an ACL misconfig (silent under-retrieval) or an index problem.
• Freshness dashboard (ingest lag), recall@k on a live sampled set, and ACL-trim counts as SLOs. Monitor for retrieval drift as the corpus grows.

Tradeoffs

Failure modes

• Cross-tenant / cross-principal leak via shared index or missing ACL filter → per-tenant namespace + server-side ACL pre-filter + CI isolation test. (Highest severity.)
• Low recall (evidence not retrieved) → hybrid (BM25 catches the IDs dense misses) + rerank; measure recall@k and gate on it.
• Stale answers → freshness SLA + incremental re-index + immediate deletion purge across all indexes.
• GraphRAG staleness / cost blowout → LightRAG incremental or scheduled rebuilds; reserve GraphRAG for stable corpora; budget the extraction pass.
• Indirect injection via poisoned document → treat retrieved text as untrusted data; scan; never execute instructions from chunks.
• Reranker timeout → degrade to RRF-fused top-k; define the fallback ladder.
• Lost-in-the-middle on big packed contexts → rerank hard, pack fewer/better chunks, best at the edges.
• Chunking regressions silently drop quality → regression gate on recall@k/groundedness.

Staff vs junior recap

• A junior does dense-only vector search; a Staff engineer does hybrid + RRF + cross-encoder rerank and can say why BM25 still matters (IDs, rare tokens) and quote recall@k.
• A junior reaches for GraphRAG to look advanced; a Staff engineer knows its build cost and staleness and reserves it for global/multi-hop questions on stable corpora (or LightRAG for fresh ones).
• A junior filters ACLs in application code after retrieval; a Staff engineer pre-filters in the index from the token and writes a CI isolation test — the shared index is the #1 leak channel.
• A junior picks a vector DB by brand; a Staff engineer picks by the ACL/transactionality/scale/ residency tradeoff and names the HNSW-vs-IVF/PQ knob.

Next: the eval and safety platform that gates all of this, in Walkthrough 05 — Eval & Safety Platform; or return to the enterprise platform that embeds this retriever.