« Phase 06 README · Warmup · Hitchhiker's · Deep Dive · Core Contributor · Staff Notes

Phase 06 — Principal Deep Dive: Advanced Retrieval (GraphRAG & RAPTOR)

The Deep Dive covered the mechanism. This is the system-design view: what it costs to run these in production, what breaks, and how to reason about whether the index-time spend is justified. The mechanism is elegant. The economics are brutal. A principal engineer owns the second part.

The indexing bill is the whole story

Flat vector RAG has a near-free index: embed each chunk once. RAPTOR and GraphRAG move intelligence to index time, and that intelligence is LLM calls. Do the arithmetic before you do anything else, because the arithmetic is what decides the architecture.

GraphRAG capacity math. Extraction is one-or-more LLM calls per chunk. Take a 50k-document corpus, ~10 chunks/document = 500k chunks. Entity/relation extraction typically runs the chunk text in and a structured triple list out — call it ~1.5k input + ~0.5k output tokens per chunk, and Microsoft's pipeline does gleanings (a second and sometimes third pass asking "did you miss any entities?"), so realistically 2–3 LLM calls per chunk. That is 1–1.5M LLM calls for extraction alone, before community detection, before a single community report is written, before you answer one question. At commodity mid-tier model prices that is a five-to-six-figure indexing run measured in days of wall-clock throughput against per-minute rate limits. Community reports add one summarization call per community across the Leiden hierarchy — hundreds to thousands more.

RAPTOR capacity math is an order of magnitude cheaper: one summary call per cluster per level, ≈ N/(b-1) calls total for N leaves. A 500k-chunk corpus with branching 5 is ~125k summary calls — still real money, but a fraction of GraphRAG's per-chunk extraction, and it buys synthesis rather than global thematic reasoning.

The principal-level sentence: "GraphRAG indexing is tokens_per_chunk × chunks × passes of LLM spend, amortized over queries." If you serve a million queries against a static corpus, the per-query amortized index cost is negligible and GraphRAG is a bargain. If you serve a thousand queries against a corpus that changes daily, you are re-paying that index bill constantly and the economics invert. The index cost is not a footnote; it is the first number on the whiteboard.

The incremental-update problem and why LightRAG exists

Full GraphRAG's community detection is a global computation. Leiden partitions the entire graph; adding one document can, in principle, shift community boundaries anywhere. The mathematically honest response to "a new document arrived" is "re-run Leiden and re-summarize affected communities" — and in the cautious limit, re-index. For a static corpus (a fixed set of filings, a shipped documentation snapshot, last year's incident retros) that is fine: you pay once. For a corpus that grows daily — which every enterprise knowledge base does — a full re-index per update window is operationally and financially non-viable.

This is the problem LightRAG is built to solve, and it is why it belongs on the same JD line as GraphRAG rather than being a lesser copy. LightRAG's incremental insert merges a new document's entities and edges into the existing graph and re-summarizes only the affected neighborhoods, not the whole partition. It trades hierarchical-community richness (LightRAG's dual-level split is flatter than Leiden's multi-level hierarchy) for the ability to absorb updates cheaply. The design question "full GraphRAG or LightRAG?" is therefore rarely about answer quality — it is about corpus freshness and the re-index cadence you can afford. Static + global questions → GraphRAG earns its heavy index. Growing + mixed questions → LightRAG or plain RAPTOR.

Staleness and the freshness/cost frontier

Even without incremental updates, precomputed structure goes stale. RAPTOR's summaries and GraphRAG's community reports are snapshots of the corpus at index time. Add ten documents that introduce a new theme and neither the tree nor the community summaries know it exists until you rebuild. This creates a freshness SLA you must state explicitly: "community summaries reflect the corpus as of the last nightly index." For a slow-moving corpus that is invisible; for a fast one it is a correctness bug waiting to surface as "the system doesn't know about the incident from this morning." The principal move is to make staleness a declared property with a refresh cadence and a cost line, not an emergent surprise.

Failure modes and blast radius

Precomputed structure means precomputed errors, and errors compound downstream. Rank them by blast radius:

• Bad entity extraction poisons the graph. A missed relation is an edge that does not exist — a multi-hop query silently returns nothing or the wrong path. A hallucinated relation is a false edge your agent will confidently traverse. Worse, extraction errors are systematic: a prompt that mis-handles a document format mis-extracts every document of that format, so the damage is correlated, not random. Extraction quality is the ceiling of the entire system; everything downstream inherits its mistakes.
• Entity-resolution failure fragments the graph. If Stripe, the gateway, and stripe_gateway become three nodes instead of one, the edges that should converge on one entity scatter across three, communities fracture, and a two-hop path that should exist is broken. This is the least glamorous and highest-leverage tuning surface in the whole system.
• Community-summary drift. The LLM writing a community report can over-generalize, editorialize, or hallucinate a theme the entities do not actually support. Because global queries answer from the summary text, a drifted summary is a wrong answer with no source chunk to fact-check against — the summary is the source now. This is why grounding and provenance (which entities, which triples produced this summary) matter more here than in flat RAG.
• RAPTOR mis-clustering buries a leaf under an unrelated summary. The collapsed-tree retrieval mode is the mitigation — because it never prunes, a mis-clustered leaf can still be retrieved directly on a specific query — which is a design reason to prefer collapsed over traversal in production, not just a benchmark result.

Cross-cutting concerns

• Cost is index-dominated (above). Budget $/index and $/reindex separately from $/query, and track them as distinct SLOs.
• Freshness is a declared SLA with a refresh cadence; LightRAG's incremental path is the lever when the SLA is tight.
• Observability must reach into the index build: token spend per stage, extraction yield (triples per chunk), community count and size distribution, and summary provenance. When a global answer is wrong, you need to trace it back to the community summary, then to the triples, then to the chunk and the extraction call. Flat RAG needs none of this; graph systems live or die by it.
• Multi-tenancy is a graph problem, not just a filter. Vectors partition cleanly by a tenant key. A shared knowledge graph does not — entity resolution across tenants can leak structure (tenant A's entity co-occurring with tenant B's in a merged community summary), so the safe default is a graph per tenant, which multiplies the index bill by tenant count. That multiplier is a capacity-planning line most teams discover in production.

"Looks wrong but intentional"

• The graph database is almost incidental. Storing embeddings in Neo4j buys nothing; the value is the community summaries. A design that "does GraphRAG" by loading vectors into a graph store and running nearest-neighbor has built a slower vector store. The store choice (Neo4j for traversal, pgvector for similarity) is an integration detail; the summaries are the feature.
• Connected components instead of Leiden in the miniature is deliberate: exact and deterministic for a well-separated corpus, and the idea (group related entities into summarizable themes) is identical. Production swaps in Leiden for hierarchical splitting of connected blobs — a fidelity upgrade, not a different concept.
• Collapsed tree scores every node — apparently wasteful versus log-depth traversal — because never pruning is worth more than the saved scoring. Robustness beats cleverness here.

When the index-time spend is justified

Reduce it to a decision. GraphRAG earns its indexing bill when: the workload is dominated by global/thematic or multi-hop questions (flat RAG structurally cannot answer them), the corpus is large and relatively static (so the index amortizes over many queries and you re-index rarely), and provenance/thematic reasoning is worth the operational weight. It does not earn it when the workload is mostly fact lookup (flat RAG is right and 100× cheaper to index), when the corpus churns daily (LightRAG's incremental path or plain RAPTOR wins), or when nobody has priced the re-index cadence. Default to flat vector RAG. Climb the ladder only when the question shape demands it — and bring the indexing number to the review. That number, not the architecture name, is the principal-level contribution.