m06 — Retrieval-Augmented Serving

A fully worked design. Search plus generation, under one latency budget, at 200 million chunks.

The finding that reorders the design: retrieval takes ~85 ms; prefilling what retrieval returned takes 207 ms. The retrieval system is not the bottleneck of the retrieval system. And a 10-chunk RAG request holds 7.2× the KV cache of a bare chat turn — so the retrieval policy is, unavoidably, a serving-capacity decision.

Table of Contents

• The Prompt
• 1. Requirements and Scope
• 2. Scale Numbers
• 3. API Surface
• 4. Data Model
• 5. High-Level Architecture
• 6. Deep Dive A: The Latency Budget Is Spent Where You Do Not Expect
• 7. Deep Dive B: Freshness Against Index Cost
• 8. Failure and Recovery
• 9. Bottlenecks and Evolution
• 10. Tradeoffs Explicitly Rejected
• The Hostile Critique
• The Revision
• References

The Prompt

"Design retrieval-augmented generation for our enterprise product. Customers connect their document stores, ask questions, and get answers grounded in their own documents with citations."

Three things in that sentence are load-bearing and easy to skim past:

1. "Their own documents" — per-tenant corpora, per-tenant access control. Not one index, many. And a retrieval bug is not a relevance problem, it is a data leak between customers.
2. "Grounded" — the answer must be supported by what was retrieved, which is a different requirement from "the model saw the documents" and needs its own mechanism.
3. "With citations" — every claim must trace to a chunk, which constrains the prompt format and the chunking.

And the thing to say in the first two minutes, because it reframes the whole design: RAG is not a search system with a model bolted on. It is a system where the search results become the model's input tokens — and tokens cost prefill time and KV memory. From §2, retrieving 10 chunks turns a 250 MiB request into a 1.77 GiB one.

The retrieval policy is a capacity decision on the GPU fleet. Anyone designing the search half without that number is designing half a system.

1. Requirements and Scope

Clarifying questions asked

"How many tenants and how big is the biggest corpus?" Assumed: 5,000 tenants, median 10k documents, largest 5M documents. A three-order-of-magnitude spread, which means one index architecture cannot serve both ends — the median tenant's index fits in a few hundred MB and the largest needs sharding. Naming that spread early prevents a design that only works for the average.

"What's the freshness requirement?" The question that decides the index architecture. Assumed minutes for edits, seconds for deletes. Deletes are the strict one and it is worth explaining why: a document revoked for access-control reasons must stop being retrievable now, and "the index refreshes hourly" is a compliance failure, not a latency one.

"Is the answer allowed to be wrong-but-fluent?" Assumed no — an ungrounded answer is worse than "I don't know", because customers cannot tell the difference and will trust it. This makes abstention a first-class output, which most RAG designs omit.

"What's the latency SLO?" Assumed TTFT p95 < 1 s end to end, matching m01. Everything — embed, search, rerank, prefill — fits in that budget, and §6 is about where it actually goes.

"Do we control the chunking?" Assumed yes. Chunking is the highest-leverage and least-discussed decision in RAG: it determines what a "citation" can point at, what fits in the budget, and whether a retrieved chunk is self-contained enough to be useful out of context.

Functional

1. Ingest tenant documents; chunk, embed, index.
2. Retrieve top-k for a query, with access control applied before ranking.
3. Rerank, assemble a prompt, generate with citations.
4. Abstain when retrieval is weak.
5. Reflect edits and deletes within the freshness SLO.

Non-functional

Explicitly out of scope

• The embedding model's training. We consume a pinned artifact — and pinning it matters (§7).
• Agentic multi-hop retrieval (retrieve → reason → retrieve). Noted in §9; it multiplies the latency budget and needs a different design.
• The generation engine — m01, ../WARMUP.md.
• Answer-quality evaluation methodology — m05.

2. Scale Numbers

The corpus. 50M documents → ~200M chunks at ~500 tokens each.

Index size, which decides where it lives:

Quantization is not an optimization here, it is what makes the index exist. 819 GB of fp32 vectors does not fit in RAM on any sane node; 12.8 GB of PQ codes fits on a laptop. The recall cost of PQ is real (a few points) and is recovered by reranking the top candidates with exact vectors — which is why the architecture has a rerank stage at all, and saying that connects two design decisions that are usually presented separately.

Now the number that reorders the design. The retrieved chunks become prompt tokens:

A 10-chunk RAG request holds 7.2× the KV of a bare chat turn. At fixed hardware that is 7.2× fewer concurrent requests. The retrieval k is a direct multiplier on the serving fleet size, and doubling k for a point of recall roughly doubles the GPU bill.

The latency budget, itemized — this is deep dive A:

embed query                          5 ms
vector search (HNSW, ef=128)        20 ms
rerank top-100 (cross-encoder)      50 ms
fetch chunk text                    10 ms
PREFILL 5,800 tokens               207 ms   <-- 62% of the pre-token budget
queue + overhead                    40 ms
                                   -----
                                   332 ms   (668 ms headroom against 1 s)

Retrieval is 85 ms; prefilling its output is 207 ms. Optimizing the vector index is optimizing 26% of the budget while the thing it feeds consumes 62%.

Index build cost:

re-embed 200M chunks at ~5,000 chunks/s/GPU = 11.1 GPU-hours
  on 16 GPUs: 42 minutes    |    on 64 GPUs: 10 minutes

Cheap enough to rebuild the whole index in under an hour — which is a genuinely important fact, because it means an embedding-model upgrade is a routine operation rather than a migration project. That shapes §7 substantially.

3. API Surface

POST /v1/answer
  { tenant_id, query, filters{}, k, cite: true, stream: true }
  -> SSE: retrieval_meta, then tokens, then citations
  -> 200 { answer, citations[{chunk_id, doc_id, span, score}], abstained: false }
  -> 200 { abstained: true, reason: "no_relevant_context", best_score: 0.31 }

POST /v1/documents        {tenant_id, doc_id, content, acl[], metadata{}}  -> 202
DELETE /v1/documents/{id}                                                  -> 204  (< 5 s visible)
GET  /v1/retrieve         # retrieval only, no generation -- for debugging and eval

Four decisions:

abstained is a normal 200 response with a reason, not an error. Abstention is a correct outcome when nothing relevant was found, and modelling it as an error means clients retry it — which is exactly wrong, since retrying will retrieve the same nothing. Making abstention first-class is what stops the system from confabulating, and it needs to exist in the API or it will not exist in the implementation.

/v1/retrieve exists separately. Retrieval quality and generation quality fail differently and must be measurable separately. Without this endpoint, "the answer was wrong" is unattributable — and the first question in every RAG post-mortem is did we retrieve the right thing?

Retrieval metadata streams first, before tokens. The client can show "searching 3 documents…" during the 207 ms prefill, which converts dead time into perceived progress. Cheap, and it is the kind of product-aware detail that distinguishes a system designer from a component designer.

DELETE is synchronous to the point of invisibility, not to the point of index removal. The distinction matters and §7 depends on it: the delete returns once the chunk is filtered out of results, which is fast; physical removal from the index happens later.

4. Data Model

document   (tenant_id, doc_id, uri, content_digest, acl[], updated_at, version)
chunk      (tenant_id, chunk_id, doc_id, doc_version, ordinal, text,
            token_count, embedding_ref, acl_digest)
index_seg  (tenant_id, seg_id, kind: hnsw|flat, vector_count, built_at,
            embed_model_ref, tombstones: roaring_bitmap)
tombstone  (tenant_id, chunk_id, deleted_at)          -- the fast-delete path
query_log  (query_id, tenant_id, query, retrieved[], scores[], answered, abstained)

acl_digest on the chunk, not just on the document. Because filtering must happen inside the search, not after it — post-filtering a top-100 can return fewer than k visible results, or zero, and the user sees "no results" for documents they can see. Pre-filtering by ACL inside the index traversal is the correct design and it requires the ACL to be available at the vector level.

doc_version on the chunk. An edited document produces new chunks with a new version; old chunks are tombstoned. This makes edits atomic at the document level — a query never sees a mix of old and new chunks of the same document, which would produce a confidently contradictory answer.

tombstones as a roaring bitmap on the segment. This is the mechanism that makes deletes fast: the graph is not modified, the result set is filtered during traversal. 200M chunks with 1% deleted is ~2M IDs, which a roaring bitmap holds in a few MB and tests in nanoseconds. HNSW cannot delete a node without degrading the graph, so nobody does — everyone tombstones, and knowing that is knowing how vector databases actually work.

query_log with retrieved[] and scores[] is not telemetry, it is the eval set. Real queries with their retrieved chunks are the only way to measure recall on the traffic you actually get. Sample them, label them, and that is the internal benchmark from §1 — otherwise you are tuning against a synthetic set that resembles nothing.

5. High-Level Architecture

  INGEST                                        QUERY
  ──────                                        ─────
  document                                      query + tenant
     │                                             │
  ┌──▼──────────────┐                     ┌────────▼─────────┐
  │ chunk           │                     │ embed (5 ms)     │
  │ (structure-aware│                     └────────┬─────────┘
  │  + overlap)     │                              │
  └──┬──────────────┘                     ┌────────▼──────────────────────┐
  ┌──▼──────────────┐                     │ SEARCH  (20 ms)                │
  │ embed (batched) │                     │  HNSW over PQ codes, ef=128    │
  └──┬──────────────┘                     │  ACL PRE-FILTER in traversal   │
  ┌──▼──────────────┐                     │  + BM25 lexical, fused (RRF)   │
  │ write chunk row │                     └────────┬───────────────────────┘
  │ + buffer segment│                     ┌────────▼─────────┐
  └──┬──────────────┘                     │ RERANK top-100   │  50 ms
     │  (every ~5 min)                    │ cross-encoder    │
  ┌──▼──────────────┐                     └────────┬─────────┘
  │ BUILD SEGMENT   │                     ┌────────▼─────────┐
  │ merge, HNSW,    │                     │ ASSEMBLE PROMPT  │  budget-aware
  │ publish atomic  │                     │ + abstain check  │
  └─────────────────┘                     └────────┬─────────┘
                                          ┌────────▼─────────┐
     DELETE ──> tombstone bitmap (< 5 s)  │ GENERATE (m01)   │  prefill 207 ms
                (no index mutation)       │ + citation bind  │
                                          └──────────────────┘

Five decisions:

1. Segments are immutable; the index is a list of segments plus tombstones. Same shape as d07 and every LSM tree. It makes deletes cheap, publishes atomic, and rebuilds safe. Mutable ANN indexes are where correctness bugs live, because a graph being modified during traversal has no clean semantics.

2. Hybrid retrieval — dense + lexical, fused. Dense embeddings fail on exact identifiers (error codes, part numbers, names) — precisely what enterprise users search for. BM25 fails on paraphrase. Reciprocal Rank Fusion combines them with one parameter and no training: score = Σ 1/(60 + rank_i). Choosing hybrid unprompted is a strong signal; dense-only is the answer of someone who has read about RAG rather than shipped it.

3. ACL filtering happens inside the traversal, not after. Post-filtering returns short or empty result sets for users with restricted views. The cost is that the index must carry ACL bits.

4. Rerank with a cross-encoder over ~100 candidates. The first stage optimizes recall cheaply over 200M; the second optimizes precision expensively over 100. This two-stage shape is what makes both PQ quantization and a small k affordable — and it is the direct answer to §2's finding that k is a GPU-fleet multiplier: better ranking lets you send fewer chunks.

5. Prompt assembly is budget-aware and it is a real component, not string concatenation. It fits chunks into a token budget, orders them, and decides whether to abstain. §6.

6. Deep Dive A: The Latency Budget Is Spent Where You Do Not Expect

The measurement first

From §2:

The retrieval pipeline is 85 ms. Prefilling what it returns is 207 ms.

The consequence, and it is the whole deep dive: the highest-leverage optimization is not a faster index. It is retrieving fewer, better chunks — because every chunk you do not send saves 21 ms of prefill and 0.15 GiB of KV.

That inverts the usual instinct, which is to raise k "to be safe". Raising k from 10 to 20 costs 178 ms of TTFT and 1.5 GiB of KV per request, and buys a few points of recall that reranking would have bought for free.

Where the budget goes as a function of k

TTFT(k) ≈ 85 ms  +  k × 500 tokens / 28,036 tok/s
        = 85 ms  +  k × 17.8 ms

KV(k)   ≈ 0.24 GiB  +  k × 0.153 GiB

Both linear in k, one in latency and one in capacity. So k is a single knob that trades recall against both SLOs simultaneously — which is the sort of clean statement that makes the tradeoff arguable with numbers rather than opinions.

The optimization that actually pays:

Chunking is free recall and it is the least-discussed lever. A chunk split mid-table or mid-sentence is unusable no matter how well it ranks. Structure-aware chunking — split on headings, keep tables intact, 10–15% overlap — raises recall at zero latency cost.

Prefix caching changes the arithmetic

The system prompt and instruction preamble are identical for every request in a tenant: m02 caches them.

But the retrieved chunks are the variable part of the prompt, and they come after the fixed part — so ordering matters enormously:

[system prompt: 500 tok]  [retrieved chunks: 5,000 tok]  [query: 300 tok]
 └── CACHEABLE ──────┘     └── varies per request ────┘

prefix cache saves 500 tokens = 18 ms of the 207 ms.

Only 9% of the prefill is cacheable in this layout. But if the same document set is retrieved often — a common enterprise pattern, where most queries hit a small popular set — reordering to put frequently-retrieved chunks in a stable position makes more of the prefix cacheable:

[system]  [top-N POPULAR chunks, stable order]  [query-specific chunks]  [query]
 └────────── cacheable when the popular set is unchanged ──────────┘

Cost: the ordering is no longer relevance-ranked, and models weight position (the "lost in the middle" effect is well documented). So this trades answer quality for TTFT and needs an A/B, not an assumption. Naming both the opportunity and its risk is better than proposing it as free.

Streaming hides some of it, and be precise about which

The 85 ms of retrieval can be overlapped with nothing — it is a hard serial dependency before the prompt exists. But:

• Embedding the query can start while the request is still being parsed.
• Reranking can start on the first ANN results rather than waiting for all of them.
• Prefill can be chunked (Sarathi) so it interleaves with other requests' decode — helping fleet TPOT, not this request's TTFT.
• The retrieval metadata frame (§3) gives the user visible progress at ~85 ms instead of a blank screen for 332 ms.

None of these reduce the 207 ms. Say so plainly — pipelining is often offered as if it removes serial work, and here it removes only the parts that were never on the critical path.

7. Deep Dive B: Freshness Against Index Cost

The two clocks, and why they need different mechanisms

These require different mechanisms, and treating them as one problem — "keep the index fresh" — produces a design that is either too slow for deletes or too expensive for edits.

Deletes: tombstones, not index mutation

HNSW is a navigable small-world graph. Removing a node breaks the paths that route through it, degrading recall for unrelated queries. There is no cheap correct delete.

DELETE chunk_id:
  1. add to the segment's roaring bitmap tombstone    (microseconds, in memory)
  2. replicate the bitmap to all query replicas       (< 1 s, small)
  3. traversal skips tombstoned IDs when collecting results
  4. physical removal happens at the next segment merge

Deletes are visible in under a second and cost nothing. The price is paid later: a segment that is 30% tombstoned wastes 30% of its traversal work and its memory.

Compaction trigger: merge when tombstones / vectors > 0.2. Rebuild that segment from live chunks only. This is exactly LSM compaction and exactly d07's segment merging — the third design in this program to arrive at the same structure, which is worth saying out loud, because recognizing a recurring primitive is the transferable skill.

One subtlety worth stating: a tombstone must be applied at traversal time, not at result time. Filtering after collecting top-k means a query where all top-k are deleted returns nothing, even though the k+1..2k results are live and relevant. Skip during traversal and keep collecting.

Edits and additions: a buffer segment

New chunks cannot go into an immutable segment. The standard structure:

query fans out over:
   [ 20 large HNSW segments  ]  built hourly/daily, 200M vectors
   [ 1 small buffer segment  ]  flat (brute-force) index, < 100k vectors
   results merged by score

The buffer is flat, not HNSW, and that is deliberate. Brute-force over 100k × 64 B PQ codes is 6.4 MB of sequential scan — under a millisecond on a modern core, and exact (recall 1.0). Below roughly 100k vectors, building a graph is pure overhead: the graph exists to avoid a scan that is already cheap.

add chunk  ->  append to buffer (visible in < 1 s)
buffer > 100k or > 5 min  ->  seal, build an HNSW segment, publish atomically

Edits = delete + add, both fast paths, and the doc_version field (§4) makes the swap atomic at the document level so a query never sees half of each version.

The embedding-model upgrade, which is the real freshness problem

A new embedding model means every vector is invalid. Old and new vectors are in different spaces and their similarities are meaningless — this is not a degradation, it is nonsense.

From §2: re-embedding 200M chunks is 11 GPU-hours = 42 minutes on 16 GPUs. Cheap. So the constraint is not compute, it is the cutover, and that is a systems problem:

1. build the new index alongside the old       (42 min, 16 GPUs, $70)
2. shadow: run both, log the retrieval delta   (measures the change before it ships)
3. A/B on live traffic by tenant               (quality is a measurement, not an assumption)
4. cut over per tenant; keep the old index for rollback
5. drop the old index after a bake period

The point: because rebuild is cheap, the safe migration is affordable. If a rebuild took a week, you would be forced into a risky in-place swap. Doing the cost arithmetic first is what tells you which migration strategy you can afford — and that ordering (cost → strategy) is the generalizable lesson.

The trap to name: embed_model_ref is on the segment (§4), and queries must be embedded with the same model as the segment they search. During migration, both models must be live and the router must pair them correctly. A mismatched query/segment pair does not error — it returns plausible, wrong results, which is the same silent-corruption class as m02's cache-key problem and deserves the same defence: put the model reference in the key and refuse to mix.

8. Failure and Recovery

The tombstone-lag row is the interesting one and it is a genuine correctness requirement. A delete is an access-control action, so serving from a replica that has not applied it is a leak, not staleness. The fix is a read-your-writes guarantee scoped to deletes:

DELETE returns tombstone_version = 4821
subsequent queries carry min_tombstone_version = 4821
router sends only to replicas with applied_version >= 4821
if none: wait (bounded), then fail closed

Fail closed, deliberately — the opposite of the availability instinct, and correct here because the failure mode being prevented is disclosure, not slowness. This is a session guarantee, the same primitive as d02 and d08, applied to a security boundary.

On the cross-tenant row: the assertion is cheap and redundant with the ACL pre-filter, and it stays anyway. Defence in depth on the one failure that is unrecoverable — you cannot un-disclose a document, and the entire product is built on the promise that you will not. A redundant check on the path that would violate it is the cheapest insurance in the design.

9. Bottlenecks and Evolution

Now: prefill of retrieved context — 62% of the pre-token budget (§6). Not the index.

Interventions in order:

1. Better chunking. Free recall, zero latency cost, and the most-neglected lever (§6). Structure-aware splitting, overlap, and contextual chunk headers (prepend the document title and section path to each chunk) so a chunk is interpretable out of context.
2. Better reranking. Buys recall at k you can afford. A stronger cross-encoder costs ~20 ms and can save 100 ms of prefill by letting k drop.
3. Adaptive k. Stop adding chunks when marginal relevance collapses — if the 4th chunk scores 0.31 against the 1st at 0.89, chunks 5–10 are noise that costs 107 ms and 0.9 GiB. Most queries need 3 chunks; a few need 15. A fixed k serves neither. Highest-value change on this list.
4. Prefix caching of the popular set (§6). Real gains where retrieval is skewed; needs an A/B because of position effects.
5. Per-tenant index tiering. Small tenants (10k chunks) do not need HNSW at all — brute force over 640 KB of PQ codes is faster than a graph traversal and exact. A 5,000-tenant fleet where the median tenant needs no index is a very different system from one index for everyone, and shape-appropriate tiering is where the operational cost actually goes.
6. Multi-hop / agentic retrieval. Retrieve, reason, retrieve again. Multiplies the budget by the number of hops — 332 ms becomes ~1 s for three hops — so it needs a different SLO and probably a different product surface. Out of scope for the interactive path, natural for a "deep research" mode.

10. Tradeoffs Explicitly Rejected

Rejected: dense-only retrieval. Fails on exact identifiers, which is what enterprise users search for. Hybrid + RRF costs 5 ms.

Rejected: fp32 or fp16 vectors in the index. 819 GB / 410 GB against 12.8 GB for PQ. The recall loss is recovered by reranking exact vectors on the top-100.

Rejected: post-filtering by ACL. Returns short or empty result sets for restricted users. Pre-filter inside traversal.

Rejected: deleting nodes from the HNSW graph. Degrades routing for unrelated queries. Tombstone + compact.

Rejected: a large fixed k "to be safe". §6 — linear in TTFT and KV. Adaptive k with a relevance floor.

Rejected: generating an answer when retrieval is weak. Produces confident, ungrounded text that users cannot distinguish from grounded text. Abstain, with the best score reported.

Rejected: one shared index across tenants with a filter. A single filter bug is a cross-tenant leak. Physical separation per tenant; the cost is many small indexes, which §9 turns into an advantage.

Rejected: rebuilding the whole index on every edit. 42 minutes on 16 GPUs is cheap for a migration and absurd per edit. Buffer + segments.

Rejected: HNSW for every tenant. Below ~100k vectors, brute force is faster and exact.

Rejected: skipping the /v1/retrieve debug endpoint. Without it, retrieval and generation failures are indistinguishable and every quality investigation stalls.

The Hostile Critique

C1. "Adaptive k stops when marginal relevance collapses. Embedding similarity scores are not calibrated — 0.31 means different things for different queries, models, and corpora. What is your threshold actually thresholding, and what happens on a query where every score is 0.75?"

C2. "You pre-filter by ACL inside the HNSW traversal. HNSW navigates by following edges to nearest neighbours. If a user can see 0.1% of the corpus, the traversal walks through overwhelmingly invisible nodes to find visible ones. What is your recall for that user, and what is your latency?"

C3. "The buffer segment is flat and brute-force, sealed at 100k vectors or 5 minutes. A tenant bulk-uploads 5 million documents. That's 20M chunks. Walk me through the next hour."

C4. "Deletes route to replicas with applied_version >= X, and fail closed if none. A replica restarts and rebuilds its tombstone state from scratch. During that window its applied_version is 0. What do your queries do, and what does that look like at the fleet level during a rolling restart?"

C5. "You say prefill is 62% of the budget and that a better reranker lets you lower k. Your reranker is a cross-encoder over 100 candidates at 50 ms. Where does it run, and what happens to that 50 ms when it's competing for the same GPUs as generation?"

C6. "Citations bind claims to chunks. The model writes a sentence synthesizing three chunks. Which chunk does it cite? And if your citation-binding check fails and you 'regenerate once, then abstain' — you've now spent two full generations and the user's latency budget is gone. What do they actually see?"

The Revision

R1 — The relevance floor must be relative and calibrated per query (answers C1)

The critique is correct and this was a real defect: raw cosine similarity is not comparable across queries. A specific query about a rare term may have a top score of 0.45 with the right answer; a vague query may score 0.80 against ten irrelevant chunks. A fixed threshold is wrong in both directions.

Change: use the shape of the score distribution, never its absolute level.

def adaptive_k(scores, max_k=15):
    top = scores[0]
    keep = [0]
    for i in range(1, min(len(scores), max_k)):
        if scores[i] < RATIO * top:       # relative drop-off, e.g. RATIO = 0.6
            break
        keep.append(i)
    return keep

def should_abstain(scores, rerank_scores):
    # Two independent signals; abstain only if BOTH agree it is weak.
    flat = (scores[0] - scores[9]) < FLATNESS_EPS      # no discrimination at all
    weak = rerank_scores[0] < RERANK_FLOOR             # calibrated: see below
    return flat and weak

The all-0.75 case the critique names is exactly the flat signal — a distribution with no discrimination means the retriever found nothing distinctive, regardless of the absolute level. Flatness is the calibration-free signal, and it is the one that generalizes.

And the cross-encoder score is calibratable, which the bi-encoder score is not: a cross-encoder is trained on relevance labels, so its output can be mapped to P(relevant) with a held-out labelled set — and recalibrated per tenant, since corpora differ. So abstention keys on the reranker, not the retriever. That is the correct division of labour and the original design had it backwards.

Cost: the calibration set must exist per tenant (a few hundred labelled query-chunk pairs) and be refreshed. For tenants without one, fall back to the flatness signal alone — weaker, but calibration-free and never absurd.

R2 — Selective ACLs need partitioned indexes, not filtered traversal (answers C2)

The critique identifies a real and well-known failure of filtered ANN search, and the original design waved at it. Filtered HNSW degrades catastrophically at low selectivity: the graph's navigation is built over all nodes, so with 0.1% visible the traversal expands enormous numbers of invisible nodes, and either recall collapses (it gives up) or latency explodes (it keeps going). This is measured and published behaviour, not a theoretical concern.

Change: choose the strategy from the selectivity, and compute the selectivity at query time.

sel = estimate_selectivity(tenant, acl)      # from a small per-ACL-group cardinality sketch

if sel > 0.10:      strategy = FILTERED_HNSW      # filtering is cheap; graph is still navigable
elif sel > 0.001:   strategy = PARTITIONED        # per-ACL-group sub-index
else:               strategy = BRUTE_FORCE        # <200k visible vectors: scan PQ codes, exact

The low-selectivity case resolves itself, which is the satisfying part: a user who can see 0.1% of 20M chunks can see 20,000 chunks — 1.3 MB of PQ codes, brute-force scannable in well under a millisecond, with recall 1.0. The hard case for the graph is the easy case for the scan.

The middle band is the genuinely hard one. Materialize sub-indexes per ACL group (not per user — users share groups, and per-user indexes would be unbounded). Most enterprises have tens of groups, not thousands, so this is bounded. The cost is index duplication for documents in multiple groups, and the mitigation is to build sub-indexes only for groups above a usage threshold, falling back to filtered traversal for the long tail.

And the lesson, which generalizes: an index structure has a selectivity range where it works, and a filter is not a free composition with it. Whenever a design filters a specialized index, ask what the filter does to the index's access pattern — the answer is often that it destroys the property the index existed for.

R3 — Bulk ingest must be a different path, with its own admission (answers C3)

The critique's scenario breaks the design as written. 20M chunks against a 100k buffer means 200 sealed segments in an hour, each triggering an HNSW build. Meanwhile the buffer is repeatedly at capacity and queries brute-force it. And segment count explodes, so every query fans out over hundreds of segments.

Change 1 — bulk is a separate path with a different structure.

POST /v1/documents/bulk   {tenant_id, manifest_uri}  ->  {job_id}

Bulk path:
  * embed in large offline batches on the eval/batch fleet, not the online one
  * build ONE large HNSW segment for the whole batch, not 200 small ones
  * publish atomically when complete
  * during the build, the documents are NOT searchable, and the job
    reports progress -- an honest "indexing 20M chunks, ~35 min remaining"

Bulk ingest does not get the 5-minute freshness SLO, and that is the right call: nobody uploading 5M documents expects them searchable in five minutes, and pretending otherwise forces the online path to absorb an offline workload. Stating which SLO does not apply is as much a part of the design as stating which does.

Change 2 — the online path gets an admission control it did not have.

if tenant.pending_chunks > BULK_THRESHOLD:      # e.g. 500k
    return 429 with a pointer to the bulk API

Without this, a client that loops over POST /v1/documents 20 million times reproduces exactly the failure regardless of the bulk path's existence. Any expensive path needs a cheap path's rate limit in front of it, or the expensive path is optional from the caller's perspective.

Change 3 — bound segment count. Merge policy targets ≤ 20 segments per tenant (tiered, like an LSM tree). Query fan-out is then bounded regardless of ingest history, which was the second-order failure the critique implies.

R4 — Tombstone state must be durable, not rebuilt (answers C4)

The critique finds a genuine availability bug with a security-shaped cause. If tombstone state is in-memory and rebuilt from a log on restart, then during a rolling restart every replica passes through applied_version = 0, and fail-closed turns a routine deploy into a total outage — while the alternative, failing open, turns it into a disclosure incident. Neither is acceptable, which means the premise is wrong.

Change: tombstones are durable, versioned, and loaded before the replica reports ready.

Tombstone bitmap is persisted WITH the segment in object storage,
versioned by tombstone_version.

Replica startup:
  1. load segments
  2. load the current tombstone bitmap snapshot   (a few MB; ~1 s)
  3. apply the delta log since the snapshot
  4. ONLY THEN report ready

A replica is never in rotation with stale tombstones, so applied_version never regresses to 0 in a serving replica. The readiness gate is the mechanism — the same pattern as any replica that must load state before serving, and the original design simply omitted it.

And the fleet-level guard the critique's framing points at: if all replicas were somehow behind, failing closed means a full outage.

if no replica satisfies min_tombstone_version:
    if age(request_delete) < 5 s:   wait up to 500 ms, then retry
    else:                            FAIL CLOSED and page

The bounded wait handles the normal race (a delete moments ago, propagation in flight); the page handles the pathological case. Fail-closed stays, because the thing being protected is disclosure — but it is now rare by construction rather than routine.

Cost: slower replica startup (~1–2 s) and tombstone snapshots in object storage. Both trivial against the alternative.

R5 — The reranker must not compete with generation, and it must be interruptible (answers C5)

The critique identifies a resource conflict the original design ignored. A cross-encoder over 100 candidates is a real forward pass; run on the generation fleet it competes for exactly the GPUs whose prefill time §6 is trying to protect — and it competes badly, because it is a small latency-critical job against large throughput-oriented ones.

Change 1 — a dedicated reranker fleet, sized independently.

The reranker is a small model (100M–500M params) that runs well on cheap GPUs (L4/A10) or even CPU with quantization. It has no business on H100s.

rerank fleet: L4 GPUs, ~$0.35/hr
  100 candidates x ~600 tokens = 60k tokens per rerank, one forward pass
  batched across concurrent queries

Cost separation is the point: the reranker's hardware is 7× cheaper per hour than the generation fleet's, and its scaling signal (queries/s) is completely different from generation's (KV occupancy). Coupling them would make both autoscalers wrong.

Change 2 — the reranker is interruptible with a deadline.

try:
    order = rerank(candidates, timeout_ms=50)
except Timeout:
    order = fusion_order        # RRF result: worse, but already computed and valid
    metrics.rerank_timeouts.inc()

A degraded ranking is far better than a blown TTFT budget, and the fusion order is a genuinely usable fallback rather than a placeholder. This makes the 50 ms a real budget rather than an average that will be exceeded under load — the difference between a latency target and a latency guarantee.

Change 3 — rerank fewer candidates under pressure. 100 → 50 halves the cost for a small recall loss. Load-shed by reducing quality gracefully instead of failing, which is the same reserved-degradation pattern as d05.

R6 — Citations are spans with confidence, and abstention has a cheaper form (answers C6)

Both halves of the critique are right, and the second exposes a latency trap the original design created for itself.

On multi-chunk synthesis: the premise that each sentence has one source chunk is wrong. Synthesis across chunks is the point of RAG.

Change 1 — citations are many-to-many, with a support score.

{ "text": "The retry limit is 5, raised from 3 in v2.1.",
  "citations": [
    {"chunk_id": "c-8821", "support": 0.91, "span": [120, 156]},
    {"chunk_id": "c-4410", "support": 0.74, "span": [8, 44]}
  ] }

Produced by scoring each generated sentence against each retrieved chunk with an NLI/entailment model — the same cross-encoder fleet from R5, reused. A sentence whose best support is below a floor is flagged as unsupported, and the answer carries a grounding_score.

Change 2 — do not regenerate. Degrade the answer. The critique is right that regenerate-then-abstain is the worst possible latency behaviour: it doubles the cost to arrive at "I don't know".

grounding check on the STREAMED output, sentence by sentence:
  * supported sentence      -> emit
  * unsupported sentence    -> emit with a visible "unverified" marker
  * >40% unsupported        -> stop the stream, emit what was verified plus
                               "I could not verify the rest from your documents"

No second generation, ever. The user gets the verified part of the answer plus an honest boundary — which is more useful than an abstention and cheaper than a retry.

Cost: the grounding check runs concurrently with generation and adds a small lag between token generation and token emission (roughly one sentence of buffering). That is the same tradeoff as output moderation in m01 §9, and the same resolution: buffer a little, accept a small leak, measure it.

And the general lesson: when a check can fail, prefer degrading the output over redoing the work. Redoing doubles the cost of the worst case, which is exactly the case that was already going badly.

References

• m01-llm-api-platform.md — the generation fleet; §2's KV multiplier lands there
• m02-kv-cache-tier.md — prefix caching, and the cache-key correctness problem §7 mirrors
• m05-eval-harness.md — how retrieval and answer quality get measured separately
• ../../systems-design/designs/d09-search-serving.md — the fan-out tail and index freshness, without the generation half
• ../../systems-design/designs/d07-log-analytics.md — immutable segments, tombstones, compaction: the same structure a third time
• ../../systems-design/WARMUP.md#47-consistency-models — session guarantees, used for delete visibility in §8
• Malkov, Y. & Yashunin, D. Efficient and robust approximate nearest neighbor search using HNSW. — the graph, and why deletes are hard
• Jégou, H. et al. Product Quantization for Nearest Neighbor Search. — the 64 B/vector representation
• Cormack, G. et al. Reciprocal Rank Fusion. SIGIR 2009 — hybrid fusion with one parameter
• Liu, N. et al. Lost in the Middle: How Language Models Use Long Contexts. — the position effect that constrains §6's cache-friendly ordering
• Gao, L. et al. Enabling Large Language Models to Generate Text with Citations. — attribution and grounding scoring