d09 — Search / Retrieval Serving

A fully worked design. Your home turf, which cuts both ways: an interviewer will push harder and expect more, and a generic answer here reads worse than a generic answer elsewhere.

This is also the portfolio-adjacent design — the one whose numbers you should be able to quote from something you have actually measured.

Table of Contents

• The Prompt
• 1. Requirements and Scope
• 2. Scale Numbers
• 3. API Surface
• 4. Data Model
• 5. High-Level Architecture
• 6. Deep Dive A: The Fan-Out Tail
• 7. Deep Dive B: Index Freshness vs Query Latency
• 8. Failure and Recovery
• 9. Bottlenecks and Evolution
• 10. Tradeoffs Explicitly Rejected
• The Hostile Critique
• The Revision
• References

The Prompt

"Design the serving side of a search system. Hundreds of millions of documents, tens of thousands of queries per second, and results have to be good — this is the product, not a feature. Updates need to show up quickly. Go."

"Results have to be good" is a latency constraint in disguise. Quality comes from ranking, ranking costs compute, and compute costs latency — so the entire design is about spending a fixed latency budget where it buys the most relevance. That framing, stated at minute two, is what separates this from a generic "shard and fan out" answer.

The second trap: candidates design the indexing pipeline because it is more concrete. The prompt says serving. Index freshness matters (§7) but the fan-out tail (§6) is where the round is won.

1. Requirements and Scope

Clarifying questions asked

"What dominates the query mix — head or tail?" Assumed a heavy head: the top 1,000 queries are ~30% of traffic, and there is a very long tail. That ratio decides whether caching is a rounding error or the main lever.

"Is this lexical, semantic, or hybrid?" Assumed hybrid — BM25 plus dense vectors — because that is the current reality and because it changes the sharding decision.

"What's the latency budget, and is it p99 or p50 that matters?" Assumed p99 < 200 ms end-to-end, and p99 is what matters because it is what users feel on the query that matters to them.

"How fresh?" Assumed tiered: new documents searchable in < 60 s, updates to existing documents < 5 min, deletions immediate (legal and trust reasons — a deleted document appearing in results is a different class of problem from a stale ranking).

Functional

1. Query → ranked results with snippets, filters, and pagination.
2. Hybrid retrieval: lexical + dense, fused.
3. Multi-stage ranking: cheap retrieval → cheaper reranking → expensive reranking on few.
4. Near-real-time indexing, immediate deletion.
5. Personalization signals in ranking.

Non-functional

Explicitly out of scope

• The crawling/ingestion pipeline upstream of indexing.
• Model training for the rankers; we serve them.
• Query understanding beyond tokenization and basic expansion.

2. Scale Numbers

Corpus. 500M docs × 2 KB = 1 TB raw. The inverted index is typically ~30% of text size = ~300 GB. Dense vectors at 768 dims × 4 B = 3 KB/doc = 1.5 TB — larger than the documents, which is the number that surprises people and drives quantization (§10).

Sharding. At ~50 GB per shard for memory-resident serving, 300 GB lexical + 1.5 TB dense (or 375 GB at int8) ≈ ~14 shards, call it 16. Each shard holds ~31M documents.

The fan-out arithmetic — the heart of the design. Every query hits every shard, so:

p99 of the SLOWEST of 16 shards, not the average.

If each shard's p99 = 50 ms and latencies are independent:
  P(at least one shard slow) = 1 - (1 - 0.01)^16 = 15%

So the SHARD p99 becomes roughly the QUERY p85.
To hit query p99 = 200 ms you need shard p99 ≈ 200 ms at the 1 - 0.01^(1/16)
level — i.e. you need each shard's p99.94, not its p99.

Fan-out converts a shard's tail into the query's median. That is the single most important number in this design and it is why §6 is a deep dive rather than a paragraph.

Replication. 20k QPS × 16 shards = 320k shard-queries/s. At ~500 shard-queries/s/replica that is 640 replicas, ×1.5 for AZ tolerance ≈ ~1,000 serving nodes. Say it: this is a large fleet, and that is why cache hit rate is worth real money.

Cache. Head queries are 30% of traffic. A result cache with a 30% hit rate removes 30% of 320k shard-queries/s = 96k/s, which is ~300 nodes. The cache is worth roughly a third of the fleet, and framing it in nodes rather than in percent is what makes the argument land.

Latency budget — allocate it explicitly, because that is the whole design:

200 ms p99 total
   5 ms   gateway + auth + cache lookup
  10 ms   query understanding, embedding the query
  60 ms   shard fan-out: retrieval + first-pass ranking   ← the tail lives here
  40 ms   merge + second-pass rerank (top ~200)
  30 ms   cross-encoder rerank (top ~20)                  ← the expensive one
  25 ms   snippet generation + response assembly
  30 ms   headroom

Note where the money goes: the cross-encoder reranks 20 documents in 30 ms while retrieval scans 500M in 60 ms. That inversion — spending a third of the budget on 20 documents — is the multi-stage ranking argument, and it is correct: relevance gains are concentrated at the top.

3. API Surface

POST /search
  {query, filters, from, size,
   user_context?,                       # for personalization
   timeout_ms?,                         # caller's budget
   quality: "full" | "fast"}            # explicit degradation
  -> {results: [{doc_id, score, snippet, explain?}],
      total_estimate, took_ms,
      partial: bool, shards_ok: 15, shards_total: 16}

POST /explain  {query, doc_id}          -> per-signal score breakdown
GET  /health   ?shard=N

Four choices worth defending:

• partial with shards_ok. If one shard is down, return results from 15 shards and say so. For search, 94% of the corpus in 200 ms is enormously better than an error — and the caller needs to know so it does not cache a degraded result as if it were complete.
• total_estimate, never an exact count. Exact counts require full evaluation of every match, which is unbounded work for a number users do not act on. Everyone who tried exact counts at scale eventually stopped.
• quality as an explicit parameter. Under load, degradation should be chosen and visible, not silent. A caller that needs speed can ask for it.
• explain. Relevance debugging is a daily activity for whoever owns quality, and a search system without it is a black box that nobody can improve.

4. Data Model

SHARDING: by document hash — a RANDOM partition, deliberately
  shard = hash(doc_id) % 16

  Every shard is a uniform random sample of the corpus, so:
    • every shard's score distribution is representative
    • top-k from each shard merges correctly
    • no shard is "the popular one"

Per shard, per segment (immutable, Lucene-style):
  inverted index   term -> postings (doc_id, tf, positions)
  doc values       columnar, for filtering and sorting
  dense vectors    HNSW graph, int8-quantized
  live docs        bitmap; deletions are a bit flip
  field stats      for BM25 (df, avgdl, N)

Per shard: a small mutable buffer + N immutable segments + background merges.

Why hash-sharding rather than semantic (by topic or language): semantic sharding is tempting because it would let you skip shards, but it destroys score comparability — BM25's IDF depends on corpus statistics, so a shard containing only medical documents computes wildly different IDFs than a general one, and the top-k from each are not on the same scale. Merging them is then statistically wrong.

Random sharding means every shard's statistics approximate the global ones, and the merge is just a k-way merge of comparable scores. The cost is that every query touches every shard — the fan-out problem in §6 — and I would rather solve that than solve score normalization across non-comparable shards.

Deletions as a bitmap. Removing a document from every postings list is O(terms in doc) with random access into immutable segments — impossible. A live-docs bitmap makes deletion a bit flip, immediately visible, with the space reclaimed at the next merge. This is why the freshness requirement can say "deletes immediate" while updates take minutes.

5. High-Level Architecture

                       query
                         │
                         ▼
              ┌────────────────────┐
              │  Gateway           │  authn, rate limit
              │  RESULT CACHE      │  ~30% hit on head queries
              └─────────┬──────────┘
                        ▼
              ┌────────────────────┐
              │ Query understanding│  tokenize, spell, expand,
              │                    │  embed (dense), classify intent
              └─────────┬──────────┘
                        ▼
        ┌───────────────────────────────────┐
        │  Broker  (scatter–gather)          │  DEEP DIVE A
        │  hedging · timeouts · partial      │
        └──┬──────┬──────┬───────────┬──────┘
           ▼      ▼      ▼           ▼
        shard0  shard1  ...       shard15      × ~60 replicas each
        ┌──────────────────────────────┐
        │ retrieve: BM25 ∪ HNSW top-k   │
        │ first-pass rank (cheap LTR)   │
        │ return top ~50 + scores       │
        └──────────────────────────────┘
           │
           ▼   merge 16 × 50 = 800 → top 200
        ┌────────────────────────┐
        │  Second-pass reranker  │  richer features, ~200 docs
        └───────────┬────────────┘
                    ▼   top 20
        ┌────────────────────────┐
        │  Cross-encoder rerank  │  expensive, tiny candidate set
        └───────────┬────────────┘
                    ▼
              snippets + response

The funnel is the design: 500M → 800 → 200 → 20, with per-document cost rising by orders of magnitude at each stage. Total cost stays bounded because the expensive model only ever sees 20 documents.

The two hard parts — say these at minute 10:

1. The fan-out tail — 16 shards means the query p99 is roughly the shard p99.94.
2. Index freshness vs query latency — every mechanism that makes the index fresher makes queries slower, and the tension is structural.

6. Deep Dive A: The Fan-Out Tail

The problem, quantified

Every query touches all 16 shards and waits for the slowest. With independent latencies:

The shard's p99 becomes the query's ~p85. To hit a 200 ms query p99 you need each shard's p99.94 under ~150 ms. That is a much harder target than "make the shard fast", and the mitigations are structural rather than about optimization.

Latencies are also not independent in practice — a GC pause, a hot neighbour, or a merge storm correlates across replicas — which makes it worse, not better.

Mitigation 1 — Hedged requests (the biggest single win)

Send to one replica; if it has not answered by p95, send the same request to a second replica and take whichever returns first.

async def query_shard(shard, req):
    first = asyncio.create_task(send(pick_replica(shard), req))
    done, _ = await asyncio.wait({first}, timeout=p95_latency)
    if done:
        return first.result()
    second = asyncio.create_task(send(pick_replica(shard, exclude=first.replica), req))
    done, pending = await asyncio.wait({first, second},
                                       return_when=asyncio.FIRST_COMPLETED)
    for t in pending:
        t.cancel()                       # cancel the loser — do not waste its work
    return done.pop().result()

Cost: ~5% extra load (only the slowest 5% get hedged). Benefit: the tail collapses, because you now need both replicas to be slow rather than one — and if the slowness is a local condition (GC, a noisy neighbour, a cold cache) the second replica is almost certainly fine.

This is Dean & Barroso's "tail at scale" result and it is the highest-leverage single technique in this design. Cancelling the loser matters: without it, hedging at high rates doubles load during exactly the periods when the system is already struggling.

Mitigation 2 — A deadline, and partial results

The broker holds a hard deadline. Shards that have not answered are abandoned, and the response says so.

deadline reached → merge what arrived → partial: true, shards_ok: 15

94% of the corpus in 200 ms beats 100% in 2 seconds, for search specifically, because relevance is a distribution and missing 6% of it rarely changes the top 10. That is a domain-specific judgement and it is worth stating as one — it would be wrong for a transactional system.

Second-order effect worth naming: results become non-deterministic under load, since which shards answer varies. That breaks caching and confuses users who reload. Mitigation: cache only complete results (partial: false), and make the degradation visible in the response.

Mitigation 3 — Make the shard itself have a short tail

The hedge treats the symptom. The causes:

• GC pauses. The dominant source of correlated tails on the JVM. Fix with a low-pause collector, off-heap index structures, and — most effective — taking a replica out of rotation during a major GC rather than serving slowly through it.
• Cold caches after a restart. A fresh replica has an empty page cache and is 10× slower. Fix: warm it with shadow traffic before adding it to rotation. Adding a cold replica to a live pool is a self-inflicted latency incident.
• Merge storms. A large segment merge saturates disk and CPU. Fix: rate-limit merges, and stagger them across replicas so no two replicas of the same shard merge simultaneously.
• Query cost variance. A query matching 50M documents costs vastly more than one matching 50. Fix: early termination — WAND/block-max WAND lets you stop scanning postings once no remaining document can enter the top-k. This is the single biggest algorithmic win for high-frequency terms, and naming it specifically is a strong signal.

Mitigation 4 — Reduce the fan-out where you can

The fan-out factor is in the exponent, so reducing it helps disproportionately. Two safe ways:

• Filter-based shard pruning: a filter on a field the corpus is also partitioned by (say, language or marketplace) means those queries touch 1 shard, not 16. This is a secondary partition on top of hash sharding, and it is worth it if such filters are common.
• Tiered indexes: a small "hot tier" of the most-frequently-retrieved documents (say 5%), queried first. If the top-k from the hot tier is confidently good — a score-gap test — skip the full fan-out entirely. This is a real technique and it can cut fan-out on a large fraction of head queries.

7. Deep Dive B: Index Freshness vs Query Latency

The structural tension: near-real-time updates mean many small segments; many small segments mean each query opens and merges results from many of them; merging segments to fix that costs I/O that competes with queries.

fresh index  → small segments → many segments → slower queries
merged index → few segments   → faster queries → merge I/O competes with serving

Every search system lives somewhere on this curve, and where it sits is a product decision.

The three-tier structure

A query searches all three and merges. The buffer is small so it is fast; the base is large but few-segmented so it is fast; the recent tier is where the cost is, and it is bounded by the merge policy.

Refresh interval is the knob, and it must be per-index rather than global: a 1 s refresh gives near-real-time at a heavy segment-count cost; 60 s is a good default; 5 min is right for a corpus that changes slowly. Setting it globally means paying the fresh-index cost for content that does not need it.

Deletions are different, and must be immediate

Updates can lag; deletions cannot — a removed document appearing in results is a legal and trust problem, not a relevance one.

So deletions bypass the pipeline: the delete propagates directly to every replica's live-docs bitmap, which is a bit flip, applied in milliseconds. The document remains in the index until the next merge reclaims its space, but it is invisible from the moment the bit flips.

The subtlety worth raising: a document deleted in the base tier but re-added in the buffer must be visible, while one deleted in the buffer must be invisible even though the base still has it. So liveness is resolved newest-tier-first, exactly like the segment-masking rule in an LSM. Getting that ordering backwards is a real bug and it is invisible in testing until someone deletes and re-adds.

Updates are delete + insert, and that has a consequence

Segments are immutable, so an update is a tombstone plus a new document. Two consequences:

1. The old and new versions coexist until merge. Queries must not return both — which the newest-first liveness rule handles.
2. A high update rate is a high garbage rate. A corpus where every document is updated daily generates a full corpus of garbage daily, so merge cost scales with the update rate, not the corpus size. That is the number that determines whether near-real-time is affordable, and it is worth asking about early.

Keeping the index consistent across replicas

Sixty replicas per shard must converge, or the same query returns different results depending on routing — which destroys user trust far more than staleness does.

Segment-based replication, not operation-based: replicas copy immutable segment files from a primary indexer rather than each independently indexing the same documents. Independent indexing is nondeterministic (merge timing, tie-breaking, floating-point accumulation), so replicas would drift. Copying files means they are bit-identical.

Cost: replicas lag by the copy time (seconds for a small segment). Benefit: identical results across replicas, and indexing CPU paid once rather than sixty times — which at this fleet size is a large saving.

8. Failure and Recovery

Deliberately accepted: under load or partial failure we return results from a subset of shards and skip expensive reranking stages. I accept that because in search, degraded relevance is almost always better than an error — a user who gets decent results does not notice, and a user who gets an error leaves. That judgement would be wrong for a transactional system, and the distinction is worth stating.

The quality degradation ladder, in the order to shed:

1. skip the cross-encoder      (top-20 rerank)     — small nDCG loss, 30 ms saved
2. skip the second-pass rerank (top-200)           — moderate loss,   40 ms saved
3. reduce per-shard top-k      (50 → 20)           — small loss,      some latency
4. accept partial shards                            — depends on which
5. serve from cache, even if stale

Note it degrades from the most expensive and least impactful downward — which requires actually knowing the nDCG contribution of each stage, which requires having measured it. That measurement is the thing that makes graceful degradation a design rather than a guess.

9. Bottlenecks and Evolution

1. The cross-encoder, immediately. It is GPU-bound and it is 30 ms of a 200 ms budget for 20 documents. Fixes in order: batch across concurrent queries (the single biggest win — GPU utilization at batch 1 is terrible, which is the same arithmetic-intensity argument as LLM decode); distill to a smaller model; cache scores for (query, doc) pairs on head queries.

2. Dense vector memory. 1.5 TB at fp32 is the largest single line item. int8 quantization gives 4× with typically < 1% recall loss — verify on your own data — bringing it to 375 GB. Beyond that, product quantization gives another 4–8× at a real recall cost, and that trade must be measured per corpus rather than assumed.

3. Fan-out at more shards. Growing the corpus means more shards, and the tail gets exponentially worse. Fixes: bigger shards (fewer, each larger — bounded by memory and by per-shard latency), or hierarchical fan-out (broker → 4 sub-brokers → 4 shards each), which turns a 16-way tail into two 4-way tails. The second is what large systems do.

4. Cache hit rate is the cheapest lever, and it is under-invested. At 30% it saves ~300 nodes. Improvements: normalize queries aggressively (case, whitespace, stopwords, synonym canonical form); cache at the retrieval level (post-fan-out, pre-rerank) as well as the result level, since retrieval results are reusable across personalization variants; and negative-cache empty results, which are common and cheap to store.

5. Personalization breaks caching, and that tension is fundamental. A per-user result set has a cache hit rate near zero. Resolution: cache the retrieval and first-pass ranking (identical across users) and apply personalization only in the final rerank on 200 documents. Personalization then costs the rerank stage rather than the whole pipeline — which is both cheaper and, in my experience, where nearly all of its lift is anyway.

At 10× corpus (5B documents): 160 shards, and the fan-out tail becomes the dominant problem. The answer is hierarchical fan-out plus aggressive tiering — a hot tier that answers most queries without full fan-out. At that point the design is more about routing than about retrieval, and saying that is the honest evolution.

10. Tradeoffs Explicitly Rejected

Rejected: semantic or topical sharding. Would let queries skip shards. Rejected because it destroys score comparability — IDF is corpus-statistical, so shards with different topical distributions produce non-comparable BM25 scores and merging them is statistically wrong. Flip condition: a hard partition users always filter on (marketplace, language) is genuinely worth a secondary partition, because those queries are single-shard by construction and the score comparison stays within a partition.

Rejected: a single-stage ranker. Simpler, one model. Rejected on the arithmetic: running a cross-encoder over even 10,000 candidates is orders of magnitude beyond the budget. The funnel is what makes an expensive model affordable — it only sees 20 documents.

Rejected: exact result counts. Rejected because it requires full evaluation of every match rather than early termination, which is unbounded work for a number users do not act on. Estimates from sampling plus corpus statistics are close enough for the "about N results" display.

Rejected: waiting for all shards. Rejected because with 16-way fan-out the query p99 becomes the shard p99.94 — you would be designing for the worst shard's worst moment. Deadline plus partial results, with the partiality surfaced.

Rejected: operation-based replication (each replica indexes independently). Simpler pipeline, no file copying. Rejected because indexing is nondeterministic in merge timing and tie-breaking, so replicas drift and the same query returns different results depending on routing — which destroys user trust more than staleness does. Segment copying also pays indexing CPU once instead of sixty times.

Rejected: a 1-second refresh interval globally. Rejected on cost: it produces enormous segment counts and merge pressure for content that mostly does not change that fast. Per-index refresh intervals let the fast-changing indexes pay for their freshness.

Rejected: fp32 dense vectors. Rejected on memory — 1.5 TB versus 375 GB at int8, for a recall loss typically under 1%. Flip condition: if measurement on this corpus showed a meaningful recall drop, fp32 for a hot subset and int8 for the tail is the compromise. Measure, do not assume — the loss is corpus-dependent.

The Hostile Critique

C1. "Hedging at p95 sends 5% extra load. During an incident every shard is slow, so p95 is exceeded on nearly every request. Walk me through what your hedging does at that moment."

C2. "You return partial: true with 15 of 16 shards. The missing shard happened to hold the single best result for that query. The user sees mediocre results and no indication that anything is wrong except a boolean they'll never look at. Is that actually better than an error?"

C3. "You cache retrieval results pre-personalization. The filters are part of the query. How many distinct filter combinations do you have, and what's your real hit rate?"

C4. "Deletes are 'immediate' via a bitmap push to 1,000 nodes. One node is network partitioned for ten minutes and keeps serving. Legal asked you to remove that document. What do you tell them?"

C5. "Segment-copy replication means 60 replicas pull the same segment from a primary indexer. A large merge produces a 40 GB segment. Do the arithmetic on that."

C6. "You degrade by skipping the cross-encoder and claim a 'small nDCG loss'. How do you know? And what does your degradation do to the A/B test that's running at the same time?"

The Revision

R1 — Hedging needs a budget, or it amplifies (answers C1)

The critique identifies a genuine feedback loop and it is the classic hedging failure: hedging is calibrated on the healthy distribution, so under system-wide slowness it fires on nearly everything and doubles load exactly when the system cannot take it.

Change: a hedge budget, exactly analogous to a retry budget.

# Hedges may never exceed 5% of base request volume, cluster-wide.
if hedge_budget.try_consume():
    send_hedge()
# else: wait for the original, no hedge

Plus two refinements:

• The p95 threshold is computed over a trailing window, so during a slow period the threshold rises with it and hedging naturally becomes rarer rather than universal.
• Hedge only when the slow replica is an outlier, not when everything is slow. If all replicas of a shard are equally slow, a hedge cannot help — the problem is not local — so do not spend the load.

Cost: during genuine widespread slowness the tail is worse, because hedging is unavailable exactly then. That is correct: when everything is slow, adding load makes it slower. The right response then is shedding (quality), not hedging.

R2 — Partial results need retry and honesty, not just a flag (answers C2)

The critique is fair. A boolean nobody reads is not a mitigation, and for a query where the missing shard held the best result, degraded output looks like bad relevance rather than partial service — which is worse, because the user blames the product.

Change, three parts:

1. Retry the missing shard once, within the deadline. A shard timing out at 150 ms with a 200 ms budget leaves 50 ms — often enough for a different replica. Try before giving up.
2. Fail rather than degrade when the loss is likely material. Estimate it: if the returned results' score distribution suggests a truncated tail — the lowest returned score is high, so there were probably better documents elsewhere — the answer is more suspect. Below a confidence threshold, return 503 and let the client retry, rather than serving results that look complete and are not.
3. Surface it to the user, not just the API. "Some results may be missing — retry" is honest and it is what a user needs to decide whether to trust what they see.

And an important consequence: never cache a partial result, never count it in relevance metrics, and never use it in an A/B test — otherwise degraded serving silently corrupts your quality measurements, which is a much longer-lived problem than the incident that caused it.

Cost: slightly lower availability (some queries now fail rather than degrade). The right trade when the degradation is severe; the wrong one when it is marginal — hence the threshold.

R3 — Cache the expensive, invariant part (answers C3)

The critique is right and the arithmetic is brutal: with 20 filterable fields the combination space is effectively unbounded, so a cache keyed on (query, filters) has a hit rate near zero for anything but the exact head.

Change: cache at the layer that is filter-independent.

Cache key:  normalized_query_text   (NOT including filters)
Cache value: the top ~1,000 doc_ids with scores, UNFILTERED

At query time: take the cached candidates, apply filters via doc-values, rerank.

• Retrieval — the expensive fan-out — is cached and reused across every filter combination of the same query text.
• Filtering 1,000 cached candidates against doc-values is ~1 ms, versus ~60 ms of fan-out.
• The hit rate is now the hit rate of the query text alone, which is the 30% figure that was claimed all along.

The correctness caveat, which must be stated: if a filter is very selective, the top 1,000 unfiltered candidates may contain no matching documents, and the cached answer would be wrong (empty). So: estimate the filter's selectivity from doc-value statistics, and if fewer than k candidates survive, fall back to a full filtered retrieval. That fallback path is where the correctness lives and it must exist.

Cost: more cache memory (1,000 IDs+scores ≈ 12 KB per entry vs ~1 KB for a result page), and a fallback path to test. Worth it — this converts the cache from decorative to load-bearing.

R4 — Deletion needs a verified, blocking path (answers C4)

The critique names a compliance failure, and "it's eventually consistent" is not an answer you can give a legal team.

Change: deletion is a two-phase, verified operation.

1. Write the tombstone to the durable delete log (replicated, ordered).
2. Push to every serving node; collect ACKs.
3. A node that has NOT acked within T seconds is REMOVED FROM ROTATION.
   It cannot serve until it has applied the delete log up to the required version.
4. Report completion only when every IN-ROTATION node has acked.

The key inversion: an unreachable node is removed from serving, not allowed to serve stale data. A partitioned node serves nothing, so it cannot serve the deleted document. Availability degrades — correctly — rather than compliance.

Plus:

• On rejoin, a node must catch up on the delete log to the current version before it may serve. Applied at startup, before health-check success.
• Audit sampling: continuously query for known-deleted documents across replicas and alarm on any hit. Verification, not assumption.
• A deletion SLA with a report naming exactly when every node acked — which is what legal actually needs.

Cost: a partition now costs capacity (nodes removed) rather than correctness. That is the right direction for deletions specifically, and it is why deletion has its own path rather than riding the normal index pipeline.

R5 — Segment distribution must be peer-to-peer (answers C5)

The arithmetic in the critique is decisive: 40 GB × 60 replicas = 2.4 TB pulled from one indexer. At 10 Gbps that is over half an hour, during which the indexer's network is saturated and merges cannot proceed.

Change, two parts:

1. Peer-to-peer distribution. Segments are chunked and distributed BitTorrent-style: replicas fetch chunks from each other, not all from the primary. Distribution time becomes O(log(replicas)) rather than O(replicas), and the primary uploads roughly once rather than sixty times.
2. Bound the merge output size. A 40 GB segment is too large regardless: it is slow to distribute, slow to warm, and it makes the merge itself a multi-hour operation that competes with serving. Cap merged segments at ~5 GB and accept more of them — the query cost of a few extra segments is far smaller than the operational cost of enormous ones.

And stagger the adoption: replicas switch to the new segment set in waves, so no more than a fraction are warming a cold segment at once. A fleet-wide simultaneous switch is a fleet-wide cold-cache event, which is the same self-inflicted latency incident as adding cold replicas.

Cost: more segments means slightly slower queries, and a peer-to-peer distribution layer to build and operate. The alternative is a half-hour distribution window during which the indexer is unusable, so the trade is clear.

R6 — Measure the degradation, and exclude it from experiments (answers C6)

The critique catches an unquantified claim and a genuine measurement hazard — and the second half is the more serious one.

Change, two parts:

Measure the ladder. Each degradation level's nDCG impact is measured offline on a held-out set and re-measured monthly:

full pipeline                nDCG@10 = 0.612   (baseline)
skip cross-encoder           nDCG@10 = 0.581   (-5.1%)   saves 30 ms
skip second-pass rerank      nDCG@10 = 0.524   (-14.4%)  saves 40 ms
per-shard top-k 50→20        nDCG@10 = 0.605   (-1.1%)   saves ~8 ms

Now the ladder is ordered by cost per nDCG point rather than by intuition — and note that reducing top-k turns out to be nearly free, so it should be shed first, before the cross-encoder. Without measurement the ordering was wrong.

Exclude degraded traffic from experiments. This is the part that matters most:

• Every response carries the pipeline version actually executed, not the one requested.
• The experiment framework excludes degraded responses from metric computation entirely.
• If degradation exceeds a small fraction of an experiment's traffic, the experiment is flagged as invalid rather than silently producing a biased result.

Why this matters more than the ladder: degradation is correlated with load, load is correlated with time of day, and time of day is correlated with user population. So degraded traffic is a biased sample, and including it does not add noise — it adds bias, which no amount of data corrects. An experiment that silently measured "treatment during peak load" versus "control during off-peak" would ship the wrong ranker with high confidence.

Cost: experiments take longer during periods with frequent degradation, and you need degradation to be rare for experimentation to be practical at all — which is itself a good forcing function.

References

• ../WARMUP.md#41-the-arithmetic — Little's law and the fan-out tail arithmetic
• d06-feature-store.md — the ranking features this serves, and point-in-time correctness for training them
• d07-log-analytics.md — segments, Bloom filters and pruning, in a different shape
• ../../coding/harness/problems/text_index/ — postings, tombstones, BM25 and segment merging as a timed 4-gate problem
• Dean, J. and Barroso, L. The Tail at Scale. CACM 2013 — hedged requests, and the fan-out arithmetic in §2. The single most relevant paper to this design
• Broder et al. Efficient Query Evaluation using a Two-Level Retrieval Process. CIKM 2003 — WAND early termination
• Ding & Suel. Faster Top-k Document Retrieval Using Block-Max Indexes. SIGIR 2011
• Robertson & Zaragoza. The Probabilistic Relevance Framework: BM25 and Beyond. 2009
• Malkov & Yashunin. Efficient and Robust Approximate Nearest Neighbor Search Using HNSW. TPAMI 2018
• Johnson, Douze, Jégou. Billion-scale similarity search with GPUs (FAISS). 2017 — product quantization and the recall/memory frontier
• Nogueira & Cho. Passage Re-ranking with BERT. 2019 — the cross-encoder stage
• Lucene documentation — segment merging, live docs, near-real-time search