« Phase 06 · Warmup · Track Overview

Core Contributor Notes — How the Real Systems Work

Table of Contents

• 1. HNSW, as implemented
• 2. Filtering: the three strategies engines actually use
• 3. Multi-tenancy in the real stores
• 4. Hybrid search as the engines ship it
• 5. Rerankers in production
• 6. Grounding checks that are not token overlap
• 7. Sharp edges
• 8. What the miniature simplifies
• 9. References

1. HNSW, as implemented

The lab does exact search. Production does not, and the structure that replaced it is worth understanding because its parameters are the ones you will be asked to tune.

HNSW builds a layered proximity graph. Each vector is inserted at a random maximum layer drawn from an exponentially decaying distribution, so upper layers are sparse and lower layers contain everything. Search starts at the top, greedily walks toward the query, descends a layer, repeats. The upper layers are "highways"; the bottom layer is local streets.

Three parameters, and they trade differently:

The one to internalize: ef_search is a runtime recall/latency dial. You can raise it for a high-stakes query and lower it under load, which makes it a degradation-ladder knob that most teams never wire up.

Memory is the constraint people underestimate. The graph itself is roughly \( M \) links per node per layer, and at M=16–64 over millions of vectors that is gigabytes on top of the vectors. Budget it explicitly, or your index does not fit the instance you sized for the vectors alone.

2. Filtering: the three strategies engines actually use

The WARMUP §7.2 filtering cliff is a real, named problem, and engines solve it three ways:

Post-filtering. Search, then filter. Simple, and it is the cliff. Some engines over-fetch by a multiplier to compensate, which works until the filter is very selective and then fails without warning.

Pre-filtering (allow-list). Compute the matching id set first, then restrict graph traversal to it. Exact recall, but if the allow-list is large the set operation dominates, and if it is tiny the graph becomes disconnected and traversal degenerates to a scan. Qdrant's approach is adaptive: it estimates filter cardinality and switches between graph traversal and a straight scan.

Partitioned indexes. One index (or namespace, or collection, or shard) per filter value. No cliff, because the filter is the choice of index. This is what the lab implements and what per-tenant isolation converges on anyway.

The practically important consequence: your isolation decision and your recall decision are the same decision. A design review that treats "how do we isolate tenants" and "why is recall bad for tenant X" as separate conversations has missed it.

3. Multi-tenancy in the real stores

Every major store has a first-class answer, and their vocabularies differ enough to cause confusion:

The pgvector row deserves attention for a bank: row-level security puts the tenant predicate in the database, so an application bug cannot bypass it. That is structurally stronger than an application-level filter and cheaper than an index per tenant — a genuinely different point on the trade curve, and the one most likely to satisfy a security review.

The "many small tenants" case is where the guidance converges: thousands of tiny namespaces are operationally painful and give each tenant a poor graph. A tenant-keyed payload index with the engine's tenant optimisation is usually better, and the isolation argument then rests on the engine enforcing the predicate during traversal rather than after it.

4. Hybrid search as the engines ship it

Most engines now ship hybrid natively, and most of them use RRF:

• Azure AI Search — vector + keyword with RRF fusion, plus an optional semantic reranker (a cross-encoder) as a second stage. Its k is fixed.
• Weaviate — hybrid with alpha blending, offering both RRF and relative-score fusion.
• Qdrant — query API with prefetch and a fusion step (RRF or DBSF).
• Elasticsearch / OpenSearch — rrf retriever combining a knn and a standard retriever.

The alpha-style score blending some engines offer is the trap from WARMUP §5.2. It works when you have tuned it for your corpus and it silently degrades when the corpus or the embedding model changes. If you use it, treat alpha as a tuned parameter with an owner and an evaluation, not a config default.

One detail worth knowing: engines differ on whether RRF is applied per shard or globally. Per-shard fusion changes results as you re-shard, which is a genuinely confusing bug to chase.

5. Rerankers in production

Three shapes:

Hosted rerank APIs (Cohere Rerank, Voyage, Jina). One call, a list of documents, scores back. Simple, and it sends your documents to a third party — a classification question, not just a cost one.

Self-hosted cross-encoders (bge-reranker, mxbai-rerank, MiniLM cross-encoders). Small models, so a GPU serves high throughput, and the data stays inside. This is usually the right answer for a bank, and it is a serving-capacity problem (Phase 05) rather than a retrieval one.

Engine-integrated semantic ranking (Azure AI Search's semantic ranker). No extra hop, and no control over the model.

Operational facts that matter:

• Rerankers have a document-length limit, often shorter than your chunks. Exceed it and the tail is silently truncated, so the reranker scores a prefix of your chunk. Check it against your chunk size.
• Batching is essential. Fifty individual calls is fifty round trips; one batched call is one.
• Score distributions are model-specific, so a floor tuned for one reranker is meaningless for another. Re-tune on model change, and treat that as a model change with an eval gate.

6. Grounding checks that are not token overlap

The lab's overlap check is a stand-in. Production uses one of:

NLI / entailment models. Split the answer into claims, and for each ask a natural-language inference model whether the retrieved context entails it. This is what RAGAS faithfulness does, and it is the closest thing to a principled measure.

LLM-as-judge. Prompt a model with the claim and the context and ask for a verdict. Flexible, more expensive, and it needs its own calibration — a judge that agrees with everything is worse than no judge. Measure judge agreement against human labels before trusting it.

Provider grounding checks. AWS Bedrock's contextual grounding check scores an answer for grounding and relevance against the source, with configurable thresholds, and can block the response. Worth reading for its API shape even if you build your own, because it separates the two scores — an answer can be perfectly grounded in the retrieved text and not answer the question.

Citation-span verification. Rather than scoring, require the generator to emit spans and then verify each span exists in the retrieved text. Cheap, deterministic, and it catches fabricated citations — which are a real failure mode and one that a scoring approach can miss entirely.

The pragmatic production shape is usually: cheap span verification on every request, expensive entailment scoring on a sample, and both feeding the same quality SLI (Phase 14).

7. Sharp edges

Cosine vs inner product vs L2. Engines expose all three. They are equivalent only for normalized vectors. Mixing normalized and unnormalized vectors in one index produces silently wrong rankings — and some embedding APIs normalize while others do not.

HNSW deletions are tombstones. Deleting from an HNSW graph does not free the node; it marks it. Recall degrades and memory does not drop until you rebuild. A corpus with high churn needs a compaction strategy, and "we delete a lot" is a real reason to prefer a different index type.

Index build time is not query time. A million-vector HNSW build is minutes to hours depending on ef_construction. Plan re-ingestion around it; a nightly full rebuild that takes six hours is a design constraint, not a detail.

Chunk-size and reranker limits interact. A 1 000-token chunk fed to a reranker with a 512-token limit is scored on half its content — silently.

Metadata filters are not free. Even with pre-filtering, a filter on an unindexed payload field is a scan. Index the fields you filter on, and know which ones those are before you go live.

Embedding APIs have input limits and batch limits, and they differ. A backfill that works on 100-chunk batches locally may fail at 1 000, and the error is often a rate limit rather than a clear message.

Normalization at query time must match ingestion time. If you normalized on write and forget on read, every score is wrong by a constant factor — which preserves ranking within one query and breaks any absolute threshold, including your relevance floor. That is a nasty bug because the symptom is "the floor stopped working".

8. What the miniature simplifies

The reasoning transfers unchanged. What the real stack adds is approximation (and with it the filtering cliff), scale (and with it ingestion as a platform), and models (and with them migrations and eval gates).

9. References

ANN and vector stores

• Malkov & Yashunin, Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs, 2016.
• Jégou, Douze & Schmid, Product Quantization for Nearest Neighbor Search, 2011.
• Qdrant documentation — multitenancy and filtering; its adaptive filtering discussion is the clearest public explanation of the cliff.
• Pinecone namespaces; Weaviate multi-tenancy; Azure AI Search index-per-tenant vs filter guidance; pgvector with Postgres row-level security.

Hybrid and reranking

• Cormack, Clarke & Büttcher, Reciprocal Rank Fusion, SIGIR 2009.
• Azure AI Search hybrid search and semantic ranker documentation; Elasticsearch rrf retriever; Weaviate hybrid alpha; Qdrant query API fusion.
• Nogueira & Cho, Passage Re-ranking with BERT, 2019; Cohere Rerank and bge-reranker documentation for the production API shapes.

Grounding and evaluation

• Es et al., RAGAS, 2023 — faithfulness, answer relevance, context precision/recall.
• AWS Bedrock contextual grounding check — grounding and relevance as separate scores with thresholds.
• Gao et al., Retrieval-Augmented Generation for Large Language Models: A Survey, 2023.

Foundations

• Robertson & Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond, 2009.
• Manning, Raghavan & Schütze, Introduction to Information Retrieval, CUP 2008.
• Weinberger et al., Feature Hashing for Large Scale Multitask Learning, ICML 2009.