Hybrid Search

Phase 9 · Document 05 · RAG Prev: 04 — Vector Databases · Up: Phase 9 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Pure semantic (vector) search has a famous blind spot: it's great at meaning but bad at exact terms — product names, error codes, SKUs, function names, rare acronyms, legal citations. Ask for "error E1234" or "the parse_header function" and embeddings may return semantically similar but wrong chunks. Hybrid search fixes this by combining dense (semantic) retrieval with sparse (keyword/BM25) retrieval, then fusing the results — capturing both "means the same" and "contains this exact token." In practice hybrid beats pure semantic on most real corpora, especially technical/enterprise ones full of identifiers, and it's a high-ROI, low-cost upgrade to recall (00 — retrieval dominates). Knowing when keyword wins, and how to fuse the two, is core retrieval literacy.

2. Core Concept

Plain-English primer: two ways to match, fused

There are two fundamentally different ways to find relevant text:

• Dense / semantic search (03): embed query and chunks, find nearest vectors. Captures meaning — "get my money back" → "refund policy". Blind to exact tokens it didn't learn to associate.
• Sparse / keyword search (BM25): classic information-retrieval scoring based on term overlap — does the chunk contain the query's words, weighted by how rare/important each word is? Captures exact matches — "E1234", "parse_header", "Section 12(b)". Blind to synonyms/paraphrase.

Hybrid search runs both and fuses their result lists, so you get meaning and exact-term matching. The classic empirical result: hybrid ≥ either alone on most corpora, with the biggest gains on technical/identifier-heavy content.

BM25 in one paragraph

BM25 (Best Match 25) is the workhorse keyword-ranking function (the thing behind Elasticsearch/Lucene). It scores a chunk for a query by summing, over the query's terms: term frequency in the chunk (more occurrences → higher, with diminishing returns) × inverse document frequency (rarer terms across the corpus count more) × a length normalization (so long chunks don't win just by being long). It's a sparse representation — a vector over the whole vocabulary that's mostly zeros (one weight per present term) — hence "sparse" vs the "dense" embedding.

Fusing the two: Reciprocal Rank Fusion (RRF)

You can't directly compare a cosine similarity (0–1) to a BM25 score (unbounded) — different scales. The robust, parameter-light standard is Reciprocal Rank Fusion (RRF): combine by rank, not raw score.

def rrf(result_lists, k=60):                    # each list = doc_ids in ranked order
    scores = {}
    for results in result_lists:                # e.g. [dense_ranked, sparse_ranked]
        for rank, doc_id in enumerate(results):
            scores[doc_id] = scores.get(doc_id, 0) + 1.0 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

A doc ranked highly by either retriever (or moderately by both) rises. RRF is scale-free, needs no tuning beyond k (≈60 is a fine default), and is the most common fusion in production. The alternative — weighted score fusion (α·normalize(dense) + (1-α)·normalize(sparse)) — can edge out RRF if you tune α, but requires per-retriever normalization and tuning.

Sparse vectors and "learned sparse"

Modern vector DBs support sparse vectors natively (a term→weight map), so you can store BM25-style sparse vectors alongside dense ones and do hybrid in one system (04). There are also learned sparse models — SPLADE is the notable one — that produce sparse term weights with term expansion (adding related terms a neural model predicts), getting some semantic benefit while staying exact-match-friendly and interpretable. Useful when you want hybrid behavior from a single sparse index.

When does keyword (sparse) win, and when dense?

Since production queries are a mix you can't predict per request, hybrid is the safe default — it covers both failure modes.

Where hybrid sits in the pipeline

Hybrid is the recall stage: cast a wide net with both retrievers, fuse, and pass the fused top-N candidates to the reranker (06) for precision, then pack (07). Retrieve more candidates than you'll keep (e.g., 50 → rerank → 5). Hybrid improves what's in the candidate pool; reranking improves the order within it.

3. Mental Model

   DENSE (embeddings [03]): meaning   + paraphrase/synonyms   - misses exact tokens
   SPARSE (BM25): term overlap        + codes/names/IDs/quotes - misses synonyms
                  (learned sparse = SPLADE: sparse + term expansion)

   HYBRID = run BOTH → FUSE → candidates
        fuse by RANK: RECIPROCAL RANK FUSION (RRF, scale-free, k≈60)  [or weighted α·dense+(1-α)·sparse]
        query type decides who wins; real traffic is mixed → hybrid is the safe default

   PIPELINE: query → [dense ∥ sparse] → RRF → top-N candidates → RERANK [06] → PACK [07]
            (retrieve wide, e.g. 50, then rerank down to 5)   hybrid boosts RECALL

Mnemonic: dense = meaning, sparse/BM25 = exact tokens; fuse by rank (RRF) to get both. Hybrid beats either alone on mixed/technical corpora — it's a recall booster feeding the reranker.

4. Hitchhiker's Guide

What to look for first: does your corpus contain identifiers/codes/names (technical/enterprise docs almost always do)? If yes, hybrid is a near-certain win over pure semantic. Then fuse with RRF and feed a reranker.

What to ignore at first: hand-tuning fusion weights and learned-sparse models. Start with dense + BM25 fused by RRF (k=60); optimize later.

What misleads beginners:

• Pure semantic for everything. It silently fails on exact-term queries — the most common "why didn't it find the obvious doc?" complaint.
• Comparing raw scores. Cosine and BM25 aren't on the same scale — fuse by rank (RRF), not raw values.
• Retrieving too few before fusion. Pull a wide candidate set from each retriever so good results survive fusion + reranking (06).
• Assuming hybrid is always better. On purely conceptual corpora the keyword side adds little; measure (09).

How experts reason: they default to hybrid (dense + BM25, RRF) for mixed/technical corpora, retrieve a wide candidate pool, then rerank (06); they reach for learned sparse (SPLADE) or tuned weighted fusion only when eval shows a gap; and they measure recall@k for dense-only vs sparse-only vs hybrid on their own queries to confirm the win.

What matters in production: recall@k uplift from hybrid (measure it), the extra latency/infra of a second index, fusion correctness (RRF), and that exact-term/identifier queries actually resolve.

How to debug/verify: build a query set that includes identifier queries (codes/names) and conceptual queries; measure recall@k for dense / sparse / hybrid; you should see hybrid ≥ both, with sparse carrying the identifier queries (09).

Questions to ask: does the vector DB support sparse/BM25 + hybrid in one system (04)? RRF or weighted fusion? how wide is the candidate pool before reranking? learned-sparse available if needed?

What silently gets expensive/unreliable: pure-semantic missing exact terms (silent recall holes), raw-score fusion (bad ordering), too-narrow candidate pools, and an extra sparse index's ops/latency if unmeasured.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Dense captures meaning; sparse/BM25 captures exact terms — each has a complementary blind spot.
• Pure semantic search fails on identifiers (codes, names, SKUs, citations) — the most common silent recall hole.
• Hybrid (dense + sparse, fused) beats either alone on most corpora, especially technical/enterprise.
• Fuse by rank, not raw score — cosine and BM25 are different scales; RRF (Σ 1/(k+rank), k≈60) is the robust default.
• Modern vector DBs support sparse + dense in one system for hybrid (04).
• Learned sparse (SPLADE) adds term expansion — sparse with some semantic benefit.
• Hybrid is the recall stage — retrieve wide, fuse, then rerank for precision (06).
• Real traffic is a mix of conceptual + exact queries → hybrid is the safe default; measure recall@k to confirm (09).

9. Observations from Real Systems

• Hybrid is the default in serious production RAG — enterprise/technical corpora are identifier-heavy, where pure semantic underperforms (00).
• Qdrant, Weaviate, Elasticsearch, pgvector, Pinecone all support hybrid (sparse + dense) with RRF or weighted fusion (04).
• The classic bug report — "it can't find the doc that literally contains the error code" — is solved by adding BM25.
• SPLADE / learned sparse is adopted when teams want hybrid behavior from one sparse index with term expansion.
• Code RAG leans heavily on sparse/keyword (exact symbol names) fused with dense (Phase 11).

10. Common Misconceptions

11. Engineering Decision Framework

SHOULD I USE HYBRID?
  corpus has identifiers/codes/names/citations (technical/enterprise)?  → YES, hybrid (high win)
  purely conceptual prose?                                              → dense may suffice; MEASURE [09]
  mixed real-world traffic (the usual case)?                            → HYBRID (safe default)

IMPLEMENT:
 1. DENSE retriever (embeddings [03]) + SPARSE retriever (BM25 / sparse vectors) — ideally one DB [04].
 2. Retrieve a WIDE pool from each (e.g., 25–50 each).
 3. FUSE with RRF (k≈60); [optional] weighted α-fusion if you'll tune.
 4. Pass fused top-N → RERANK [06] → PACK [07].
 5. MEASURE recall@k: dense vs sparse vs hybrid on YOUR queries (incl. identifier queries) [09].
 6. If gaps remain → learned sparse (SPLADE) or tuned weights.

12. Hands-On Lab

Goal

Show that hybrid beats pure semantic on exact-term queries by measuring dense vs sparse vs RRF-fused recall on your corpus.

Prerequisites

• The embedded chunks from 03/04; a BM25 lib (rank_bm25) or a DB with sparse support (Qdrant); a query set that includes identifier queries (codes/names) and conceptual queries.

Steps

1. Dense retriever: top-k by cosine (from 04).
2. Sparse retriever: build a BM25 index over the same chunks (rank_bm25 or the DB's sparse vectors); top-k by BM25.
3. Identifier query test: run a query for an exact token that appears in one chunk (e.g., an error code or function name). Observe dense often misses it, sparse nails it.
4. Fuse (RRF): combine the two ranked lists with the rrf() function from §2; retrieve fused top-k.
5. Measure: on the labeled query set, compute recall@k for dense-only, sparse-only, hybrid. Expect hybrid ≥ both, with sparse carrying the identifier queries and dense the conceptual ones.
6. Feed the reranker: take the fused top-25 and (optionally) rerank to top-5 to preview the recall→precision handoff (06).

Expected output

A recall@k table (dense / sparse / hybrid) and a concrete identifier-query example where hybrid wins because BM25 found the exact token — demonstrating the complementary blind spots.

Debugging tips

• Hybrid worse than dense → fusion bug (comparing raw scores instead of ranks) or too-narrow pools.
• Sparse finds nothing → tokenization mismatch (e.g., identifiers split oddly); check the BM25 tokenizer.

Extension task

Try weighted fusion (α·normalize(dense)+(1-α)·normalize(sparse)), sweep α, and compare to RRF; or add SPLADE learned-sparse and compare.

Production extension

Run hybrid natively in your vector DB (single index for dense + sparse), wire RRF, retrieve wide → rerank (06), and track recall@k by query type on a dashboard (09).

What to measure

recall@k for dense/sparse/hybrid (split by identifier vs conceptual queries); fusion correctness; latency of the extra retriever.

Deliverables

• A dense vs sparse vs hybrid recall@k comparison (split by query type).
• An identifier-query example proving the hybrid win.
• A fusion implementation (RRF) + a note on when you'd tune weights/use SPLADE.

13. Verification Questions

Basic

1. What does dense search capture that sparse misses, and vice versa?
2. What is BM25, in one sentence?
3. Why fuse by rank (RRF) instead of raw scores?

Applied 4. Give three query types where keyword search beats semantic, and why. 5. Why is hybrid the safe default for mixed production traffic?

Debugging 6. RAG can't find a doc that literally contains the queried error code. Fix? 7. Hybrid underperforms dense alone. Two likely causes.

System design 8. Design hybrid retrieval (dense + sparse + RRF) feeding a reranker for a technical docs corpus.

Startup / product 9. Why does adding hybrid often deliver an outsized recall improvement for enterprise/technical RAG at low cost?

14. Takeaways

1. Dense = meaning, sparse/BM25 = exact terms — complementary blind spots.
2. Pure semantic fails on identifiers; hybrid beats either alone on mixed/technical corpora.
3. Fuse by rank with RRF (k≈60) — don't compare raw cosine vs BM25 scores.
4. Hybrid is the recall stage — retrieve wide, fuse, then rerank for precision (06).
5. Measure recall@k (dense/sparse/hybrid) on your own queries, including identifier queries.

15. Artifact Checklist

• A dense vs sparse vs hybrid recall@k comparison (by query type).
• An RRF fusion implementation.
• An identifier-query example proving the hybrid win.
• A wide-candidate-pool → rerank handoff (06).
• A note on when to use weighted fusion / SPLADE.

Up: Phase 9 Index · Next: 06 — Reranking