Reranking

Phase 9 · Document 06 · RAG Prev: 05 — Hybrid Search · Up: Phase 9 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Retrieval (03–05) is optimized for recall — cast a wide net so the right chunk is somewhere in the top 25–50. But the generator can only use a handful of chunks (07), and the order matters. Reranking is the precision stage: take the wide candidate pool and re-score each chunk against the query with a far more accurate (but slower) model, so the truly best 3–5 rise to the top. It's one of the highest-ROI additions to a RAG pipeline — a cheap cross-encoder reranker routinely turns mediocre retrieval into good retrieval by fixing the ordering that fast vector/keyword search gets approximately right. Skipping it is the most common reason "we retrieve the right doc but it's buried at rank 12 and the model never sees it."

2. Core Concept

Plain-English primer: re-score the shortlist, accurately

Retrieval and reranking use two different model architectures with a deliberate speed/accuracy split:

• Bi-encoder (retrieval): embeds the query and each chunk separately into vectors, then compares by cosine (03). Because chunk vectors are precomputed and indexed, search over millions is fast — but the query and chunk never "see" each other, so the relevance estimate is coarse.
• Cross-encoder (reranking): feeds the query and a chunk together into a model that attends across both and outputs a single relevance score. Far more accurate (it can judge whether this chunk actually answers this query) — but it must run once per (query, chunk) pair at query time, so it's too slow to run over the whole corpus.

The pipeline exploits both: bi-encoder retrieves wide and cheap (recall), cross-encoder reranks the shortlist accurately (precision).

query → RETRIEVE top-50 (bi-encoder, fast, approximate) [03–05]
      → RERANK those 50 with a cross-encoder (slow, accurate) → keep top-5
      → PACK top-5 into the prompt [07] → GENERATE [08]

Why it works: precision@k is what the generator needs

Vector/keyword search gets the candidate set roughly right but the order only approximately — the best chunk is often at rank 8 or 12, not 1. Since you only pass a few chunks to the generator, what matters is precision@k (are the top few actually the most relevant?). The cross-encoder, seeing query+chunk jointly, reorders so the genuinely-best land in the top-k you keep. Empirically this is a large quality jump for little cost — and it decouples retrieval breadth from generation budget: retrieve 50 for recall, keep 5 for the prompt.

from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
def rerank(query, candidates, top_k=5):
    scores = reranker.predict([(query, c["content"]) for c in candidates])  # joint scoring
    return [c for c, _ in sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)[:top_k]]

The two knobs: candidate count (N) and kept count (k)

• N (retrieve-then-rerank pool): how many candidates the reranker scores. Bigger N → higher chance the true best is in the pool (recall) but more reranker calls → more latency/cost. Typical N = 20–100.
• k (kept after rerank): how many you pass to the generator. Smaller k → less noise + cheaper generation, but risk dropping a needed chunk. Typical k = 3–8.

Tune both against eval (09): raise N until recall@N plateaus; set k to the smallest that keeps faithfulness high.

Hosted vs open rerankers

Decision drivers: quality (eval on your data), latency (it's on the hot path), cost (per-pair/per-call), privacy (open/self-host vs API — Phase 5.01), and multilingual. A small open cross-encoder is a great default; Cohere/BGE for higher quality.

The cost is real — budget it

Reranking adds latency (N forward passes of the cross-encoder) and possibly API cost to every query. It's usually worth it, but: keep N reasonable, run the reranker on GPU or use a fast model for low-latency paths, and consider reranking only when the retrieval scores are ambiguous. It's a precision-for-latency trade you measure (Phase 7.08).

Where it sits

Reranking is the bridge between hybrid retrieval (05, which produces the wide fused candidate pool) and context packing (07, which arranges the kept top-k in the prompt). Retrieval maximizes recall; reranking maximizes precision; packing optimizes presentation. All three serve the generator (08).

3. Mental Model

   BI-ENCODER (retrieval): embed query & chunks SEPARATELY → cosine → FAST over millions, coarse [03]
   CROSS-ENCODER (rerank): query + chunk TOGETHER → joint relevance score → ACCURATE, slow (per pair)

   retrieve WIDE (N≈20–100, recall) ──rerank (cross-encoder)──► keep TOP-k (k≈3–8, precision) → pack [07]
        decouples retrieval breadth from generation budget; fixes ORDER (best chunk was at rank 8, not 1)
   KNOBS: N ↑ recall (↑latency/cost) · k ↓ noise (risk dropping a needed chunk) → tune vs eval [09]
   cost = N cross-encoder passes/query (precision-for-latency trade) → measure [7.08]

Mnemonic: retrieve wide with a fast bi-encoder (recall), then re-score the shortlist with a slow accurate cross-encoder (precision), keep the top few. It fixes the ordering — high ROI, real latency cost.

4. Hitchhiker's Guide

What to look for first: are you retrieving wide then reranking to a few, or passing raw vector top-k straight to the generator? Adding a reranker is usually the single biggest quality jump after fixing chunking.

What to ignore at first: LLM-as-reranker and exotic models. Start with a small open cross-encoder (or Cohere Rerank), N≈30, k≈5; tune later.

What misleads beginners:

• Skipping reranking. Vector top-k order is approximate; the best chunk is often in the pool but not at the top — the generator never sees it (00).
• Reranking a tiny pool. If you only retrieve 5 and rerank 5, you can't recover a chunk that was at rank 20 — retrieve wide (05).
• Confusing reranker with retriever. A cross-encoder can't search the corpus (too slow) — it only reorders a shortlist.
• Ignoring latency. N cross-encoder passes add real latency on the hot path — budget it (Phase 7.08).
• Keeping too many (large k). More chunks = more noise + cost + lost-in-the-middle (07).

How experts reason: they retrieve wide (hybrid, N≈20–100) → rerank with a cross-encoder → keep a small k, choosing the reranker by eval quality × latency × cost × privacy, tuning N to a recall plateau and k to the smallest that holds faithfulness, and measuring the precision@k uplift vs no-rerank (09). On latency-critical paths they use a fast/GPU reranker or rerank conditionally.

What matters in production: precision@k / faithfulness uplift from reranking, added latency (p95), reranker cost, and the N/k settings tuned to your SLO and eval.

How to debug/verify: measure recall@N (is the right chunk in the pool?) and precision@k / nDCG with vs without reranking (09); if recall@N is low, fix retrieval (05) — reranking can't add what wasn't retrieved.

Questions to ask vendors: quality on retrieval benchmarks/your data? latency per N pairs? cost per call/pair? multilingual? open-weight/self-hostable (Phase 5.01)?

What silently gets expensive/unreliable: no reranking (buried best chunks), too-small candidate pool (can't recover good chunks), reranker latency on the hot path, and large k (noise + cost).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Reranking is the precision stage — it reorders a wide candidate pool so the truly-best chunks land in the small top-k the generator uses.
• Bi-encoders retrieve (fast, separate encoding); cross-encoders rerank (accurate, joint encoding) — the deliberate speed/accuracy split.
• You must retrieve wide (N) then rerank to a few (k) — a cross-encoder is too slow to search the whole corpus.
• It's high-ROI: a cheap cross-encoder often delivers a large precision@k/faithfulness jump for little cost.
• N and k are the knobs: raise N to a recall plateau; set k to the smallest that holds faithfulness (09).
• Reranking can't recover what retrieval missed — if recall@N is low, fix retrieval first (05).
• It adds latency/cost (N forward passes/query) — budget it on the hot path (Phase 7.08).
• Options: small open cross-encoders (default), BGE/Cohere/Voyage/Jina (stronger), or LLM-as-reranker (flexible, slow).

9. Observations from Real Systems

• Retrieve-then-rerank is the standard production RAG pattern — wide hybrid retrieval → cross-encoder rerank → small top-k (05, 07).
• Cohere Rerank is a popular drop-in hosted reranker; BGE reranker the popular open-weight choice.
• Adding a reranker is frequently the second cheap win (after chunking) on a struggling RAG system (02).
• The classic symptom it fixes: "the right doc is retrieved but ranked low, so the generator never sees it."
• Latency-sensitive products use fast/GPU rerankers or rerank conditionally to keep p95 acceptable (Phase 7.08).

10. Common Misconceptions

11. Engineering Decision Framework

ADD RERANKING (almost always worth it after chunking):
 1. RETRIEVE WIDE: hybrid top-N (N≈20–100) for recall.                         [05]
 2. RERANK: cross-encoder scores each (query, chunk) jointly → reorder.
      reranker = eval quality × latency × cost × privacy:
        default → small open cross-encoder (ms-marco-MiniLM)
        higher quality → BGE / Cohere / Voyage / Jina
        flexible/edge cases → LLM-as-reranker (slow/pricey)
 3. KEEP TOP-k (k≈3–8) → pack [07] → generate [08].
 4. TUNE: N up to a recall@N plateau; k = smallest holding faithfulness.       [09]
 5. MEASURE precision@k / nDCG with vs without reranking; budget latency.      [09,7.08]
 6. If recall@N low → fix RETRIEVAL first (rerank can't recover misses).        [05]

12. Hands-On Lab

Goal

Prove reranking's precision@k uplift: retrieve wide, rerank with a cross-encoder, and measure the quality jump vs raw vector order.

Prerequisites

• The retrieval from 04/05; pip install sentence-transformers; ~10–15 labeled queries with known relevant chunks (gold).

Steps

1. Baseline (no rerank): retrieve top-5 by vector/hybrid; record precision@5 and whether the gold chunk is rank-1.
2. Retrieve wide: retrieve top-30 candidates (the pool).
3. Rerank: score all 30 with cross-encoder/ms-marco-MiniLM-L-6-v2; keep top-5.
4. Measure uplift: recompute precision@5 (and nDCG@5 / MRR) after reranking; compare to baseline. Expect a clear improvement and the gold chunk moving toward rank-1.
5. Sweep N: rerank pools of N=10/30/50; show recall@N plateaus (more candidates stop helping) and find your N.
6. Sweep k + latency: vary k=3/5/8 and record faithfulness vs noise; time the reranker per query (N passes) to see the latency cost.

Expected output

A before/after table: precision@k, nDCG@k, gold-chunk rank, reranker latency — demonstrating reranking lifts precision for a measurable latency cost, plus chosen N/k.

Debugging tips

• No uplift → recall@N is the ceiling (gold not in the pool); widen retrieval/fix hybrid (05).
• Latency too high → smaller reranker, smaller N, GPU, or conditional reranking.

Extension task

Compare the open cross-encoder vs Cohere Rerank (or BGE) on quality + latency + cost for your corpus.

Production extension

Wire reranking after hybrid retrieval in your pipeline; expose N/k as config; track precision@k and reranker p95 latency on a dashboard (Phase 7.08).

What to measure

precision@k, nDCG@k, MRR, gold-chunk rank (with/without rerank); recall@N vs N; reranker latency; reranker comparison.

Deliverables

• A before/after rerank quality comparison (precision@k / nDCG@k).
• Chosen N and k, justified by recall plateau + faithfulness.
• A reranker comparison (open vs hosted) on quality/latency/cost.

13. Verification Questions

Basic

1. What's the difference between a bi-encoder and a cross-encoder, and why use each where?
2. Why can't a cross-encoder be the retriever?
3. What do N and k control?

Applied 4. Why retrieve 50 and keep 5 instead of retrieving 5 directly? 5. Pick a reranker for (a) a low-latency chat, (b) a private corpus, (c) max quality. Justify.

Debugging 6. Reranking gives no quality uplift. What's the likely ceiling, and where do you fix it? 7. p95 latency jumped after adding reranking. Three mitigations.

System design 8. Design retrieve-wide → rerank → keep-k with tuned N/k and a latency budget for a docs assistant.

Startup / product 9. Why is reranking one of the highest-ROI additions to a RAG product, and what's its cost trade-off?

14. Takeaways

1. Reranking is the precision stage — reorder a wide candidate pool so the best chunks reach the small top-k the generator uses.
2. Bi-encoder retrieves (fast); cross-encoder reranks (accurate) — retrieve wide (N), keep few (k).
3. It's high-ROI but can't recover what retrieval missed — fix recall first (05).
4. Tune N (to a recall plateau) and k (smallest holding faithfulness); measure precision@k/nDCG uplift (09).
5. Budget the latency/cost (N passes/query); choose the reranker by quality × latency × cost × privacy.

15. Artifact Checklist

• A before/after rerank comparison (precision@k / nDCG@k, gold-chunk rank).
• Chosen N and k justified by recall plateau + faithfulness.
• A reranker comparison (open vs hosted) on quality/latency/cost.
• A reranker latency measurement on the hot path.
• Reranking wired between hybrid retrieval (05) and packing (07).

Up: Phase 9 Index · Next: 07 — Context Packing