Hitchhiker's Guide to Lab 02 — RAG Pipeline

"The answer to the ultimate question of infrastructure diagnostics is... in the right chunk of text."

This guide explains why every component of the RAG pipeline works the way it does. Read it alongside the code.

Table of Contents

• The 30-Second Mental Model
• Section 1 — Text Chunking
  • Why chunk at all?
  • Character vs token vs semantic chunking
  • The overlap math
  • Chunk size effect on retrieval
• Section 2 — Embeddings
  • What all-MiniLM-L6-v2 actually is
  • What is contrastive learning?
  • Cosine similarity formula
  • Why cosine, not L2 or dot product?
• Section 3 — Vector Search in ChromaDB
  • HNSW — Hierarchical Navigable Small World
  • Key HNSW hyperparameters
  • Why 22 chunks is enough for high recall
• Section 4 — The RAG Prompt Pattern
  • Why the system prompt matters
  • The citation format design
  • Temperature = 0.3 for factual Q&A
  • Hybrid search (extending this lab)
• Section 5 — Evaluation Methodology
  • Why a golden set?
  • RAGAS — the production evaluation framework
  • recall@k, MRR, and nDCG — precise definitions with worked examples
  • Retrieval metrics vs end-to-end answer metrics — they dissociate
• Section 6 — Production Considerations
  • Embedding model air-gap
  • ChromaDB persistence and size limits
  • Concurrent reads
  • What not to log
• Section 7 — Digital Twin Integration
• Section 8 — Interview Cheat-Sheet
  • 5 Phrases That Land Well in Interviews
• Section 9 — Hybrid Retrieval: Dense + Sparse + RRF
  • BM25 from first principles
  • Why dense and sparse fail differently
  • Reciprocal Rank Fusion — merging incomparable rankings
  • Implementing hybrid retrieval offline
• Section 10 — Rerankers: Late Precision
  • Bi-encoder vs cross-encoder — the structural difference
  • The two-stage pattern and its latency budget
  • Offline rerankers for the air gap
• Section 11 — Context Engineering (2025-2026)
  • Contextual retrieval — fixing the orphan-chunk problem
  • Late chunking — embed the document, then split
  • Parent-document retrieval — search small, read big
• Section 12 — GraphRAG and Structured Corpora
  • The failure class: global questions
  • How GraphRAG actually works
  • The honest cost accounting
  • The lighter alternative that usually suffices
• Section 13 — Embedding Models in 2026
  • What actually matters when choosing
  • Matryoshka representation learning — truncatable vectors
  • The offline-servable model classes
• Section 14 — Agentic RAG
  • Query decomposition
  • The self-correction loop
  • When it pays for itself — and how to keep it bounded
• References

The 30-Second Mental Model

OFFLINE (once)             ONLINE (every query)
──────────────────         ──────────────────────────────────────────────
  docs                       question
    │                           │
    ▼                           ▼ embed (same model, ~1ms)
  chunk text               query_vector  [384-dim float32]
    │                           │
    ▼ embed (~30ms/chunk)       ▼ cosine search (HNSW, ~40ms)
  chunk_vectors            top-k chunks
    │                           │
    ▼ upsert                    ▼ format numbered citations
  ChromaDB                  RAG prompt
                                │
                                ▼ LLM (~15s)
                              answer with [Source: filename]

The key insight: you only embed each document once, but retrieve from it thousands of times. The cost of offline embedding is amortized to near-zero per query. The LLM — not retrieval — is the latency and cost bottleneck.

Section 1 — Text Chunking

Why chunk at all?

LLMs have finite context windows (32K–128K tokens). Even within that limit, long contexts degrade answer quality — models lose focus on the relevant passage when buried in irrelevant text. Chunking lets you extract only the relevant pieces (e.g., 4 × 1000-char chunks ≈ 600 tokens vs the 41,000 chars in all 5 docs combined).

Character vs token vs semantic chunking

This lab uses recursive character splitting (the chunk_text() function in ingest.py), which is the sweet spot for technical documents.

The overlap math

Given chunk size $C$ and overlap $O$, a document of $D$ characters produces:

$$N_{\text{chunks}} = \left\lceil \frac{D - O}{C - O} \right\rceil$$

For $D = 8000$, $C = 1000$, $O = 150$:

$$N = \left\lceil \frac{8000 - 150}{1000 - 150} \right\rceil = \left\lceil \frac{7850}{850} \right\rceil = \lceil 9.24 \rceil = 10 \text{ chunks}$$

The overlap ensures that a sentence split across a boundary is fully captured by at least one chunk. Without overlap, a fact at the boundary of two chunks might be truncated in both — making it unretrievable.

Chunk size effect on retrieval

The fundamental tension:

• Too small (< 200 chars): chunks contain single sentences. High precision but low recall — a multi-sentence fact is split, so a single query may only retrieve the first sentence without the critical qualifier.

• Too large (> 2000 chars): chunks contain multiple facts. High recall but low precision — retrieving the right fact also brings in a paragraph of unrelated text, diluting the prompt context.

• Sweet spot (~800–1200 chars): captures a complete "thought unit" (a paragraph, a procedure step, a table) while staying focused.

Empirically test this with eval.py --retrieval-only at chunk sizes 300, 500, 1000, 1500.

Section 2 — Embeddings

What all-MiniLM-L6-v2 actually is

It is a 22M-parameter BERT-based encoder fine-tuned by Microsoft using contrastive learning on 1 billion sentence pairs from diverse sources (Reddit, StackOverflow, news articles, scientific papers, etc.).

Architecture: 6-layer transformer → pooling → L2-normalise → 384-dimensional vector

The 384-dim output is deliberately compact (vs 768 for BERT-base or 1536 for text-embedding-3-small). This reduces memory and search latency with only a small quality degradation for semantic similarity tasks.

What is contrastive learning?

During training, pairs of semantically similar sentences are pushed together in vector space, while dissimilar sentences are pushed apart. The loss function (a form of InfoNCE) is:

$$\mathcal{L} = -\log \frac{\exp(\text{sim}(a_i, p_i) / \tau)}{\sum_{j} \exp(\text{sim}(a_i, p_j) / \tau)}$$

Where $a_i$ is an anchor sentence, $p_i$ is a positive (semantically similar) example, $p_j$ are negative examples (other sentences in the batch), and $\tau$ is a temperature scaling parameter.

The result: sentences with similar meaning land close together in the 384-dim space, regardless of exact wording. "What does RTU stand for?" and "RTU means Remote Terminal Unit" will have high cosine similarity even though they share almost no words.

Cosine similarity formula

$$\text{cosine_similarity}(A, B) = \frac{A \cdot B}{|A| \cdot |B|}$$

Since all-MiniLM-L6-v2 L2-normalises its outputs ($|A| = |B| = 1$), this simplifies to:

$$\text{cosine_similarity}(A, B) = A \cdot B$$

Cosine distance (what ChromaDB returns) is:

$$d_\text{cosine} = 1 - \text{cosine_similarity}(A, B)$$

This is why rag.py converts: similarity = 1.0 - distance.

Why cosine, not L2 or dot product?

For sentence embeddings where all vectors are unit-length, cosine and dot product are equivalent. Cosine is the standard because it's robust to unnormalised models.

Section 3 — Vector Search in ChromaDB

HNSW — Hierarchical Navigable Small World

ChromaDB uses HNSW for approximate nearest neighbour (ANN) search. Understanding HNSW at interview-depth:

Core idea: build a multi-layer graph where each node (chunk) is connected to its nearest neighbours. The top layers have long-range connections (fast navigation); the bottom layer has short-range connections (fine-grained search).

Layer 2 (sparse): A ──── E
                  │
Layer 1:          A ─── C ─── E
                  │     │
Layer 0 (dense):  A─B─C─D─E─F─G

Query traversal: enter at a random node in the top layer, greedily move to the neighbour closest to the query vector, descend when no closer neighbour exists, continue at the next layer.

Complexity: $O(\log N)$ for search (vs $O(N)$ for brute-force). At N=22 chunks, brute-force is fine; HNSW matters at N > 10,000.

Key HNSW hyperparameters

To configure in ChromaDB:

collection = client.get_or_create_collection(
    "domain_docs",
    metadata={
        "hnsw:space": "cosine",
        "hnsw:M": 32,              # more connections = better recall
        "hnsw:ef_search": 50,      # broader beam during query
    }
)

Why 22 chunks is enough for high recall

The evaluation shows ~95% retrieval hit rate at N=22 with top_k=4. This works because:

1. The corpus is domain-specific — embeddings from different domains (SCADA vs pump stations vs digital twins) are naturally well-separated in the 384-dim space.
2. 22 chunks gives each query 5× more chunks than it retrieves (top_k=4). There's room for false positives without missing the true positive.

At N > 10,000 chunks, you'd need to tune ef_search and consider filtering by metadata (e.g., where={"source": "scada-overview.md"}) to maintain precision.

Section 4 — The RAG Prompt Pattern

Why the system prompt matters

The system prompt in rag.py does three critical things:

1. Anti-hallucination constraint: "Use ONLY the information provided. Do not use prior knowledge."
2. Citation instruction: "Always cite the source document with [Source: filename]."
3. Graceful degradation: "If not in context, say 'I don't have that information in the provided documents.'"

Without (1), a model trained on Wikipedia will happily answer "What is SCADA?" from its training data — bypassing your retrieval entirely. This makes evaluation meaningless and defeats the purpose of RAG.

Without (3), the model may hallucinate plausible-sounding but wrong infrastructure data. In a digital twin context, a made-up pressure threshold could cause a spurious alarm — or mask a real one.

The citation format design

[1] Source: scada-overview.md (chunk 0)
SCADA stands for Supervisory Control and Data Acquisition...

[2] Source: pipeline-monitoring.md (chunk 2)
Normal operating pressure for a water distribution system...

The numbered format lets the LLM reference [1] or [2] in its answer rather than copy-pasting filenames. This produces clean, audit-friendly output like:

"SCADA stands for Supervisory Control and Data Acquisition [1]. Typical pipeline operating pressure is 40–80 PSI [2]."

Temperature = 0.3 for factual Q&A

The eval harness uses temperature=0.0 for reproducible results. The server defaults to 0.3 for natural-sounding answers that are still grounded.

Hybrid search (extending this lab)

Pure vector search is sometimes outperformed by keyword (BM25) search for exact technical terms (acronyms, part numbers, threshold values). The solution is hybrid search:

1. Get top-20 from vector search
2. Get top-20 from BM25 keyword search
3. Fuse using Reciprocal Rank Fusion:

$$\text{RRF}(d) = \sum_{r \in \text{rankers}} \frac{1}{k + r(d)}$$

where $k = 60$ (constant), $r(d)$ is the rank of document $d$ in ranker $r$.

Combine the fused list to top-4. Hybrid search consistently outperforms either approach alone, especially for specific numerical queries like "100 PSI alarm threshold".

Section 5 — Evaluation Methodology

Why a golden set?

You can't tell if a RAG system is working by eyeballing a few responses. You need a golden question set — questions with known expected answers and expected source documents — to measure quality quantitatively and detect regressions when you change chunking, embedding models, or prompt templates.

The eval.py harness measures two things:

Retrieval hit rate: does the expected document appear in top-k results?

$$\text{Retrieval Hit Rate} = \frac{\text{questions where correct doc retrieved}}{N}$$

This measures your retrieval system independently of the LLM. If this is low, fix chunking, try a different embedding model, or increase top_k.

Answer hit rate: does the LLM answer contain the expected keyword?

$$\text{Answer Hit Rate} = \frac{\text{questions where answer contains expected keyword}}{N}$$

This measures end-to-end pipeline quality. If retrieval is high but answer is low, the problem is the LLM's instruction-following or your prompt template — not retrieval.

RAGAS — the production evaluation framework

RAGAS provides four standardised metrics:

Overall RAG Score ≈ Faithfulness × Context_Recall × Answer_Relevance

Faithfulness is the most important metric for safety-critical applications. A faithfulness score < 0.8 means the model is adding information from its training data rather than the retrieved context.

Install and run:

pip install ragas

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall, context_precision
from datasets import Dataset

data = {
    "question": ["What does SCADA stand for?"],
    "answer": ["SCADA stands for Supervisory Control and Data Acquisition"],
    "contexts": [["SCADA stands for Supervisory Control and Data Acquisition..."]],
    "ground_truth": ["Supervisory Control and Data Acquisition"],
}
evaluate(Dataset.from_dict(data), metrics=[faithfulness, answer_relevancy])

recall@k, MRR, and nDCG — precise definitions with worked examples

Hit rate is binary and order-blind: it can't tell "right chunk at rank 1" from "right chunk at rank 4", and it can't handle partially-relevant chunks. Production retrieval work uses three sharper instruments — each answers a different question. The worked examples below use a tiny golden set of 3 queries whose ranked top-4 results are:

recall@k — of all relevant chunks, what fraction made it into the top k?

\[ \text{recall@}k ;=; \frac{1}{|Q|} \sum_{q \in Q} \frac{|\text{relevant}_q \cap \text{top-}k_q|}{|\text{relevant}_q|} \]

Each query above has one relevant chunk, so recall@4 = (1 + 1 + 0) / 3 = 0.67, while recall@1 = (1 + 0 + 0) / 3 = 0.33. Use recall@k when the LLM will read all k chunks anyway — presence in the window is what matters, position less so.

MRR (Mean Reciprocal Rank) — how high is the first relevant hit?

\[ \text{MRR} ;=; \frac{1}{|Q|} \sum_{q \in Q} \frac{1}{\text{rank}_q^{\text{first relevant}}} \]

with the reciprocal defined as 0 on a miss. Here: Q1 → 1/1, Q2 → 1/3, Q3 → 0, so MRR = (1 + 0.333 + 0) / 3 = 0.44. MRR only credits the first relevant result — the right metric for single-fact Q&A (one correct chunk suffices) and for UIs that display a single "best source" citation.

nDCG@k (normalized Discounted Cumulative Gain) — is the whole ordering good, with graded relevance?

The two metrics above treat relevance as binary. nDCG lets you grade it — "this chunk fully answers (rel = 3), that one partially (rel = 1), that one not at all (rel = 0)":

\[ \text{DCG@}k = \sum_{i=1}^{k} \frac{2^{rel_i} - 1}{\log_2(i+1)}, \qquad \text{nDCG@}k = \frac{\text{DCG@}k}{\text{IDCG@}k} \]

Worked example — retrieved chunks with graded relevance (3, 0, 1) in positions 1–3:

\[ \text{DCG} = \frac{7}{\log_2 2} + \frac{0}{\log_2 3} + \frac{1}{\log_2 4} = 7 + 0 + 0.5 = 7.5 \]

The ideal ordering (3, 1, 0) gives IDCG = 7 + 1/1.585 + 0 ≈ 7.63, so nDCG = 7.5 / 7.63 ≈ 0.98. The log denominator encodes "a good chunk at rank 1 is worth more than the same chunk at rank 4"; dividing by the ideal ordering's score makes results comparable across queries with different numbers of relevant chunks. nDCG is the standard metric for evaluating rerankers (Section 10), whose entire job is ordering.

The golden-set discipline, restated as process: the eval set lives in version control next to the code; every retrieval failure observed in production becomes a new golden case; relevance judgments are fixed before experiments, never adjusted to flatter a change; and the whole suite reruns on every change to chunking, embedder, top-k, or prompt. It is the retrieval analogue of a unit-test suite — 20–50 well-chosen queries catch most regressions. You are building a tripwire, not publishing a benchmark.

Retrieval metrics vs end-to-end answer metrics — they dissociate

Retrieval metrics (recall@k, MRR, nDCG) and answer metrics (answer hit rate, faithfulness, human grades) measure different subsystems, and they routinely move independently. All four quadrants occur in practice, and each has a different diagnosis:

A concrete dissociation you can reproduce with this lab: raise top_k from 4 to 12. recall@k rises monotonically — adding results can only help a recall metric. Answer quality often falls: more distractor chunks dilute the prompt, and the one relevant chunk lands mid-context where models attend worst. The retrieval dashboard says "improvement"; users see a regression.

So: measure both, on every change, and never let one proxy for the other. Retrieval metrics tell you where to work (chunking, embeddings, fusion vs prompt, model); answer metrics tell you whether users are actually better off.

Section 6 — Production Considerations

Embedding model air-gap

all-MiniLM-L6-v2 is downloaded from HuggingFace Hub on first use. For a production air-gapped deployment:

# On a machine with internet:
pip install huggingface_hub
python -c "from huggingface_hub import snapshot_download; snapshot_download('sentence-transformers/all-MiniLM-L6-v2')"
# → Downloads to ~/.cache/huggingface/hub/models--sentence-transformers--all-MiniLM-L6-v2/

# Package the cache dir, transfer to air-gapped machine, restore at same path
# Then load with:
model = SentenceTransformer('/path/to/cached/model')  # or set HF_HOME env var

ChromaDB persistence and size limits

PersistentClient(path="chroma_db/") writes to SQLite. Practical limits:

For the digital twin use case (continuous sensor ingestion), you'll need TTL-based eviction: delete chunks older than 30 days, or use a separate short-term vs long-term collection.

Concurrent reads

ChromaDB's PersistentClient supports concurrent reads from multiple threads/processes via SQLite's WAL mode. The FastAPI server handles concurrency correctly because:

• Embeddings are generated per-request (sentence-transformers is thread-safe)
• ChromaDB reads use WAL (multiple readers, one writer)
• The pipeline is stateless per-query

For write concurrency (parallel ingestion), use a queue and single writer process.

What not to log

The audit log in rag_server.py intentionally logs source filenames but not:

• Raw retrieved text (may contain PII from future document additions)
• Full question text (truncated to 80 chars)
• LLM answer text

For compliance (GDPR, ISO 27001), store logs with TTL and restrict access to the log file.

Section 7 — Digital Twin Integration

This lab is the AI assistant layer of the 4-layer Digital Twin architecture described in docs/digital-twin-concepts.md:

4. Visualisation ◄── anomaly alerts, NL explanations  ◄──┐
3. Analytics     ◄── predictive maintenance insights  ◄──┤
2. Model         ◄── state estimator, physics model   ◄──┤
1. Data          ◄── sensors, RTUs, PLCs              ◄──┘
                      ▲ live sensor feed                   ▲
                      └── RAG pipeline answers operator ───┘
                          questions about any layer

Concrete use cases enabled by this RAG pipeline:

1. Operator knowledge lookup (NL query → doc answer):

"Why is my pump showing high vibration?" → Retrieves pump-station-ops.md → "Bearing vibration > 10 mm/s indicates worn bearings. Schedule replacement within 24 hours."

2. Alarm explanation (alarm code → contextual doc):

"Alarm code P-HIGH-001 has triggered. What should I do?" → The alarm code can be pre-mapped to a question; RAG provides the response from the SOP.

3. Sensor calibration guidance (on-demand procedure):

"How do I calibrate the flow meter on pump 3?" → Retrieves sensor-types.md → step-by-step calibration procedure.

4. Change documentation (query new docs as they're added): When a new SOP or equipment manual is added to docs/, re-run ingest.py. The pipeline immediately starts answering questions from it — no retraining required.

Section 8 — Interview Cheat-Sheet

5 Phrases That Land Well in Interviews

1. "Retrieval hit rate and answer hit rate measure different failure modes — one is a retrieval problem, the other is a prompt adherence problem."

2. "The key property of good chunking is that each chunk contains exactly one retrievable fact — not zero, not five. Overlap prevents boundary-split facts from being unretrieval."

3. "Faithfulness score below 0.8 is a red flag — the model is drawing on training data rather than retrieved context, which is dangerous in safety-critical applications like infrastructure monitoring."

4. "For air-gapped deployments, the embedding model must be pre-cached. The first-download-from-HuggingFace assumption breaks in a SCADA environment with no outbound internet."

5. "RAG and fine-tuning are not mutually exclusive — a fine-tuned model with better domain vocabulary produces better embeddings and more coherent answers when combined with RAG than a base model does."

Section 9 — Hybrid Retrieval: Dense + Sparse + RRF

Section 4 previewed hybrid search in four lines. This section builds it from the ground up, because for industrial corpora it is not an optional extension — pure dense retrieval is the single most common reason a RAG system that shines in the demo fails in the plant.

BM25 from first principles

BM25 ("Best Matching 25", the Okapi ranking function) is not legacy technology to be tolerated — it is thirty years of information-retrieval theory compressed into one scoring function over exact term matches. For a query \( q \) with terms \( t \), document \( d \), term frequency \( f(t,d) \), document length \( |d| \), average document length \( \overline{|d|} \), corpus size \( N \), and \( n_t \) documents containing \( t \):

\[ \text{BM25}(q,d) = \sum_{t \in q} \underbrace{\ln!\left(\frac{N - n_t + 0.5}{n_t + 0.5} + 1\right)}{\text{IDF — term rarity}} \cdot \underbrace{\frac{f(t,d),(k_1+1)}{f(t,d) + k_1\left(1 - b + b,\frac{|d|}{\overline{|d|}}\right)}}{\text{saturating TF, length-normalised}} \]

Each mechanism has one job:

• IDF (rarity weighting): the corpus itself decides which words carry information. "the" appears in every document → IDF ≈ 0, contributes nothing. "NPSH" appears in 2 of 500 documents → IDF = ln(498.5/2.5 + 1) ≈ 5.3, dominates the score. No training, no embeddings — pure counting.
• TF saturation: the term-frequency fraction rises with \( f(t,d) \) but asymptotes at \( k_1 + 1 \). The first occurrence of "cavitation" is strong evidence the document is about cavitation; the tenth adds almost nothing. Without saturation, a document that stuffs a keyword 50 times would beat every genuinely relevant document. \( k_1 \) (typically 1.2–2.0) sets how quickly the curve flattens.
• Length normalisation: dividing TF by a factor that grows with \( |d| / \overline{|d|} \) counters the fact that a 40-page manual mentions everything once by accident. \( b \in [0,1] \) dials the strength: b = 0 ignores length entirely, b = 1 normalises fully; b = 0.75 is the decades-tested default.

Why dense and sparse fail differently

The two retrievers are not "old vs new" — they are orthogonal sensors with disjoint blind spots:

The dense failure deserves a mechanism, not a hand-wave: an embedding model can only give stable meaning to strings it saw patterns for in training. A valve tag like VLV-3042-B gets shredded by the BPE tokenizer into arbitrary subword fragments ("VL", "V-", "30", "42") whose combined embedding is essentially noise — two different part numbers can land closer together in vector space than the query and its own target document. BM25 doesn't care: the token either matches or it doesn't, and because part numbers are rare, IDF makes a match decisive.

A digital-twin corpus is the worst case for either retriever alone: prose SOPs (dense territory) saturated with tags, alarm codes, and thresholds (sparse territory) — often in the same sentence. So you run both.

Reciprocal Rank Fusion — merging incomparable rankings

You now have two ranked lists. The tempting mistake is mixing scores: \( \alpha \cdot \text{cosine} + \beta \cdot \text{BM25} \). But cosine lives in roughly [0, 1] while BM25 is unbounded and corpus-dependent (0 to 40+, shifting whenever the corpus changes) — any linear combination mixes incompatible units, and whatever α you tuned breaks on the next re-ingest. Ranks, unlike scores, are dimensionless. Reciprocal Rank Fusion uses only positions:

\[ \text{score}(d) = \sum_{r \in \text{rankers}} \frac{1}{k + \text{rank}_r(d)} \]

with \( k = 60 \) by convention (Cormack et al., 2009). Why k matters: with k = 0, rank 1 scores 1.0 vs rank 2's 0.5 — one ranker's top pick dominates everything. With k = 60, rank 1 scores 1/61 vs rank 2's 1/62 — nearly equal, so a document must place decently in multiple lists to win. RRF rewards consensus over any single ranker's enthusiasm. Worked example:

C7 — good in both lists — beats C3 (one ranker's favourite) and C9 (found by only one ranker). Agreement between independent evidence sources is itself evidence; that is the entire philosophy of RRF, and why it consistently matches or beats trained fusion on out-of-domain data.

Implementing hybrid retrieval offline

Everything needed is pip-installable and runs with zero network access — rank_bm25 is pure Python (SQLite FTS5 is the heavier-duty alternative, already inside Python's stdlib sqlite3). The key discipline: the sparse index and ChromaDB must share the same chunk IDs, and both get rebuilt by the same ingest.py run.

from rank_bm25 import BM25Okapi
from collections import defaultdict
import numpy as np

# ingest time — same chunks, same IDs as ChromaDB
tokenized = [c.lower().split() for c in chunk_texts]   # keep it simple; don't split
bm25 = BM25Okapi(tokenized)                            # hyphenated part numbers apart!

# query time
def hybrid_search(query, top_k=4, fetch_k=20, k_rrf=60):
    dense_ids = collection.query(query_texts=[query], n_results=fetch_k)["ids"][0]
    sparse_order = np.argsort(-bm25.get_scores(query.lower().split()))[:fetch_k]
    scores = defaultdict(float)
    for rank, cid in enumerate(dense_ids, start=1):
        scores[cid] += 1.0 / (k_rrf + rank)
    for rank, idx in enumerate(sparse_order, start=1):
        scores[chunk_ids[idx]] += 1.0 / (k_rrf + rank)
    return sorted(scores, key=scores.get, reverse=True)[:top_k]

One tokenisation warning: naive .split() keeps VLV-3042-B intact (good), but if you add stemming/punctuation stripping later, verify identifiers survive — destroying them silently removes the exact advantage you added BM25 for. And as always: validate the hybrid pipeline against pure-dense on the golden set (Section 5) before declaring victory.

Section 10 — Rerankers: Late Precision

Bi-encoder vs cross-encoder — the structural difference

Everything so far uses a bi-encoder: query and document are encoded independently into vectors, and relevance is a single dot product. That independence is what makes vector search fast — documents are embedded once, offline, and query time is one cheap comparison per candidate. But it imposes a structural ceiling: the document was compressed into 384 floats before your query existed. The encoder had to guess, blind, which aspects of the chunk would ever matter. All query–document interaction is squeezed through one number.

A cross-encoder removes that bottleneck. It concatenates the pair — [CLS] query [SEP] chunk — and runs the combined sequence through the full transformer. Now every attention layer lets every query token attend to every chunk token: "112 PSI" in the query lines up against "threshold of 110 PSI" in the chunk; a "not" flips the meaning of a procedure step; a unit mismatch (bar vs PSI) surfaces. The output head reads the final representation and emits a single relevance score. This computes interaction features that a bi-encoder structurally cannot represent — not because the bi-encoder is badly trained, but because its architecture forces all document information through a query-agnostic bottleneck first.

bi-encoder (retrieval):              cross-encoder (reranking):
query ─► encoder ─► v_q ─┐           [ query ; chunk ] ─► one encoder,
chunk ─► encoder ─► v_d ─┴► v_q·v_d     full cross-attention ─► score
(chunk encoded blind, once,          (joint encoding, per pair,
 cacheable → fast, coarse)            nothing cacheable → slow, precise)

The price of precision: nothing is precomputable. Scoring N chunks costs N full forward passes per query. You cannot cross-encode a million chunks. You can absolutely cross-encode fifty.

The two-stage pattern and its latency budget

That asymmetry dictates the standard 2025+ architecture — retrieve wide, then rerank narrow:

query ─► hybrid retrieval (Section 9) ─► top-50 candidates   [recall stage]
      ─► cross-encoder scores all 50  ─► top-5 to the LLM    [precision stage]

Stage 1 is judged on recall@50 ("did the truth make the pool?" — cheap to make large); stage 2 on nDCG@5 ("is the truth at the top?" — Section 5's ordering metric, which is why nDCG exists in your toolbox). Indicative latency on lab-class hardware with a 0.3–0.6B reranker:

Read the last two rows together: against a local LLM's tens of seconds, even a CPU reranker is a rounding error. The latency argument that kills rerankers in a 100 ms web-search SLA simply does not apply to air-gapped RAG. Size fetch_k to your box and move on.

A free bonus: the reranker's score is a usable relevance signal. If the best of 50 candidates scores below a threshold you calibrated on the golden set, the honest response is Section 4's graceful degradation — "I don't have that information" — instead of feeding the LLM top-ranked garbage.

Offline rerankers for the air gap

The BGE-reranker class is the current default for local deployments: bge-reranker-v2-m3 (~2.3 GB, multilingual including Arabic — directly relevant to this JD), bge-reranker-base/-large (smaller, En/Zh-centric), and the tiny ms-marco-MiniLM cross-encoders (~90 MB, English, surprisingly strong). All are plain HuggingFace models: pre-cache them with the exact snapshot_download pattern from Section 6 and load from the local path.

from sentence_transformers import CrossEncoder
reranker = CrossEncoder("/models/bge-reranker-v2-m3")            # pre-cached, offline

pairs   = [(query, chunk_text[cid]) for cid in candidate_ids]    # top-50 from Section 9
scores  = reranker.predict(pairs)                                # one score per pair
top5    = [cid for _, cid in sorted(zip(scores, candidate_ids), reverse=True)[:5]]

Evaluate the stage in isolation: same golden set, nDCG@5 with and without reranking. Typical result on identifier-heavy technical corpora: hybrid retrieval fixes the misses, the reranker fixes the ordering, and the combination is worth more than either.

Section 11 — Context Engineering (2025-2026)

Chunking as described in Section 1 has a structural flaw that the field spent 2024–2025 fixing: a chunk is embedded knowing nothing outside its own characters. Three techniques attack that flaw from different angles.

Contextual retrieval — fixing the orphan-chunk problem

The failure, concretely: chunk 7 of pump-station-ops.md reads "Its alarm threshold defaults to 110 PSI and may only be raised with engineering approval." Whose threshold? The pronoun's referent lives in chunk 6; the station series lives in the document title. Embedded in isolation, the vector encodes "threshold, PSI, approval" — but no pump, no station. The query "discharge pressure alarm threshold for B-series booster stations" sails right past it. Chunking amputated the context; the embedding never saw it.

The fix (Anthropic's "contextual retrieval", September 2024) is almost embarrassingly direct: at index time, for each chunk, hand an LLM the full document plus the chunk and ask for 1–2 sentences situating the chunk; prepend that blurb to the chunk before embedding it (and before BM25-indexing it):

[generated context]  This chunk is from the pump-station operations manual,
in the section on discharge-pressure alarms for B-series booster stations.
[original chunk]     Its alarm threshold defaults to 110 PSI and may only …

Why it works: the blurb restores exactly the information chunking destroyed — document identity, section topic, pronoun referents — as text, so both the dense vector and the sparse index carry it. Anthropic's reported numbers: contextual embeddings cut retrieval failures ~35%; combined with contextual BM25, ~49%; with reranking on top, ~67%.

The cost, honestly: one LLM call per chunk, at index time only — queries pay nothing. On this track's air-gapped stack that means an overnight batch on the Lab 1 model, and prompt caching (see the Lab 1 guide, §13.4) makes it cheap: the long document is the shared cached prefix across all of that document's chunk-calls, so each call re-prefills only the chunk and instruction. And because this lab's chunk IDs are content hashes, re-ingestion only re-contextualises chunks that actually changed.

Late chunking — embed the document, then split

The standard pipeline splits first and embeds each piece in isolation. Late chunking (Jina AI, 2024) inverts the order:

1. Run the entire document through a long-context embedding model once. A transformer encoder produces one contextualised vector per token — and each token's vector is already conditioned on the whole document via attention. The token "Its" literally carries information about the pump it refers to, because attention resolved the reference during the forward pass.
2. Then apply chunk boundaries — and mean-pool each span's token vectors into one chunk embedding.

Same number of stored vectors, same query path, same top-k — but every chunk vector has inherited document-level context, with zero extra LLM calls. The requirements: an embedding model with a long max sequence (8k+: BGE-M3, nomic-embed, the jina-embeddings line — Section 13) and access to token-level outputs rather than only the pooled sentence vector.

How the two techniques compare — they solve the same disease in different organs:

They compose: teams increasingly run late chunking (nearly free) and add contextual blurbs where the corpus is pronoun-heavy or the sparse side needs the boost.

Parent-document retrieval — search small, read big

Section 1 presented chunk size as a tension: small chunks match precisely, large chunks give the LLM enough context. That's a false forced choice — decouple the search granularity from the read granularity:

• Index small: embed child chunks of ~300 chars (one fact each — maximum retrieval precision), each carrying its parent section's ID in metadata.
• Feed big: at query time, match against children, collect their (deduplicated) parents, and put the parent sections (~1500–3000 chars) into the prompt.

collection.add(
    ids=[child_id],
    documents=[child_text],                       # what gets embedded & matched
    metadatas=[{"parent_id": section_id,          # what gets retrieved & shown
                "source": filename}],
)
# query: match children → {parent_ids} → fetch parent text from a doc store

The only cost is prompt tokens. Mental model for the three techniques together: contextual retrieval and late chunking fix the vector (what the chunk means); parent-document retrieval fixes the payload (what the LLM gets to read).

Section 12 — GraphRAG and Structured Corpora

The failure class: global questions

Everything in this guide so far shares one assumption: the answer lives in a few chunks that are semantically similar to the question. Some questions violate that shape entirely:

• "Which pump stations share the same SCADA vendor?"
• "What failure modes recur across all of this year's incident reports?"
• "Summarise our chlorination practices across all sites."

These are aggregation (global) questions: the answer is a join or a fold over dozens of chunks scattered through the corpus. No single chunk contains it — so no chunk is particularly similar to the question, and vector top-k returns k fragments that merely share vocabulary with it. The LLM then does something worse than failing: it confidently summarises a biased sample of 4 chunks as if it were the whole corpus. This is not "top-k needs tuning". Top-k is the wrong shape for the question.

How GraphRAG actually works

GraphRAG (Microsoft Research, 2024) restructures the corpus at index time so that global structure exists before any question arrives:

INDEX TIME
  chunks ──LLM──► (entity, relation, entity) triples + descriptions
  triples ──────► knowledge graph
  graph ──Leiden──► communities (hierarchical clusters of related entities)
  communities ──LLM──► written summaries, bottom-up (station → region → corpus)

QUERY TIME
  local search:  match query entities → traverse neighbours → pull their
                 chunks + relations → answer   (entity-centric questions)
  global search: map-reduce over community summaries — each summary answers
                 partially (map), an LLM merges them (reduce)   (corpus-wide questions)

The LLM extraction pass is the heart of it: an LLM reads every chunk and emits entities ("Pump Station 7", "VendorX SCADA v4") and relations ("uses", "supplied-by", "located-in"). Community detection (the Leiden algorithm — modularity-based graph clustering) then finds densely-connected groups, and pre-written community summaries give global questions something retrievable that no original chunk ever was.

The honest cost accounting

• Index-time LLM cost explodes. Every chunk gets at least one extraction call; total generation is a multiple of your corpus token count. On an air-gapped 30B at ~12 tok/s, a 10,000-chunk corpus is days of GPU time — repeated on re-index.
• Maintenance is the real bill. A document update invalidates extracted entities, community membership, and every summary above them. Incremental updating exists but is fiddly; many teams just re-index on a schedule, paying the full cost again.
• The graph is only as good as the extractor. A mid-size local model mis-extracts entities and relations, and those errors are baked invisibly into the graph — you audit prose badly, and graphs worse.

The lighter alternative that usually suffices

Before reaching for GraphRAG, ask: is the relational structure already explicit somewhere? In a digital twin, it is — the asset registry already knows pump → station → vendor → maintenance history. Then the right architecture is routing, not graph extraction:

• Aggregation questions → SQL over the asset database via function calling — exactly what this track's Lab 4 builds (lab-04 function calling). "Which stations share a SCADA vendor" is a one-line GROUP BY, computed exactly, in milliseconds, with zero hallucination surface.
• Scoped semantic questions → metadata filters + vectors (where={"site": "..."}) — the Section 3 pattern.
• Prose questions → hybrid retrieval + reranking (Sections 9–10).

The decision rule: GraphRAG earns its cost only when relationship-heavy questions target unstructured prose whose structure you cannot extract once into tables. If a nightly ETL job can produce the table, the table wins — cheaper, deterministic, auditable. In this track's context, the function-calling database already embodies the lightweight answer.

Section 13 — Embedding Models in 2026

What actually matters when choosing

The reflex — "pick the top of the MTEB leaderboard" — is wrong for the same reason trusting quantization charts is wrong: public retrieval benchmarks are English-web-heavy, are explicit optimisation targets (with all the contamination that implies), and contain approximately zero Arabic SCADA procedures. A model's benchmark rank and its performance on your corpus are correlated, not interchangeable. Your golden set (Section 5) is the only leaderboard that matters. What to actually weigh:

Matryoshka representation learning — truncatable vectors

Normally you cannot shorten an embedding: information is spread across all dimensions in no particular order, so truncating 1024 → 256 destroys the geometry. Matryoshka Representation Learning (MRL) makes truncation a designed-in feature by changing the training objective: the loss is applied not just to the full vector but to a nested family of prefixes —

\[ \mathcal{L}{\text{MRL}} = \sum{m \in {64, 128, 256, \dots, d}} c_m , \mathcal{L}_{\text{contrastive}}\big(v[1{:}m]\big) \]

— so the first 64 dims alone must already work as an embedding, then the first 128 must work better, and so on. The optimizer has no choice but to pack the most discriminative directions into the earliest coordinates: importance-ordered dimensions, coarse-to-fine, like a progressive JPEG (or the nesting dolls of the name). At deployment you truncate to your budget and L2-renormalise:

• 1024 → 256 dims = 4× less vector storage, ~4× faster brute-force scoring, smaller HNSW graph
• typical cost: 1–2% retrieval quality — verified on your golden set, as always

Practical consequence for this stack: a strong 1024-dim MRL model truncated to 256 often beats all-MiniLM-L6-v2's native 384 dims at comparable serving cost — you get big-model semantics at small-model prices.

The offline-servable model classes

The 2026 local-deployable landscape, properties over hype — every one of these is downloadable to the HF cache and served offline exactly like Section 6:

Two operational contracts that bite in production:

1. Instruction prefixes are part of the model. E5/BGE-class models were trained with role prefixes ("query:" vs "passage:"). Indexing with the prefix and querying without it (or vice versa) silently costs you several points of recall — the same "same model, same convention, both sides" rule as Section 2, one level deeper.
2. Changing the embedder means re-indexing everything. Vectors from different models live in unrelated spaces; a collection with mixed embeddings is garbage that still returns results. Budget the re-embed (and re-eval) into any upgrade plan.

Section 14 — Agentic RAG

Everything above is a pipeline: one pass, fixed stages. Agentic RAG turns it into a loop — the LLM inspects its own retrieval results and decides to search again, differently. Two patterns cover most of the value:

Query decomposition

Multi-hop questions — "Is pump 3's current vibration above the limit in its maintenance manual?" — need two different retrievals (live reading; documented limit) that no single query surfaces together. An LLM call splits the question into sub-queries, each is retrieved independently (Sections 9–10), and the answer is synthesised over the union.

The self-correction loop

question ─► retrieve ─► draft answer
                ▲            │
                │            ▼
         re-retrieve ◄─ critique: "list claims in the draft NOT supported
         (targeted        by the provided chunks; emit search queries
          queries)         that would verify them"
                             │
                     all supported? ──► final answer (with citations)

The critique step is the same local LLM under a different prompt — a rubric asking it to find unsupported claims and generate verification queries. This catches the classic single-pass failure: retrieval was partially right, the draft filled the gap from pretraining (Section 5's most dangerous quadrant), and nothing downstream noticed.

When it pays for itself — and how to keep it bounded

Each iteration multiplies LLM calls 3–5×; on Lab 1 hardware at 10–15 tok/s, a looped answer takes minutes, not seconds. That trade is right for high-stakes, low-QPS work — maintenance planning, incident retrospectives, compliance answers — where a wrong answer costs hours. It is wrong for interactive lookups where single-pass hybrid+rerank at ~90% quality returns in seconds. And loops need guardrails, because an LLM will happily retrieve forever:

• Hard iteration cap (2–3). Non-negotiable; enforced in code, not in the prompt.
• Fixed-point detection: if re-retrieval returns only already-seen chunk IDs, more looping cannot add evidence — stop and answer with explicit caveats.
• Log every iteration (queries issued, chunks added, critique verdict) — agentic failures are debugging nightmares without the trace.

The eval discipline is unchanged: agentic mode must beat single-pass on the golden set by enough to justify its latency. Measure the delta; don't assume it.

References

Foundations

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — arXiv:2005.11401 — https://arxiv.org/abs/2005.11401
• Reimers & Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks — arXiv:1908.10084 — https://arxiv.org/abs/1908.10084
• Malkov & Yashunin, Efficient and robust approximate nearest neighbor search using HNSW graphs — arXiv:1603.09320 — https://arxiv.org/abs/1603.09320

Hybrid retrieval & reranking

• Robertson & Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond — Foundations and Trends in IR, 2009 — https://www.staff.city.ac.uk/~sbrp622/papers/foundations_bm25_review.pdf
• Cormack, Clarke & Büttcher, Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods — SIGIR 2009 — https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf
• Nogueira & Cho, Passage Re-ranking with BERT — arXiv:1901.04085 — https://arxiv.org/abs/1901.04085

Context engineering & GraphRAG

• Anthropic, Introducing Contextual Retrieval (engineering post, Sept 2024) — https://www.anthropic.com/news/contextual-retrieval
• Günther et al. (Jina AI), Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models — arXiv:2409.04701 — https://arxiv.org/abs/2409.04701
• Edge et al. (Microsoft), From Local to Global: A Graph RAG Approach to Query-Focused Summarization — arXiv:2404.16130 — https://arxiv.org/abs/2404.16130

Embedding models

• Kusupati et al., Matryoshka Representation Learning — arXiv:2205.13147 — https://arxiv.org/abs/2205.13147
• Wang et al., Text Embeddings by Weakly-Supervised Contrastive Pre-training (E5) — arXiv:2212.03533 — https://arxiv.org/abs/2212.03533
• Chen et al., BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity — arXiv:2402.03216 — https://arxiv.org/abs/2402.03216
• Nussbaum et al., Nomic Embed: Training a Reproducible Long Context Text Embedder — arXiv:2402.01613 — https://arxiv.org/abs/2402.01613

Evaluation & context behaviour

• Es et al., RAGAS: Automated Evaluation of Retrieval Augmented Generation — arXiv:2309.15217 — https://arxiv.org/abs/2309.15217
• Liu et al., Lost in the Middle: How Language Models Use Long Contexts — arXiv:2307.03172 — https://arxiv.org/abs/2307.03172

Tools & docs

• ChromaDB documentation — https://docs.trychroma.com
• rank_bm25 — https://github.com/dorianbrown/rank_bm25
• sentence-transformers (CrossEncoder) — https://sbert.net