Lab 02 — RAG Pipeline Over Domain Documents

Goal: Build a fully local, air-gapped Retrieval-Augmented Generation pipeline that answers questions from a corpus of infrastructure documents — with source citations, 22-question evaluation, and a FastAPI service endpoint.

Files in This Lab

Prerequisites

1. Ollama running (same as Lab 1)

ollama serve &
ollama list    # confirm qwen3-coder:30b is present

2. Python packages

/Users/s0x/anaconda3/bin/pip install chromadb sentence-transformers fastapi uvicorn openai httpx

3. Embedding model cached

sentence-transformers downloads all-MiniLM-L6-v2 (~90 MB) on first use from HuggingFace. This must happen once while you have internet. After that, it's cached at ~/.cache/huggingface/hub/ and works offline.

# Pre-cache the model (one-time, requires internet)
/Users/s0x/anaconda3/bin/python -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('all-MiniLM-L6-v2'); print('cached')"

Step 1 — Ingest Documents

cd "AI Specialist/lab-02-rag-pipeline"

/Users/s0x/anaconda3/bin/python ingest.py

Expected output:

Found 5 document(s) in docs/
Loading embedding model: all-MiniLM-L6-v2 ...
Embedding model loaded.

Opening ChromaDB at: chroma_db/
Processing: digital-twin-concepts.md
  4 chunks  (8,241 chars total)
  Embedding 4 chunks ... done

Processing: pipeline-monitoring.md
  4 chunks  (7,892 chars total)
  Embedding 4 chunks ... done

Processing: pump-station-ops.md
  5 chunks  (9,104 chars total)
  Embedding 5 chunks ... done

Processing: scada-overview.md
  4 chunks  (7,618 chars total)
  Embedding 4 chunks ... done

Processing: sensor-types.md
  5 chunks  (8,347 chars total)
  Embedding 5 chunks ... done

Ingestion complete.
  22 chunks written from 5 document(s)
  Collection 'domain_docs' now has 22 total chunks

This creates chroma_db/ — a local persistent vector database. Ingestion is idempotent: re-running it updates existing chunks in place.

Experimenting with chunk size (Lab objective):

# Re-ingest with smaller chunks and compare eval results
/Users/s0x/anaconda3/bin/python ingest.py --chunk-size 500 --chunk-overlap 80 --reset
/Users/s0x/anaconda3/bin/python eval.py --retrieval-only

# And with larger chunks
/Users/s0x/anaconda3/bin/python ingest.py --chunk-size 1500 --chunk-overlap 200 --reset
/Users/s0x/anaconda3/bin/python eval.py --retrieval-only

Step 2 — Query via CLI

/Users/s0x/anaconda3/bin/python query.py "What is a SCADA system?"

Expected output:

Question: What is a SCADA system?
──────────────────────────────────────────────────────────────────────
SCADA stands for Supervisory Control and Data Acquisition. It is a
category of software and hardware systems that allow industrial
organisations to control processes locally or at remote locations,
monitor, gather, and process real-time data [Source: scada-overview.md].

──────────────────────────────────────────────────────────────────────
Sources (4 chunks retrieved):
  [1] scada-overview.md  chunk 0  similarity=82.4%
  [2] scada-overview.md  chunk 1  similarity=71.3%
  [3] digital-twin-concepts.md  chunk 3  similarity=58.1%
  [4] pump-station-ops.md  chunk 4  similarity=41.2%

Latency: 12450ms total (retrieval 42ms + LLM 12408ms) | Tokens: 487→68

Interactive mode:

/Users/s0x/anaconda3/bin/python query.py
>> What does NPSH stand for?
>> How does a digital twin differ from a simulation?
>> quit

With chunk preview (see what was retrieved):

/Users/s0x/anaconda3/bin/python query.py --verbose "What causes sensor drift?"

Step 3 — Run the API Server

/Users/s0x/anaconda3/bin/uvicorn rag_server:app --host 0.0.0.0 --port 8001 --reload

Expected startup:

INFO:     Loading RAG pipeline (ChromaDB + embedding model)...
INFO:     RAG pipeline ready. 22 chunks indexed.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8001

Endpoints

GET /health

curl http://localhost:8001/health
# {"status":"ok","chunks_indexed":22}

GET /stats

curl http://localhost:8001/stats
# {"collection":"domain_docs","total_chunks":22,"embed_model":"all-MiniLM-L6-v2","llm_default":"qwen3-coder:30b"}

POST /query

curl -s -X POST http://localhost:8001/query \
  -H "Content-Type: application/json" \
  -d '{
    "question": "What are the safety thresholds for motor vibration at a pump station?",
    "max_tokens": 200
  }' | python -m json.tool

Expected shape:

{
  "id": "a3f7c2d1",
  "answer": "The safety threshold for pump vibration is 10 mm/s RMS. A reading of 4–7 mm/s triggers a warning, and above 10 mm/s triggers a critical alarm and immediate shutdown [Source: pump-station-ops.md].",
  "sources": [
    {
      "source": "pump-station-ops.md",
      "chunk_index": 4,
      "similarity": 0.8741,
      "preview": "The following thresholds are typical for a medium-sized water pump station..."
    }
  ],
  "prompt_tokens": 512,
  "completion_tokens": 54,
  "latency_ms": 14230.0,
  "retrieval_ms": 38.4,
  "llm_ms": 14191.6,
  "timestamp": "2026-05-27T23:30:00+00:00"
}

Step 4 — Run the Tests

In a second terminal (while the server is running):

/Users/s0x/anaconda3/bin/python test_lab2.py

Expected output:

[PASS] Health endpoint — {'status': 'ok', 'chunks_indexed': 22}
[PASS] Chunks indexed > 0 — 22 chunks
[PASS] Stats endpoint — {'collection': 'domain_docs', 'total_chunks': 22, ...
[PASS] Query returns 200
[PASS] Answer non-empty — SCADA stands for Supervisory Control and Data Acq...
[PASS] Sources returned — 4 sources
[PASS] Latency logged — 12450ms
[PASS] Retrieval time present — 42ms
[PASS] Source has filename — scada-overview.md
[PASS] Source has similarity — 0.824
[PASS] Source has preview — SCADA stands for Supervisory Control and Data Ac...
[PASS] SCADA question hits scada doc — scada-overview.md ...
[PASS] Out-of-scope: 200 response
[PASS] Out-of-scope: model declines gracefully — I don't have that information...
[PASS] top_k=2 returns 2 sources — got 2 sources

Result: 15/15 tests passed

Step 5 — Evaluate

Fast: retrieval only (no LLM — completes in ~30 seconds)

/Users/s0x/anaconda3/bin/python eval.py --retrieval-only

Full: retrieval + answer quality (~8 minutes with 30B model)

/Users/s0x/anaconda3/bin/python eval.py

Quick 5-question check

/Users/s0x/anaconda3/bin/python eval.py --retrieval-only --questions 5

Expected retrieval-only output:

Evaluating 22 questions | mode=retrieval-only | top_k=4
Collection: 22 chunks indexed

[✓—] Q01: What does SCADA stand for?  (sim=0.87)
[✓—] Q02: What communication protocols does SCADA use?  (sim=0.79)
[✓—] Q03: What is the role of an RTU in a SCADA system?  (sim=0.76)
...
[✓—] Q22: How do digital twins use sensor data?  (sim=0.71)

──────────────────────────────────────────────────────────────────────
Retrieval hit rate: 21/22 = 95%
Elapsed: 28.3s

Observations

Retrieval latency is negligible

From real runs:

retrieval_ms: 38–65ms    ← ChromaDB cosine search over 22 chunks (HNSW)
llm_ms:       10–25s     ← LLM generation (warm 30B model)
total:        10–25s

The LLM dominates latency. This means improving retrieval quality (better chunking, more documents, reranking) is nearly free — you get better answers at almost no latency cost.

Similarity scores tell you a lot

Chunks with similarity > 0.75 are highly relevant; 0.50–0.75 are topically related; < 0.50 are barely related and may introduce noise into the prompt. The --verbose flag on query.py lets you inspect retrieved text directly.

The out-of-scope test reveals prompt adherence

The system prompt instructs the model to respond with a specific phrase when the answer isn't in the context. The test checks for this. The degree to which the model follows this instruction is a measurable property — critical in production where you don't want hallucinated infrastructure data.

Chunk size trade-offs

Run eval.py --retrieval-only after re-ingesting at each size to see the effect concretely.

Extension Exercises

1. Add a PDF: find any public infrastructure specification (e.g., EPA guidance doc). Copy it to docs/, run pip install pypdf, re-run ingest.py. Query it.

2. Implement hybrid search: add a BM25 keyword index alongside the vector index. Fuse results with Reciprocal Rank Fusion (see HITCHHIKERS-GUIDE section 4).

3. Add a reranker: install sentence-transformers cross-encoder (cross-encoder/ms-marco-MiniLM-L-6-v2). After retrieving top-10, rerank to top-4 before building the prompt.

4. Connect to Lab 1 server: change OLLAMA_BASE_URL in rag.py to http://localhost:8000/v1 to route through your Lab 1 FastAPI wrapper instead of directly to Ollama.

5. Evaluate chunk sizes systematically: run eval.py --retrieval-only for chunk sizes 300, 500, 800, 1000, 1500. Plot retrieval hit rate vs chunk size.