Lab 01 — RAG over Azure AI Search

Goal. Build a grounded banking Q&A pipeline: ingest policy/product PDFs → chunk → embed → index in Azure AI Search → retrieve with hybrid (BM25 + vector) + semantic reranking → generate a cited, grounded answer with Azure OpenAI. Then prove hybrid+semantic beats pure-vector on your own eval set. This is JD4's core "THE MUST." Read K02 first.

Stack. Azure OpenAI (text-embedding-3-large, gpt-4o) · Azure AI Search (vector + semantic) · azure-search-documents · keyless auth (DefaultAzureCredential). Local fallback. Local embeddings (sentence-transformers) + a local vector store (Chroma/FAISS) + Ollama for generation; you still implement hybrid (do a keyword pass + vector pass and RRF-fuse them yourself) so the concept transfers.

Run it (offline, no Azure)

pip install -r requirements.txt        # only pytest; the pipeline is pure stdlib
python run.py                          # sample answers + the A/B retrieval table
python run.py "How long do I have to dispute a transaction?"   # ask your own
pytest -q                              # 9 tests = the lab's success criteria

Everything runs deterministically with no Azure account, no network, no model downloads. The "vector" retriever is a deterministic TF-IDF + synonym embedder standing in for text-embedding-3-large; the LLM is a deterministic extractive generator that can only emit text from the retrieved sources (grounded by construction) and abstains otherwise.

Code tour (what each file teaches)

File	What it implements (and the K02 section it mirrors)
data.py	Synthetic banking corpus + a 20-question eval set (mixed keyword/semantic), with per-chunk `acl`/`language` metadata
rag.py	Chunking (§3), TF-IDF embeddings (§4), BM25 (§5), RRF fusion (§6), semantic rerank (§7), grounded+citing+abstaining generation (§10), security trimming (§9) — plus `Azure*` provider stubs showing the one-class swap to go live
run.py	CLI demo: sample answers, the security-trimming abstention, and the mode-vs-recall@5 A/B table
test_rag.py	Encodes the success criteria: BM25 ranks exact identifiers, vectors match paraphrase, RRF rewards agreement, answers are cited, abstains out-of-corpus, staff docs hidden from retail, hybrid ≥ its parts

To go live on Azure: swap Embedder→AzureOpenAIEmbedder, LocalLLM→AzureOpenAILLM, LocalHybridIndex→AzureAISearchIndex (each stub in rag.py shows the real SDK call). The pipeline logic is unchanged.

Steps

Corpus. Collect 8–12 banking-style docs (use public bank product T&Cs, fee schedules, an FAQ, an account-agreement PDF as stand-ins). Include at least one scanned PDF (sets up Lab 03).
Chunk. Recursive/structure-aware chunking ~500 tokens, 15% overlap. Attach metadata to every chunk: source, page, section, language, and a fake acl group label (sets up security trimming).
Embed. text-embedding-3-large (3072-dim). Note the dimension — it sizes your index.
Index. Create an AI Search index with: content (searchable → BM25), contentVector (vector, HNSW/cosine), filterable source/language/acl, retrievable page. Add a semantic configuration. (Bonus: use integrated vectorization — an indexer + embedding skill — so Azure embeds for you.)
Retrieve. Implement a query that runs hybrid (keyword + vector) with queryType=semantic, an acl filter, top=5. Inspect the fused + reranked results and their scores/captions.
Generate. Build the grounding prompt (K06 §6): "answer only from SOURCES, cite [source:page], abstain if absent." Call gpt-4o, temperature 0.
Cite & abstain. Return the answer with citations; verify it says "I don't have that" for an out-of-corpus question.
A/B the retrieval. On a 20-question eval set (you know the answers), compare pure vector vs hybrid vs hybrid+semantic: measure recall@5 (right chunk retrieved?) and answer groundedness (eyeball or use Lab 06).

Measurable result

A table: retrieval mode × {recall@5, groundedness, avg latency}. You should see hybrid+semantic ≥ hybrid > pure vector on recall@5, and be able to explain why (exact identifiers + paraphrase + cross-encoder reordering — K02 §6–7).

Stretch

Add query rewriting for multi-turn follow-ups; re-measure.
Add an Arabic doc + Arabic question; confirm multilingual retrieval works (K05 §6).
Add security trimming: run the same query as two users with different acl groups; confirm they get different results.

Talking point

"The quality ceiling of RAG is set by retrieval, not the LLM — so I default to hybrid + semantic reranking with security-trimmed filters and a grounding-with-citations prompt, and I prove the choice with recall@k and groundedness on an eval set."

Resume bullet

"Built an Azure AI Search RAG pipeline (hybrid BM25+vector with semantic reranking, integrated vectorization, security-trimmed retrieval) over banking documents with Azure OpenAI grounded generation and citations; A/B-tested retrieval modes, improving recall@5 from 0.7 (pure vector) to 0.9 (hybrid+semantic)."

AI Specialist