Knowledge 02 — RAG & Azure AI Search

Goal of this module. Go from "RAG means stuff documents into the prompt" to being able to design, build, tune, and defend a production banking retrieval pipeline on Azure AI Search — chunking, embeddings, vector + keyword + hybrid + semantic reranking, filtering, citations, and the failure modes. This is one of JD4's three "THE MUST" anchors ("scalable Generative AI / RAG pipelines using Azure OpenAI and Azure AI Search"). Expect the deepest technical probing here.

Table of Contents

1. What RAG is and why a bank needs it

RAG = Retrieval-Augmented Generation. Instead of relying on the model's frozen weights, you retrieve relevant text from an external knowledge store at query time and put it in the prompt, then the model generates an answer grounded in that text.

Why a bank needs it specifically:

  • The bank's knowledge (product terms, policies, procedures, fee schedules, regulations) is private (never in the model's training data) and changes (a fine-tune would be stale and un-citable).
  • Answers must be traceable to a source — RAG gives you the exact document/section the answer came from, which is mandatory for audit and customer trust.
  • It bounds hallucination: the model answers from provided text and is instructed to abstain otherwise (K01 §7).

The mental model: RAG turns "what does the model remember?" into "what can the model find, then phrase?" The quality ceiling of a RAG system is set by retrieval, not the LLM. A perfect model over wrong context gives a confident wrong answer. So most of your engineering effort goes into retrieval.

2. The RAG pipeline, end to end

Two phases — offline ingestion and online query:

OFFLINE (ingestion / indexing)
  raw docs (PDF, DOCX, scans)
    → extract text/layout (Document Intelligence for scans/forms — K04)
    → clean & normalize
    → CHUNK into passages (with metadata: source, page, section, language, ACL)
    → EMBED each chunk (text-embedding-3-large)
    → UPSERT into Azure AI Search index (vector field + text fields + filters)

ONLINE (query time)
  user question
    → (optional) rewrite / translate / expand query
    → EMBED the query
    → RETRIEVE: hybrid (BM25 + vector) top-k  → RRF fuse
    → RERANK with semantic ranker → top-n (e.g. 3–5)
    → BUILD prompt: system + grounding instructions + chunks(+citations) + question
    → GENERATE with Azure OpenAI → answer + citations
    → EVALUATE/guard (groundedness) → return

Every arrow is a tunable decision. The interview tests whether you know why each exists.

3. Chunking: the decision that makes or breaks retrieval

Why chunk at all? You can't embed or retrieve a 50-page PDF as one vector — meaning gets averaged into mush, and you'd stuff the whole thing into context. You split documents into passages small enough to be specific but large enough to be self-contained.

The core trade-off:

  • Too small (e.g. 1 sentence): high precision per chunk but loses surrounding context; the answer may be split across chunks; retrieval misses because a single sentence lacks signal.
  • Too large (e.g. whole page): each chunk dilutes (one vector covers many topics), retrieval precision drops, and you waste context tokens.
  • Sweet spot for prose: ~300–800 tokens with 10–20% overlap so a fact straddling a boundary survives in at least one chunk.

Chunking strategies (know all four):

  1. Fixed-size + overlap — simplest; split every N tokens with M overlap. Good baseline.
  2. Recursive / structure-aware — split on natural boundaries (headings, paragraphs, sentences) first, only falling back to fixed size. Keeps semantic units intact. (LangChain RecursiveCharacterTextSplitter.)
  3. Semantic chunking — split where embedding similarity between consecutive sentences drops (topic shift). Higher quality, more compute.
  4. Layout-aware — for banking forms/statements, use Document Intelligence layout (K04) to chunk by table/section/field so a statement row stays whole. Often the right call for structured financial docs.

Always attach metadata to each chunk: source_doc, page, section, language, effective_date, and access-control labels (who may see it). Metadata powers citations, filtering, security trimming, and freshness (§9).

4. Embeddings and the vector index (HNSW)

Embedding. Each chunk → a vector (e.g. 3072-dim with text-embedding-3-large) capturing meaning (K01 §3). The query is embedded the same way; retrieval finds chunks whose vectors are nearest the query vector by cosine similarity.

The scaling problem. Comparing a query to every chunk vector (exact / brute-force kNN) is O(N·d) — fine for thousands, too slow for millions. So we use ANN (Approximate Nearest Neighbor) indexes.

HNSW (Hierarchical Navigable Small World) — the ANN algorithm Azure AI Search uses. Intuition: build a multi-layer graph where each node (vector) links to its near neighbors; search greedily hops through the graph from a coarse top layer down to fine layers, getting close to the true nearest neighbors in logarithmic-ish time. Parameters:

  • m — neighbors per node (higher = better recall, more memory).
  • efConstruction — build-time breadth (higher = better graph, slower build).
  • efSearch — query-time breadth (higher = better recall, slower query). You trade recall vs latency/memory. Defaults are sane; tune only with eval evidence.

Distance metric: cosine for normalized text embeddings (Azure AI Search supports cosine/dotProduct/Euclidean). For OpenAI embeddings, cosine.

5. Keyword search (BM25): still essential

Vectors are great at meaning but bad at exactness: account numbers, product codes ("Account 4521-A"), proper nouns, acronyms, and rare tokens. Two texts about different account numbers have near-identical embeddings.

BM25 (Best Matching 25) — the classic keyword ranking function Azure AI Search uses for full-text. It scores a document by term frequency (how often the query terms appear) × inverse document frequency (how rare/discriminative those terms are), with length normalization. It nails exact-token matches that vectors miss.

The lesson: keyword and vector retrieval fail in complementary ways. Keyword misses paraphrases; vector misses exact identifiers. So you combine them.

6. Hybrid search and Reciprocal Rank Fusion

Hybrid search runs both BM25 (keyword) and vector (semantic) retrieval, then fuses the two ranked lists into one. Azure AI Search does this natively and fuses with Reciprocal Rank Fusion (RRF).

RRF, concretely. For each document, sum 1 / (k + rank) across the lists it appears in (k a small constant, e.g. 60; rank its position in that list). A doc ranked high in either list scores well; a doc ranked high in both wins. RRF needs no score calibration between the two systems (it uses ranks, not raw scores) — which is why it's robust.

\[ \text{RRF}(d) = \sum_{l \in \text{lists}} \frac{1}{k + \text{rank}_l(d)} \]

Why hybrid is the default for enterprise/banking: it captures both "card declined ≈ transaction failed" (vector) and "Account 4521-A" / "IBAN" / "Reg D" (keyword). On virtually every enterprise benchmark, hybrid + semantic reranking beats pure vector. If asked "vector or keyword?", the senior answer is "hybrid, fused with RRF, then semantically reranked."

7. The semantic ranker (reranking)

Hybrid retrieval gives you a good top-50. The semantic ranker is a second-stage cross-encoder model (Microsoft's, deep-learning-based) that re-scores those candidates by reading the query and each passage together (not as separate vectors), producing a far more accurate relevance ordering. It then returns the top results and can extract semantic captions/answers (highlighted snippets).

Why two stages? A cross-encoder reading every query-document pair is accurate but expensive — you can't run it over millions of docs. So: cheap retrieve (bi-encoder vectors + BM25) narrows millions → ~50; expensive rerank (cross-encoder) orders 50 → 5. This retrieve-then-rerank pattern is standard and is what Azure's semantic ranker implements.

Impact: semantic reranking typically lifts the right chunk into the top-3, which is what the LLM actually reads. It directly raises groundedness and answer accuracy. It is a paid feature/tier in AI Search — worth it for banking quality.

8. Azure AI Search concretely: indexes, fields, integrated vectorization

The objects:

  • Index — your searchable collection; defined by a schema of fields.
  • Field — typed; flags: searchable (full-text/BM25), filterable, sortable, facetable, retrievable, plus vector fields (with a dimension + a vectorSearchProfile → algorithm config like HNSW).
  • Semantic configuration — declares which fields are title/content/keywords for the semantic ranker.
  • Indexer + data source + skillset — optional pull pipeline: an indexer crawls a data source (Blob, Cosmos, SQL), runs a skillset (e.g. OCR, text split, embedding skill), and writes documents — this is integrated vectorization (Azure embeds your chunks for you, so you don't run a separate embedding job).

Minimal index (conceptually):

{
  "fields": [
    { "name": "id", "type": "Edm.String", "key": true },
    { "name": "content", "type": "Edm.String", "searchable": true },          // BM25
    { "name": "contentVector", "type": "Collection(Edm.Single)",
      "dimensions": 3072, "vectorSearchProfile": "hnsw-cosine" },             // vector
    { "name": "source", "type": "Edm.String", "filterable": true, "retrievable": true },
    { "name": "page", "type": "Edm.Int32", "retrievable": true },
    { "name": "language", "type": "Edm.String", "filterable": true },          // en / ar
    { "name": "acl", "type": "Collection(Edm.String)", "filterable": true }    // security trimming
  ],
  "semantic": { "configurations": [ { "name": "default", "prioritizedFields": {
      "prioritizedContentFields": [ { "fieldName": "content" } ] } } ] }
}

Query (hybrid + semantic), conceptually: send the user text (for BM25), the query vector (for ANN), set queryType=semantic with the semantic config, apply filter, request top results + captions. Azure runs BM25 + vector, RRF-fuses, semantically reranks, and returns ranked passages with scores and captions.

Integrated vectorization is the senior shortcut: configure an indexer + embedding skill so ingestion (chunk → embed → index) and query-time embedding are handled by Azure — less glue code, fewer moving parts in a regulated pipeline.

9. Filtering, security trimming, and freshness

In a bank these are not optional:

  • Security trimming (the big one). A user must only retrieve documents they're authorized to see. Store ACL labels (groups/roles) on each chunk and apply a filter at query time: acl/any(g: search.in(g, 'group1,group2')). Never rely on the LLM to "not mention" forbidden content — filter it out of retrieval entirely. This is a top interview probe.
  • Metadata filters — restrict by product, region, language eq 'ar', document type. Combine with vector search (pre-filter narrows the candidate set).
  • Freshness — store effective_date; filter out superseded policy versions or boost recent ones. A bank answering with last year's fee schedule is a compliance problem.
  • Multi-tenancy — partition indexes or filter by tenant for isolation.

10. Generation: grounding, citations, and the prompt

Retrieval found the chunks; now the LLM must answer only from them and cite. The grounding prompt pattern (K06):

SYSTEM:
You are a banking assistant. Answer ONLY using the SOURCES below.
If the answer is not in the SOURCES, say you don't have that information
and offer to connect a human agent. Never invent figures, rates, or policies.
Cite each fact as [source:page].

SOURCES:
[doc_A:p3] <chunk text>
[doc_B:p1] <chunk text>

USER: <question>

Key practices:

  • Citations map each claim to source:page so the answer is auditable and the user can verify. Many banking UIs show the source snippet.
  • Abstention is a feature: "I don't have that" beats a confident hallucination.
  • Don't over-stuff: top 3–5 reranked chunks usually beat top 20 (cost + lost-in-the-middle).
  • Live numbers (balances, today's rate) are not in RAG docs — those come from a tool call (K03). RAG is for documents/policies; tools are for systems of record.

11. Evaluating and tuning RAG

You can't tune what you don't measure. Build an eval set of (question, ideal answer, expected source) pairs and score (K08):

Retrieval metrics:

  • Recall@k / hit-rate — is the right chunk in the top-k? (If not, generation can't succeed.)
  • MRR / nDCG — is it ranked high?
  • Context precision — are the retrieved chunks actually relevant (not noise)?

Generation metrics (RAGAS / Foundry evaluators):

  • Groundedness / Faithfulness — is every claim supported by the retrieved context? (the headline banking metric.)
  • Answer relevance — does it actually address the question?
  • Context recall — did retrieval bring back everything needed?

Tuning loop: if groundedness is low but recall is high → fix the prompt/generation (stronger grounding instructions, fewer/cleaner chunks). If recall is low → fix retrieval (chunking, hybrid weights, reranking, query rewriting). Diagnosing which half is broken is the skill.

12. Advanced patterns

  • Query rewriting / expansion — an LLM rewrites a vague/contextual follow-up ("what about for premium?") into a standalone query before retrieval; or expands synonyms/acronyms. Big wins in multi-turn chat.
  • Multilingual retrieval — for Arabic↔English, multilingual embeddings let an English query hit Arabic docs and vice-versa; optionally translate the query. Index a language field.
  • HyDE (Hypothetical Document Embeddings) — embed a hypothetical answer the LLM drafts, not the raw question, to better match document-style text.
  • Parent-document / small-to-big — retrieve on small precise chunks but feed the LLM the larger parent section for context.
  • Agentic RAG — let an agent decide whether/what to retrieve, issue multiple queries, and combine — instead of a fixed single retrieval (K03). Azure's newer agentic retrieval in AI Search does query planning over the index.
  • Reranking models — beyond Azure's semantic ranker you can use Cohere/cross-encoder rerankers via Foundry.

13. Common misconceptions

  • "Vector search alone is best." Hybrid (BM25+vector) + semantic rerank wins on enterprise data; pure vector misses exact identifiers.
  • "Bigger chunks give more context, so they're better." They dilute embeddings and hurt precision; ~300–800 tokens + overlap is the prose sweet spot.
  • "More retrieved chunks = better answers." Top 3–5 reranked usually beat top 20 (cost + lost-in-the-middle + noise).
  • "RAG fixes hallucination completely." It bounds it; you still need abstention + groundedness evaluation. Bad retrieval → confident wrong answers.
  • "The LLM can be told not to reveal unauthorized docs." No — security-trim at retrieval with filters; never trust the model as an access-control layer.
  • "Put live balances in the vector index." No — those are tool calls to systems of record; RAG is for documents/policies.
  • "Azure Cognitive Search." It's Azure AI Search now.

14. Interview Q&A

Q: Design a RAG pipeline for a bank's customer-support assistant on Azure. Ingest policy/product PDFs → Document Intelligence for scans → layout/recursive chunking ~500 tokens w/ overlap + metadata (source, page, language, ACL) → embed with text-embedding-3-large → Azure AI Search index (vector + searchable text + filters + semantic config), ideally via integrated vectorization. Query: rewrite follow-ups → hybrid (BM25+vector, RRF) → semantic rerank → top-5 → grounding prompt with citations and abstention → Azure OpenAI → groundedness eval/guardrail → return with sources. Security-trim by ACL filter; live figures via tool calls, not RAG. Evaluate recall@k + groundedness in CI.

Q: Vector vs keyword vs hybrid — defend your choice. Keyword (BM25) nails exact identifiers (account numbers, codes) but misses paraphrase; vector captures meaning but misses exact tokens. They fail complementarily, so hybrid fuses both with RRF (rank-based, no score calibration needed), then a cross-encoder semantic ranker reorders the top candidates for accuracy. Hybrid+semantic is the enterprise default.

Q: Why retrieve-then-rerank instead of just retrieving more? Bi-encoders (vectors) are cheap and scale to millions but score query and doc independently, so ranking is approximate. Cross-encoders read query+doc together for accurate relevance but are too expensive to run over everything. So retrieve cheaply to ~50, rerank expensively to ~5. It lifts the right chunk into the top-k the LLM reads, raising groundedness.

Q: How do you stop a customer retrieving documents they're not allowed to see? Security-trim at retrieval: store ACL/group labels on each chunk and apply a filter constraining results to the user's groups. Enforce in the retrieval query, not the prompt — the LLM is never an access-control boundary. Combine with private networking and per-tenant isolation.

Q: Groundedness is low but the right chunk is being retrieved. What's wrong and how do you fix it? It's a generation problem, not retrieval: the prompt isn't constraining the model to the sources, or you're stuffing too many noisy chunks (lost-in-the-middle), or temperature is too high. Fix: stronger "answer only from SOURCES / cite / abstain" instructions, fewer cleaner reranked chunks, temperature ~0, and a groundedness evaluator gate in CI.

Q: How do you handle Arabic and English in the same RAG system? Multilingual embeddings (the -3 models) so cross-language retrieval works; a language metadata field for filtering/boosting; optional query translation; ensure Document Intelligence/Vision OCR supports Arabic for ingestion; include Arabic Q&A in the eval set; RTL-correct rendering. (K05.)

15. References

  • RAG — Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (NeurIPS 2020).
  • BM25 — Robertson & Zaragoza, "The Probabilistic Relevance Framework: BM25 and Beyond" (2009).
  • Reciprocal Rank Fusion — Cormack, Clarke & Büttcher (SIGIR 2009).
  • HNSW — Malkov & Yashunin, "Efficient and robust approximate nearest neighbor search using HNSW graphs" (2016/2018).
  • Cross-encoder reranking — Nogueira & Cho, "Passage Re-ranking with BERT" (2019); Reimers & Gurevych, Sentence-BERT (2019).
  • HyDE — Gao et al., "Precise Zero-Shot Dense Retrieval without Relevance Labels" (2022).
  • Azure AI Search — Microsoft Learn: Vector search, Hybrid search & RRF, Semantic ranking, Integrated vectorization, Agentic retrieval.
  • RAGAS — Es et al., "RAGAS: Automated Evaluation of Retrieval Augmented Generation" (2023).
  • Lost in the middle — Liu et al. (2023).