Knowledge 04 — Azure AI Document Intelligence (IDP)

Goal of this module. Go from "OCR reads text from images" to being able to design an Intelligent Document Processing (IDP) pipeline for a bank — KYC docs, statements, cheques, trade-finance paperwork — using Azure AI Document Intelligence, with confidence handling, human review, and feeding the output into RAG/agents. JD4: "Develop document processing pipelines using Azure AI Document Intelligence."

Table of Contents

• 1. The problem: documents are the bank's lifeblood
• 2. OCR vs Layout vs Extraction (the three levels)
• 3. Azure AI Document Intelligence: the model families
• 4. Under the hood: how it works
• 5. Confidence scores and human-in-the-loop
• 6. Custom and composed models
• 7. The end-to-end IDP pipeline for a bank
• 8. Document Intelligence + RAG + LLMs
• 9. Arabic and multilingual documents
• 10. Common misconceptions
• 11. Interview Q&A
• 12. References

1. The problem: documents are the bank's lifeblood

A bank runs on documents: account-opening forms, passports/Emirates IDs, salary certificates, bank statements, cheques, invoices, trade-finance letters of credit, loan applications, signed agreements. Most arrive as scans, photos, or PDFs — unstructured pixels, not data. To use them (onboard a customer, reconcile a statement, clear a cheque) you must turn them into structured fields reliably and auditably.

That conversion is IDP — Intelligent Document Processing. Doing it by hand is slow, costly, and error-prone; doing it with naive OCR loses structure (tables, key-value pairs, checkboxes). Azure AI Document Intelligence (formerly Form Recognizer) is Azure's managed IDP service.

2. OCR vs Layout vs Extraction (the three levels)

These are commonly conflated; an interviewer will check you separate them:

1. OCR (Optical Character Recognition) — pixels → text + word/line bounding boxes. Answers "what characters are on this image and where?" It does not understand meaning or structure. (Azure's Read model / the Read API.)
2. Layout — OCR plus structure: paragraphs, tables (rows/cells), selection marks (checkboxes), reading order, headings, page geometry. Answers "what's the shape of this document?" Critical for statements (tables) and forms. (Azure's prebuilt-layout.)
3. Extraction — Layout plus semantics: named fields with values and confidence — InvoiceTotal, AccountNumber, IBAN, CustomerName, line items. Answers "what does this document mean?" Either via prebuilt models (invoice, receipt, ID, etc.) or custom models you train, or newer generative field extraction.

Rule of thumb: OCR for "give me the text," Layout for "give me text + tables + structure (great for chunking RAG)," Extraction for "give me the specific fields my system needs."

3. Azure AI Document Intelligence: the model families

The senior move: use prebuilt when one fits (invoice, ID), Layout + LLM extraction for flexibility, and custom/composed only when you have a stable bespoke form at volume worth training for.

4. Under the hood: how it works

Document Intelligence is a pipeline of deep models:

1. Image preprocessing — deskew, de-noise, orientation detection (you can help with OpenCV first — K05).
2. OCR/text detection — a vision model locates and reads text + bounding polygons, handling print and handwriting across many languages.
3. Layout analysis — models detect tables (cell grid), selection marks, reading order, and document structure.
4. Field extraction — for prebuilt/custom, a model maps detected text+layout to named fields (e.g. learns that the number top-right near "IBAN" is the IBAN), returning each with a confidence.
5. Output — JSON with text, boundingRegions (page + polygon for every element → enables highlight/verify UIs and audit), tables, key-value pairs, and typed fields with confidences. Layout can emit Markdown (handy for RAG/LLM consumption).

Async API shape: you POST the document (or a URL/SAS to Blob), get an operation id, then poll until the analysis result is ready. The SDK wraps this as a poller.

from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.identity import DefaultAzureCredential
client = DocumentIntelligenceClient(endpoint, DefaultAzureCredential())  # keyless
poller = client.begin_analyze_document("prebuilt-layout", body=doc_bytes)
result = poller.result()
for table in result.tables:        # statement rows
    ...
for kv in result.key_value_pairs:  # form fields
    ...

5. Confidence scores and human-in-the-loop

Every extracted field comes with a confidence (0–1). This is the heart of a bank-grade pipeline, because you cannot blindly trust extraction for KYC or payments.

The pattern: confidence-thresholded straight-through processing (STP) with human review.

• Field confidence ≥ threshold (e.g. 0.95 for high-risk fields like IBAN/amount) → auto-accept (straight-through).
• Field confidence < threshold → route to a human to verify/correct in a review UI that shows the field highlighted on the original image (using boundingRegions).
• Track the STP rate (% auto-processed) as your business KPI; tune thresholds per field by risk.

Why this matters in banking: regulators and risk teams require that high-impact extractions are validated; a wrong IBAN moves money to the wrong place. The design answer is never "trust the model" — it's "risk-weighted thresholds + human review of low-confidence + full audit of corrections." Document Intelligence Studio / labeling tools support building this loop.

6. Custom and composed models

When prebuilt models don't fit your bespoke forms:

• Custom extraction — label ~5+ sample documents (fields, tables) in Document Intelligence Studio; train a model that learns your layout. Two flavors: template (consistent fixed layout — fast, few samples) and neural (varying layouts — more robust, more samples).
• Custom classification — first classify the incoming doc type (passport vs statement vs LC) so you route to the right extractor.
• Composed model — bundle several custom models; the service auto-routes each document to the matching one. Ideal for a mixed intake queue.

Lifecycle discipline (senior signal): version your custom models, keep a labeled eval set, and re-validate accuracy when retraining — same model-risk hygiene as any ML in a bank (K09).

7. The end-to-end IDP pipeline for a bank

intake (email/upload/scanner) → Blob Storage (raw, immutable)
   → [optional] OpenCV preprocess (deskew, denoise, crop) — K05
   → classify document type (custom classifier / LLM)
   → route to extractor (prebuilt-id / prebuilt-invoice / custom / layout+LLM)
   → fields + confidences (JSON)
   → confidence gate:  high → straight-through ; low → human review queue
   → validation (checksums: IBAN mod-97, date sanity, cross-field rules)
   → persist structured record (Cosmos/SQL) + link to source + audit log
   → downstream: index for RAG (K02) / trigger agent workflow (K03) / post to core systems

Every box emits to audit logs; every field links back to its boundingRegion on the source image so any decision is traceable. This diagram, narrated, is a complete interview answer to "build us a KYC onboarding pipeline."

8. Document Intelligence + RAG + LLMs

Three powerful combinations the interview rewards:

1. Document Intelligence as the RAG ingester — use Layout (Markdown output) to extract clean, structure-aware text from scanned/complex PDFs, then chunk by section/table → far better retrieval than dumping raw OCR (K02 §3). This is the canonical "RAG over scanned banking PDFs" answer.
2. Layout + LLM for flexible extraction — run Layout, then ask an LLM (structured output) to pull arbitrary fields from the layout text. Flexible, no model training, good for long-tail document types.
3. Document Intelligence as an agent tool — an agent calls a extract_document(blob_uri) tool to read an uploaded statement, then reasons over the structured result (K03).

9. Arabic and multilingual documents

JD4's Arabic requirement hits hardest here, because OCR is where script matters:

• Read/Layout support Arabic print and (increasingly) handwriting; verify the language is supported for your model/version.
• RTL (right-to-left) reading order, connected scripts (Arabic letters join and change shape by position), and diacritics complicate OCR and downstream text handling — test on real samples, don't assume.
• Mixed Arabic/English documents (common in Gulf banking) need correct per-region language handling; the service returns detected language per line.
• Numbers may be Eastern Arabic numerals (٠١٢٣) — normalize to Western digits before validation/math.
• Always eval on real Arabic documents with native-speaker review; English benchmarks don't transfer.

10. Common misconceptions

• "OCR and Document Intelligence are the same." OCR is level 1 (text+boxes); Document Intelligence adds layout (tables/structure) and extraction (named fields + confidence).
• "Just OCR the PDF and feed it to the LLM." You lose tables/structure and get worse RAG; use Layout for structure-aware chunking.
• "Trust the extracted fields." Use confidence thresholds + human review for high-risk fields; STP only above threshold.
• "Form Recognizer." It's Document Intelligence now.
• "One custom model for everything." Classify first, then route to the right prebuilt/custom model (composed).
• "Arabic works the same as English." RTL, connected script, diacritics, and Eastern numerals need explicit testing/normalization.

11. Interview Q&A

Q: Design a KYC document-processing pipeline for customer onboarding. Intake to immutable Blob → optional OpenCV preprocessing → custom classifier routes (passport/Emirates ID/salary cert/statement) → prebuilt-idDocument for IDs, custom/neural for bespoke forms, Layout for statements → fields + confidences → confidence gate (high-risk fields like ID number/IBAN at ≥0.95 auto-accept, else human-review queue with highlighted source) → validation (ID checksum, IBAN mod-97, expiry sanity, cross-field consistency) → persist structured record + source link + audit → push to core/AML systems. KPIs: STP rate, extraction accuracy, review turnaround. Everything keyless (Managed Identity), private-networked, audited.

Q: How do you decide prebuilt vs custom vs Layout+LLM? Prebuilt if a model fits (invoice/ID/receipt) — zero training, maintained by MS. Custom (template/neural) when you have a stable bespoke form at volume worth labeling/training. Layout+LLM for flexible/long-tail extraction without training. Compose custom models behind a classifier for mixed intake.

Q: How do you handle low-confidence extractions in a regulated context? Risk-weighted confidence thresholds per field; below threshold → human review in a UI that highlights the field on the original via bounding regions; log every correction; track STP rate; re-tune thresholds and retrain with corrected samples. Never auto-process a low-confidence money/identity field.

Q: How does Document Intelligence improve a RAG pipeline over scanned PDFs? Raw OCR loses tables and structure, producing poor chunks. Layout returns structure-aware text (Markdown, tables, sections) so you chunk by semantic unit, keep table rows intact, and retain page references for citations — materially higher retrieval quality and auditability than dumping OCR text.

Q: What changes for Arabic documents? Confirm Arabic support for the model/version; expect RTL reading order, connected script, and diacritics to stress OCR (test on real samples); normalize Eastern Arabic numerals to Western before validation/math; handle mixed Arabic/English per-line language; evaluate with native-speaker review.

12. References

• Azure AI Document Intelligence — Microsoft Learn: What is Document Intelligence, Layout model, Prebuilt models, Custom models (template/neural), Composed models, Read/OCR, Markdown output.
• Document Intelligence Studio — labeling, training, and human-review tooling docs.
• Read OCR / language support — Microsoft Learn OCR language support tables.
• IDP patterns — Azure Architecture Center, Automate document processing reference architectures.
• IBAN validation — ISO 13616 (mod-97 check); used as a downstream validation example.
• Related: K02 — RAG & Azure AI Search, K05 — Vision & Multimodal.