Lab 03 — Document Intelligence Pipeline

Goal. Build an IDP pipeline that turns a bank statement (and a KYC ID) into structured JSON using Azure AI Document Intelligence, with confidence-based human review and validation, then index the result for RAG (Lab 01) and expose it as an agent tool (Lab 02). JD4: document processing pipelines using Azure AI Document Intelligence. Read K04 first.

Stack. Azure AI Document Intelligence (prebuilt-layout, prebuilt-invoice/prebuilt-idDocument, optional custom) · Blob Storage · keyless auth. Local fallback. Tesseract OCR + a layout heuristic for text; the pipeline shape (classify → extract → confidence gate → validate → persist) is what matters and transfers.

Run it (offline, no Azure)

pip install -r requirements.txt        # only pytest; pipeline is pure stdlib
python run.py                          # process 4 synthetic docs + STP rate
pytest -q                              # 9 tests = success criteria

The extractor is a deterministic stand-in returning the same shape as Azure DI (fields + confidence + bounding regions); the confidence gate, validation (real IBAN mod-97 + balance math), human-review queue, and audit are the bank-grade parts and run identically offline.

Code tour

Steps

1. Intake. Drop sample docs (a statement PDF, an ID image, an invoice) into Blob (immutable container).
2. Classify. Route by document type (a simple classifier or an LLM call) → choose the extractor.
3. Extract.
   • Statement → prebuilt-layout → pull the transactions table (rows/cells) + account header fields.
   • ID → prebuilt-idDocument → name, ID number, expiry.
   • Invoice → prebuilt-invoice → vendor, total, line items. Capture each field's confidence and bounding region.
4. Confidence gate. Set per-field thresholds by risk (e.g. IBAN/amount ≥ 0.95). High → auto-accept; low → push to a human-review queue with the field highlighted on the source image (use bounding regions). Track the STP rate.
5. Validate. Deterministic checks: IBAN mod-97, date sanity, statement balance math (opening + sum(txns) == closing), ID expiry not past.
6. Persist. Structured record → Cosmos/SQL, linked to the source blob; audit log every extraction + correction.
7. Feed downstream. Chunk the layout (Markdown) text → index for RAG (K04 §8); expose an extract_document(blob_uri) agent tool (Lab 02).

Measurable result

A per-field accuracy + confidence report on ~10 documents, your STP rate at the chosen thresholds, and a working human-review step for low-confidence fields. Show the statement balance-math validation catching a deliberately corrupted row.

Stretch

• Train a custom/neural model on a bespoke form and compose it behind the classifier (K04 §6).
• Add an Arabic statement/ID and verify OCR + Eastern-numeral normalization (Lab 04).
• Use Layout → LLM structured-output extraction for a long-tail document type without training.

Talking point

"For a bank I never auto-trust extraction on money/identity fields — I use risk-weighted confidence thresholds with human review of low-confidence fields (highlighted on the source via bounding regions), deterministic validation like IBAN mod-97 and balance math, and a full audit of every field and correction. Layout also gives me structure-aware text that makes downstream RAG far better than raw OCR."

Resume bullet

"Built an Azure AI Document Intelligence IDP pipeline (classify → prebuilt/layout extraction → confidence-gated straight-through processing with human review → IBAN/balance validation → audited structured store) feeding both a RAG index and an agent tool; achieved 90%+ straight-through rate while routing low-confidence money fields to review."