Lab 04 — Multimodal + Arabic

Goal. Process an Arabic (+English) cheque or ID image: preprocess with OpenCV, OCR with Azure AI Vision / Document Intelligence, reason with GPT-4o vision, and produce RTL-correct, numeral-normalized structured output — handling the script pitfalls a Gulf bank cares about. JD4: multimodal and multilingual (Text + Vision, English/Arabic) and Computer Vision using OpenCV. Read K05 first.

Stack. OpenCV (opencv-python-headless) · Azure AI Vision Read / Document Intelligence (Arabic) · Azure OpenAI gpt-4o (vision) · optional YOLO/Custom Vision for ROI detection. Local fallback. OpenCV + Tesseract with the Arabic language pack; a local multimodal model for the reasoning step.

Run it (offline)

pip install -r requirements.txt        # numpy + opencv-python-headless + pytest
python run.py                          # deskew a synthetic cheque + Arabic logic + words-vs-figures
python run.py --save                   # also writes before/after PNGs to ./out/
pytest -q                              # 9 tests = success criteria

The OpenCV preprocessing (deskew/denoise/threshold/perspective) and the Arabic logic (Eastern-numeral normalization, amount-in-words parsing, words-vs-figures control) are real and run offline. OCR + GPT-4o vision are pluggable (a FakeOCR cheque result for tests; the docstrings show the Azure Vision / GPT-4o calls).

Code tour

Steps

1. Preprocess (OpenCV). Take a phone photo of a cheque/ID. Detect the document contour → perspective-correct & crop → deskew → denoise → adaptive threshold. Show before/after; this alone should lift OCR accuracy.
2. Detect ROIs (optional). YOLO/Custom Vision to locate the amount box, payee line, signature, MICR/MRZ; crop each for targeted OCR.
3. OCR (Arabic+English). Run Vision Read / Document Intelligence with Arabic support; get text + boxes + per-line detected language.
4. Normalize. Convert Eastern Arabic numerals (٠١٢٣) → Western; fix RTL/bidirectional string order so mixed Arabic/Latin renders correctly; strip/keep diacritics as needed.
5. Reason (GPT-4o vision). Send the cleaned image + instructions: extract payee, amount-in-figures, amount-in-words, date as strict JSON; instruct "null if illegible, do not guess." Cross-check figures vs words.
6. Validate. amount-in-words == amount-in-figures? date valid? signature ROI present? Confidence gate → human review on mismatch.
7. Output. RTL-correct structured record + the source crops + audit; hand to the clearing workflow/agent.

Measurable result

A working flow that takes a skewed Arabic cheque photo and returns correct, numeral-normalized structured fields, with the OpenCV before/after and a caught words-vs-figures mismatch. Be able to name five concrete ways Arabic changed the implementation (K05 §6).

Stretch

• Build a tiny Arabic eval set (5 cheques) with native review; compare OCR engines.
• Add signature presence/anomaly detection (YOLO ROI + heuristic).
• Compare Document Intelligence vs Vision Read vs GPT-4o-only on the same images and discuss the trade-off.

Talking point

"Arabic isn't a localization checkbox — it changes tokenization cost, OCR difficulty (RTL, connected/position-dependent letters, diacritics), numeral normalization, bidi rendering, and demands its own eval set with native reviewers. I preprocess deterministically with OpenCV first, OCR with an Arabic-capable engine, then use the multimodal LLM only for judgment — and I still validate money fields with words-vs-figures and confidence gating."

Resume bullet

"Built a multimodal Arabic/English cheque-processing pipeline: OpenCV preprocessing (perspective-correct, deskew, denoise, threshold) → Arabic-capable OCR → GPT-4o vision structured extraction with Eastern-numeral normalization and RTL handling → words-vs-figures validation and confidence-gated human review."