Knowledge 05 — Vision, Multimodal & Multilingual (incl. Arabic)

Goal of this module. Cover the JD4 lines "Implement multimodal and multilingual AI solutions (Text + Vision, English/Arabic)" and "Build Computer Vision workflows using OpenCV" (+ the "good to have" YOLO/deep-learning). Go from "what is a pixel" to designing an Arabic cheque/ID processing flow that combines OpenCV preprocessing, Azure AI Vision OCR, YOLO detection, and GPT-4o multimodal reasoning — with the RTL/script pitfalls a Gulf bank cares about.

Table of Contents

1. The three ways to "do vision" — and when each

A senior engineer chooses the cheapest tool that solves the task, not the fanciest:

  1. Classic CV (OpenCV) — deterministic pixel operations: resize, deskew, threshold, edge/contour detection, crop. No ML, fast, free, explainable. Use for preprocessing and simple geometric tasks (find the document boundary, straighten a scan, isolate the signature box).
  2. Specialized DL models (YOLO, Azure Vision) — trained detectors/OCR/classifiers. Use when you need to find/recognize specific things (detect the cheque's MICR line, read text, detect a stamp/logo) at high accuracy and speed.
  3. Multimodal LLM (GPT-4o vision) — a generalist that reasons about an image in natural language. Use when the task is open-ended/contextual ("describe what's wrong with this submitted ID photo", "extract these fields and explain anomalies") or when you want one model for image+text together. Most flexible, most expensive, least deterministic.

The pipeline principle: combine them. OpenCV cleans the image → a detector/OCR extracts structured signals → an LLM reasons over the result. Don't send a crooked, noisy 12-megapixel phone photo straight to GPT-4o; preprocess first (cheaper, more accurate).

2. OpenCV: classic computer vision from first principles

What an image is. A digital image is a grid of pixels; a grayscale pixel is one intensity (0–255), a color pixel is three (BGR in OpenCV's convention). So an image is a NumPy array of shape (height, width, channels). Every OpenCV op is array math.

The preprocessing operations you'll actually use in banking (and what they do):

  • Grayscale (cv2.cvtColor) — drop color; most document tasks don't need it and it's 3× less data.
  • Resize / DPI normalization — OCR wants a consistent scale (~300 DPI equiv).
  • Denoising (cv2.GaussianBlur, fastNlMeansDenoising) — remove scanner/camera speckle that confuses OCR.
  • Thresholding / binarization (cv2.threshold, adaptive / Otsu) — turn the image black-and-white so text separates cleanly from background. Otsu auto-picks the split; adaptive handles uneven lighting (phone photos).
  • Deskew — detect the dominant text angle (Hough lines / minAreaRect) and rotate so lines are horizontal. Crooked scans wreck OCR; this is the single highest-value preprocessing step.
  • Edge detection (cv2.Canny) + contours (cv2.findContours) — find shapes/boundaries: locate the document edges in a photo to crop & perspective-correct (cv2.getPerspectiveTransform + warpPerspective) so you get a flat, cropped scan from a phone snapshot.
  • Morphology (erode/dilate/open/close) — clean up binary images, connect broken strokes, remove specks.
  • ROI extraction — slice the array to a region (the signature box, the MICR strip, the photo on an ID) for targeted downstream processing.

Why it matters: OCR/extraction accuracy is dominated by image quality. A 10-line OpenCV preprocessing chain (deskew → denoise → adaptive threshold → crop) often lifts downstream extraction accuracy more than swapping models. Interviewers love a candidate who reaches for cheap deterministic preprocessing before expensive models.

import cv2, numpy as np
img  = cv2.imread("cheque.jpg")
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
gray = cv2.fastNlMeansDenoising(gray, h=10)
bw   = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
                             cv2.THRESH_BINARY, 31, 10)
# deskew via minAreaRect of foreground pixels, then warpAffine ...

3. YOLO and object detection

Object detection = find what is in an image and where (bounding boxes + class labels), unlike classification (one label for the whole image) or OCR (text). YOLO ("You Only Look Once") is the dominant real-time detector family (Ultralytics' v8/v11 are current).

Why "You Only Look Once": earlier detectors ran a classifier over many region proposals (slow). YOLO makes detection a single forward pass of a convolutional network that simultaneously predicts boxes + class probabilities over a grid — hence real-time speed. It outputs boxes with confidence, and Non-Max Suppression (NMS) removes duplicate overlapping boxes for the same object.

Banking uses: detect and localize the signature, stamp, photo, MRZ, MICR line, or logo on a document; detect tampering regions; count/locate items; queue/branch analytics from cameras. You'd fine-tune YOLO on a few hundred labeled banking samples for a custom object (e.g. "find the cheque's signature region"), then crop that ROI for OCR/verification.

When YOLO vs Azure Vision vs Document Intelligence: Document Intelligence already handles document fields; reach for YOLO when you need a custom object detector it doesn't provide, real-time video, or on-prem/edge inference. It's "good to have" in JD4 — know what it is and when it fits, you won't be asked to train one live.

4. Azure AI Vision

Azure AI Vision (part of Azure AI services, formerly Computer Vision/Cognitive Services) is the managed vision API. Key capabilities:

  • Read (OCR) — robust printed + handwritten text extraction across many languages (incl. Arabic), with word/line bounding boxes and reading order. The general-purpose OCR you call when you don't need full document layout.
  • Image Analysis 4.0 (Florence foundation model) — captions, tags, objects, people, smart crops, background removal, OCR, and dense/region captions in one model.
  • Face (gated/limited-access for verification) — detection/verification; relevant to ID selfie-match in KYC but access-restricted and compliance-sensitive.
  • Video Retrieval / spatial analysis — search and analyze video.
  • Custom Vision — train your own image classifier/detector with few samples (a no-code alternative to YOLO for simple cases).

Vision OCR vs Document Intelligence OCR: they share OCR tech; Document Intelligence adds layout/tables/fields/confidence and is the right call for documents (statements/forms), while AI Vision Read is for general images/scene text. For JD4 banking docs, default to Document Intelligence; use AI Vision for general image understanding.

5. Multimodal LLMs (GPT-4o vision)

What "multimodal" means here. GPT-4o / GPT-4.1 accept images alongside text in the same prompt and reason jointly. Under the hood the image is encoded into tokens (vision encoder → embeddings the transformer attends to like text tokens), so an image costs tokens proportional to its resolution/tiling — a real cost factor.

When to use it (vs OCR/detection):

  • The task needs understanding/judgment, not just extraction: "Does this submitted utility bill show a name and address matching the application, and is anything suspicious?" — a single GPT-4o call with the image + instructions + structured output.
  • You want one model to read, interpret, and respond in language (and in Arabic).
  • Flexible/long-tail document types where training a custom extractor isn't worth it (pair with Document Intelligence Layout for accuracy on the structured bits).

When not to: high-precision field extraction at scale where a tuned extractor is cheaper and more reliable, or where you need exact bounding boxes/confidences for audit. Multimodal LLMs can also hallucinate about images — so the same grounding/verification discipline applies, and for money/identity fields you still validate.

resp = client.chat.completions.create(model="gpt-4o-prod", messages=[
  {"role":"system","content":"Extract fields as JSON; if illegible, mark null. Do not guess."},
  {"role":"user","content":[
     {"type":"text","text":"Extract payee, amount, date from this cheque."},
     {"type":"image_url","image_url":{"url": data_uri}}]}],
  response_format={"type":"json_object"}, temperature=0)

6. Multilingual AI and the Arabic problem

The Arabic requirement is a strong tell that this is a Gulf/MENA bank, and it touches every layer:

  • Tokenization cost — Arabic uses more tokens per word than English (often ~2×), raising cost/latency and tightening the context budget (K01 §2). Budget for it.
  • Script properties — Arabic is RTL (right-to-left), letters are cursive/connected and change shape by position (initial/medial/final/isolated), and diacritics (harakat) may be present or absent. These stress OCR, tokenizers, and any string handling.
  • RTL rendering & mixed text — UI must render RTL correctly and handle bidirectional (Arabic + Latin/numbers in one line) without mangling order. A subtle but common production bug.
  • Numerals — Eastern Arabic numerals (٠١٢٣٤٥٦٧٨٩) vs Western (0123456789); normalize before validation/math.
  • Dialects vs MSA — Modern Standard Arabic (formal/written) differs from Gulf/Egyptian/Levantine dialects (spoken). A customer may type dialect; ensure your model/prompts handle it and test it.
  • Embeddings & retrieval — use multilingual embeddings so Arabic queries hit Arabic (and cross to English) docs; index a language field (K02 §12).
  • Generation — instruct the model to answer in the user's language; detect language (Azure AI Language) or let the model mirror it. Keep the system prompt in English but allow Arabic responses.
  • Translation — Azure AI Translator for Arabic↔English when you must normalize to one pivot language (e.g. for downstream English-only systems) — but prefer native handling for user-facing text to preserve nuance.
  • Evaluation — you must evaluate with native Arabic test cases and reviewers; English metrics don't transfer, and groundedness/quality can silently degrade in Arabic.

The interview soundbite: "Arabic isn't a localization checkbox — it changes tokenization cost, OCR difficulty (RTL/connected/diacritics), numeral normalization, RTL/bidi rendering, dialect handling, multilingual embeddings, and demands its own eval set with native reviewers."

7. A banking vision pipeline: the Arabic cheque

phone photo / scan of an Arabic+English cheque
  → OpenCV: detect document contour → perspective-correct → deskew → denoise → adaptive threshold
  → YOLO/Custom Vision (optional): locate ROIs (amount box, payee line, signature, MICR)
  → Azure AI Vision Read / Document Intelligence Layout: OCR text (Arabic+English) per ROI, with boxes
  → normalize: Eastern→Western numerals, RTL/bidi-correct strings, parse amount/date
  → GPT-4o (optional): cross-check legal vs courtesy amount, flag inconsistencies, structured JSON
  → validation: amount-in-words == amount-in-figures? date valid? signature present?
  → confidence gate: high → straight-through ; low → human review (highlight ROI on image)
  → persist structured record + source crop + audit log → clearing workflow / agent (K03)

This single diagram demonstrates OpenCV + detection + OCR + multimodal LLM + Arabic handling + confidence/human-review — i.e. every vision/multimodal/multilingual line of the JD at once.

8. Common misconceptions

  • "Send the raw photo to GPT-4o." Preprocess with OpenCV first — cheaper and far more accurate.
  • "OCR understands documents." OCR gives text+boxes; structure/fields need Layout/extraction; meaning/judgment needs an LLM.
  • "Multimodal LLMs don't hallucinate about images." They do — validate money/identity fields and use structured output + abstention.
  • "Arabic is English right-to-left." It's connected/cursive, position-dependent shaping, diacritics, Eastern numerals, dialects, and bidi — each a real engineering concern.
  • "Use multilingual? Just translate everything to English." Translation loses nuance for user-facing text and adds a failure point; prefer native multilingual models, translate only when a downstream system demands one language.
  • "YOLO is always best for finding things in docs." Document Intelligence already finds document fields; YOLO is for custom objects/real-time/edge.

9. Interview Q&A

Q: A customer uploads a blurry phone photo of an Arabic ID. Walk me through processing it. OpenCV first: find the document contour, perspective-correct and crop, deskew, denoise, adaptive threshold. Then Document Intelligence prebuilt-idDocument (or Vision Read for raw OCR) with Arabic support → fields + confidence + bounding boxes. Normalize Eastern→Western numerals and bidi strings. Validate (ID checksum, expiry); cross-check the ID photo for KYC if in scope (gated Face/face-match, compliance permitting). Confidence gate: low → human review with the field highlighted on the image. Audit everything. Optionally GPT-4o to flag anomalies/tampering.

Q: When do you use OpenCV vs Azure Vision vs a multimodal LLM? OpenCV for deterministic preprocessing/geometry (deskew, crop, threshold, ROI) — cheap, explainable. Azure Vision/Document Intelligence for robust OCR/structure/fields at scale. Multimodal LLM when the task needs reasoning/judgment over the image or open-ended extraction. Combine: OpenCV cleans → detector/OCR extracts → LLM reasons.

Q: What makes Arabic harder than English for an AI banking assistant? More tokens/word (cost/latency/context); RTL and bidirectional layout; cursive position-dependent letter shapes and diacritics stressing OCR; Eastern Arabic numerals needing normalization; MSA-vs-dialect variation in user input; need for multilingual embeddings and language-aware retrieval; and a mandatory Arabic eval set with native reviewers because English metrics don't transfer.

Q: How does YOLO work and when would you use it here? YOLO does detection in a single forward pass of a CNN that predicts bounding boxes + class probabilities over a grid, with NMS removing duplicate boxes — hence real-time speed. In banking I'd fine-tune it to localize custom objects (signature/stamp/MICR/tampering regions) or for real-time camera analytics, then crop ROIs for OCR/verification — when Document Intelligence's built-in document fields aren't enough.

Q: How do you keep multimodal extraction trustworthy for payments? Treat the LLM/OCR output as a proposal, not truth: cross-field validation (amount-in-words vs figures, checksums), confidence thresholds with human review of low-confidence high-risk fields, structured output with abstention ("null if illegible, don't guess"), full audit linking each field to its source crop, and Arabic-inclusive evaluation.

10. References

  • OpenCV — official docs/tutorials (image processing, thresholding, contours, geometric transforms); Bradski & Kaehler, Learning OpenCV.
  • YOLO — Redmon et al., "You Only Look Once" (2016); Ultralytics YOLOv8/v11 docs; Non-Max Suppression.
  • Azure AI Vision — Microsoft Learn: Image Analysis 4.0 (Florence), Read OCR, Custom Vision, language support.
  • GPT-4o / multimodal — Azure OpenAI vision/GPT-4o docs; OpenAI GPT-4o system card.
  • Arabic NLP/OCR — Habash, Introduction to Arabic Natural Language Processing (2010); Unicode Bidirectional Algorithm (UAX #9).
  • Azure AI Translator / AI Language — Microsoft Learn (translation, language detection, PII).
  • Related: K04 — Document Intelligence, K02 — RAG & AI Search.