System-Design Walkthroughs

The interview's centerpiece is "design us a [banking AI system] on Azure." This file gives you one reference architecture and four use-case variants, each with the structure to talk through, the trade-offs, and the safeguards. Learn to draw the stack and narrate it. Depth is in the knowledge modules.

Table of Contents

0. How to run a system-design answer

  1. Clarify first (30 seconds). Customer-facing or internal? Scale (users/QPS)? Languages (Arabic?)? Latency SLO? Regulatory/residency constraints? What's the system of record? — Asking these is the senior signal.
  2. State the approach in one sentence, then draw the stack.
  3. Walk the data flow end to end (request → response).
  4. Call out 2–3 key trade-offs with your choice and why.
  5. Address the banking safeguards (grounding, tool-not-model, access-trim, human-in-loop, audit, residency, cost).
  6. Mention eval + observability — how you'd know it works and keep it working.

Spend most time on retrieval quality, agent control, and safety — that's what they're probing.

1. The reference architecture (memorize this)

Almost every banking GenAI design is a specialization of this. Draw it; then for each use case, highlight the relevant parts.

[ Channels: Web / Mobile / Teams / WhatsApp / IVR ]  (Arabic + English)
                     │
            [ APIM ]  auth (Entra), throttle, audit choke point
                     │
        [ FastAPI BFF ]  async, SSE streaming, Pydantic, Managed Identity
                     │
   ┌─────────────[ LangGraph agent / Foundry Agent Service ]─────────────┐
   │  state · routing · tools · human-in-the-loop · checkpoints (Cosmos) │
   └───────┬───────────────┬────────────────┬──────────────┬────────────┘
           │               │                │              │
   [ Azure OpenAI ]  [ Azure AI Search ]  [ Tools ]   [ Content Safety ]
   gpt-4o / mini     hybrid+semantic      core-banking  Prompt Shields,
   embeddings        (security-trimmed)   (MCP/REST)    groundedness,
                                          via OBO auth   PII redaction
           │               │
   [ Ingestion: Document Intelligence · AI Vision · OpenCV/YOLO ]
           │
   [ Data: Cosmos (state/vectors) · Blob (docs, immutable audit) · SQL ]
           │
   [ Platform/Trust: Entra · Managed Identity · Key Vault · Private Endpoints ]
   [ Delivery: Git · CI/CD (eval gate) · Bicep · ACR · Container Apps · App Insights/Monitor ]

The narration spine: "Requests come through APIM (auth/throttle/audit) to an async FastAPI BFF, which drives a LangGraph agent. The agent reasons with Azure OpenAI, grounds in security-trimmed hybrid retrieval from AI Search, and calls tools to systems of record for live data — never inventing money facts. Content Safety guards in and out, high-risk actions pause for human approval, everything is audited, and we deploy via CI/CD with an eval gate behind Private Endpoints with Managed Identity."

2. Use case A: Customer Conversational AI

Prompt: "Design a customer-facing assistant that answers product/policy questions and handles simple account queries, in Arabic and English, for millions of customers."

Approach: RAG for knowledge + a lightly agentic layer for account tools, on the reference stack.

Flow:

  1. Channel → APIM → FastAPI (/chat/stream, SSE). Authenticate the customer (Entra B2C / bank IdP); derive their identity + entitlements.
  2. Route intent (cheap gpt-4o-mini, structured output): knowledge question → RAG; account query → tool; out-of-scope/advice → refuse; complex → escalate.
  3. RAG (knowledge): query rewrite → hybrid+semantic AI Search, security-trimmed + language filter → top-5 → grounding prompt with citations + abstention → gpt-4o.
  4. Tool (account): get_balance/list_transactions via OBO auth to core banking — live data from the system of record, never the model.
  5. Guardrails: Content Safety + Prompt Shields + PII redaction in/out; groundedness check before responding.
  6. State in Cosmos (partition by conversationId); audit everything.

Trade-offs to voice: hybrid+semantic vs pure vector (recall vs cost); model tiering (mini for routing, 4o for answers — cost at millions of users); PTUs for predictable latency; how much agency (start RAG+tools, not full autonomy).

Safeguards: grounding+citations+abstention; tool-not-model for figures; security-trimmed retrieval; Arabic handling (multilingual embeddings, RTL, dialect, Arabic eval); deflection-to-human path; disclosed AI use.

Success metrics: groundedness, answer relevance, deflection/containment rate, CSAT, cost/conversation, p95 latency — stratified by language.

3. Use case B: KYC / Document Intelligence onboarding

Prompt: "Automate customer onboarding: ingest IDs, salary certificates, and statements, extract and verify data, with minimal manual effort but full compliance."

Approach: an IDP pipeline (K04) with confidence-gated straight-through processing and human review, feeding downstream AML/core systems.

Flow:

  1. Upload → Blob (immutable). Optional OpenCV preprocessing for phone photos.
  2. Classify document type → route to extractor.
  3. Extract: prebuilt-idDocument (IDs), prebuilt-layout (statements/tables), custom/neural (bespoke forms) → fields + confidence + bounding regions.
  4. Confidence gate: high-risk fields (ID number, IBAN, amount) ≥ 0.95 → straight-through; else → human review queue with field highlighted on source.
  5. Validate: ID checksum, IBAN mod-97, expiry, statement balance math; cross-doc consistency (name matches across docs).
  6. Screen: push identity to AML/sanctions/PEP screening (humans decide).
  7. Persist structured record + source links + audit; trigger account creation in core system.

Trade-offs: prebuilt vs custom vs Layout+LLM; confidence thresholds (STP rate vs error risk); how much the LLM judges vs deterministic validation.

Safeguards: human review of low-confidence money/identity fields; deterministic validation; full audit of fields + corrections; never auto-decide eligibility — AML/onboarding decisions stay human-governed; residency for sensitive ID data.

Metrics: STP rate, field-level extraction accuracy (esp. IBAN/ID/amount), review turnaround, onboarding time, error/escalation rate.

4. Use case C: Agentic workflow automation (dispute handling)

Prompt: "Automate disputed-transaction handling end to end with an agent."

Approach: a LangGraph agent (K03) with durable state and human approval — the canonical "agentic banking" answer.

Flow / graph:

  1. intake — capture the dispute (channel/agent), identify customer.
  2. identify_txn — tool: transaction lookup (system of record).
  3. retrieve_policy — RAG over dispute/chargeback policy (cited).
  4. assess_eligibility — LLM reasons + deterministic rule checks (time window, amount limits, category).
  5. draft_resolution — propose outcome + rationale + citations.
  6. human_review — interrupt before any refund/credit; an ops agent approves/declines (logged).
  7. execute — tool: file the dispute / issue provisional credit (idempotent).
  8. notify — customer update (Arabic/English). State (txn, policy refs, eligibility, draft, approval) carried through; checkpointed to Cosmos so a crash mid-flow resumes; everything audited.

Trade-offs: single agent vs supervisor+workers; how much is LLM judgment vs deterministic rules (hard money rules in code); where the human gate sits (every refund vs above a threshold).

Safeguards: human-in-the-loop on money movement; least-privilege idempotent tools (a retried "credit" can't double-pay); live amounts from systems of record; immutable audit of the full trajectory; step/cost caps; treat any document text as untrusted (injection).

Metrics: task success rate, tool-call accuracy, cycle time, % auto-resolved vs escalated, cost/case, rework rate.

5. Use case D: Multimodal Arabic cheque processing

Prompt: "Process scanned/photographed Arabic+English cheques for clearing."

Approach: the vision pipeline from K05: OpenCV → OCR → GPT-4o vision → validation.

Flow:

  1. Image → Blob. OpenCV: contour detect → perspective-correct → deskew → denoise → adaptive threshold.
  2. Optional YOLO/Custom Vision ROIs (amount box, payee, signature, MICR).
  3. OCR (Vision Read / Document Intelligence, Arabic) → text + boxes + per-line language.
  4. Normalize: Eastern→Western numerals, RTL/bidi order.
  5. GPT-4o vision: extract payee, amount-in-figures, amount-in-words, date → strict JSON; "null if illegible."
  6. Validate: words == figures? date valid? signature present? → confidence gate → human review on mismatch.
  7. Persist + audit → clearing workflow.

Trade-offs: Document Intelligence vs Vision vs GPT-4o-only; how much OpenCV preprocessing; YOLO ROI vs whole-image OCR.

Safeguards: words-vs-figures cross-check; confidence gating + human review; Arabic eval set with native review; full audit with source crops; fraud/anomaly signals flagged not auto-decided.

Metrics: field accuracy (amount especially), STP rate, fraud-flag precision/recall, throughput.

6. Cross-cutting: scale, cost, HA, security

Have these ready for "how does this scale to millions / stay up / stay cheap / stay secure":

  • Scale & latency: stateless FastAPI on Container Apps autoscaling on concurrency (≥2 zonal replicas); PTUs for predictable model capacity; cache embeddings and frequent answers; AI Search replicas/partitions for query load; Cosmos autoscale RU/s with a good partition key.
  • Cost: model tiering (mini routes/extracts, 4o answers, o-series only for hard planning); prompt caching; trim retrieved context; track $/conversation as a first-class metric; PTU reservations at steady volume.
  • HA/DR: multi-zone replicas, multi-region failover behind APIM/Front Door, retries/circuit breakers/idempotency, health probes, graceful degradation; documented DR runbook.
  • Security (layers): Entra + Managed Identity + least-privilege RBAC + OBO; Private Endpoints / no public egress; Key Vault; Content Safety + Prompt Shields; security-trimmed retrieval; PII redaction; Defender + Sentinel; immutable audit; residency-pinned regions.
  • Evaluation/observability: CI eval gate (groundedness/safety/task success) + online continuous eval → Azure Monitor alerts; full distributed traces (prompt→retrieval→tool→model) with tokens/latency/cost; user-feedback loop into the eval set.

Close any design with: "and I'd prove it works with an eval gate on groundedness and task success, monitor it continuously, and audit every decision — because it's a bank."