Knowledge 00 — Azure AI Platform Foundations

Goal of this module. Start from "I have an Azure account and nothing else" and end able to provision, secure, scale, and cost-model the entire Azure AI stack a banking GenAI app runs on — and explain every choice to a security architect. This is the layer JD4 calls "Microsoft Azure Cloud experience … is THE MUST." If you can only deeply learn one module before the interview, the interviewer will probe this hardest, because it's what separates "used the OpenAI API" from "ships on Azure in a bank."

Table of Contents

1. The mental model: resources, deployments, and the control/data plane

Before any AI, understand the shape of Azure itself, because the interview will assume it.

  • Tenant — your organization's identity boundary in Microsoft Entra ID (formerly Azure Active Directory). Every user, app, and service has an identity here.
  • Subscription — a billing and access boundary. Costs roll up to a subscription.
  • Resource group — a folder of resources with a shared lifecycle (deploy/delete together). A banking project is usually one resource group per environment (dev/test/prod).
  • Resource — a provisioned thing: an Azure OpenAI account, an AI Search service, a Cosmos DB account, a Storage account, a Key Vault, a Container App.

Two planes act on every resource — a distinction interviewers love:

  • Control plane (Azure Resource Manager, ARM) — create / configure / delete resources, set RBAC, networking. Authenticated by Entra ID; described by Bicep/ARM/Terraform.
  • Data plane — the actual use of the resource: calling the chat completions endpoint, querying the search index, reading a Cosmos document. Authenticated either by keys (a shared secret — avoid in banking) or by Entra ID tokens via Managed Identity (preferred).

Why this matters for a bank: you provision (control plane) with least-privilege RBAC and IaC so it's auditable and reproducible; you call (data plane) with Managed Identity so there are no secrets in code. "Keyless" is a banking expectation, not a nice-to-have.

2. Azure OpenAI Service from first principles

What it is. A first-party Azure resource that gives you OpenAI's models (GPT-4o, GPT-4.1, o-series reasoning models, text-embedding-3-*, DALL·E, Whisper, etc.) running inside Azure's compliance, networking, and identity boundary, with Microsoft's enterprise data-handling commitments. It is not api.openai.com. Same models, different operational/legal envelope — which is exactly why banks use it.

Why it exists. A bank cannot send customer PII to a public consumer API with consumer terms. Azure OpenAI gives: data stays in your chosen region, your prompts/outputs are not used to train the foundation models, Private Endpoints keep traffic off the public internet, Entra ID governs access, and you get an SLA and enterprise support. That bundle is the product.

Under the hood — the deployment concept (this trips people up).

  1. You create an Azure OpenAI resource in a region.
  2. Inside it, you create a deployment: you pick a base model (e.g. gpt-4o), a model version, a deployment name (you choose it, e.g. gpt-4o-prod), and a capacity type (Standard/PAYG, Provisioned/PTU, Global vs Regional, Data Zone).
  3. Your code calls the deployment name, not the model name:
from openai import AzureOpenAI
client = AzureOpenAI(
    azure_endpoint="https://my-aoai.openai.azure.com",
    api_version="2024-10-21",          # the data-plane API version (pin it)
    azure_ad_token_provider=token_provider,  # Managed Identity, NOT a key
)
resp = client.chat.completions.create(
    model="gpt-4o-prod",               # <- the DEPLOYMENT name, not "gpt-4o"
    messages=[{"role": "user", "content": "..."}],
)

Deployment SKUs (memorize the trade-off):

  • Standard (PAYG) — pay per token, shared capacity, latency varies with global load. Great for dev and spiky low volume.
  • Global Standard — routed to global capacity for best availability/price; data processed in the geography per the data-zone option.
  • Data Zone Standard — processing pinned to a data zone (e.g. EU) for residency, still pooled.
  • Provisioned Throughput (PTU) — you reserve units of throughput → predictable latency and capacity, billed monthly (or via reservations). Banks at scale buy PTUs so a viral morning doesn't throttle the assistant.

Versioning discipline (a senior signal): pin the api-version and the model version explicitly. Auto-update model versions can silently change behavior — unacceptable when you've validated a model for a regulated use case. You upgrade deliberately, re-running your eval suite (K08).

3. Capacity & cost: TPM, RPM, PTU, and quotas

This is the most common "do you actually run this in prod?" probe.

  • TPM (Tokens Per Minute) — the primary quota unit for Standard deployments. When you create a deployment you assign it a TPM budget from your regional quota. Every request consumes prompt + completion tokens against the per-minute bucket.
  • RPM (Requests Per Minute) — Azure derives a coupled RPM limit (historically ~6 RPM per 1,000 TPM). You can hit the RPM wall before the TPM wall with many tiny requests.
  • 429 Too Many Requests — what you get when you exceed TPM/RPM. Production code must handle it: exponential backoff with jitter, honoring the Retry-After header, and ideally a queue. The SDK retries some of this; you still design for it.
  • PTU (Provisioned Throughput Unit) — reserved capacity. You buy N PTUs; each model has a throughput-per-PTU. Gives consistent latency and isolates you from neighbor load. Costed as a monthly commitment; PTU reservations discount it further. The sizing question — "how many PTUs for 500 req/s at 800 in / 400 out tokens?" — is a real interview exercise (see Lab 05 and K07).

Cost intuition. Your bill ≈ (input tokens × input price + output tokens × output price) summed over traffic. Output tokens cost more than input. Levers to cut it:

  • Prompt caching (repeated system prompts / long context cached for a discount).
  • Model tiering — use a small/cheap model (gpt-4o-mini/4.1-mini) for routing, classification, and extraction; reserve the big model for final reasoning.
  • Trim retrieved context to the top-k that actually helps (more context ≠ better, and it's tokens you pay for).
  • PTUs once volume is steady and predictable.

4. Azure AI Foundry / Microsoft Foundry

What it is. The unified platform/portal/SDK for building, evaluating, deploying, and governing AI on Azure — the successor umbrella over the older "Azure AI Studio." Rebranded "Microsoft Foundry" in 2026. Key pieces:

  • Model catalog — Azure OpenAI models plus open and partner models (Llama, Mistral, Cohere, DeepSeek, etc.) deployable as serverless APIs or managed compute.
  • Foundry Agent Service — a managed runtime for production agents (GA), built on the OpenAI Responses API, wire-compatible with OpenAI agents. It provides hosted execution, tools (including built-in ones like file search, code interpreter, Azure AI Search, and your custom/MCP tools), threads/state, memory (procedural/user/session), and observability via Azure Monitor. This is Azure's answer to "I don't want to host my own agent loop."
  • Evaluators / Observability — built-in and custom evaluators (groundedness, relevance, coherence, safety) you can run offline or as continuous evaluation on sampled production traffic (K08).
  • Prompt flow — a tool for authoring, testing, and CI-gating LLM pipelines.
  • Content Safety integration, governance, and a project/hub structure for team collaboration.

Why it matters for JD4: the JD names "Azure AI Foundry" as a MUST. The signal they want: you know Foundry is where you manage models, run evals, and optionally host agents — not just a fancy playground. Saying "I'd host the agent in Foundry Agent Service for managed memory + observability, or self-host the LangGraph loop in a Container App when I need full control" is exactly the senior nuance they're after.

5. The wider Azure AI services family

"Azure AI services" (formerly Cognitive Services) is the family of prebuilt AI APIs. JD4 names four you must know:

ServiceWhat it doesJD4 use
Azure OpenAILLMs, embeddings, vision, speech-to-textReasoning, RAG, agents
Azure AI Search (was Cognitive Search)Retrieval engine: keyword + vector + hybrid + semantic rankerThe "R" in RAG (K02)
Azure AI Document Intelligence (was Form Recognizer)OCR + layout + key-value + table + custom extractionKYC/statement IDP (K04)
Azure AI VisionImage analysis, Read OCR, video retrieval, Florence modelsCheque/ID images, OCR (K05)

Adjacent ones worth a sentence: Azure AI Language (NER, PII detection, sentiment, key-phrase, language detection, conversational language understanding), Azure AI Translator (Arabic↔English), Azure AI Speech (ASR/TTS for IVR). A multi-service "Azure AI services" resource can expose several of these under one endpoint/key.

6. Identity & secrets: Entra ID, Managed Identity, Key Vault

The banking rule: no secrets in code, ever. Here's the chain:

  • Microsoft Entra ID — the identity provider. Issues OAuth2 tokens. Every human and service principal authenticates here.
  • Managed Identity — an Entra identity automatically managed for an Azure resource (e.g. your Container App). Your app gets a token with no stored credential; Azure rotates it. There are system-assigned (tied to one resource's lifecycle) and user-assigned (shareable across resources) variants.
  • RBAC role assignment — you grant the Managed Identity a least-privilege role on each resource, e.g. Cognitive Services OpenAI User on the AOAI resource, Search Index Data Reader on AI Search, a data-plane role on Cosmos. The app can do exactly those actions and nothing else.
  • Azure Key Vault — for the few secrets you genuinely can't avoid (a third-party API key, a connection string). Apps read them via Managed Identity at runtime; secrets never land in Git or env files committed to source.
# Keyless auth pattern — the senior default
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
cred = DefaultAzureCredential()  # uses Managed Identity in Azure, your login locally
token_provider = get_bearer_token_provider(
    cred, "https://cognitiveservices.azure.com/.default")
# pass token_provider to AzureOpenAI(...) — no api_key anywhere

Why interviewers care: a leaked key in a banking repo is a reportable security incident. Demonstrating the Managed-Identity + Key-Vault + RBAC pattern instantly reads as "has shipped in an enterprise."

7. Networking: Private Endpoints and the bank's "no public egress" rule

By default, Azure PaaS resources have public endpoints (reachable from the internet, but still auth-gated). Banks usually forbid that. The tools:

  • Private Endpoint — gives the resource a private IP inside your VNet; traffic never traverses the public internet. Combined with Public Network Access = Disabled on the resource, the only way to reach Azure OpenAI / AI Search / Cosmos is from inside the network.
  • VNet integration — your compute (Container Apps environment / AKS) joins the VNet so it can reach those private endpoints.
  • Private DNS zones — resolve the resource's hostname to the private IP.
  • NSGs / Azure Firewall — control and log egress; a bank often forces all outbound through a firewall for inspection.
  • API Management (APIM) in front of your FastAPI / AOAI — central auth, rate-limiting, logging, and a single audited choke point.

The interview-ready sentence: "In a bank I disable public network access on the AOAI, Search, and Cosmos resources, expose them via Private Endpoints into the app's VNet, put APIM in front for throttling and audit, and force egress through Azure Firewall."

8. Content Safety & data-handling commitments

Azure AI Content Safety is a service (and a built-in filter on AOAI) that screens text/images:

  • 4 harm categories — hate, sexual, violence, self-harm — each scored severity 0–7. You set thresholds; content over threshold is blocked or flagged. Runs on input (user prompt) and output (model completion).
  • Prompt Shields — detects jailbreak attempts and indirect prompt injection (malicious instructions hidden in retrieved documents — very relevant to RAG over uploaded customer docs).
  • Groundedness detection — flags ungrounded (hallucinated) claims relative to provided sources.
  • Protected material detection — flags copyrighted text/code.

Data-handling commitments (the compliance pitch you should be able to recite): with Azure OpenAI, prompts and completions are not used to train Microsoft/OpenAI foundation models; data is processed and (for abuse monitoring) optionally stored in your geography; you can apply for abuse-monitoring opt-out for sensitive workloads so no human review/retention occurs. These commitments are why a bank's CISO approves Azure OpenAI over a consumer API.

9. Regions, residency, and availability

  • Region — a geographic Azure location (e.g. UAE North, West Europe, Sweden Central). Not every model is in every region; you pick regions where (a) your data must reside and (b) the model you need is available.
  • Data residency — banks often must keep data in-country or in-region (e.g. a UAE bank → UAE North, or an EU data zone). Use Regional or Data Zone deployments, not unrestricted Global, when residency is mandated.
  • Availability / HA — design for the model endpoint being unavailable: multi-region deployments with a router/failover, retries, and graceful degradation (e.g. fall back to a smaller model or a templated response). PTUs in two regions behind APIM is a common HA pattern.
  • Quotas are per-region — your TPM budget is regional; multi-region also multiplies headroom.

10. Putting it together: a reference resource diagram

Resource Group: bank-assistant-prod  (one per env; deployed via Bicep)
├── Microsoft Entra ID app + Managed Identity (user-assigned)
├── Azure OpenAI account  [Public access: DISABLED, Private Endpoint]
│     ├── deployment: gpt-4o-prod            (PTU)
│     ├── deployment: gpt-4o-mini-router     (PAYG, cheap routing/classification)
│     └── deployment: text-embedding-3-large (PAYG)
├── Azure AI Search        [Private Endpoint] — index: kb-policies, semantic config
├── Azure AI Document Intelligence  [Private Endpoint]
├── Azure AI Vision        [Private Endpoint]
├── Cosmos DB (NoSQL)      [Private Endpoint] — conversations, state, vectors
├── Storage (Blob)         [Private Endpoint] — raw docs, audit logs (immutable)
├── Key Vault              [Private Endpoint] — the few real secrets
├── Container Apps Env (VNet-integrated)
│     └── FastAPI app (Managed Identity → RBAC to AOAI/Search/Cosmos)
├── Container Registry (ACR) — app images
├── API Management — auth, throttle, audit choke point
├── Azure Monitor / App Insights / Log Analytics — traces, metrics, audit
└── Azure AI Foundry project — model mgmt, evaluators, (optional) Agent Service

If you can draw and narrate this, you have demonstrated the "Strong Microsoft Azure Cloud experience … THE MUST."

11. Common misconceptions

  • "Azure OpenAI is just a proxy to OpenAI." No — different endpoint, different data/legal/network envelope, different auth, deployment-name indirection, regional quotas. The operational model is distinct.
  • "I call the model by name gpt-4o." No — you call your deployment name. Forgetting this fails the first hands-on test.
  • "Keys are fine." In a bank, prefer Managed Identity; keys are a finding.
  • "Public endpoint + auth is enough." Banks usually require Private Endpoints / no public egress.
  • "PTU is just more expensive PAYG." PTU is reserved capacity for predictable latency, a different SLA story — the reason latency-sensitive prod buys it.
  • "Cognitive Search / Form Recognizer." Renamed to AI Search / Document Intelligence; using old names signals you've been away from the platform.

12. Interview Q&A

Q: Walk me through how you'd deploy GPT-4o for a banking app on Azure. Create an Azure OpenAI resource in a residency-compliant region; create a gpt-4o deployment (PTU for predictable latency at scale, plus a mini deployment for routing); pin model + API version; disable public access and expose via Private Endpoint; authenticate the app with Managed Identity + least-privilege RBAC (Cognitive Services OpenAI User); front it with APIM for throttling/audit; wire Content Safety in/out; emit traces to App Insights; gate releases on an eval suite. Cost-control via model tiering + prompt caching + PTU reservations.

Q: PAYG vs PTU — when each? PAYG (Standard) for dev and spiky low volume — pay per token, shared, variable latency. PTU once volume is steady and you need predictable latency/throughput and isolation from neighbor load — reserved monthly capacity, cheaper per token at scale, the right call for a customer-facing banking assistant.

Q: How do you keep customer PII safe with Azure OpenAI? Azure OpenAI's data commitments (no training on your data, regional processing, abuse-monitoring opt-out available); Private Endpoints (no public traffic); Managed Identity (no keys); PII redaction before the prompt where feasible (K08); content safety + groundedness; full audit logging to immutable storage; residency-pinned regions.

Q: You're getting 429s in production. What do you do? Short term: exponential backoff + jitter honoring Retry-After, a request queue, and shedding non-critical load. Diagnose whether it's the TPM or RPM ceiling. Medium term: raise quota, split load across regions, move hot paths to PTU for guaranteed capacity, and cut tokens (caching, context trimming, model tiering).

Q: What's the difference between the control plane and data plane here, and why does it matter for compliance? Control plane = ARM (create/configure/RBAC/networking), described as IaC for auditability and reproducibility. Data plane = the actual model/search/db calls, authenticated by Entra tokens via Managed Identity. Separating them lets you give developers IaC-reviewed provisioning while runtime access is least-privilege and keyless — both auditable, which a bank's regulators require.

13. References

  • Azure OpenAI Service — Microsoft Learn: What is Azure OpenAI Service, Deployment types, Quotas and limits, Provisioned throughput. learn.microsoft.com/azure/ai-services/openai.
  • Azure AI Foundry / Microsoft Foundry — Foundry Agent Service overview, What's new in Microsoft Foundry (2026); built on the OpenAI Responses API.
  • Microsoft Entra ID & Managed Identity — Managed identities for Azure resources; Azure RBAC docs.
  • Azure Key Vault — Best practices for secrets management.
  • Private Link / Private Endpoints — Azure Private Link docs; Configure private endpoints for Azure AI services.
  • Azure AI Content Safety — Harm categories, Prompt Shields, Groundedness detection.
  • Azure AI services overview (formerly Cognitive Services) — Microsoft Learn.
  • Data, privacy, and security for Azure OpenAI Service — Microsoft Learn (data-handling commitments, abuse monitoring opt-out).
  • Bicep / Azure Resource Manager — Infrastructure as code on Azure docs.