Knowledge 07 — Production Engineering (FastAPI · Docker · Containers · Cosmos DB · CI/CD)

Goal of this module. The JD's whole right-hand side: "Develop REST APIs using FastAPI and deploy using Docker and Azure Containers," "Integrate Azure Cosmos DB," "Implement CI/CD pipelines and Git-based workflows." Go from "I can write a Flask route" to shipping and operating a banking-grade AI service on Azure — async APIs, streaming, containers, a correctly-modeled Cosmos data layer, and an automated, eval-gated pipeline.

Table of Contents

1. FastAPI: why it's the default for AI services

FastAPI is a Python web framework built on ASGI (async) and Pydantic (typed validation). It's the default for LLM services because:

  • Async-native — an LLM call spends seconds waiting on the network; async lets one worker handle many concurrent requests while they wait (§2). Flask (WSGI) blocks a worker per request.
  • Streaming — first-class support for streaming responses (SSE), essential for token-by-token chat UX.
  • Typed contracts — request/response models are Pydantic; you get validation + auto-generated OpenAPI/Swagger docs for free.
  • Dependency injection — clean auth, rate-limit, and client-lifecycle wiring.

JD4 also lists Streamlit/Flask as "good to have" for rapid PoC — the honest framing: Streamlit/Flask to demo an idea in an afternoon; FastAPI for the production API.

from fastapi import FastAPI, Depends
from pydantic import BaseModel
app = FastAPI()

class ChatRequest(BaseModel):
    message: str
    conversation_id: str | None = None
    language: str = "en"

@app.post("/chat")
async def chat(req: ChatRequest, user=Depends(verify_entra_token)):
    ...  # async LLM/agent call; returns grounded answer + citations

2. Async, streaming (SSE), and concurrency

Why async matters here. A chat request might take 2–8s (LLM decode). With sync workers, 50 concurrent users need 50 blocked workers. With async, while request A awaits the model, the event loop serves B, C, … — far higher concurrency per instance, lower cost. Use async SDK clients (AsyncAzureOpenAI, async Cosmos client, httpx) and never block the event loop with sync I/O or CPU-heavy work (offload CPU work to a thread/process pool).

Streaming (Server-Sent Events). For chat, stream tokens as they're generated so the user sees output immediately (perceived latency ≪ total latency). FastAPI returns a StreamingResponse that yields chunks as the model produces them:

from fastapi.responses import StreamingResponse
@app.post("/chat/stream")
async def stream(req: ChatRequest):
    async def gen():
        async for chunk in client.chat.completions.create(..., stream=True):
            if delta := chunk.choices[0].delta.content:
                yield f"data: {delta}\n\n"
    return StreamingResponse(gen(), media_type="text/event-stream")

Resilience patterns to bake in: timeouts on every external call; retry with exponential backoff + jitter on 429/5xx (honor Retry-After); a circuit breaker so a dead dependency fails fast; idempotency keys for any state-mutating endpoint (critical in banking — a retried "transfer" must not double-execute); graceful degradation (smaller model / templated fallback) when the primary model is unavailable.

3. Pydantic and contract-first APIs

Pydantic validates and coerces data against typed models at the boundary. Benefits in a banking service:

  • Reject malformed input early with clear 422 errors — never let unvalidated data reach the model or the database.
  • Typed LLM output — validate the model's structured JSON against a Pydantic model; on failure, retry/repair (K06 §5).
  • Self-documenting — /docs (Swagger) and /openapi.json are generated, useful for the API consumers and for APIM import.
  • Settings management — BaseSettings reads config/secrets (from env/Key Vault) with validation; no stray os.getenv.

Contract-first means the Pydantic schemas are the API spec — front-end and back-end agree on types, and changes are reviewable in Git.

4. Docker and the container supply chain

Why containers. A container packages your app + exact dependencies + runtime into an immutable image that runs identically on your laptop, CI, and Azure — solving "works on my machine" and enabling reproducible, auditable deploys (a bank wants to know exactly what's running).

The image, well-built:

  • Multi-stage build — a builder stage installs deps; the final stage copies only what's needed → smaller, fewer CVEs.
  • Slim base (python:3.12-slim), non-root user, pinned dependency versions (lockfile), .dockerignore to keep secrets/junk out.
  • No secrets in the image — inject at runtime via Managed Identity/Key Vault.
  • Health endpoints (/health, /ready) for orchestrator probes.
FROM python:3.12-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
FROM python:3.12-slim
RUN useradd -m app
COPY --from=builder /install /usr/local
COPY . /app
WORKDIR /app
USER app
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Supply chain (banking cares): push to Azure Container Registry (ACR); scan images for vulnerabilities (Defender for Containers / Trivy); sign images; pin/track base-image versions. The registry + scan + sign chain is part of the audit story.

5. Azure Container options: ACA vs AKS vs ACI

The JD says "Azure Container Services" — know the family and when each:

  • Azure Container Apps (ACA) — serverless containers with scale-to-zero, built-in ingress, revisions, KEDA autoscaling (e.g. scale on concurrent requests/queue length), Dapr. The default for an AI microservice: managed, cheap at low/variable load, VNet-integratable. Best first answer.
  • Azure Kubernetes Service (AKS) — full Kubernetes; choose when you need fine-grained control, GPU node pools, complex multi-service topologies, or an existing k8s platform. More ops burden.
  • Azure Container Instances (ACI) — single-container, no orchestration; for simple jobs/burst tasks.
  • ACR — the registry feeding all of the above.

Senior answer: "ACA for the API by default (serverless, autoscale, VNet); AKS if the bank already standardizes on Kubernetes or I need GPU/complex topology." Pair with autoscaling rules tied to real signals and min replicas ≥ 2 across zones for HA.

6. Azure Cosmos DB from first principles

What it is. A globally-distributed, multi-model NoSQL database. For JD4 you use the NoSQL (Core/Document) API: JSON documents in containers, queried with a SQL-like dialect. ("Document DB" in the JD = this.) Why a bank picks it for an AI app: low-latency global reads/writes, elastic scale, schema-flexible JSON (great for conversation/state), tunable consistency, and native vector search.

The three concepts that decide success or failure:

  1. Partitioning & the partition key. Cosmos shards a container across logical partitions by a partition key you choose. All docs with the same key value live together. This choice is the #1 design decision — get it wrong and you can't fix it without a migration:

    • Pick a key with high cardinality and even access so load spreads (e.g. conversationId or userId), avoiding a hot partition (one key getting all traffic).
    • Keep most queries single-partition (you provide the partition key) — cross-partition queries fan out, cost more RU, and scale worse.
    • A logical partition has a 20 GB limit — don't pick a key that grows unboundedly per value.

  2. Request Units (RU/s) — the currency. Every operation costs RUs (a point read of a 1KB doc ≈ 1 RU; queries/writes cost more). You provision RU/s (manual or autoscale) or go serverless. Exceed your RU/s and you get 429 rate-limited (the SDK retries with backoff). Cost and throughput are RU/s; model your access patterns to minimize RU (point reads > queries, right indexes, avoid cross-partition).

  3. Consistency levels. Cosmos offers five (Strong → Bounded Staleness → Session → Consistent Prefix → Eventual). Session (default) is usually right for a chat app (read-your-own-writes per session). A bank moving money in Cosmos would consider Strong, trading latency for correctness — but money usually lives in the core banking system, not Cosmos.

Native vector search. Cosmos DB NoSQL supports vector indexing (DiskANN) and VectorDistance() in queries — so you can store embeddings alongside documents and do similarity search without a separate vector store. Useful for conversation memory or smaller RAG corpora colocated with app data; Azure AI Search remains the heavier-duty retrieval engine (K02).

# point read = cheapest; always pass the partition key
item = await container.read_item(item="msg-123", partition_key="conv-abc")

7. Data modeling for a banking AI app

What you actually store, and how:

  • Conversations / messages — container partitioned by conversationId (or userId); each turn a document {conversationId, role, content, timestamp, citations, toolCalls}. Point-read the conversation by partition key. Use TTL to auto-expire transient data per retention policy.
  • Agent state / checkpoints — LangGraph checkpoints persisted to Cosmos for durability/resume (K03 §5), partitioned by threadId.
  • Audit log — immutable record of every prompt, retrieval, tool call, and decision. Often written to append-only Blob (immutable/WORM) or a dedicated audit store, not just Cosmos, to satisfy regulators (tamper-evidence).
  • Extracted documents / KYC records — structured output of Document Intelligence (K04), linked to source blobs.
  • Embeddings/vectors — in Azure AI Search (primary RAG) and/or Cosmos vectors (colocated memory).
  • Embed vs reference — denormalize (embed) data read together (a message with its citations); reference data that's large/shared. Model around access patterns, not entity-relationship normalization (this is NoSQL, not SQL).

You'll also use SQL (JD requirement) for relational/reporting data and T-SQL/Azure SQL where it fits; know joins, indexes, and how to query both worlds.

8. CI/CD and Git workflows

Git workflow (banking flavor): trunk-based or GitHub-Flow with short-lived feature branches → PR with required reviews (and code owners) → protected main. Conventional commits; no direct pushes to main; signed commits where required.

CI (on every PR):

  1. Lint + type-check (ruff, mypy).
  2. Unit/integration tests (mock Azure deps).
  3. Build the Docker image; scan it (Trivy/Defender).
  4. AI eval gate — run the groundedness/accuracy/safety eval set (K08); fail the build if quality regresses. This is the AI-specific CI step seniors call out — you gate model/prompt/retrieval changes on measured quality, not just code tests.
  5. IaC validation (Bicep what-if).

CD (on merge):

  1. Push image to ACR.
  2. Deploy to a staging environment; run smoke + eval tests.
  3. Progressive rollout to prod — ACA revisions with traffic splitting (canary: 5% → 50% → 100%) or blue/green; automated rollback on health/eval alarms.
  4. Infrastructure as Code (Bicep/Terraform) provisions everything reproducibly — no click-ops in a bank; the environment is in Git and reviewed.

Tooling: GitHub Actions or Azure DevOps Pipelines (both common in MS-shop banks); OIDC federated credentials so the pipeline authenticates to Azure without stored secrets.

9. Observability and reliability

You can't operate what you can't see — and a bank requires you to prove what happened:

  • Tracing — distributed traces across API → agent → tools → model (OpenTelemetry → Application Insights). Capture prompt, retrieval, tool calls, tokens, latency, cost per request. Foundry/Agent Service emit traces to Azure Monitor (K00, K08).
  • Metrics — latency (p50/p95/p99), throughput, error rate, token usage & $/request, 429 rate, RU consumption, cache hit-rate. Dashboards + alerts.
  • Logging & audit — structured logs; immutable audit of AI decisions for compliance; PII-aware logging (redact before logging — K08).
  • SLOs & HA — define latency/availability SLOs; multi-zone (min 2 replicas), multi-region for DR; health/readiness probes; autoscale on real load; PTUs for predictable model capacity (K00 §3).
  • Cost monitoring — token cost is a first-class metric; alert on spend anomalies; tier models and cache to control it.

10. Common misconceptions

  • "Flask is fine for the LLM API." Sync/WSGI blocks a worker per slow LLM call; FastAPI/async handles many concurrent waits and streams. Flask/Streamlit are for PoCs.
  • "Any partition key works." The partition key is the make-or-break Cosmos decision — high cardinality, even access, single-partition queries, ≤20GB/partition. You can't easily change it later.
  • "Cosmos throughput is just storage." It's RU/s; exceed it → 429. Model access patterns to minimize RU.
  • "Put secrets in env vars / the image." Use Managed Identity + Key Vault; OIDC in CI — no stored secrets.
  • "CI is just unit tests." For AI you add an eval gate that fails on quality regression, plus image scanning and IaC validation.
  • "Deploy straight to prod." Canary/blue-green with automated rollback; IaC, not click-ops.
  • "One big replica is HA." HA needs ≥2 replicas across zones, health probes, retries, and a DR region.

11. Interview Q&A

Q: Why FastAPI over Flask for this service? LLM calls are I/O-bound and slow; FastAPI's async serves many concurrent requests per worker while they await the model and streams tokens (SSE) for responsive chat. It also gives typed Pydantic contracts and auto OpenAPI. Flask/Streamlit are great for quick PoCs, not the production async API.

Q: How do you choose a Cosmos partition key for a chat app? Pick a high-cardinality key matching the dominant access pattern — usually conversationId (or userId) — so load spreads evenly, reads are single-partition point reads (cheap RU), and no partition is hot or exceeds 20GB. It's effectively irreversible, so I model access patterns first. RU/s is provisioned (autoscale) and I design queries to minimize RU.

Q: Walk me through your CI/CD for an AI service in a bank. Short-lived branches → PR with required reviews on protected main. CI: lint/type-check, tests (mocked Azure), Docker build + vulnerability scan, AI eval gate (groundedness/safety must not regress), Bicep what-if. CD on merge: push to ACR, deploy to staging, smoke+eval, then canary to prod via ACA revisions with traffic splitting and automated rollback. Everything via IaC; pipeline auths to Azure with OIDC — no stored secrets.

Q: How do you make the service highly available and resilient? ≥2 replicas across availability zones (ACA/AKS), health/readiness probes, autoscale on concurrency; timeouts, retries with backoff honoring Retry-After, circuit breakers, idempotency keys for mutating calls; graceful degradation to a smaller model/templated response; multi-region + PTUs for model capacity; DR runbook. Observability (App Insights traces/metrics, alerts) to detect and roll back fast.

Q: How do you stream responses and why? Return a StreamingResponse (SSE) that yields model deltas as they're generated, so the user sees tokens immediately — perceived latency is the time-to-first-token, not the full completion. Requires async clients end-to-end and careful error handling mid-stream.

Q: When ACA vs AKS? ACA by default — serverless, scale-to-zero/burst, KEDA autoscale, managed ingress, VNet-integratable: ideal for an AI microservice. AKS when the bank already runs Kubernetes, or I need GPU node pools/complex multi-service topology and full control, accepting more ops overhead.

12. References

  • FastAPI — official docs (async, dependencies, StreamingResponse); Starlette/ASGI; Uvicorn.
  • Pydantic — official docs (v2 validation, BaseSettings).
  • Docker — official docs (multi-stage builds, best practices); Azure Container Registry docs.
  • Azure Container Apps / AKS / ACI — Microsoft Learn (ACA scaling/revisions/KEDA; AKS; ACI).
  • Azure Cosmos DB — Microsoft Learn: Partitioning, Request Units, Consistency levels, Vector search (DiskANN), NoSQL query.
  • CI/CD — GitHub Actions / Azure DevOps docs; Bicep docs; OpenID Connect federation to Azure.
  • Observability — OpenTelemetry; Azure Monitor / Application Insights docs.
  • Reliability — Azure Architecture Center, Reliability/Well-Architected Framework; Nygard, Release It! (circuit breakers, bulkheads).