System Design — Reference Azure Architectures

Worked, principal-level designs that tie the whole track together. Each one names the control plane (identity, RBAC, policy, landing zone) and the data plane (network, compute, events, storage), then quantifies the SLA, the cost, and the tradeoff per Well-Architected pillar. Use them as templates; the capstone (P15) builds one.

The design method (every time)

  1. Workload facts first: RPS, payload, p99 latency SLO, availability target, data residency, compliance regime, team size.
  2. Two planes, always:
    • Control plane — which tenant/MG/subscription, RBAC model, policy baseline, IaC + pipeline, secrets strategy.
    • Data plane — VNet/subnets/NSG/Private Endpoints, compute, eventing, storage, egress.
  3. Quantify: compose the SLA (serial ∏Aᵢ; redundant 1−∏(1−Aᵢ)), estimate $/month, size instances/partitions, set the retry budget.
  4. Tradeoff per pillar: Reliability, Security, Cost, Operational Excellence, Performance — name what you trade and write the ADR sentence.
  5. Fail it: blast radius, throttling limits, DR/region-failover, rollback.

Design 1 — Secure event-driven serverless API (the canonical platform)

Requirement: a public REST API, 2k RPS, p99 < 300 ms, 99.95% target, PII at rest, EU data residency, deploy 20×/day.

Control plane

  • Landing zone: workload subscription under the Corp MG; policy baseline denies public storage/SQL and non-EU regions; RBAC via Entra groups, least privilege.
  • IaC: Terraform (azurerm) in Git; CI deploys via OIDC federation (no secret); what-if/plan gate + manual approval to prod.

Data plane

  • Edge: Front Door (global, WAF) → APIM (JWT validation: aud/iss/scp, rate-limit/quota) → Functions (Flex Consumption, VNet-integrated).
  • Async: API drops work on Service Bus; a Function consumes peek-lock with DLQ; an Event Grid topic fans out domain events to subscribers.
  • Data: Cosmos DB (EU region, zone-redundant) via Private Endpoint; secrets/keys in Key Vault (RBAC, purge protection); compute uses Managed Identity (no secrets).
  • Observability: App Insights distributed tracing correlated by operation_Id; Log Analytics + KQL alerts on p95 latency and DLQ depth.

Quantify

  • SLA: Front Door (99.99) × APIM (99.95) × Functions (99.95) × Cosmos (99.99 zone-redundant) ≈ 99.88% serial — below target ⇒ add regional redundancy on the weakest link or relax the SLO; show the math, then decide.
  • Retry: Service Bus consumer backoff + jitter, MaxDelivery 5 → DLQ; client honors 429.

Tradeoffs

  • Cost vs latency: Flex/Premium kills cold starts but costs more than Consumption — justify by the p99 SLO.
  • Security vs simplicity: Private Endpoints + no public PaaS adds DNS complexity; worth it for PII.

Design 2 — Multi-subscription landing zone & governance (P05)

Requirement: onboard 50 teams safely; enforce guardrails; central networking/logging.

  • MG hierarchy: Root → Platform (Identity/Management/Connectivity) + Landing Zones (Corp/ Online) + Sandbox + Decommissioned.
  • Policy-as-code initiative at each MG: deny disallowed regions, require diagnostic settings, enforce HTTPS/TLS, audit untagged resources; exemptions time-boxed in Git.
  • Subscription vending: pipeline creates a subscription in the right MG with budgets, RBAC groups, hub-spoke peering, and a Private DNS link pre-wired.
  • Hub-and-spoke: shared Azure Firewall + DNS in the hub; spokes route 0.0.0.0/0 via UDR to the firewall; central Log Analytics.

Tradeoff: central control vs team velocity — guardrails as deny for the non-negotiables, audit for the rest, self-service inside the guardrails.

Design 3 — Zero-downtime deploy with rollback (P08, P14)

  • Blue-green slots behind APIM/Front Door; deploy green, smoke-test, swap; health gate on error-rate/latency triggers auto-rollback (swap back).
  • Canary alternative: shift 5% → 25% → 100% with KQL health gates between steps; abort on regression.
  • DB changes: expand-migrate-contract (backward-compatible schema) so a rollback is safe.

Tradeoff: blue-green = instant rollback, double the resources during deploy; canary = smaller blast radius, slower and needs solid health signals.

Design 4 — Disaster recovery & multi-region (P14)

  • Active-passive across a region pair: primary serves; secondary warm with geo-replicated Cosmos/storage and IaC-reproducible infra; Front Door fails over on health probe.
  • RPO/RTO drive the spend: synchronous replication (low RPO, high cost) vs async (cheaper, data-loss window). State the numbers.
  • Test the failover (game day) — an untested DR plan is a hope, not a control.

Anti-patterns the panel listens for

  • Designing only the data plane and forgetting identity/RBAC/policy/landing zone.
  • "We'll use Kubernetes" with no workload numbers to justify it over Container Apps/Functions.
  • Public PaaS endpoints for PII data; secrets in app settings instead of Key Vault + MI.
  • Claiming "exactly-once" delivery; quoting an SLA without composing the dependency chain.
  • No rollback story, no DR test, no cost estimate.