« All Roles

Principal Azure Cloud Engineer & Architect — Curriculum

Target Role: Azure Cloud Engineer → Principal Cloud Architect — secure, event-driven, serverless platforms on Microsoft Azure (the reference JD — jd.md)

Also prepares you for:

• Microsoft / consulting partners — Cloud Solution Architect, Azure Platform Engineer
• Any enterprise running a regulated, multi-subscription Azure estate: landing zones, Entra ID, policy-as-code, private networking, event-driven serverless, and 24×7 ops
• The Azure certification ladder used as a floor, not a ceiling: AZ-104 (Administrator), AZ-204 (Developer), AZ-305 (Solutions Architect Expert), AZ-500 (Security), AZ-700 (Networking) — this track goes far deeper than any of them, because it makes you build the mechanisms the exams only name.

Duration: ~32 weeks core (≈8 months) — extendable with the capstones Seniority Target: Principal / Staff IC — the escalation point for the hardest cloud failures and the person who defines the platform standards the org builds on Goal: Make you the engineer who can architect an Azure platform at the system level and debug it at the token, route-table, manifest-digest, message-lock, checkpoint, and KQL-query level — across IaC, identity, governance, networking, containers, CI/CD, APIs, events, serverless, secrets, observability, and reliability.

Why This Curriculum Exists

Most "learn Azure" material teaches you to operate the portal: which blade to click, which az command to copy. That gets you to senior. It does not get you to principal, because the principal's job is to make correct tradeoffs under conflicting requirements and to debug the layer everyone else stands on — and you cannot do either if the service is a black box.

So this track is built on two non-negotiables:

1. Every Azure mechanism in the JD gets implemented, not just described. The ARM deployment dependency graph, Terraform's plan/diff/replace engine, the OAuth2 + OIDC + JWT validation flows, the RBAC + Azure Policy evaluator, the NSG + effective- routes engine, an OCI/ACR registry with garbage collection, a CI/CD DAG with approval gates and OIDC federation, an API gateway with scope authorization and rate limiting, a Service Bus broker with peek-lock/sessions/dead-letter, the Functions scale controller and Durable replay engine, Key Vault envelope encryption and Managed- Identity token acquisition, a KQL query engine, and the reliability primitives (SLA composition, retry+jitter, circuit breaker) — you build a working, tested miniature of each.
2. Everything is taught to the depth interviewers and incidents actually demand — the token claims, the priority ordering, the longest-prefix match, the lock-expiry off-by-one, the SLA arithmetic, the failure signatures. We assume you are building the platform that serves millions of requests with security and availability guarantees, not a demo resource group.

How Each Phase Teaches (the "many ways of explaining")

Every phase deliberately explains the same material through several lenses, because principal-level understanding is the intersection of all of them:

The lab engineering standard is documented in LAB-STANDARD.md. Read it before you start the first lab.

The Phases

Use the sidebar for the full clickable table of contents, and see GLOSSARY.md / CHEATSHEET.md for the running references.

How to Use This Track

1. Start here, then read Phase 00 end to end — it builds the mental model (control plane, Well-Architected pillars, the tradeoff dials) that every later phase plugs into.
2. For each phase: read WARMUP.md (professor), skim HITCHHIKERS-GUIDE.md (the numbers), do the lab (lab.py red → green against test_lab.py), then read BROTHER-TALK.md.
3. Work phases roughly in order — identity (P03–P05) and networking (P06) underpin everything after them.
4. Use interview-prep/ and system-design/ as running references; the capstone (P15) ties the whole stack together.

Cross-Cutting Themes (the principal's lenses)

• Control plane vs data plane — ARM/RBAC/Policy decide what exists and who may touch it; the blob, message, secret, and token are a separate system with separate identity, throttling, and SLA.
• Identity is the perimeter — every lab that touches auth defaults to deny, validates the token's claims, and prefers a Managed Identity over a secret.
• Idempotency everywhere — ARM PUT, Terraform apply, Service Bus dedup, Event Grid retry, Durable replay — the same operation twice must converge, not duplicate.
• Quantify the tradeoff — SLA composition, retry budgets, partition/shard sizing, and the monthly bill are arithmetic you do before the design review, not after the incident.

Azure Cloud Engineer / Principal Cloud Architect

Secure, Event-Driven, Serverless Platforms on Microsoft Azure — IaC, Identity, Networking, and Operations

Role Summary

We are looking for an engineer who can design, build, operate, and evolve secure, scalable, and reliable solutions on Microsoft Azure — and who understands the platform deeply enough to be the escalation point when it breaks at 2 a.m. and the architect when the org decides what to build next.

This is not a "click in the portal" role and not a "run one Terraform module" role. It is about owning the cloud platform layer itself: the Infrastructure-as-Code that provisions everything, the identity fabric (Microsoft Entra ID) that authenticates and authorizes every call, the landing zones and governance that hundreds of engineers deploy into, the networking substrate that isolates and connects workloads, the event-driven and serverless application patterns the business runs on, and the observability and operational playbooks that keep it all available, secure, and fast.

The reference responsibilities (verbatim from the requisition):

1. Provide troubleshooting and operational support for Azure cloud services and infrastructure.
2. Design, implement, and maintain Infrastructure as Code (IaC) solutions using Terraform, HashiCorp CDKTF, and ARM Templates.
3. Configure and manage Azure Landing Zones, Microsoft Entra ID (Azure AD), Azure Policies, RBAC, networking, and identity integrations.
4. Build, deploy, and manage containerized applications using Docker and Azure Container Registry (ACR).
5. Develop and maintain CI/CD pipelines to automate application builds, testing, and deployments.
6. Develop and integrate RESTful APIs using JSON and implement secure authentication and authorization mechanisms (OAuth2, OpenID Connect).
7. Design and implement event-driven architectures using Azure Event Grid, Azure Service Bus, and Azure Logic Apps.
8. Integrate serverless applications with Azure security services, including Azure Key Vault, Managed Identities, and Microsoft Entra ID.
9. Monitor, troubleshoot, and optimize cloud infrastructure to ensure high availability, security, and performance.
10. Collaborate with cross-functional teams to deliver secure, scalable, and reliable cloud solutions.

What "Principal-Level" Means Here

A senior engineer uses these services correctly. A principal understands them under the hood — well enough to predict failure, defend a design with numbers, and debug across every layer of the stack:

• Control plane vs data plane. You know that every Azure write goes through Azure Resource Manager (ARM) as an idempotent PUT against a resource provider, that the deployment engine builds a dependency graph from dependsOn and implicit references, and that the data plane (the blob you read, the message you receive, the secret you fetch) is a different system with different identity, throttling, and SLA.
• Identity is the new perimeter. You can draw the OAuth2 authorization-code+PKCE flow and the client-credentials flow from memory, validate a JWT's iss/aud/exp/scp claims by hand, explain how a Managed Identity acquires a token from IMDS with no secret, and reason about RBAC evaluation (role assignments, NotActions, deny assignments, scope inheritance) and Azure Policy effects as code.
• Networking is software-defined. You can evaluate an NSG rule set in priority order, compute effective routes with longest-prefix matching across system/UDR/BGP routes, and explain why a Private Endpoint changes DNS resolution and how that breaks (and fixes) connectivity.
• Events and serverless are distributed systems. You know Service Bus peek-lock, sessions, duplicate detection, and dead-lettering; Event Grid's at-least-once delivery with exponential-backoff retry; the Functions scale controller; and Durable Functions' event-sourced, replay-based determinism.
• Reliability and cost are quantitative. You compose SLAs (serial vs redundant), size retries with backoff + jitter and a retry budget, run a circuit breaker, honor 429 / Retry-After, and put numbers on availability zones, paired regions, and the monthly bill before anyone asks.

The Stack You Will Master

ARM / Bicep / Azure Resource Manager   Terraform (AzureRM provider) / CDKTF
Microsoft Entra ID (Azure AD)          OAuth2 / OIDC / JWT / Conditional Access
RBAC / Azure Policy / Management Groups Cloud Adoption Framework / Landing Zones
VNet / NSG / UDR / Private Link / DNS   Azure Firewall / ExpressRoute / Front Door
Docker / ACR / AKS / Container Apps     Azure DevOps / GitHub Actions / OIDC federation
API Management / REST / JSON            Key Vault / Managed Identities / IMDS
Event Grid / Service Bus / Logic Apps   Azure Functions / Durable Functions
Azure Monitor / Log Analytics / KQL     Application Insights / distributed tracing
Availability Zones / paired regions     Well-Architected Framework / FinOps

What Success Looks Like

You are the person who can architect an Azure platform at the system level and debug it at the token, route table, manifest digest, message lock, checkpoint, and KQL-query level — and who leaves behind landing zones, IaC, policy-as-code, and runbooks that let hundreds of other engineers ship securely without reinventing the platform.

Everything in this curriculum is built, not just described: you implement working, test-verified miniatures of ARM's deployment graph, Terraform's plan engine, the OAuth2 token flows, the RBAC/Policy evaluator, the NSG/route engine, an OCI registry, a CI/CD DAG, an API gateway, a Service Bus broker, the Functions runtime, Key Vault envelope encryption, a KQL engine, and the reliability primitives — so the mechanisms become muscle memory instead of folklore.

Azure Lab Engineering Standard

Every lab in this track builds a runnable, test-verified miniature of a real Azure mechanism — the deployment graph, the token flow, the RBAC evaluator, the NSG engine, the message broker — so the internals become muscle memory instead of portal folklore. You do not "deploy a VNet"; you build the engine that decides whether a packet is allowed. That is what makes the knowledge defensible in a principal interview and useful at 2 a.m.

Why miniatures (and not live Azure)

A live Azure subscription is non-deterministic, costs money, requires credentials, and hides the mechanism behind a REST API. A lab that reimplements the algorithm — NSG priority evaluation, longest-prefix routing, JWT validation, peek-lock redelivery, Durable replay — is offline, deterministic, free, and teaches the part interviewers actually probe. Each lab README ends with a "How this maps to real Azure" section that ties the miniature back to the production service, its limits, and its az/Terraform equivalent.

Required files (per lab)

The runnable core is Python (stdlib + pytest), offline, deterministic. No network, no Azure SDK, no credentials, no Date.now()/random without a seed. Multi-language or real-az/Terraform work appears only as build specs in the README "Extensions" section for the learner's own subscription.

The test contract

import importlib, os
lab = importlib.import_module(os.environ.get("LAB_MODULE", "lab"))

Run both ways; both must pass:

pytest test_lab.py -v                       # against your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # against the reference

Test taxonomy every flagship lab includes: happy path; malformed / out-of-range input (raises ValueError); deny-by-default / least-privilege behavior where identity or policy is involved; boundary cases (the off-by-one that interviewers probe — R+W>N strictness, NSG priority ties, lock-expiry exactly at the boundary, longest-prefix /32 vs /0); idempotency / replay / duplicate where the service guarantees it; and deterministic output (same input → same bytes).

The four teaching docs (per phase)

Every phase explains the same material through four lenses, because principal-level understanding is their intersection:

Doc conventions (mdBook-compatible)

• WARMUP.md opens with a Table of Contents of working anchor links, kept in sync with the headings.
• MathJax for any math (\( … \) inline, $$ … $$ block). Standard Markdown only.
• No bare angle brackets in prose — wrap <like-this> in backticks so mdBook does not eat them as HTML.
• File references use relative links. Code fences are language-tagged.

Definition of done

A lab is complete only when: the reference suite passes (LAB_MODULE=solution), the learner lab.py passes after the TODOs are filled in, python solution.py prints a sensible worked example, the README's success criteria are testable, and the "How this maps to real Azure" section is accurate about the production service and its limits.

Glossary — Every Azure Term in the JD, Defined Precisely

The JD names ~140 Azure technologies and concepts. This glossary defines each to the depth an interviewer probes, and points to the phase where you build it. If a term in the JD is not here, that is a bug — open it as a gap.

Legend: → PNN = the phase that implements/teaches it.

A. Control Plane & Resource Management

• Azure Resource Manager (ARM) — the single control-plane API and deployment engine in front of every Azure write. A request is an idempotent PUT/PATCH/DELETE to a URI like /subscriptions/{s}/resourceGroups/{rg}/providers/{ns}/{type}/{name}?api-version=…. ARM authenticates (Entra token), authorizes (RBAC + deny assignments), validates against Policy, then routes to the resource provider. → P00, P01
• Resource provider (RP) — a service (e.g. Microsoft.Storage, Microsoft.Compute, Microsoft.Network) that owns a set of resource types and implements their control- plane CRUD. Must be registered on a subscription before use. → P01
• Resource ID — the canonical, hierarchical identifier: /subscriptions/{sub}/resourceGroups/{rg}/providers/{namespace}/{type}/{name}. The unit of RBAC scope, Policy scope, tagging, and locks. → P00
• Resource group (RG) — a lifecycle and scope boundary; resources in one RG share a region for metadata (not necessarily for the resource) and are deleted together. → P01
• Management group (MG) — a container above subscriptions forming a tree (root → MGs → subscriptions → RGs → resources). Policy and RBAC assigned at an MG flow down by inheritance. Max depth 6 levels (excluding root/subscription). → P05
• ARM template — a declarative JSON document ($schema, parameters, variables, resources, outputs) describing desired state. The deployment engine resolves dependsOn + implicit reference()/resourceId() dependencies into a DAG and deploys in topological order. → P01
• Bicep — a typed DSL that transpiles to ARM JSON; same engine, nicer syntax, implicit dependency detection via symbolic references. → P01
• Deployment mode — Incremental (default; leaves un-templated resources) vs Complete (deletes anything in the RG not in the template — a foot-gun). → P01
• What-if — ARM's dry-run that diffs desired vs current state and classifies each change (Create/Modify/Delete/NoChange/Ignore). → P01
• Deployment stack — a resource that manages a set of resources as a unit with managed deletion and deny-settings (drift protection). → P01, P05
• Idempotency — applying the same desired state twice yields the same result and no duplicate side effects. ARM PUT is idempotent by design; the whole IaC model depends on it. → P01, P02

B. Infrastructure as Code

• Terraform — declarative IaC with a state file (the recorded reality), a plan (desired vs state diff → create/update/replace/delete), and apply (executes the plan in dependency order). The azurerm provider maps HCL resources to ARM calls. → P02
• State — Terraform's source of truth mapping config addresses to real resource IDs + attributes. Stored remotely (Azure Storage backend with blob lease locking) to allow team collaboration. Drift = real ≠ state. → P02
• Plan graph / dependency ordering — Terraform builds a DAG from references and depends_on, walks it to order create/destroy; destroys run in reverse. → P02
• ForceNew / replace — a change to an immutable attribute requires destroy-then-create (-/+), not in-place update. Recognizing which attributes are ForceNew is principal- level Terraform. → P02
• CDKTF (Cloud Development Kit for Terraform) — author Terraform in TypeScript/Python/ etc.; synthesizes to Terraform JSON, then uses the normal plan/apply engine. → P02
• Drift detection — comparing real state to recorded state to find out-of-band changes (terraform plan with refresh, or a deployment stack's deny-settings). → P02
• Module / provider / backend — reusable config unit; the plugin that talks to a target API; where state lives. → P02

C. Identity — Microsoft Entra ID (Azure AD)

• Microsoft Entra ID — Azure's cloud identity provider (formerly Azure AD): tenants, users, groups, service principals, app registrations, and the OAuth2/OIDC token service. The control-plane authentication authority. → P03
• Tenant — an isolated Entra directory instance; identified by a tenant ID (GUID). The token issuer (iss) is scoped to a tenant. → P03
• App registration vs service principal — the registration is the global app definition (in its home tenant); the service principal is the local identity/credential instance in a tenant that gets role assignments. → P03
• OAuth2 — the authorization framework (RFC 6749). Grants used in Azure: authorization code (+ PKCE) for users, client credentials for daemons, on-behalf-of for middle tiers, device code for input-constrained devices. Issues access tokens. → P03
• OpenID Connect (OIDC) — an identity layer on top of OAuth2 that adds the ID token (a JWT proving who the user is) and a discovery document (/.well-known/openid-configuration). → P03
• PKCE (Proof Key for Code Exchange) — RFC 7636: the public client sends code_challenge = BASE64URL(SHA256(code_verifier)); at token exchange it presents the code_verifier. Defeats authorization-code interception. → P03
• JWT (JSON Web Token) — base64url(header).base64url(payload).base64url(signature). Validation = verify signature against the issuer's JWKS key (by kid), then check iss, aud, exp, nbf, and authorization claims (scp for delegated scopes, roles for app roles). → P03, P09
• Claims — iss (issuer), aud (audience = your API's app ID URI), sub (subject), exp/nbf/iat (validity window), scp (space-delimited delegated scopes), roles (app roles), appid/azp (client), oid/tid (object/tenant). → P03
• JWKS / kid — the issuer's published JSON Web Key Set; the token header's kid selects which public key validates the signature, enabling key rotation. → P03
• Conditional Access — policy engine that gates token issuance on signals (user, device compliance, location, risk) → grant/block/require MFA. → P03
• Managed Identity — an Entra service principal that Azure manages for you so code holds no secret. System-assigned (tied to one resource's lifecycle) or user-assigned (standalone, attachable to many). Tokens are fetched from IMDS. → P12
• IMDS (Instance Metadata Service) — the non-routable 169.254.169.254 endpoint a VM/ Function calls to get a Managed-Identity access token, no credential required. → P12
• Workload identity federation — exchanging a trusted external OIDC token (e.g. GitHub Actions, a Kubernetes service account) for an Entra token with no stored secret, by matching the federated credential's issuer/subject/audience. → P08, P12

D. Authorization — RBAC & Policy

• Azure RBAC — authorization model: a role assignment binds a security principal (user/group/SP/MI) to a role definition at a scope. Allow-based; assignments are additive and inherited down the scope hierarchy. → P04
• Role definition — a named set of permissions: Actions, NotActions (subtracted), DataActions, NotDataActions, and AssignableScopes. Effective control-plane permission = Actions − NotActions, matched with wildcards (*, Microsoft.Storage/*/read). → P04
• Deny assignment — an explicit deny that overrides role assignments (used by Azure Blueprints/managed apps/deployment stacks). Evaluation: deny wins. → P04
• Control plane vs data plane authorization — Actions govern managing the resource (ARM); DataActions govern the data inside it (e.g. read a blob, get a secret). A storage Owner is not automatically a Blob Data Reader. → P04, P12
• Azure Policy — governance engine evaluating resources against rules. Effects: audit, deny, append, modify, auditIfNotExists, deployIfNotExists (DINE), disabled. Evaluated at create/update (control plane) and on a compliance scan. → P04
• Policy alias — maps a policy field to a provider property path (e.g. Microsoft.Storage/storageAccounts/supportsHttpsTrafficOnly) so rules can read resource properties. → P04
• Initiative (policy set) — a bundle of policy definitions assigned together (e.g. a regulatory baseline). → P04, P05
• Exemption / exclusion — a documented, time-boxed carve-out from a policy/initiative at a scope (notScopes). → P05

E. Governance at Scale — Landing Zones

• Cloud Adoption Framework (CAF) — Microsoft's guidance for governed cloud adoption; the Azure Landing Zone (ALZ) is its reference architecture. → P05
• Azure Landing Zone (ALZ) — a pre-built, governed environment: a management-group hierarchy (Platform / Landing Zones / Sandbox / Decommissioned), policy-as-code baselines, RBAC, hub networking, logging, and a subscription-vending process. → P05
• Subscription vending — the automated, policy-compliant allocation of a new subscription into the right MG with budgets, RBAC, and networking pre-wired. → P05
• Policy-as-code — policy/initiative definitions and assignments managed in Git and deployed via pipelines, so governance is reviewed and versioned. → P05

F. Networking

• Virtual network (VNet) — an isolated L3 network in a region with a private address space (CIDR), subdivided into subnets. Software-defined: there is no physical switch you can see. → P06
• Network security group (NSG) — a stateful, priority-ordered allow/deny firewall on a subnet or NIC. Rules (100–4096) are evaluated low-number-first; first match wins; default rules allow VNet-in/out and Azure LB and deny the rest. Stateful: the return packet is auto-allowed. → P06
• User-defined route (UDR) / route table — overrides Azure's system routes. Next-hop chosen by longest-prefix match; ties break UDR > BGP > system. Used to force traffic through an NVA/firewall. → P06
• System routes / BGP — Azure's default routes (VNet-local, internet, etc.) and dynamically learned routes (via ExpressRoute/VPN). → P06
• Private Endpoint / Private Link — a private IP in your subnet that maps to a PaaS resource (storage, SQL, Key Vault), pulling it onto your VNet; it rewrites DNS so the public FQDN resolves to the private IP via a Private DNS zone. Misconfigured DNS is the #1 Private Endpoint failure. → P06, P12
• Service endpoint — an alternative that extends a VNet identity to a PaaS service over the Azure backbone (no private IP). → P06
• Azure Firewall / NVA — a managed (or third-party) stateful firewall used as the hub's egress/inspection point, targeted by UDRs. → P06
• Hub-and-spoke — a topology with shared services (firewall, DNS, gateways) in a hub VNet peered to many spoke workload VNets. → P06
• VNet peering / ExpressRoute / VPN Gateway — VNet-to-VNet private connectivity; private dedicated circuit to on-prem; encrypted tunnel to on-prem. → P06
• Azure Front Door / Application Gateway / Load Balancer — global L7 CDN+WAF; regional L7 WAF/router; regional L4 distributor. → P06, P09

G. Containers

• Docker image — an ordered stack of read-only layers + a config; identified by a content-addressed digest (sha256:…). A manifest lists the config + layer digests; a tag is a mutable pointer to a digest. → P07
• OCI distribution — the registry API: blobs (layers/config) are content-addressable; pulling resolves tag → manifest → layer digests, fetching only missing layers. → P07
• Azure Container Registry (ACR) — managed OCI registry with geo-replication, ACR Tasks (build/patch on push), content trust (Notary signing), soft-delete, and garbage collection of untagged manifests. → P07
• AKS (Azure Kubernetes Service) — managed Kubernetes: control plane (managed) + node pools; the scheduler bin-packs pods onto nodes by requests, respecting limits, taints/tolerations, affinity, and topology. → P07
• Azure Container Apps (ACA) / Container Instances (ACI) — serverless containers (KEDA-driven scale-to-zero) and single-shot container groups; the non-Kubernetes paths. → P07

H. CI/CD & Supply Chain

• Azure Pipelines / GitHub Actions — CI/CD as a DAG of stages → jobs → steps with dependsOn, conditions, matrix fan-out, environments, and approval gates. → P08
• OIDC deployment federation — the pipeline presents its OIDC token to Entra and gets an access token via workload-identity federation — no stored cloud secret. The modern default for CI deploys. → P08, P12
• Deployment strategy — blue-green (swap two environments), canary (shift a % then ramp), rolling — with a health gate that triggers automatic rollback. → P08, P14
• Artifact / SBOM / provenance — the built unit, its bill of materials, and signed evidence of how/where it was built (supply-chain integrity). → P08

I. APIs & Edge

• REST / JSON — resource-oriented HTTP: methods (GET/POST/PUT/PATCH/DELETE), status codes, idempotency, pagination, and JSON bodies. → P09
• Azure API Management (APIM) — the API gateway: products/APIs/operations, a policy pipeline (inbound → backend → outbound → on-error) for JWT validation, rate-limit/quota, caching, transformation, and routing. → P09
• Scope / app-role authorization — the API validates the token's scp (delegated) or roles (app) claim against the operation's requirement before letting the call through. → P09
• Token-bucket rate limiting — capacity + refill-rate algorithm bounding request rate; the basis of rate-limit and quota policies; returns 429 Too Many Requests + Retry-After. → P09, P14

J. Eventing & Messaging

• Azure Service Bus — enterprise message broker: queues + topics/subscriptions, peek-lock (lock → process → complete/abandon/dead-letter), sessions (FIFO per session key), duplicate detection (dedup window by MessageId), scheduled and deferred messages, transactions, and dead-letter queues (DLQ) on max-delivery or TTL expiry. At-least-once by default. → P10
• Azure Event Grid — a push event-routing service (publish/subscribe) using CloudEvents; at-least-once delivery with exponential-backoff retry and dead-lettering to storage; filters by event type/subject. → P10
• Event Hubs — high-throughput event streaming (partitioned log, consumer offsets); the Kafka-shaped sibling (named here for contrast). → P10
• Logic Apps — a serverless workflow orchestrator (connectors + triggers + actions) for integration; the low-code complement to Functions. → P10
• Delivery semantics — at-most-once (fire&forget), at-least-once (retry → may duplicate; the practical default), exactly-once-effect (at-least-once + idempotent consumer / dedup). → P10

K. Serverless Compute

• Azure Functions — event-driven serverless compute: a trigger (queue, HTTP, timer, Event Grid) invokes a function; bindings declaratively wire inputs/outputs. The scale controller adds/removes instances based on event source backlog. → P11
• Hosting plans — Consumption (scale-to-zero, pay-per-exec), Flex Consumption / Premium (pre-warmed, VNet), Dedicated. Cold start is the Consumption tradeoff. → P11
• Durable Functions — stateful orchestration on Functions via event sourcing: the orchestrator function is replayed from its history on each event, so it must be deterministic (no DateTime.Now, no random, no I/O outside activities). Patterns: function chaining, fan-out/fan-in, async HTTP, monitor, human interaction. → P11

L. Secrets & Encryption

• Azure Key Vault — managed store for secrets, keys, and certificates, with soft-delete + purge protection (recoverable deletes), versioned objects, and two authorization models: RBAC (recommended) or legacy access policies. → P12
• Envelope encryption — encrypt data with a DEK (data encryption key), then encrypt (wrap) the DEK with a KEK (key encryption key) held in Key Vault/HSM; rotate the KEK without re-encrypting data. → P12
• Managed HSM / key rotation / versioning — FIPS 140-2 L3 single-tenant HSM; automated key rotation creating new versions while old versions still decrypt. → P12

M. Observability & Operations

• Azure Monitor — the umbrella: Metrics (numeric time series), Logs (Log Analytics workspace), Alerts, and Application Insights (APM + distributed tracing). → P13
• KQL (Kusto Query Language) — the read query language over Log Analytics: where/project/summarize … by/bin()/join/top/render. The principal's incident tool. → P13
• Application Insights / distributed tracing — request/dependency telemetry correlated by operation_Id across services; sampling controls cost/cardinality. → P13
• Alert rule — threshold/aggregation over a metric or log query window → action group (page/email/webhook). → P13

N. Reliability, HA & Cost

• Availability zone (AZ) — physically separate datacenter(s) within a region; spreading replicas across zones survives a datacenter failure. → P14
• Region pair / paired region — two regions with platform-coordinated updates and (for some services) geo-replication for disaster recovery. → P14
• SLA composition — multiply availabilities of serial dependencies; combine redundant components as 1 − ∏(1 − Aᵢ). More serial dependencies → lower composite SLA. → P00, P14
• Retry with backoff + jitter / retry budget — bounded exponential backoff with randomized jitter to avoid thundering herds, capped by a budget so retries can't amplify an outage. → P14
• Circuit breaker — closed → (failures exceed threshold) → open (fail fast) → half-open (probe) → closed; protects a struggling dependency. → P14
• Throttling / 429 / Retry-After — Azure services rate-limit; well-behaved clients honor Retry-After and back off rather than hammering. → P14
• Well-Architected Framework (WAF) — the five pillars: Reliability, Security, Cost Optimization, Operational Excellence, Performance Efficiency — the rubric every design review scores against. → P00, P15
• FinOps — quantifying and optimizing cloud spend (reservations, right-sizing, autoscale, lifecycle tiers, scale-to-zero) as an engineering discipline. → P15

Cheat Sheet — The Numbers, Formulas & Decision Rules

One page of the things a principal Azure engineer recites without looking. Each links to the phase that derives it. Memorising the number is worthless without the why — but in an interview or an incident, having both is the difference between "sounds senior" and "is senior."

The control-plane request (P00–P01)

client → Entra (authenticate: JWT) → ARM (authorize: RBAC − NotActions, then deny wins)
       → Policy (deny/append/modify/audit) → resource provider (idempotent PUT)

Rule: every Azure write is one idempotent PUT to a resource ID. Control plane (manage the resource) and data plane (the blob/secret/message) are different systems with different identity, throttling, and SLA. Owner ≠ Blob Data Reader.

Resource ID shape:

/subscriptions/{sub}/resourceGroups/{rg}/providers/{namespace}/{type}/{name}

RBAC evaluation (P04)

effective control-plane perms = union(Actions) − union(NotActions)     # across all assignments
data-plane perms              = union(DataActions) − union(NotDataActions)
final = (matched by an allow) AND NOT (matched by any deny assignment)  # DENY ALWAYS WINS

Wildcards: * matches one+ segments; Microsoft.Storage/*/read matches any read under the RP. Assignments are additive and inherited down MG → Sub → RG → resource. There is no "deny by role"; you remove access by not assigning or via a deny assignment.

Azure Policy effects (P04), precedence high→low

disabled  <  audit/auditIfNotExists  <  append/modify  <  deployIfNotExists  <  deny

deny blocks the write at the control plane; audit only flags; deployIfNotExists (DINE) and modify need a managed identity with rights to remediate.

Management-group inheritance (P05)

effective at a scope = (assignments at this scope) ∪ (everything inherited from ancestors)
                       minus notScopes/exemptions

MG tree max depth 6 levels (under root). A subscription lives in exactly one MG.

OAuth2 / OIDC / JWT (P03, P09)

auth-code + PKCE:  challenge = BASE64URL(SHA256(verifier))   # method S256
client-credentials: client_id + secret/cert → access token (no user, app perms in `roles`)

JWT validation order (fail closed on any miss):

1. parse header → pick key by `kid` from JWKS
2. verify signature
3. iss == expected issuer (tenant)
4. aud == your API's app-ID-URI / client-id
5. nbf ≤ now ≤ exp           (apply ±clock-skew, default 300 s)
6. authorize: scp (delegated) or roles (app) contains required value

scp = space-delimited delegated scopes (user present). roles = app roles (daemon). Never trust a token you didn't validate aud on — that's the confused-deputy.

Networking (P06)

NSG: rules 100–4096, evaluated lowest priority number first, first match wins; defaults allow VirtualNetwork↔VirtualNetwork and AzureLB, deny the rest. Stateful — return traffic is auto-allowed (don't write a reverse rule).

Effective route next-hop:

choose route with LONGEST PREFIX match to dest IP
tie-break: UDR  >  BGP  >  system
0.0.0.0/0 is the default; a /32 UDR beats it

Subnet math: a /26 = 64 addresses, −5 reserved by Azure (.0 network, .1 gateway, .2/.3 Azure DNS-ish, .255 broadcast) → 59 usable. Smallest usable subnet is /29.

Private Endpoint: it rewrites DNS — the public FQDN must resolve to the private IP via a Private DNS zone (e.g. privatelink.blob.core.windows.net). Broken DNS = #1 failure.

Containers / OCI (P07)

tag  → manifest digest (sha256)  → [config digest, layer digest, layer digest, …]
pull = resolve tag → fetch manifest → fetch only MISSING layer blobs (content-addressed)

Garbage collection = mark all blobs referenced by a live manifest, sweep the rest. Tags are mutable; digests are immutable — pin prod by digest, not :latest.

CI/CD (P08)

Pipeline = DAG of stages (dependsOn + condition). A stage runs iff all deps succeeded and its condition is true. OIDC federation beats stored secrets: pipeline OIDC token → Entra (match issuer/subject/audience) → access token, nothing to leak. Rollout: blue-green (instant swap/rollback), canary (% then ramp, gate on health).

Messaging / eventing (P10)

Service Bus peek-lock:

receive → message LOCKED for lockDuration → complete | abandon | dead-letter
lock expires → message redelivered, DeliveryCount++
DeliveryCount > MaxDeliveryCount → moved to DLQ

Dedup: within DuplicateDetectionWindow, a repeat MessageId is dropped. Sessions = FIFO + single-consumer per session key. Event Grid: at-least-once, exponential-backoff retry (to ~24 h), then dead-letter to storage. Delivery semantics: at-most (no retry) → at-least (retry, may dup) → exactly-once effect (at-least + idempotent/dedup consumer).

Serverless (P11)

Functions scale controller adds instances from event backlog (queue depth / lag), up to the plan max; Consumption scales to zero (cold start is the cost). Durable orchestrators are replayed from history every event → must be deterministic: no clock, no random, no I/O outside activities; use context APIs only.

Key Vault & Managed Identity (P12)

Managed Identity token:  code → IMDS 169.254.169.254 → access token (NO secret)
Envelope encryption:     data --DEK--> ciphertext ;  DEK --KEK(in Vault)--> wrapped DEK
rotate KEK ⇒ rewrap DEK only (data untouched)

Soft-delete + purge-protection: a deleted secret/key is recoverable for the retention window and cannot be purged early when purge-protection is on. Prefer RBAC over legacy access policies.

Observability / KQL (P13)

T | where Timestamp > ago(1h)
  | summarize p95 = percentile(DurationMs, 95), n = count() by bin(Timestamp, 5m), Service
  | top 10 by p95 desc

summarize … by groups; bin(t, 5m) buckets time; sampling trades cost vs fidelity.

Reliability & SLA math (P00, P14)

serial dependencies:   A_total = ∏ Aᵢ            (more hops → lower SLA)
redundant copies:      A = 1 − ∏ (1 − Aᵢ)        (parallel raises SLA)

Three serial 99.9% services → 99.9%³ ≈ 99.7% (≈2× the downtime). Two redundant 99% → 1 − 0.01² = 99.99%.

Retry: exponential backoff base · 2ⁿ + full jitter random(0, backoff), capped, under a retry budget (e.g. ≤10% of calls) so retries can't amplify an outage. Honor 429 + Retry-After. Circuit breaker: closed → open (fail fast) → half-open (probe) → closed.

The five Well-Architected pillars (P00, P15)

Reliability · Security · Cost Optimization · Operational Excellence · Performance Efficiency

Rule: name the pillar you are trading, quantify both ends, then choose — and write the ADR. "Best architecture" is meaningless without the workload's weights.

Interview Prep — Principal Azure Engineer / Architect

A running reference for the questions a strong panel actually asks — and the level of answer that separates senior from principal. Pair each section with the phase that builds the mechanism, so your answer is "I implemented this," not "I read about it."

How principal interviews differ

A senior candidate explains what a service does. A principal explains how it works under the hood, when it breaks, what it costs, and what they'd trade. Every answer below should end with a tradeoff and, where possible, a number.

1. Control plane & IaC (P00–P02)

• "Walk me through what happens when you run terraform apply against Azure." — HCL → plan (state vs desired diff → create/update/replace/delete; ForceNew ⇒ replace), DAG ordering, each resource = an ARM PUT (idempotent) → RP. State lock via blob lease. Drift = real ≠ state.
• "ARM template vs Bicep vs Terraform vs CDKTF — when each?" — Same engine underneath; choose by team (HCL multi-cloud → Terraform; Azure-only typed → Bicep; programmatic synth → CDKTF). Defend with blast radius, state ownership, and drift story, not taste.
• "Incremental vs Complete deployment mode — what's the foot-gun?" — Complete deletes un-templated resources in the RG. Name a time it would have wiped something.
• "How do you make a deployment idempotent and safe to re-run?" — PUT desired state, no imperative side effects, what-if/plan first, deployment stacks with deny-settings.

2. Identity, OAuth2 & JWT (P03)

• "Draw the authorization-code + PKCE flow." — verifier/challenge (S256), redirect with code, back-channel token exchange with verifier, ID token (OIDC) + access token.
• "How do you validate a JWT? In what order, and what fails closed?" — kid → JWKS key → signature → iss → aud → nbf/exp (±skew) → scp/roles. Skipping aud = the confused-deputy.
• "Client credentials vs on-behalf-of vs managed identity — when?" — daemon vs middle- tier delegation vs secret-less Azure workload.
• "scp vs roles?" — delegated (user present) vs application (daemon) authorization.

3. Authorization — RBAC & Policy (P04–P05)

• "A user is Owner on the subscription but can't read a blob. Why?" — control-plane vs data-plane: Owner grants Actions, not DataActions; needs Storage Blob Data Reader.
• "How does deny win?" — deny assignments override role assignments; walk the evaluation.
• "How would you enforce 'no public storage accounts' across 200 subscriptions?" — Azure Policy deny at a management group, policy-as-code in Git, exemptions time-boxed.
• "Effective permissions at a resource?" — union of inherited assignments minus NotActions, minus deny.

4. Networking (P06)

• "A VM can't reach a storage account over its Private Endpoint. Diagnose." — DNS: the FQDN must resolve to the private IP via the Private DNS zone; check NSG, then UDR next-hop, then the zone link. (Almost always DNS.)
• "Two NSG rules match a packet — which wins?" — lowest priority number; first match; stateful return.
• "Force all egress through a firewall — how?" — UDR 0.0.0.0/0 → NVA next-hop; mind longest-prefix and asymmetric routing.
• "Size a VNet for 3 subnets of ~500 hosts each." — /23 per subnet (512 − 5), parent /21; show the −5 reserved.

5. Containers & CI/CD (P07–P08)

• "Why pin by digest, not :latest?" — tags are mutable; digests are immutable + content-addressed; reproducibility + supply-chain.
• "How does a registry save space across images?" — shared content-addressed layers; GC marks live-manifest blobs and sweeps the rest.
• "Deploy from CI without storing a cloud secret." — OIDC workload-identity federation (issuer/subject/audience match) → short-lived token.
• "Blue-green vs canary — pick one and defend." — instant rollback vs gradual exposure; tie to blast radius and the health gate.

6. Events & serverless (P10–P11)

• "Exactly-once with Service Bus?" — there is no exactly-once delivery; you get at-least-once + an idempotent/dedup consumer = exactly-once effect. Peek-lock, DeliveryCount, DLQ.
• "Event Grid vs Service Bus vs Event Hubs?" — reactive push routing vs enterprise broker (ordering/sessions/transactions) vs high-throughput stream.
• "Why must a Durable orchestrator be deterministic?" — it's replayed from history; a clock/random/IO call would diverge on replay. Side effects go in activities.
• "Cold start — when does it matter and how do you fix it?" — Consumption scale-to-zero; Premium/Flex pre-warmed instances, or keep-warm.

7. Secrets & reliability (P12, P14)

• "How does a Function authenticate to Key Vault with no secret?" — Managed Identity → IMDS token → Key Vault data-plane (RBAC). Envelope encryption + KEK rotation.
• "Compose the SLA of: Front Door → APIM → Function → Cosmos." — multiply the serial availabilities; show where redundancy raises it; name the weakest link.
• "A dependency is failing — retries are making it worse. What do you do?" — backoff + jitter under a retry budget, circuit breaker, honor 429/Retry-After, shed load.

The system-design loop (use in every round)

1. Clarify the workload: RPS, payload size, latency SLO, availability target, data residency, compliance.
2. Sketch control plane (identity, RBAC, policy, landing zone) and data plane (network, compute, events, storage) — principals always draw both.
3. Quantify: SLA composition, cost/month, partition/instance sizing, retry budget.
4. Name the tradeoff per Well-Architected pillar and write the ADR sentence.
5. Fail it: what's the blast radius, the throttling limit, the DR story, the rollback?

See ../system-design/README.md for worked reference designs.

System Design — Reference Azure Architectures

Worked, principal-level designs that tie the whole track together. Each one names the control plane (identity, RBAC, policy, landing zone) and the data plane (network, compute, events, storage), then quantifies the SLA, the cost, and the tradeoff per Well-Architected pillar. Use them as templates; the capstone (P15) builds one.

The design method (every time)

1. Workload facts first: RPS, payload, p99 latency SLO, availability target, data residency, compliance regime, team size.
2. Two planes, always:
   • Control plane — which tenant/MG/subscription, RBAC model, policy baseline, IaC + pipeline, secrets strategy.
   • Data plane — VNet/subnets/NSG/Private Endpoints, compute, eventing, storage, egress.
3. Quantify: compose the SLA (serial ∏Aᵢ; redundant 1−∏(1−Aᵢ)), estimate $/month, size instances/partitions, set the retry budget.
4. Tradeoff per pillar: Reliability, Security, Cost, Operational Excellence, Performance — name what you trade and write the ADR sentence.
5. Fail it: blast radius, throttling limits, DR/region-failover, rollback.

Design 1 — Secure event-driven serverless API (the canonical platform)

Requirement: a public REST API, 2k RPS, p99 < 300 ms, 99.95% target, PII at rest, EU data residency, deploy 20×/day.

Control plane

• Landing zone: workload subscription under the Corp MG; policy baseline denies public storage/SQL and non-EU regions; RBAC via Entra groups, least privilege.
• IaC: Terraform (azurerm) in Git; CI deploys via OIDC federation (no secret); what-if/plan gate + manual approval to prod.

Data plane

• Edge: Front Door (global, WAF) → APIM (JWT validation: aud/iss/scp, rate-limit/quota) → Functions (Flex Consumption, VNet-integrated).
• Async: API drops work on Service Bus; a Function consumes peek-lock with DLQ; an Event Grid topic fans out domain events to subscribers.
• Data: Cosmos DB (EU region, zone-redundant) via Private Endpoint; secrets/keys in Key Vault (RBAC, purge protection); compute uses Managed Identity (no secrets).
• Observability: App Insights distributed tracing correlated by operation_Id; Log Analytics + KQL alerts on p95 latency and DLQ depth.

Quantify

• SLA: Front Door (99.99) × APIM (99.95) × Functions (99.95) × Cosmos (99.99 zone-redundant) ≈ 99.88% serial — below target ⇒ add regional redundancy on the weakest link or relax the SLO; show the math, then decide.
• Retry: Service Bus consumer backoff + jitter, MaxDelivery 5 → DLQ; client honors 429.

Tradeoffs

• Cost vs latency: Flex/Premium kills cold starts but costs more than Consumption — justify by the p99 SLO.
• Security vs simplicity: Private Endpoints + no public PaaS adds DNS complexity; worth it for PII.

Design 2 — Multi-subscription landing zone & governance (P05)

Requirement: onboard 50 teams safely; enforce guardrails; central networking/logging.

• MG hierarchy: Root → Platform (Identity/Management/Connectivity) + Landing Zones (Corp/ Online) + Sandbox + Decommissioned.
• Policy-as-code initiative at each MG: deny disallowed regions, require diagnostic settings, enforce HTTPS/TLS, audit untagged resources; exemptions time-boxed in Git.
• Subscription vending: pipeline creates a subscription in the right MG with budgets, RBAC groups, hub-spoke peering, and a Private DNS link pre-wired.
• Hub-and-spoke: shared Azure Firewall + DNS in the hub; spokes route 0.0.0.0/0 via UDR to the firewall; central Log Analytics.

Tradeoff: central control vs team velocity — guardrails as deny for the non-negotiables, audit for the rest, self-service inside the guardrails.

Design 3 — Zero-downtime deploy with rollback (P08, P14)

• Blue-green slots behind APIM/Front Door; deploy green, smoke-test, swap; health gate on error-rate/latency triggers auto-rollback (swap back).
• Canary alternative: shift 5% → 25% → 100% with KQL health gates between steps; abort on regression.
• DB changes: expand-migrate-contract (backward-compatible schema) so a rollback is safe.

Tradeoff: blue-green = instant rollback, double the resources during deploy; canary = smaller blast radius, slower and needs solid health signals.

Design 4 — Disaster recovery & multi-region (P14)

• Active-passive across a region pair: primary serves; secondary warm with geo-replicated Cosmos/storage and IaC-reproducible infra; Front Door fails over on health probe.
• RPO/RTO drive the spend: synchronous replication (low RPO, high cost) vs async (cheaper, data-loss window). State the numbers.
• Test the failover (game day) — an untested DR plan is a hope, not a control.

Anti-patterns the panel listens for

• Designing only the data plane and forgetting identity/RBAC/policy/landing zone.
• "We'll use Kubernetes" with no workload numbers to justify it over Container Apps/Functions.
• Public PaaS endpoints for PII data; secrets in app settings instead of Key Vault + MI.
• Claiming "exactly-once" delivery; quoting an SLA without composing the dependency chain.
• No rollback story, no DR test, no cost estimate.

Phase 00 — The Azure Principal: Control Plane, Well-Architected & Tradeoffs

Difficulty: ⭐⭐☆☆☆ (conceptual) → the judgment it builds is ⭐⭐⭐⭐⭐ Estimated Time: 1 week (12–18 hours) Prerequisites: none — this is the orientation that makes the other 15 phases cohere

Why This Phase Exists

Before you touch Terraform, Entra ID, NSGs, or Service Bus, you have to understand what job you are actually being hired to do on Azure — because at the principal level the job is not "operate the portal," it is "make correct tradeoffs under conflicting requirements, and debug the layer everyone else stands on."

The reference JD says it plainly: this is "not a 'click in the portal' role and not a 'run one Terraform module' role. It is about owning the cloud platform layer itself." And the hardest part is "making correct tradeoffs" and being "the escalation point when it breaks at 2 a.m."

Two truths organise everything that follows, and this phase plants both:

1. There is exactly one control plane. Every write to Azure — a VM, a role assignment, a policy, a private endpoint — is an idempotent PUT to a resource ID through Azure Resource Manager (ARM). Authenticate (Entra token) → authorize (RBAC, then deny wins) → validate (Policy) → route to the resource provider. If you hold this one request shape, every later phase is "what does this provider do once ARM hands it the request?"
2. Every design is a tradeoff across five pillars. The Well-Architected Framework — Reliability, Security, Cost Optimization, Operational Excellence, Performance Efficiency — is the rubric every design review scores against. You cannot max all five; the principal names the pillar being traded, quantifies both ends with numbers (SLA math, the bill), chooses, and writes the ADR.

So Phase 00 builds the mental model and the calculator for those tradeoffs. Every later phase plugs a real Azure service into a slot in the model you build here. Skip this phase and you will learn 15 services and still not be able to architect — which is exactly the failure mode that keeps strong engineers stuck at senior.

Concepts

• Azure Resource Manager (ARM) as the single control plane. Every write is an idempotent PUT to a resource ID; ARM authenticates, authorizes (RBAC − NotActions, then deny wins), validates (Policy), then routes to the resource provider. Re-issuing the same PUT converges to the same state — the property the entire IaC model depends on.
• Control plane vs data plane. ARM/RBAC/Policy decide what exists and who may manage it (Actions); the blob, secret, message, and token are a separate system with its own identity, throttling, and SLA (DataActions). A storage Owner is not automatically a Blob Data Reader. This single distinction explains a huge fraction of "I have access but I'm getting 403" incidents.
• The resource ID hierarchy and scopes. The canonical ID /subscriptions/{s}/resourceGroups/{rg}/providers/{ns}/{type}/{name} is the unit of RBAC scope, Policy scope, tagging, and locks. Scopes nest Management Group → Subscription → Resource Group → resource, and assignments inherit downward. Resource providers (Microsoft.Storage, Microsoft.Network, …) own the types and must be registered.
• The Well-Architected Framework five pillars — Reliability, Security, Cost Optimization, Operational Excellence, Performance Efficiency — and how every design trades them against each other.
• SLA composition math. Serial dependencies multiply (A = ∏ Aᵢ — more hops, lower SLA); redundant components combine (A = 1 − ∏(1 − Aᵢ) — zones/regions raise it); and (1 − A) × 43 200 converts a nines-target into downtime minutes/month. Availability zones and paired regions are how you buy the redundancy term.
• Cost modeling basics & "spend the headroom." Compute hours × price, storage GB-month, and egress GB are the first-order bill; unused reliability or latency headroom is a cost lever, not a virtue.
• The Architecture Decision Record (ADR). The lightweight artifact that makes a tradeoff durable and reviewable: name the pillar, quantify both ends, choose, and record what would change your mind. A design review (interview Round 5) is a room pressure-testing your ADRs.

Labs

Lab 01 — WAF Tradeoff Modeler (flagship, implemented)

→ Lab folder: lab-01-waf-tradeoff-modeler/

Integrated-Scenario Suggestions (carried through the whole track)

These scenarios are the spine of the capstone (Phase 15). Keep them in mind from now — each later phase contributes a real, tested component, and each one forces the five-pillar tradeoff you model here:

1. A regulated multi-subscription landing zone — management-group hierarchy, policy-as- code baselines, RBAC inheritance, and subscription vending (P04–P05). Tradeoff: security & governance ↔ developer velocity.
2. A secure event-driven serverless platform — Functions + Durable + Service Bus + Event Grid behind Private Endpoints, secrets in Key Vault via Managed Identity (P10–P12). Tradeoff: reliability (exactly-once-effect) ↔ operational simplicity.
3. A global API front door — APIM with JWT scope authorization and token-bucket rate limiting, multi-region with Front Door (P09). Tradeoff: performance/availability ↔ cost (egress, duplicated compute).
4. A private-networking hub-and-spoke — NSGs, UDRs, Azure Firewall, Private DNS (P06). Tradeoff: security isolation ↔ operability (the #1 Private Endpoint DNS failure).
5. An observability + reliability backbone — Azure Monitor, KQL, alert rules, retry + circuit breaker + SLA composition (P13–P14). Tradeoff: cost (telemetry volume) ↔ fidelity / time-to-detect.

For each, write the one-paragraph WAF tradeoff now (which pillar dominates, what you'd trade, the number that decides it). You'll refine it as the matching phase lands.

Guides in This Phase

• HITCHHIKERS-GUIDE.md — the 30-minute orientation; read first
• WARMUP.md — the full zero-to-principal primer; read slowly
• BROTHER-TALK.md — the candid, off-the-record version

Key Takeaways

• The job is tradeoffs across five pillars, made on one control plane, encoded into platforms and defended in ADRs — not portal clicks.
• Every Azure write is one idempotent PUT to a resource ID. Control plane (manage) and data plane (the blob/secret/message) are different systems with different identity, throttling, and SLA. Owner ≠ Blob Data Reader.
• SLA and cost are arithmetic you do before the review, not after the incident: serial multiplies, redundant combines, nines convert to minutes, and the bill has a shape.
• "Make it reliable / cheap / fast" is not a requirement until you attach numbers to the pillars; this phase's lab is where you learn to attach them.

Deliverables Checklist

• Lab 01 implemented; all tests pass against solution.py and your lab.py.
• You can draw the control-plane request path (Entra → ARM → RBAC → Policy → RP) from memory and parse any resource ID into its scope.
• You can compose a serial and a redundant SLA in your head and convert it to minutes/month (99.9 % → 43.2; 99.99 % → 4.32).
• You can take any "design X on Azure" prompt and immediately name the dominant pillar and the pillar you'd trade.
• You can write a one-page ADR for a real decision (template in WARMUP.md).

Warmup Guide — The Azure Principal: Control Plane, Well-Architected & Tradeoffs

Zero-to-principal primer for Phase 00. We start from "what even is Azure when you strip the portal away" and end with the arithmetic and the artifacts that turn architectural opinion into engineering: the single control plane and the resource-ID address space, the control-plane/data-plane split, the five Well-Architected pillars, SLA composition and downtime math, first-order cost modeling, and the Architecture Decision Record. Every later phase plugs a real Azure service into a slot in the model built here.

Table of Contents

• Chapter 1: Azure Resource Manager — The One Control Plane
• Chapter 2: The Resource ID — Azure's Address Space
• Chapter 3: Control Plane vs Data Plane
• Chapter 4: The Well-Architected Framework — Five Pillars
• Chapter 5: SLA Composition — The Reliability Arithmetic
• Chapter 6: Availability Zones & Paired Regions
• Chapter 7: Cost Modeling and "Spend the Headroom"
• Chapter 8: The ADR — Making a Tradeoff Durable
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Azure Resource Manager — The One Control Plane

From zero. When you click "Create a virtual machine" in the portal, run az vm create, terraform apply, or New-AzVM, none of those is "talking to a VM." They are all generating one thing: an HTTPS request to a single deployment-and-management API called Azure Resource Manager (ARM). The portal is a JavaScript app that calls ARM. The CLI is a Python app that calls ARM. Terraform's azurerm provider is a Go binary that calls ARM. There is exactly one front door, and everything in this curriculum sits behind it.

What it is. ARM is the control-plane API and deployment engine in front of every Azure write. A request is an HTTP method against a resource URI with an API version:

PUT https://management.azure.com
    /subscriptions/{sub}/resourceGroups/{rg}
    /providers/Microsoft.Storage/storageAccounts/{name}?api-version=2023-01-01
Authorization: Bearer <Entra access token>
Content-Type: application/json

{ "location": "eastus", "sku": { "name": "Standard_LRS" }, "kind": "StorageV2" }

Why it exists. Before ARM (the "classic"/ASM era), each service had its own management API, its own permissions model, and no shared notion of grouping, tagging, templating, or dependency ordering. ARM unifies all of that: one auth model (Entra), one authorization model (RBAC + Policy), one identity for resources (the resource ID), one grouping primitive (resource groups), one templating/deployment engine, and one audit log (Activity Log). The payoff is that governance, automation, and reasoning generalise across every service.

How it works under the hood. A single write travels a fixed pipeline. Memorise it; it is the spine of phases 01–05:

client (portal / az / Terraform / SDK)
   │  PUT /subscriptions/.../{type}/{name}?api-version=…   + Bearer token
   ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ Azure Resource Manager (management.azure.com)                            │
│                                                                          │
│  1. AUTHENTICATE  — validate the Entra (Azure AD) JWT: signature, iss,   │
│                     aud=https://management.azure.com, exp/nbf            │
│  2. AUTHORIZE     — RBAC: union(Actions) − union(NotActions) across all  │
│                     role assignments at this scope and every ancestor;   │
│                     then DENY ASSIGNMENTS — a deny always wins           │
│  3. POLICY        — Azure Policy: deny / append / modify / audit /       │
│                     deployIfNotExists evaluated against the request      │
│  4. ROUTE         — hand the validated request to the resource provider  │
└─────────────────────────────────────────────────────────────────────────┘
   │
   ▼
Resource Provider (Microsoft.Storage)  → reconciles desired state, returns 200/201

Idempotency is the load-bearing property. ARM PUT is idempotent by design: the body describes the desired state of the resource, not a delta. Issue the same PUT twice and the second one is a no-op convergence to the same state — no duplicate resource, no error. Formally, if apply is the state transition and s the desired state, then

$$ \text{apply}(\text{apply}(x, s), s) = \text{apply}(x, s) $$

— applying twice equals applying once. This is why Terraform, Bicep, and ARM templates can re-run safely, why a retried deployment doesn't double your VMs, and why "declarative infrastructure" works at all. (POST actions — start/stop/regenerate-key — are the non-idempotent exceptions; they are operations, not state.)

Production significance. When something fails to deploy, the question is which stage of that pipeline rejected it: a 401 is authentication (token), a 403 is authorization (RBAC or a deny assignment or a Policy deny), a 409 is a conflict, a 429 is throttling, and a 5xx from the provider is the service itself. Knowing the pipeline turns a vague "it won't deploy" into a one-line diagnosis.

Common misconceptions.

• "The portal does special things the API can't." No — the portal is strictly a client of ARM. Anything it does, the API does.
• "PUT and POST are interchangeable." No — PUT is idempotent desired-state; POST is a non-idempotent action. Retrying a POST can double-execute.
• "ARM stores my data." No — ARM manages the resource's existence and configuration (control plane). Your blobs, secrets, and messages live in the data plane (Chapter 3).

Chapter 2: The Resource ID — Azure's Address Space

From zero. Every resource in Azure has exactly one canonical name — a path, like a file path, called the resource ID:

/subscriptions/{subscriptionId}/resourceGroups/{rgName}/providers/{namespace}/{type}/{name}

A concrete example:

/subscriptions/0000…/resourceGroups/rg-prod/providers/Microsoft.Storage/storageAccounts/stprod001

Why it exists. The resource ID is not just a name — it is simultaneously the unit of RBAC scope, Policy scope, tagging, locks, and the audit trail. When you grant a role, assign a policy, place a delete-lock, or read the Activity Log, you do it at a resource ID (or a prefix of one). One address space, reused for everything.

How it works under the hood — the hierarchy and scopes. Read the segments as key/value pairs:

Above subscriptions sit management groups (P05), giving the full inheritance chain:

Tenant Root MG  ──▶  Management Group  ──▶  Subscription  ──▶  Resource Group  ──▶  Resource
(/providers/Microsoft.Management/managementGroups/{name})        (this is where the ID address space starts)

A role or policy assigned at any node inherits down to every descendant. Assign Reader at the subscription and the principal can read every RG and resource under it. This is the single most important fact for reasoning about "who can touch this?": you cannot answer it by looking at the resource alone — you must look up the chain.

Nested types. Some resources live inside others, and the ID nests the type path:

/subscriptions/0000…/resourceGroups/rg-net
  /providers/Microsoft.Network/virtualNetworks/vnet-hub/subnets/snet-app

Here the type is the slash-joined path virtualNetworks/subnets and the name is the leaf (snet-app), not the parent (vnet-hub). Subnets, blob containers, Key Vault secrets, and SQL databases are all nested types. Your lab's parse_resource_id must keep the leaf name and join the type path — exactly the rule production tooling uses.

Resource providers. The {namespace} (Microsoft.Storage, Microsoft.Compute, Microsoft.Network, …) names the resource provider (RP) — the service that owns those types and implements their control-plane CRUD. An RP must be registered on a subscription before its types can be created (az provider register --namespace Microsoft.Storage); the classic "the subscription is not registered to use namespace X" error is a missing registration.

Production significance. Parsing and classifying a resource ID is the first move in every governance and incident task: what scope is this, what inherits onto it, and which RP do I escalate to? The lab makes you build that parser so the structure is muscle memory.

Common misconceptions.

• "The resource group's region is where my resource runs." No — the RG region is where the RG's metadata lives; the resource has its own location.
• "GUIDs in the ID are the names." The subscription is a GUID; resource and RG names are human strings you choose. Don't confuse the two.
• "A subnet's name is the vnet." No — it's the leaf segment after subnets/.

Chapter 3: Control Plane vs Data Plane

From zero. There are two completely separate systems hiding behind one resource:

• The control plane manages the resource's existence and configuration: create the storage account, change its SKU, set its firewall, delete it. This is ARM (Chapter 1), authorized by RBAC Actions.
• The data plane is the stuff inside the resource: the blobs in the account, the secrets in the vault, the messages in the queue, the rows in the database. This is a different endpoint (stprod001.blob.core.windows.net, not management.azure.com), with its own throttling, its own SLA, and its own authorization via RBAC DataActions.

Why it exists. Managing a resource and using its data are genuinely different concerns with different blast radii. The person who provisions a storage account (an infra engineer) is often not the person who should read customer blobs (an application). Splitting the authorization lets you grant one without the other.

How it works under the hood. A role definition carries four lists:

Actions          — control-plane verbs allowed     (e.g. Microsoft.Storage/storageAccounts/read)
NotActions       — control-plane verbs subtracted
DataActions      — data-plane verbs allowed         (e.g. .../blobServices/containers/blobs/read)
NotDataActions   — data-plane verbs subtracted

Effective permission is union(Actions) − union(NotActions) for the control plane and the DataActions equivalent for the data plane — evaluated independently. The canonical trap:

So a storage-account Owner can change the firewall, regenerate keys, and delete the account — but is not automatically allowed to read a blob with their Entra identity. (They can grab the account key — a control-plane action — and use that, which is exactly why key-based access bypasses data-plane RBAC and why principals disable shared-key auth.)

Production significance. A large share of "I'm an Owner but I get 403 Authorization Permission Mismatch" tickets are this exact split: control-plane access, no DataAction. The fix is to assign a data role (Storage Blob Data Reader/Contributor, Key Vault Secrets User), not to escalate the control-plane role. Recognising it on sight is a principal signal.

Common misconceptions.

• "Owner can do anything." On the control plane, yes; on the data plane, only via keys, not via their Entra identity by default.
• "It's the same SLA." No — control plane and data plane publish separate SLAs and have separate throttling limits.

Chapter 4: The Well-Architected Framework — Five Pillars

From zero. The Azure Well-Architected Framework (WAF) is Microsoft's rubric for "is this a good architecture?", expressed as five pillars. It exists because "good" is meaningless without axes, and these are the five every design review scores against:

1. Reliability — does it stay up and recover? (SLA, redundancy, failover, retries, no single points of failure). Quantified by the math in Chapter 5.
2. Security — is it safe? (identity as the perimeter, least privilege, encryption, network isolation, secrets in Key Vault, deny-by-default).
3. Cost Optimization — is the spend justified? (right-sizing, reservations/savings plans, autoscale, scale-to-zero, lifecycle tiering, egress awareness — Chapter 7).
4. Operational Excellence — can a team run it? (IaC, CI/CD, monitoring, runbooks, incident response — the pillar that dominates total cost over a system's life).
5. Performance Efficiency — does it meet its latency/throughput targets and scale without over-provisioning? (the right SKU, caching, async, partitioning).

Why it exists and why it's a tradeoff model. You cannot maximise all five — they pull against each other:

So the WAF is not a checklist you "pass" — it is a scoring function whose weights come from the workload. A payments system weights Reliability and Security; an internal batch tool weights Cost. The principal move is: name the pillar you are trading, quantify both ends, choose, and write the ADR (Chapter 8). The lab's resolve_tradeoff is this scoring function, and its headline test proves the consequence: the same two architectures flip winners when the weights change.

The math the lab uses. Score each architecture $A$ on each pillar $p \in P$ as $s_{A,p} \in [0,1]$, with workload weights $w_p \ge 0$. The weighted, normalised score is

$$ \text{Score}(A) = \sum_{p \in P} s_{A,p} \cdot \frac{w_p}{\sum_{q} w_q} $$

The winner is the higher score; the deciding pillar is the one with the largest weighted advantage for the winner, $\arg\max_p (s_{\text{win},p} - s_{\text{lose},p}) \cdot \frac{w_p}{\sum_q w_q}$ — the one sentence you put in the ADR ("we chose multi-region because reliability dominated for this workload").

Common misconceptions.

• "There's a best architecture." No — there's a best architecture for a given set of weights. Asking "what's the best?" without the workload is a category error.
• "Security and cost are separate teams' problems." No — they are pillars you trade in one design, simultaneously.

Chapter 5: SLA Composition — The Reliability Arithmetic

From zero. A Service Level Agreement (SLA) is a published availability number — App Service 99.95 %, zone-redundant SQL 99.99 %, a single VM with premium SSD 99.9 %. The catch: Azure publishes a number per service, never a composite for your topology. You compose it by hand. This is the single most useful piece of arithmetic in a design review.

Serial dependencies multiply. If a request must traverse several components and any one being down fails the request (a hard dependency chain — web app → SQL → Key Vault), the availabilities multiply:

$$ A_{\text{serial}} = \prod_{i} A_i $$

Three 99.9 % services in series:

$$ 0.999 \times 0.999 \times 0.999 = 0.999^3 \approx 0.997002999 $$

— roughly 99.7 %, about 3× the downtime of a single component. Every hop you add to the request path lowers the composite SLA. This is the quantitative reason "just add another microservice" is never free.

Redundant copies combine. If you have several interchangeable copies and the system is down only when all of them are down (active/active across zones), the un-availabilities multiply:

$$ A_{\text{redundant}} = 1 - \prod_{i} (1 - A_i) $$

Two 99 % copies:

$$ 1 - (1 - 0.99)^2 = 1 - 0.01^2 = 1 - 0.0001 = 0.9999 $$

— two cheap 99 % things give you 99.99 %. Redundancy is how you buy nines.

Nines → minutes (the number that ends the argument). Convert availability to downtime per month (30-day month = 43,200 minutes):

$$ \text{downtime}_{\text{min/month}} = (1 - A) \times 43{,}200 $$

The leap from three to four nines is not a config flag — it's multi-AZ or multi-region, a different architecture and a different bill. Knowing the minutes makes the conversation concrete: "you're asking for four nines; that's 4 minutes a month; here's the redundancy that buys it and here's what it costs."

Production significance. Before you commit to an SLA in a contract, you compose it: list the serial path, multiply, then decide which links need a redundant pair to hit the target. The lab makes you compute exactly this.

Common misconceptions.

• "My SLA is the weakest component's SLA." No — serial is the product, which is worse than the weakest link.
• "Redundant copies are independent." Rarely fully — a shared region, shared dependency, or correlated failure breaks independence, so 1 − ∏(1 − Aᵢ) is an upper bound. Say so in the interview; it's the honest, principal-level caveat (and the lab's extension).
• "99.9 % and 99.99 % are basically the same." They are 43 minutes vs 4 minutes a month — a 10× difference and usually a different architecture.

Chapter 6: Availability Zones & Paired Regions

From zero. The redundant-SLA math in Chapter 5 only helps if you have somewhere to put the copies that fails independently. Azure gives you three tiers of blast-radius isolation:

• Availability Zone (AZ) — physically separate datacenter(s) within one region, with independent power, cooling, and network. Spreading replicas across ≥2 zones survives a datacenter-level failure and is the cheapest way to add the redundant term (the cross-zone latency is sub-millisecond; egress between zones in-region is cheap or free for many services). This is how you go from 99.9 % to 99.99 % within one region.
• Region — a geography (East US, West Europe). Multi-region active/active survives an entire region outage but adds full cross-region egress cost and replication complexity — the most expensive reliability you can buy.
• Paired region — most Azure regions have a designated pair in the same geography with platform-coordinated maintenance (the pair isn't updated simultaneously) and, for some services, built-in geo-replication for disaster recovery (e.g. GRS storage, geo-redundant backups). The pair is your DR target, not necessarily your active/active target.

How it maps to the math. "Zone-redundant" services apply Chapter 5's redundant formula for you across zones and publish the higher composite (e.g. zone-redundant SQL 99.99 % vs 99.9 % single-zone). When you go multi-region yourself, you apply the formula — and you discount it for correlation, because a region pair shares a geography and some platform fate.

Production significance. The reliability ladder is: single instance → multi-AZ (zone-redundant) → multi-region active/passive (failover) → multi-region active/active. Each rung adds a redundant term and a cost/operability term. The principal picks the lowest rung that hits the workload's downtime target, because every rung above it is paid for forever.

Common misconception. "Multi-region is always more reliable than multi-AZ." For most workloads multi-AZ already hits four nines at a fraction of the cost and complexity; multi-region buys you region-outage survival specifically, and you should only pay for it if a region outage is in your threat model and your data layer can actually replicate that far without breaking consistency.

Chapter 7: Cost Modeling and "Spend the Headroom"

From zero. You cannot be the cost-optimization person WAF asks for without arithmetic. The first-order monthly bill has three terms, and most real bills are dominated by one of them:

$$ \text{cost}{\text{month}} = \underbrace{H \cdot p{\text{vcpu}}}{\text{compute}} + \underbrace{G_s \cdot p{\text{storage}}}{\text{storage}} + \underbrace{G_e \cdot p{\text{egress}}}_{\text{egress}} $$

where $H$ = compute (vCPU-)hours, $G_s$ = stored GB-months, $G_e$ = egress GB, and the prices approximate Azure list ($p_{\text{storage}}\approx$0.018$/GB-month LRS hot, $p_{\text{egress}}\approx$0.087$/GB internet after the free tier).

The terms to watch.

• Compute is usually the headline, and the levers are right-sizing the SKU, reserved instances / savings plans (up to ~72 % off for a 1–3 year commit), autoscale, and scale-to-zero (Consumption Functions/Container Apps) — trading cold-start latency for near-zero idle cost.
• Storage is cheap per GB but accumulates; lifecycle tiering (hot → cool → archive) and not hoarding redundant copies (LRS vs GRS vs ZRS is a 1–2× multiplier) are the levers.
• Egress is the ambush. Ingress is free; egress (data leaving Azure, or crossing regions) is billed per GB and is what makes chatty multi-region designs quietly expensive. A cross-region call pattern can make egress dominate a bill that "looked like compute."

"Spend the headroom." This is the principal mindset that separates cost-aware design from cheese-paring. If your reliability target is met with headroom, don't buy more nines. If your latency SLA is 15 minutes and you consume 40 seconds, the other ~14 minutes is a cost lever — batch writes harder, use spot/low-priority compute, compact lazily, scale to zero. Juniors optimise the dimension they can see; principals find the unused headroom in one pillar and convert it into savings in another. The same logic runs in reverse: if you're over budget, look for the pillar you're over-delivering on and trade it down.

Production significance. "Make it cheaper" becomes concrete the moment you (a) tag every resource with an owner so the bill is a map, and (b) write the three-term model so you can see which term to attack. Optimising compute when egress dominates is the classic wasted sprint.

Common misconceptions.

• "Storage is the big cost." Usually compute is, then sometimes egress; raw storage is rarely the headline (small files and request costs aside).
• "Reserved instances lock me in dangerously." They commit spend, not a specific VM; savings plans are even more flexible. For steady-state baseline load, not reserving is leaving 30–70 % on the table.
• "Cheaper architecture = better." Only if operability holds — an architecture that needs a dedicated on-call team is not cheaper.

Chapter 8: The ADR — Making a Tradeoff Durable

From zero. A principal's decisions outlive their memory of them and must survive their absence. The Architecture Decision Record (ADR) is the one-page artifact that makes a tradeoff reviewable, teachable, and reversible-on-purpose. It is the output of everything above — you do the SLA math and the cost model, then you record the call.

The template you'll reuse all track:

# ADR-NNN: <short title>
Status: proposed | accepted | superseded by ADR-MMM
Date: YYYY-MM-DD

## Context
The workload's requirements as numbers on the five pillars: target SLA (and the
composed math), budget, latency target, security/compliance constraints, who operates it.

## Decision
What we chose, in one sentence — and the DECIDING PILLAR.

## Alternatives Considered
Each option, and the pillar it traded the wrong way for OUR weights.

## Consequences
What gets better, what gets worse, what we must now monitor, and the number (or
event) that would make us revisit this.

A worked one-liner the lab can emit: "For the payments workload (weights: reliability 4, security 2, cost 1), we chose multi-region active/active over single-region because reliability dominated — composing to 99.99 % (4.3 min/month) vs the single region's 99.9 % (43 min/month). We accept ~2× compute + egress cost and higher operational load; we revisit if a minute of downtime costs less than $X or if the data layer can't sustain cross-region consistency."

Why it matters for your career. An architecture review (interview Round 5, and your day job) is a room full of people pressure-testing your ADRs. The discipline the ADR enforces is exactly Chapter 4's: name the pillar, quantify both ends, choose, and write down what would change your mind. The person whose decisions are legible gets trusted with bigger ones.

Common misconception. "ADRs are bureaucratic overhead." A good ADR is one page and takes ten minutes; it saves the hour-long re-litigation six months later when someone asks "why did we do it this way?" — and you point at the numbers instead of arguing from memory.

Lab Walkthrough Guidance

Lab 01 — WAF Tradeoff Modeler, suggested order (matches the file top-to-bottom):

1. ResourceId + parse_resource_id (Ch. 1–2). Split on /, strip the leading slash, reject empty/odd segments, then read subscriptions/{s} → optional resourceGroups/{rg} → optional providers/{ns}/{type}/{name}[/{type2}/{name2}…]. For nested types, join the type segments with / and keep the leaf name. Raise ValueError on every malformed shape — interviewers will hand you a bad ID to see if you validate.
2. resource_scope_level (Ch. 2). Parse, then classify by the deepest field set: provider → resource; resource_group → resourceGroup; else subscription.
3. SLA functions (Ch. 5): serial_sla (product), redundant_sla (1 − ∏(1 − aᵢ)), downtime_minutes_per_month ((1 − A) × 43200). Validate each availability in [0, 1]. The exact-number tests (0.999³ ≈ 0.997002999, two 0.99 → 0.9999) are the boundary probes.
4. monthly_cost_usd (Ch. 7): three terms, all inputs non-negative.
5. Architecture.__post_init__ (Ch. 4): all five pillars present, none unknown, each in [0, 1].
6. resolve_tradeoff (Ch. 4): weight-normalised score per architecture; winner + deciding pillar; tie within tol. Make test_weights_flip_the_winner pass — and make sure you understand why it's the point of the whole lab.

Success Criteria

You are ready for Phase 01 when you can, from memory:

1. Draw the control-plane request pipeline (Entra authenticate → ARM authorize via RBAC, deny-wins → Policy → resource provider) and say which stage a 401 / 403 / 409 / 429 came from.
2. Explain why every ARM write is an idempotent PUT to a resource ID, and what idempotency buys IaC.
3. Parse any resource ID into subscription / RG / provider / type / name (including a nested type) and name its scope and what inherits onto it.
4. State why a storage Owner cannot read a blob with their Entra identity, and the fix.
5. List the five WAF pillars and give a real tradeoff between two of them.
6. Compose a serial and a redundant SLA in your head and convert it to minutes/month.
7. Write the three-term cost model and name which term ambushes multi-region designs.
8. Write a one-page ADR that names the pillar, quantifies both ends, and states what would change your mind.

Interview Q&A

Q: Walk me through what happens between az storage account create and the account existing. The CLI acquires an Entra access token (audience https://management.azure.com) and issues an idempotent PUT to the resource ID /subscriptions/…/providers/Microsoft.Storage/ storageAccounts/{name}?api-version=…. ARM authenticates the token (signature, iss, aud, exp), authorizes via RBAC — union(Actions) − union(NotActions) across every assignment at this scope and its ancestors, then deny assignments where deny wins — evaluates Azure Policy (a deny effect blocks the write here), and routes the validated request to the Microsoft.Storage resource provider, which reconciles desired state and returns 201/200. Re-issuing the same PUT converges with no duplicate — that's the idempotency the whole IaC model rests on.

Q: I'm an Owner on a storage account but I get 403 reading a blob. Why? Control plane vs data plane. Owner grants Actions: * — manage the account, set its firewall, regenerate keys — but no DataActions, so your Entra identity can't read blob data directly. The fix isn't a bigger control-plane role; it's a data role like Storage Blob Data Reader. (You could pull the account key — a control-plane action — and use shared-key auth, which bypasses data-plane RBAC; that's exactly why hardened environments disable shared-key access and rely on DataActions + Managed Identity.)

Q: A team wants 99.99 % availability. How do you respond? First I make it concrete: 99.99 % is 4.32 minutes of downtime a month versus 99.9 %'s 43 minutes — a 10× ask. Then I compose their actual topology: if the request path is web → SQL → Key Vault, that's serial, so the composite is the product — three 99.9 % services already fall to ~99.7 %, worse than the target, before we add redundancy. So I find which links need a redundant pair (zone-redundant SQL gets that link to 99.99 % via 1 − ∏(1 − Aᵢ)), price the multi-AZ option, and only reach for multi-region if a region outage is in scope — because that's the most expensive nine and it adds egress and operational load. I write the ADR with the composed number and the cost.

Q: How do you make a "make it cheaper" mandate concrete? Attribute first — tag every resource with an owner so the bill becomes a map — then write the three-term model (compute + storage + egress) to see which term dominates. The levers are mechanical per term: compute falls with right-sizing + reservations/savings plans + autoscale + scale-to-zero; storage with lifecycle tiering and the right redundancy SKU; egress by killing chatty cross-region patterns. Crucially I hunt for unused headroom — if the SLA or latency target has slack, I spend it: batch harder, use spot, compact lazily. Most "expensive" systems are over-delivering on a pillar nobody is paying for.

Q: Why is "what's the best architecture?" a bad question? Because "best" is undefined without the workload's pillar weights. The Well-Architected pillars conflict — multi-region buys reliability and costs you cost and operability — so the same two designs genuinely flip winners when the weights change (my lab proves this with a test). The right question is "best for these requirements," and the principal's job is to extract those requirements as numbers, score the options, and record the deciding pillar in an ADR. Anyone who answers "best architecture" without asking about the workload is reciting patterns, not engineering.

Q: Difference between a senior and a principal Azure engineer, in one sentence? A senior deploys a correct architecture for a given design; a principal defines the design by composing the SLA, modeling the cost, naming the pillar being traded, and recording the decision in an ADR — so hundreds of other engineers inherit correct tradeoffs by default.

References

• Azure Resource Manager overview — the control plane, request pipeline, resource providers.
• Resource ID & scopes and Understand scope — the address space and inheritance.
• Control plane and data plane operations — the split this phase hammers.
• Azure Well-Architected Framework — the five pillars and the design-review methodology.
• WAF Reliability pillar and target metrics / availability math — SLA composition and downtime targets.
• Azure SLAs for Online Services — the published per-service numbers you compose by hand.
• Availability zones & region pairs and cross-region replication — the redundancy you buy nines with.
• Azure pricing calculator and bandwidth (egress) pricing — the cost model's real numbers.
• Michael Nygard, Documenting Architecture Decisions (2011) — the ADR pattern.
• The track's own CHEATSHEET.md and GLOSSARY.md — the running references.

🛸 Hitchhiker's Guide — Phase 00: The Azure Principal

Read this if: you can click "Create resource" in the portal but "architect our Azure platform and defend it in a design review" still sounds like a different job. It is. This is the compressed tour the WARMUP derives slowly. Skim it, then read the WARMUP.

0. The 30-second mental model

Everything you do to Azure is one idempotent PUT to a resource ID through one control plane (ARM) — authenticate (Entra), authorize (RBAC, then deny wins), validate (Policy), route to the provider. And every design is a tradeoff across five Well-Architected pillars that you settle with numbers (SLA math, the bill) and record in an ADR. One sentence to tattoo: you don't click resources, you compose tradeoffs on a single control plane — and "best architecture" is meaningless without the workload's weights.

1. The control-plane request, one breath

client (portal / az / Terraform) ── PUT /subscriptions/.../{type}/{name} + Bearer token ──▶ ARM
  ARM:  authenticate (Entra JWT)  →  authorize (RBAC − NotActions, then DENY wins)
        →  Policy (deny/append/modify/audit)  →  resource provider (idempotent PUT)

A 401 is auth, a 403 is RBAC/deny/Policy, a 409 is conflict, a 429 is throttling, a 5xx is the provider. That mapping turns "it won't deploy" into a one-line diagnosis.

2. The resource ID is the whole address space

/subscriptions/{sub}/resourceGroups/{rg}/providers/{namespace}/{type}/{name}

It's simultaneously the RBAC scope, Policy scope, tag target, lock target, and audit key. Scopes nest MG → Subscription → RG → resource, and assignments inherit downward — so you can never answer "who can touch this?" from the resource alone; look up the chain. Nested types: …/virtualNetworks/vnet/subnets/snet → type is virtualNetworks/subnets, name is the leaf snet.

3. Control plane ≠ data plane (the trap)

4. The numbers to tattoo on your arm

5. The five pillars (score every design on these)

Reliability · Security · Cost Optimization · Operational Excellence · Performance Efficiency

You can't max all five. Name the pillar you're trading, quantify both ends, choose, write the ADR. Multi-region buys Reliability + Performance, costs Cost + Operational Excellence. Scale-to-zero buys Cost, costs Performance (cold start). There is no "best architecture" — only "best for these weights."

6. The reliability ladder (buy the lowest rung that hits the target)

single instance  →  multi-AZ (zone-redundant)  →  multi-region active/passive  →  active/active
   99.9%                99.99% (cheap nines)         region-outage survival         most expensive nine

Multi-AZ already hits four nines cheaply; multi-region is for region-outage survival specifically — and you discount the redundant-SLA formula for correlation (copies aren't truly independent).

7. az one-liners worth muscle memory

az account show --query id -o tsv                          # current subscription id
az resource list --query "[].id" -o tsv                    # every resource id (the address space)
az resource show --ids "<resource-id>"                     # round-trip a parsed id
az role assignment list --scope "<resource-id>" -o table   # who can touch this (at this scope)
az role assignment create --assignee <oid> --role "Storage Blob Data Reader" \
   --scope "<account-id>"                                  # grant a DATA role, not Owner
az provider register --namespace Microsoft.Storage         # fix "not registered to use namespace"
az policy assignment list --scope "<scope-id>" -o table    # what policy gates this scope

8. War-story shapes you'll relive

• "I'm Owner but I get 403 on the blob." → control-plane role, no DataAction. Assign a data role; don't escalate to a bigger control-plane role.
• "Our 'three nines' SLA isn't being met." → the path is serial, so the composite is the product (≈99.7 %), not 99.9 %. Add a redundant pair on the weak link, not more retries.
• "The bill doubled and nobody added VMs." → cross-region/egress traffic. Ingress is free; egress isn't. Look at the bandwidth line, not compute.
• "Multi-region is down even though we have two regions." → the copies weren't independent (shared dependency / paired-region fate). The 1 − ∏(1 − Aᵢ) number was an upper bound.
• "The subscription isn't registered to use namespace Microsoft.X." → a resource provider that was never registered. One command fixes it.

9. Vocabulary that signals you've held the pager

• Control plane / data plane — manage the resource vs use its data; different identity, throttling, SLA.
• Idempotent PUT — desired-state write that converges, not duplicates; the basis of IaC.
• Resource ID / scope — the address that is the RBAC/Policy unit; inheritance flows down.
• Deny wins — a deny assignment overrides any allow; RBAC is allow-additive otherwise.
• Serial vs redundant SLA — multiply vs 1 − ∏(1 − Aᵢ); the only two SLA operations.
• Deciding pillar — the one-sentence reason a tradeoff went the way it did; goes in the ADR.
• Spend the headroom — convert unused reliability/latency margin into cost savings.
• ADR — the durable record of a decision and what would reverse it.

10. Beginner mistakes that mark you in interviews

1. Saying "best architecture" without asking the workload's pillar weights.
2. Thinking Owner can read data — forgetting the control/data-plane split.
3. Quoting an SLA as the weakest link's number instead of the product of the serial path.
4. Saying "99.9 % vs 99.99 %, basically the same" — it's 43 vs 4 minutes, a 10× ask and a different architecture.
5. Optimising compute when egress dominates the bill (optimise the term that's actually big).
6. Treating multi-region as "free reliability" instead of the most expensive, correlation- discounted nine.
7. Making decisions in chat instead of ADRs — so they're re-litigated every quarter.

11. How this phase pays off later

• The control-plane request is P01–P05's spine (ARM deploy graph, Terraform plan, RBAC, Policy, landing zones all live on it).
• The resource ID + scope model is exactly how P04/P05 reason about inheritance.
• The control/data-plane split recurs in P09 (API auth), P10 (Service Bus), P12 (Key Vault).
• The SLA + cost math is Round-1 mental arithmetic and the backbone of P14 (reliability) and P15 (FinOps capstone).
• The ADR habit is Round-5 (architecture leadership) and your day job as a principal.

Now read the WARMUP slowly, then build the modeler. You'll use its functions as back-of-envelope checks for the rest of the track.

👨🏻 Brother Talk — Phase 00, Off the Record

No slides, no certification rubric. This is me, your brother, telling you the things people only say after the second coffee. Read it once now, and again the week before your interview.

Listen. You've probably been told that becoming a principal Azure engineer means collecting certifications — AZ-104, then 204, then 305, 500, 700 — and then you're "principal." It doesn't work like that. I've watched people with the full alphabet of badges still get stuck at senior, because nobody told them the secret:

The portal is the easy part. The judgment is the job.

Here's the thing that took me too long to learn. When a director says "architect our Azure platform," they are not testing whether you know which blade to click or which az flag to pass. They assume you can find the button. What they're watching is: do you compose the SLA before you promise four nines? Do you say "that's 4 minutes a month, here's the multi-AZ that buys it and here's the bill"? Do you name the pillar you're trading instead of hand-waving "it'll be reliable and cheap"? That move — pricing a tradeoff out loud — is the whole game. Everyone else in the room is reciting service names. You're doing engineering.

So let me give you the mindset shifts, brother to brother.

Internalise that there's one front door, and it changes everything. The single most freeing realization in this whole track is that every Azure write — VM, role assignment, policy, private endpoint — is the same shape: an idempotent PUT to a resource ID through ARM. Once you hold that, the 15 services ahead stop being 15 manuals. Each one is just "what does this provider do after ARM hands it the request, and what's its data plane?" People who never grasp this learn Azure as a thousand disconnected portal recipes and drown. People who grasp it learn one system with fifteen plugins. Be the second kind. That's what this phase is for.

Stop saying "highly available" and "scalable." I mean it — those words are banned for you now. Every time you want to say "highly available," say a number: "99.99 %, which is 4.32 minutes a month, composed from a zone-redundant data layer." Every time you want to say "scalable," say "sized to 6× average, sheds load via the rate limiter past that." The moment you do this you sound three levels more senior, because you've revealed that availability is a budget you compose, not an adjective you sprinkle. A junior wants everything multi-region. A principal knows multi-region is the most expensive nine on the menu and orders it only where a region outage is actually in the threat model.

Fall in love with the boring pillar: operational excellence. Everyone optimizes performance and reliability because they benchmark well and demo nicely. Nobody brags about "this is easy to debug at 2 a.m." But operability is the pillar that decides whether your platform survives, because a system runs for years and gets debugged thousands of times. The "elegant" private-endpoint-everywhere design that takes a DNS PhD to troubleshoot is a liability. The boring design any on-call can reason about is an asset. Choose boring on purpose — it's a flex, not a compromise. (And when the WAF review asks you to score it, the boring one quietly wins the operational-excellence pillar that everyone else forgot to weight.)

Learn the control-plane/data-plane split until it's reflex. This is the one that makes you look senior in seconds. The moment someone says "I'm an Owner but I'm getting 403 on the blob," and you say "right, Owner is control plane — you need a data role like Storage Blob Data Reader, or you're falling back to the account key" — you've just diagnosed in one breath what others debug for an hour. It's not because you're smarter. It's because you hold one distinction they never internalized. Collect distinctions like that. They're the whole signal.

Write the ADR even when nobody asks. Here's a career hack that feels like overhead and pays off enormously: when you make a real architecture call, write the one-pager — context as numbers on the five pillars, the decision, the alternatives, the deciding pillar, and what would change your mind. Do it even for decisions nobody requested documentation for. Six months later when someone asks "why are we single-region?", you don't defend it from memory and emotion — you point at the doc and say "here was the composed SLA and the cost; have the numbers changed?" You become the person whose decisions are legible. That person gets trusted with bigger decisions. That's the ladder, and it's quieter and surer than chasing another badge.

The honest truth about this role's difficulty: it's hard because it's wide, not because any one piece is deep beyond reach. Nobody is born knowing ARM and Entra and networking and Service Bus and Durable Functions. The people who got here didn't have bigger brains; they had a framework — one control plane, one address space, five pillars, two SLA formulas, a three-term bill — that let them absorb each new service as "oh, this is the messaging slot, here's its data plane, here's how it scores on the pillars." That framework is this entire phase. Internalize it and the other 15 phases stop being 15 unrelated services and become one platform you already understand.

One more thing, and it's the most important. You don't have to be the smartest person in the room to be the principal in the room. You have to be the one who turns vague fear — "will this scale? is this secure? what will it cost?" — into specific, answerable questions with numbers attached. Calm comes from numbers. When the room is panicking about "can this survive a region outage," and you say "the data layer can't replicate that far without breaking consistency, so active/active is off the table; here's the multi-AZ that gets us 99.99 % and here's the egress cost if we want DR to the paired region" — you're not smarter than them. You just did the arithmetic they were too anxious to do. That's leadership in this job. It's learnable. It starts with the lab in this phase.

Go build the modeler. Then come find me in Phase 01 — that's where we make the deployment graph and idempotency click, the stuff everyone name-drops and quietly doesn't understand.

— your brother 👨🏻

Lab 01 — WAF Tradeoff Modeler

Phase: 00 — The Azure Principal: Control Plane, Well-Architected & Tradeoffs Difficulty: ⭐⭐☆☆☆ (the arithmetic is easy; the judgment is ⭐⭐⭐⭐⭐) Time: 3–4 hours

Round-1 of an Azure architecture interview — and every design review you will ever run — is back-of-envelope arithmetic worn as architecture. This lab builds the calculator: parse an ARM resource ID and classify its scope, compose SLAs (serial multiply, redundant combine), turn availability into downtime minutes/month, model a monthly bill, and run a weighted Well-Architected tradeoff resolver whose entire point is that the same two architectures flip winners when the workload's pillar weights change — which is why "best architecture" is a meaningless phrase.

What you build

• parse_resource_id / ResourceId — turn the canonical ARM URI /subscriptions/{s}/resourceGroups/{rg}/providers/{ns}/{type}/{name} into a frozen dataclass; support subscription-, resource-group-, and resource-level IDs and nested types (virtualNetworks/vnet/subnets/sub); reject malformed input with ValueError.
• resource_scope_level — classify any ID as subscription / resourceGroup / resource — the unit of RBAC, Policy, tagging, and locks, and the first move in every "who can touch this?" question.
• serial_sla / redundant_sla / downtime_minutes_per_month — the SLA arithmetic a principal does before the design review: serial dependencies multiply (∏ Aᵢ), redundant copies combine (1 − ∏(1 − Aᵢ)), and (1 − A) × 43200 is the minutes/month that make a nines-target real.
• monthly_cost_usd — a first-order bill (compute hours + storage GB-month + egress GB) so you can see which term dominates before you optimise the wrong one.
• Architecture (five WAF pillars) + resolve_tradeoff — score two designs under the workload's weights and return the winner and the deciding pillar (the one sentence for your ADR).

Key concepts

Files

Run

pip install -r requirements.txt
pytest test_lab.py -v                       # against your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # against the reference
python solution.py                          # the worked example

Success criteria

• All tests pass against your implementation (and LAB_MODULE=solution).
• You can explain why parse_resource_id rejects an odd number of path segments (ARM IDs are key/value pairs — a dangling key has no value).
• You can explain why a nested ID's resource_type is virtualNetworks/subnets while its name is the leaf (snet-app), not the parent vnet.
• You can explain why test_serial_sla_exact asserts three 99.9 % services compose to 0.997002999 (≈ 3× the downtime of one), and why two 99 % redundant copies give 0.9999.
• You can explain why test_weights_flip_the_winner is the whole point of the lab.
• Given any "design X on Azure" prompt you can, in two minutes, name the dominant pillar, write a serial+redundant SLA estimate, and a monthly bill on paper.

How this maps to real Azure

Limits of the miniature (be honest in the interview): availabilities are not independent (a region outage takes correlated copies down together, so redundant_sla over-estimates real multi-region availability); real bills add reserved-instance and savings-plan discounts, request/transaction costs, and tiered egress; and a WAF score is a structured conversation, not a number you can sum mechanically across every workload.

Extensions (build these on your own subscription)

• Add availability_zone_sla(component_sla, zones, correlation) that introduces a correlation factor so redundant copies aren't treated as fully independent — the honest version of the multi-region math.
• Add egress_cost_tiered(gb) with Azure's banded egress pricing (first 100 GB free, then banded) and show how a chatty cross-region call pattern dominates the bill.
• Extend resolve_tradeoff to N architectures returning a ranked list with the pairwise deciding pillar — then defend the ranking against a friend who changes the weights.
• Write the matching ARM/Bicep: a Microsoft.Network/virtualNetworks/subnets resource and confirm az resource show --ids returns the same nested ID your parser produced.
• Emit an ADR from a TradeoffResult (context = the weights, decision = the winner, consequence = the pillar you traded away) using the template in WARMUP.md Ch. 7.

Interview / resume

• Talking points: "Walk me through what happens between az ... create and the resource existing." "Why is Owner not enough to read a blob?" "Compose the SLA of a web app calling SQL calling Key Vault — show me the number." "Defend single-region over multi-region with numbers."
• Resume bullet: Built a Well-Architected decision-support model that parses ARM resource-ID scopes, composes serial/redundant SLAs into downtime budgets, estimates monthly cost, and produces explainable, weight-driven architecture tradeoffs for design reviews and ADRs.

Phase 01 — Azure Resource Manager & the Deployment Engine

Difficulty: ⭐⭐⭐☆☆ (the algorithm) → ⭐⭐⭐⭐⭐ (the judgment about modes, idempotency, and async) Estimated Time: 1 week (12–18 hours) Prerequisites: Phase 00 (control plane vs data plane, the resource-ID shape, the Well-Architected lens). Comfortable reading JSON. No live Azure subscription required for the lab.

Why This Phase Exists

Phase 00 established the single most important sentence in this entire track: every write in Azure is one idempotent PUT to a resource ID, routed through Azure Resource Manager. This phase opens that sentence up and builds the machine inside it.

Here is the thing almost nobody can actually explain in an interview, even people who've used Azure for years: when you click "Create" in the portal, or run terraform apply, or az deployment group create, what literally happens? They can name the services. They cannot draw the request pipeline (authenticate → authorize → validate → route to the resource provider), they cannot explain how ARM knows to build the virtual network before the network interface that sits in it, and they go pale when you ask why re-running the same deployment is safe but --mode Complete once deleted their production database.

That last one is not hypothetical. The Complete-mode story is a real, recurring outage shape: an engineer deploys a template that's missing a resource someone added by hand, in Complete mode, and ARM — doing exactly what it was told — tears that resource down to make the resource group match the template. The blast radius is "everything in the RG not in this file." Understanding why that's the defined behavior (not a bug) is the difference between fearing the tool and wielding it.

So this phase makes you build the deployment engine — the dependency graph, the topological scheduler, the idempotent PUT, the mode logic, the what-if diff — because once you've implemented the algorithm, ARM stops being magic and Terraform's plan engine (Phase 02), CloudFormation, Pulumi, and Bicep all reveal themselves as the same graph algorithm with different front-end syntax. This is the keystone phase for all of IaC.

What "Principal-Level" Means Here

A senior engineer writes a working ARM/Bicep template and runs the deploy. A principal understands the engine well enough to:

• Predict the deploy order from the template without running it — by reading the implicit reference()/resourceId() edges, not just the dependsOn they typed.
• Debug a stuck or failed deployment by reasoning about provisioningState, the async-operation poll, and which resource in the graph is blocking its dependents.
• Choose the deployment mode deliberately and explain the blast radius of each — and never reach for Complete mode without a reason and a what-if.
• Defend idempotency as an architecture property: why CI can re-run the same template on every commit, why drift converges, and where idempotency quietly breaks (a resource with a generated name, a reference() to something deleted out-of-band).
• Read a what-if diff like a code review and catch the Delete nobody intended before it ships.

Concepts

• The ARM control-plane request pipeline — every write is a REST call to …/providers/{ns}/{type}/{name}?api-version=…; ARM authenticates it (validates the Entra JWT), authorizes it (RBAC Actions − NotActions, then deny assignments win), validates it (Azure Policy can deny/modify/append), and routes it to the resource provider that owns the type. Get this pipeline cold; it's the spine of P03, P04, and P05.
• api-version — every RP exposes a dated contract; the request must name one, and the version pins the request/response schema. New properties only exist on new versions.
• Asynchronous operations — most provisioning is long-running: ARM returns 201/202 with a provisioningState (Accepted → Running → Succeeded|Failed) and an Azure-AsyncOperation/Location URL the client polls until terminal. "The deploy is done" means the poll reached Succeeded, not that the PUT returned.
• ARM template anatomy — $schema, contentVersion, parameters, variables, resources, outputs, and the template functions (resourceId(), reference(), concat(), parameters(), [...] expressions) the engine evaluates at deploy time.
• The dependency graph — the heart of it. Edges come from explicit dependsOn and the implicit dependencies ARM infers when one resource's property reference()s or resourceId()s another. ARM topologically orders the graph, deploys independent resources in parallel, and detects cycles.
• Idempotency — PUT of desired state (not an imperative command) means re-applying converges: Create if absent, Modify if drifted, NoChange if identical. This is the property the entire IaC discipline stands on.
• Deployment modes — Incremental (default; adds/updates, leaves un-templated resources alone) vs Complete (makes the RG match the template exactly — deletes the rest). The foot-gun, and the cleanup tool.
• What-if — the dry-run diff that classifies every resource as Create/Modify/Delete/NoChange/Ignore without touching anything. Your PR-time safety net.
• Bicep — a typed DSL that transpiles to ARM JSON; same engine, symbolic references (so dependencies are inferred from vnet.id, not hand-written dependsOn), modules, loops, and far less ceremony.
• Deployment stacks & deny-settings — a stack manages a set of resources as a unit, with managed deletion (drop a resource from the stack → it's deleted) and deny-settings that block out-of-band changes (drift protection at the control plane).

Labs

Lab 01 — ARM Deployment Engine (flagship, implemented)

→ Lab folder: lab-01-arm-deployment-engine/

Integrated-Scenario Suggestions (carried through the whole track)

The phases compound. Keep these in mind as you build — each plugs the deployment engine into a larger system:

1. Landing-zone bootstrap — a deployment that lays down a management-group subtree, policy assignments, and a hub VNet. The order (policy before the resources it governs; networking before the workloads) is pure dependency-graph reasoning. (→ P05)
2. Idempotent CI deploy — a pipeline that runs what-if on PR (the diff is the review) and deploy on merge, safely re-applying on every commit because the PUT is idempotent. (→ P08)
3. Drift control — a deployment stack with denySettings that blocks out-of-band portal edits; reconcile detected drift by re-deploying desired state. (→ P05, P02 for the Terraform-state version of the same idea)
4. Multi-resource provisioning order — a VM that needs a NIC that needs a subnet that needs a VNet, plus a Key Vault the VM reads a secret from — the canonical "why did this deploy in that order" graph. (→ P06, P12)
5. Cross-RP routing — a single template touching Microsoft.Storage, Microsoft.Network, and Microsoft.Compute, each routed to a different resource provider, each with its own api-version and async behavior. (→ all later phases)

Guides in This Phase

• HITCHHIKERS-GUIDE.md — the 30-minute orientation; read first
• WARMUP.md — the full primer; read slowly
• BROTHER-TALK.md — the candid version

Key Takeaways

• Everything in Azure is a PUT through ARM, and the deploy engine is, at its core, a dependency-graph topological scheduler over idempotent PUTs. Build that once and every IaC tool — Bicep, Terraform, Pulumi, CloudFormation — becomes a front-end you already understand.
• Implicit dependencies are real edges. reference() and resourceId() create ordering constraints you never typed; the engine (and you) must find them.
• Idempotency is the load-bearing property of IaC. Desired-state PUTs converge, which is why you can re-run a template on every commit and why drift is recoverable.
• Mode is a blast-radius decision. Incremental is additive and safe-ish; Complete makes the RG exactly the template — and deletes the rest. Never run Complete without a what-if you've read.
• Async is the default. "The PUT returned" is not "the resource is ready"; the provisioningState poll is. Debugging stuck deploys lives here.

Deliverables Checklist

• Lab 01 implemented; all 25 tests pass against solution.py and your lab.py
• You can draw the ARM request pipeline (authenticate → authorize → validate → route) from memory
• Given a template, you can list its implicit + explicit edges and write the deploy order without running it
• You can explain Incremental vs Complete and the exact blast radius of each
• You can explain why PUT is idempotent and trace a re-deploy to all-NoChange
• You can describe the provisioningState async-poll loop

Warmup Guide — Azure Resource Manager & the Deployment Engine

Zero-to-principal primer for Phase 01. We start from "what even is ARM" and end with you able to draw the control-plane request pipeline, predict a deployment's order from implicit references alone, explain the Complete-mode foot-gun, and reason about the async-operation poll — the things interviewers probe and incidents turn on. Every mechanism is opened under the hood: the pipeline, the graph algorithm, the idempotent PUT, the diff.

Table of Contents

• Chapter 1: What ARM Is — The One API in Front of Everything
• Chapter 2: The Control-Plane Request Pipeline
• Chapter 3: ARM Template Anatomy
• Chapter 4: The Dependency Graph — Explicit and Implicit Edges
• Chapter 5: Topological Ordering and Cycle Detection
• Chapter 6: Idempotency — The PUT That Converges
• Chapter 7: Deployment Modes — Incremental vs Complete
• Chapter 8: What-If — The Dry-Run Diff
• Chapter 9: Asynchronous Operations and provisioningState
• Chapter 10: Bicep — The Typed Transpiler
• Chapter 11: Deployment Stacks and Deny-Settings
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: What ARM Is — The One API in Front of Everything

From zero. When you create anything in Azure — through the portal, the az CLI, PowerShell, a Terraform run, a GitHub Action — you are not talking to "the storage service" or "the VM service" directly. You are talking to one front door: Azure Resource Manager (ARM). It is a single, global REST API and deployment engine that sits in front of every Azure resource.

What it is. ARM is the control plane: the system that decides what resources exist, what their configuration is, and who may manage them. Every management operation — create, read, update, delete, tag, lock, assign a role — is an HTTP request to an ARM endpoint whose URL is the resource's canonical ID:

PUT https://management.azure.com
      /subscriptions/{subId}
      /resourceGroups/{rg}
      /providers/{namespace}/{type}/{name}
      ?api-version=2023-01-01

That hierarchy — subscription → resource group → provider/type/name — is the resource ID, and it's the unit of RBAC scope, Policy scope, tagging, and locks (Phase 00 drilled this). Memorize the shape; you'll read it in error messages for the rest of your career.

Why it exists. Before ARM (the "classic"/ASM era), every Azure service had its own API, its own auth, its own deployment story. There was no single place to apply governance, no atomic "deploy these ten things together," no uniform RBAC. ARM unifies all of it: one auth model (Entra), one authorization model (RBAC + Policy), one deployment grammar (templates), and one idempotent verb (PUT desired state). Every hard problem in this curriculum — identity, governance, networking, secrets — plugs into this pipeline.

The control-plane / data-plane split (recall from P00). ARM is the control plane: it creates the storage account, it does not read your blob. Reading the blob is the data plane — a different endpoint ({account}.blob.core.windows.net), different auth (a SAS token or a data-plane RBAC role), different throttling, different SLA. A PUT to create Key Vault goes through ARM; getting a secret out of it does not. Conflating the two is the single most common Azure misconception; keep them in separate boxes in your head.

Common misconception. "The portal does its own thing." No — the portal is just a JavaScript client of the same ARM REST API you could call with curl. There is no privileged back channel. Everything the portal can do, ARM can do, and so can you with a token.

Chapter 2: The Control-Plane Request Pipeline

From zero. A request doesn't go straight from your az command to the disk that stores your VM. It passes through a pipeline of gates, in a fixed order. This is the single most valuable diagram in the phase:

                        ┌─────────────────────────────────────────────┐
  az / portal / TF ───▶ │              Azure Resource Manager          │
   (a signed request)   │                                             │
                        │  1. AUTHENTICATE  validate the Entra JWT     │
                        │       (signature, iss, aud, exp)             │
                        │  2. AUTHORIZE     RBAC: Actions − NotActions  │
                        │       then DENY assignments win              │
                        │  3. VALIDATE      Azure Policy effects        │
                        │       (deny / modify / append / audit)       │
                        │  4. ROUTE         to the resource provider    │
                        └──────────────────────┬──────────────────────┘
                                               │
                                               ▼
                        ┌─────────────────────────────────────────────┐
                        │   Resource Provider (Microsoft.Storage, …)   │
                        │   does the idempotent PUT, returns           │
                        │   provisioningState (async)                  │
                        └─────────────────────────────────────────────┘

Walk each gate under the hood:

1. Authenticate. Your token is a JWT issued by Microsoft Entra ID. ARM verifies the signature against Entra's published keys (JWKS, selected by the kid in the header), and checks iss (issuer = your tenant), aud (audience = https://management.azure.com), and the validity window (nbf ≤ now ≤ exp). No valid token → 401. You build this validator in Phase 03; here, just know it's gate one.
2. Authorize. ARM computes your effective permissions: the union of Actions across all your role assignments at this scope and every ancestor scope, minus NotActions. Then — deny assignments win — any explicit deny overrides an allow. No matching allow, or a matching deny → 403. You build this in Phase 04.
3. Validate. Azure Policy evaluates the request against assigned rules. A deny effect blocks the write (RequestDisallowedByPolicy); modify/append mutate the request (e.g. add a required tag) before it proceeds; audit just records non-compliance. Policy runs at create/update time, here, on the control-plane path. You build this in Phase 04 too.
4. Route. Only now does ARM hand the request to the resource provider — the service (Microsoft.Storage, Microsoft.Network, Microsoft.Compute) that owns this resource type and implements its actual CRUD. The RP must be registered on the subscription, or you get MissingSubscriptionRegistration.

Why this order. It is fail-closed and cheapest-check-first: reject the unauthorized caller before spending a cycle on policy, reject the policy-violating request before bothering the resource provider. Identity is the perimeter; it's gate one for a reason.

Production significance. Almost every "I can't deploy" ticket is one of these four gates: a 401 (bad/expired token), a 403 (RBAC or a deny assignment — and "I'm Owner" doesn't help if a deny assignment from a deployment stack or managed app is blocking you), a RequestDisallowedByPolicy, or a MissingSubscriptionRegistration. Knowing the pipeline is the triage tree.

Common misconception. "RBAC and Policy are the same thing." No. RBAC answers who may perform this action; Policy answers is this resource configuration allowed. You can have permission to create a storage account (RBAC says yes) and still be blocked because Policy requires HTTPS-only and your request didn't set it (deny). Two different gates, both must pass.

Chapter 3: ARM Template Anatomy

From zero. A template is a JSON document describing desired state — the resources you want to exist, declaratively, not the steps to create them. You hand ARM the "what"; ARM figures out the "how" and the "in what order." The top-level shape:

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": { "location": { "type": "string", "defaultValue": "eastus" } },
  "variables":  { "saName": "[concat('stg', uniqueString(resourceGroup().id))]" },
  "resources": [
    {
      "type": "Microsoft.Storage/storageAccounts",
      "apiVersion": "2023-01-01",
      "name": "[variables('saName')]",
      "location": "[parameters('location')]",
      "sku": { "name": "Standard_LRS" },
      "kind": "StorageV2"
    }
  ],
  "outputs": { "saId": { "type": "string", "value": "[resourceId('Microsoft.Storage/storageAccounts', variables('saName'))]" } }
}

Each section, under the hood:

• $schema — the template-language version; lets the engine and tooling validate the document shape.
• parameters — the typed inputs callers supply at deploy time (with optional defaults, allowed values, constraints). The seam between the template and the environment.
• variables — computed, reusable values; pure functions of parameters and other variables, resolved at the start of the deployment.
• resources — the heart: the array of desired resources, each with a type, apiVersion, name, and properties.
• outputs — values returned after deploy (a connection string, a resource ID) for the caller or a parent/nested deployment to consume.

The [...] expressions and template functions. Anything in [ ] is an expression ARM evaluates at deploy time. The functions that matter for this phase:

• resourceId('type', 'name') — builds a resource's full ID. Crucially, when you use it to point one resource at another, ARM infers an implicit dependency (Chapter 4).
• reference('name') — reads runtime properties of another resource (its endpoints, its generated keys). Because it must read a resource that exists, it also creates an implicit dependency.
• dependsOn — the explicit ordering you write by hand: "deploy these first."
• parameters(), variables(), concat(), uniqueString() — value plumbing.

Why declarative. Because declarative + idempotent (Chapter 6) is what makes the file re-runnable. An imperative script ("create X, then create Y") run twice creates two X's; a declarative desired-state template run twice converges to the same single X. That property is the whole reason IaC exists.

Common misconception. "Order in the resources array is the deploy order." It is not. ARM ignores array position and deploys in dependency order (Chapter 5). Resources with no dependency relationship deploy in parallel.

Chapter 4: The Dependency Graph — Explicit and Implicit Edges

This is the chapter the lab is built on, and the one most engineers get wrong.

From zero. ARM cannot deploy a network interface before the subnet it lives in, or a VM before its NIC. It learns these ordering constraints by building a directed graph: a node per resource, an edge from a resource to each resource it depends on. Edges come from two sources:

1. Explicit edges — dependsOn. You literally list the names:

{ "type": "…/virtualMachines", "name": "vm1", "dependsOn": ["nic1"] }

2. Implicit edges — the ones you didn't type. When a resource's property references another resource via reference() or resourceId(), ARM infers the dependency automatically:

{
  "type": "…/networkInterfaces", "name": "nic1",
  "properties": {
    "ipConfigurations": [{
      "properties": { "subnet": { "id": "[resourceId('…/subnets', 'vnet1', 'subnet1')]" } }
    }]
  }
}

You never wrote "dependsOn": ["vnet1"] — but ARM adds that edge, because resourceId names vnet1, and you can't get a subnet's ID until the VNet exists. This is why a correct template often has almost no dependsOn: the references already encode the order.

Under the hood — what the lab does. The engine scans every string value in a resource's properties (recursing into nested dicts and lists, because real properties nest deeply) for the pattern reference('NAME') or resourceId('NAME'), pulls out NAME, and unions those with the explicit dependsOn list:

deps(resource) = set(resource.dependsOn) ∪ { NAME : reference('NAME') or resourceId('NAME') ∈ properties }

A resource never depends on itself (ARM drops a self-edge; a self-edge would be a trivial cycle). And every referenced name must resolve to a resource in the template — a reference to a name that isn't there is ARM's deploy-time InvalidTemplate: the resource '…' is not defined in the template error, which the lab raises as a ValueError.

Production significance. When you over-specify dependsOn (adding edges the references already imply), you don't break correctness — but you can serialize a deployment that could have run in parallel, making it slower. When you under-specify (a hidden ordering the references don't capture, e.g. an eventual-consistency timing dependency), you get flaky deploys. The principal reads the graph, not the dependsOn list.

Common misconception. "I need dependsOn everywhere." Usually the opposite: prefer references (Bicep makes this automatic and symbolic). Reach for explicit dependsOn only for an ordering ARM can't infer from a reference.

Chapter 5: Topological Ordering and Cycle Detection

From zero. Given the dependency graph, the engine must produce an order in which every resource comes after all of its dependencies. That ordering is a topological sort, and it exists iff the graph is a DAG (directed acyclic graph). A cycle — A dependsOn B, B dependsOn A — has no valid order and is un-deployable.

Under the hood — Kahn's algorithm. The lab implements the classic queue-based topo sort:

1. in_degree[n] = number of n's dependencies          # how many edges point OUT to deps
2. ready = every node with in_degree 0                 # nothing left to wait for
3. while ready is non-empty:
       n = pop the SMALLEST-named node from ready       # deterministic tie-break
       emit n
       for each m that depends on n:
           in_degree[m] -= 1
           if in_degree[m] == 0: add m to ready
4. if we emitted fewer nodes than the graph has → a CYCLE remains

Two design choices in that pseudocode are worth their own paragraph:

Determinism via tie-break. At any step, several nodes may be "ready" at once (all their deps are done). Real ARM deploys all of them in parallel — order among them is undefined. But an undefined order is untestable, so the lab always pops the alphabetically smallest ready node. That gives one canonical, reproducible order: the same template always yields the same plan. Internalize that the determinism is a lab affordance for testability — production ARM parallelises ties. (The Extensions section shows how to emit parallel "waves" instead.)

Cycle detection for free. If the graph has a cycle, the nodes in the cycle never reach in_degree 0 (each is waiting on the next), so they're never emitted. After the loop, if len(emitted) < len(graph), the leftover nodes are the cycle — the lab raises ValueError("cycle: ...") naming them. This is the same mechanism ARM uses to reject a circular dependsOn.

A worked trace. Graph: vnet1 → ∅, subnet1 → {vnet1}, nic1 → {subnet1}, vm1 → {nic1, storage1}, storage1 → ∅.

Order: storage1 → vnet1 → subnet1 → nic1 → vm1. Every resource lands after its deps.

Complexity. \( O(V + E) \) — linear in resources plus edges. ARM deployments are small (tens to low hundreds of resources), so this is instant; the value is correctness, not speed.

Common misconception. "Topo sort gives one answer." A DAG generally has many valid topological orders; the tie-break picks one. What's invariant is the partial order (dependencies before dependents), not the total order.

Chapter 6: Idempotency — The PUT That Converges

From zero. An operation is idempotent if doing it twice has the same effect as doing it once. PUT (set this resource to this desired state) is idempotent; POST (create a new thing) generally is not. ARM resources are deployed with PUT, and that single fact is what makes Infrastructure-as-Code work.

Under the hood — the three outcomes. For each resource, the resource provider compares the desired state in the template against the actual state of the resource and converges:

The lab's deploy does exactly this against a state dict (name → properties): absent → Create, present-but-different → Modify, identical → NoChange.

Why it matters — the convergence property. Because a PUT of already-matching state is a NoChange, you can re-run the same template as many times as you want and it stays correct. This is the property your CI/CD depends on (Phase 08): every commit can re-apply the full template; nothing duplicates; drift gets reconciled back to desired state. Run the lab's idempotency test and watch a second deploy of the same template return all NoChange — that is IaC's foundational guarantee, in code.

$$ \text{deploy}(T, \text{deploy}(T, S).\text{state}) ;\Rightarrow; \text{every action} = \texttt{NoChange} $$

Where idempotency quietly breaks (principal territory). It is not magic:

• A resource with a generated/random name (uniqueString(newGuid()) done wrong) makes every deploy Create a new one — non-idempotent by construction.
• A reference() to a resource someone deleted out-of-band fails the whole deploy.
• Some properties are immutable — changing them isn't a Modify, it forces a delete-and-recreate (this is Terraform's ForceNew; you'll build it in Phase 02).
• A PATCH-style partial update can behave differently from a full PUT of desired state.

Common misconception. "Idempotent means it does nothing the second time." No — it means the end state is the same. The second run still checks every resource; it just finds them already correct and reports NoChange. "Same result," not "no work."

Chapter 7: Deployment Modes — Incremental vs Complete

From zero. When you deploy a template to a resource group, you pick a mode that decides what happens to resources in the RG that aren't in your template.

Incremental (the default). ARM adds the resources in the template and updates the ones that exist — and leaves everything else alone. Your template is a superset instruction: "make sure these exist and look like this." Resources in the RG not mentioned in the template are untouched.

Complete. ARM makes the resource group match the template exactly — which means it deletes every resource in the RG that is not in the template. Your template becomes the whole truth of the RG.

RG before:   [ storage1, vnet1, rogue_db ]
Template:    [ storage1, vnet1 ]

Incremental → RG after: [ storage1, vnet1, rogue_db ]   (rogue_db untouched)
Complete    → RG after: [ storage1, vnet1 ]             (rogue_db DELETED)

The lab models this precisely: in Complete mode, any name in state but not in the template is given a Delete action and removed from the final state; in Incremental mode it's never touched.

Production significance — the foot-gun. Complete mode is how production resources get deleted by a "routine" deploy. The classic incident: someone added a resource by hand (or another team's template owns it), you deploy your template — which doesn't mention it — in Complete mode, and ARM dutifully deletes it. This is not a bug; it is the defined contract. The defenses: (1) Incremental is the default — stay there unless you have a reason; (2) always run what-if first (Chapter 8) and read the Deletes; (3) use resource locks (CanNotDelete) on anything precious; (4) prefer deployment stacks (Chapter 11) for managed, intentional deletion.

When Complete is the right tool. When you want the RG to be exactly the template — e.g. an ephemeral environment you own end-to-end, or cleaning up resources a previous template version created and the new one dropped. Intentional, scoped, what-if'd.

Common misconception. "Complete redeploys everything from scratch." No — it doesn't re-create the resources that match; those are still NoChange/Modify. It only adds the delete behavior for un-templated extras. Complete = Incremental + delete-the-rest.

Chapter 8: What-If — The Dry-Run Diff

From zero. Before you let ARM change anything, you want to see what it would do. What-if is ARM's dry run: it computes the diff between your template (desired) and the current state (actual) and reports, per resource, a change type — without deploying anything.

The change classification mirrors the lab's Change objects:

Under the hood. What-if runs the same classification logic as deploy — build the graph, order it, compare desired vs actual per resource — but writes the results to a list instead of to state. The non-negotiable property: it must not mutate anything. The lab asserts this explicitly: what_if simulates against a throwaway copy so the real state is byte-identical before and after. A "dry run" that mutates state is a bug, and a dangerous one.

Production significance. What-if is your PR-time safety net and the thing that catches the Complete-mode Delete before it ships. The principal workflow (Phase 08): what-if on the pull request (the diff is the review — reviewers read the Creates, Modifies, and especially the Deletes), then deploy on merge. A what-if that surfaces an unexpected Delete has paid for itself a hundred times over.

Common misconception. "What-if is exact." It's a strong predictor, not a contract — some changes only manifest when the resource provider actually evaluates the request (api-version quirks, computed defaults). Read it as "here's what ARM intends," and treat surprising Deletes as a stop sign.

Chapter 9: Asynchronous Operations and provisioningState

From zero. Most Azure resources don't provision instantly — a VM, an AKS cluster, a SQL database take minutes. So ARM operations are asynchronous: the PUT returns quickly with "I've accepted this, it's in progress," and you poll until it's done.

Under the hood — the poll loop. The PUT returns 201 Created or 202 Accepted with:

• a provisioningState on the resource: Accepted → Creating/Running → Succeeded (or Failed/Canceled), and
• an Azure-AsyncOperation (or Location) header — a URL the client GETs repeatedly, honoring the Retry-After interval, until the state is terminal.

client: PUT resource …                       → 202 Accepted, Azure-AsyncOperation: <url>
client: GET <url>   (wait Retry-After)        → { "status": "InProgress" }
client: GET <url>   (wait Retry-After)        → { "status": "InProgress" }
client: GET <url>   (wait Retry-After)        → { "status": "Succeeded" }   ← terminal

az and Terraform hide this loop behind their progress spinner, but it's what they're doing. "The deploy finished" means the poll reached a terminal state — not that the PUT returned.

Why it matters — ordering and debugging. Because the engine deploys dependencies before dependents (Chapter 5), it must wait for a dependency's poll to reach Succeeded before starting its dependents — otherwise the NIC would try to attach to a subnet that's still Creating. And when a deploy is "stuck," the principal looks at which resource's provisioningState is non-terminal and why (a quota error, a policy modify, a dependency that Failed), rather than staring at the spinner. The lab abstracts the poll away (resources "complete" instantly) so the graph logic is naked; the Extensions add a provisioningState state machine so you can model the real loop.

Common misconception. "If the CLI command returned 0, the resource is ready." Only if the CLI waited for the poll (most do by default; --no-wait does not). With --no-wait you've only submitted the operation — the resource may still be Creating.

Chapter 10: Bicep — The Typed Transpiler

From zero. ARM JSON is verbose, expression-heavy, and painful to write by hand. Bicep is a domain-specific language that compiles ("transpiles") down to ARM JSON. It is not a different engine — bicep build emits the same template the ARM deployment engine consumes. Same pipeline, same graph, same modes; nicer front-end.

The same storage account, in Bicep:

param location string = 'eastus'

resource sa 'Microsoft.Storage/storageAccounts@2023-01-01' = {
  name: 'stg${uniqueString(resourceGroup().id)}'
  location: location
  sku: { name: 'Standard_LRS' }
  kind: 'StorageV2'
}

output saId string = sa.id

What Bicep buys you, under the hood:

• Symbolic references and automatic dependencies. You write sa.id, not resourceId(...). Because sa is a symbol, Bicep infers the dependency graph from the symbol references — you almost never write dependsOn. (This is Chapter 4's implicit edges, made first-class.)
• Types and IntelliSense. The compiler knows each resource type's schema for the chosen api-version and catches typos and invalid properties before deploy.
• Modules, loops, conditionals. First-class module, for, and if instead of copy-paste and copy blocks.

Production significance. Bicep is Microsoft's recommended authoring experience for native Azure IaC, and the what-if/modes/stacks story is identical (it's the same engine). But — and this is the principal point — it's still ARM underneath. When a Bicep deploy fails, you debug the ARM error, the ARM graph, the ARM provisioningState. Bicep is ergonomics; the engine you built in the lab is the truth.

Common misconception. "Bicep is a competitor to ARM." It compiles to ARM. "Bicep vs Terraform" is a real comparison (different engines, different state models — Phase 02); "Bicep vs ARM" is not (same engine).

Chapter 11: Deployment Stacks and Deny-Settings

From zero. A normal deployment is fire-and-forget: ARM applies it and forgets which resources came from which template. A deployment stack is a resource that remembers the set of resources it manages and governs their lifecycle as a unit.

Two principal features, under the hood:

• Managed deletion. Drop a resource from the stack's template and re-deploy → the stack deletes that resource (you choose detachResources to leave it, or deleteResources to tear it down). This is a safer, scoped version of Complete mode: the blast radius is "resources this stack manages," not "everything in the RG."
• Deny-settings (drift protection). A stack can apply a deny assignment to its managed resources — denySettingsMode: denyDelete or denyWriteAndDelete — that blocks out-of-band changes at the control plane. Someone tries to delete or edit a stack-managed resource in the portal → ARM denies it (deny assignments win, Chapter 2). The desired state is protected by ARM itself, not by hope.

Production significance. This closes the IaC loop: not just "deploy desired state," but "keep the real world equal to desired state and reject drift." It's the control-plane analog of Terraform's drift detection (Phase 02) — except enforced by ARM rather than detected after the fact. For a landing zone (Phase 05) where hundreds of engineers share an estate, deny-settings are how you stop a well-meaning portal click from deleting shared infrastructure.

Common misconception. "A deployment stack is just a deployment." A deployment is an event (it happened, then it's history). A stack is a resource (it persists, tracks its managed set, enforces deny-settings, and governs deletion on the next update).

Lab Walkthrough Guidance

Lab 01 — ARM Deployment Engine, suggested order (each step has tests waiting):

1. _implicit_refs_in_value + the regex (Ch. 4). Write the regex that captures NAME from reference('NAME') and resourceId('NAME'). Recurse into dicts and lists — test_extract_implicit_resourceid_and_nested proves nested properties count.
2. extract_dependencies (Ch. 4). Union dependsOn with the implicit refs; raise on a missing name; drop the self-edge. Tests: explicit+implicit union, self-reference dropped, missing-name raises.
3. build_dependency_graph (Ch. 4). Map name → deps; raise on a duplicate name and on any reference to a name not in the template. Tests cover both unknown dependsOn and unknown implicit references.
4. topological_order (Ch. 5). Kahn's algorithm, tie-break by name for determinism, raise ValueError("cycle: ...") if not every node is emitted. Tests: order respects deps, deterministic tie-break, cycle + self-cycle raise.
5. deploy (Ch. 6–7). Validate the mode; topo-order; classify Create/Modify/NoChange against a copy of state (don't mutate the caller's dict); in Complete mode, Delete un-templated extras (sorted). Tests: all-Create from empty, idempotent re-deploy = all-NoChange, Modify on changed props, Incremental leaves extras, Complete deletes them, bad mode raises, state not mutated.
6. what_if (Ch. 8). Same classification, returned as Change objects, mutating nothing. Tests assert the input state is byte-identical after the call.

Run pytest test_lab.py -v after each step; commit when a section goes green. Then python solution.py to see the worked plan: graph, deploy order, all-Create, idempotent re-deploy, a Modify via what-if, and a Complete-mode Delete.

Success Criteria

You are ready for Phase 02 when you can, from memory:

1. Draw the ARM request pipeline: authenticate (JWT) → authorize (RBAC, deny wins) → validate (Policy) → route (resource provider) — and name the HTTP error each gate throws.
2. Given a template, list its implicit (reference/resourceId) and explicit (dependsOn) edges and write a valid deploy order.
3. Run Kahn's algorithm on paper and explain how the leftover-nodes test detects a cycle.
4. State why an ARM PUT is idempotent and trace a re-deploy to all NoChange.
5. Explain Incremental vs Complete and the exact blast radius of each — and why what-if is the defense.
6. Describe the provisioningState async-poll loop and why dependents wait for a dependency's Succeeded.
7. Explain that Bicep transpiles to ARM and that deployment stacks + deny-settings protect against drift.

Interview Q&A

Q: Walk me through exactly what happens when I run az deployment group create. The CLI acquires an Entra token and sends a deployment PUT to ARM. ARM authenticates the JWT (signature, iss, aud, exp), authorizes via RBAC — union of Actions minus NotActions across my assignments and inherited scopes, with deny assignments overriding — and validates against Azure Policy (which may deny or modify the request). Then ARM parses the template, builds a dependency graph from explicit dependsOn and implicit reference()/resourceId() edges, topologically orders it, and deploys independent resources in parallel — each routed to its resource provider as an idempotent PUT. Most are async, so ARM returns 202 and polls provisioningState to Succeeded before starting dependents. If I used the default Incremental mode, un-templated resources are left alone; in Complete mode they'd be deleted.

Q: I never wrote a dependsOn, but my NIC deployed after my subnet. How? Implicit dependencies. The NIC's subnet.id property uses resourceId(...) (or in Bicep, a symbolic subnet.id reference) naming the subnet. ARM infers an edge from that reference — you can't compute a subnet's ID until the VNet/subnet exists — so it orders the subnet first automatically. Most correct templates have very few hand-written dependsOn precisely because references already encode the order; explicit dependsOn is for orderings ARM can't infer from a reference.

Q: Incremental vs Complete — which is the default, and when would Complete bite you? Incremental is the default: it adds and updates the template's resources and leaves everything else in the RG untouched. Complete makes the RG match the template exactly — it deletes any resource not in the template. It bites you when something exists that your template doesn't mention — a hand-created resource, another team's resource — and you deploy in Complete mode: ARM deletes it, by design. Defenses: stay on Incremental, always run what-if and read the Deletes, lock precious resources, and prefer deployment stacks for intentional, scoped deletion.

Q: Why is an ARM deployment idempotent, and why do you care? Because resources are deployed with PUT of desired state, not imperative commands. The resource provider compares desired to actual and converges: Create if absent, Modify if drifted, NoChange if identical. So re-running the same template is safe — it doesn't duplicate anything; matching resources are NoChange. I care because it's the foundation of CI/CD: every commit can re-apply the full template, drift reconciles back to desired state, and a retry after a transient failure just resumes convergence. The classic break is a resource with a randomly generated name — that Creates a new one every run and isn't idempotent.

Q: A deployment is stuck. How do you debug it? I look at the dependency graph and the provisioningState of each resource. A "stuck" deploy is usually one resource whose async operation hasn't reached a terminal state — because of a quota limit, a policy modify/deny, a resource-provider error, or a dependency that Failed and is blocking its dependents (they correctly wait for the dependency's Succeeded). I pull the deployment operations (az deployment operation group list) to find the specific resource and its error code, rather than watching the spinner.

Q: What's the difference between ARM, Bicep, and Terraform here? ARM is the engine and the JSON template language — the control-plane API every Azure write goes through. Bicep is a typed DSL that transpiles to ARM JSON: same engine, same graph, same modes, nicer syntax with symbolic references and automatic dependencies. Terraform is a different engine (Phase 02) with its own state file, plan/apply model, and ForceNew replace semantics — it maps HCL to ARM calls via the azurerm provider but doesn't use ARM's template engine. So "Bicep vs ARM" is front-end vs engine; "Terraform vs ARM/Bicep" is two genuinely different deployment engines.

Q: How does what-if work, and why trust it? It runs the same desired-vs-actual classification as a real deploy — build the graph, compare each resource — but writes the result to a diff instead of to Azure, mutating nothing. It classifies each resource as Create/Modify/Delete/NoChange/Ignore. I trust it as a strong predictor and a safety net — especially for catching an unintended Delete before a Complete-mode deploy — but not as a hard contract, since some changes only surface when the resource provider actually evaluates the request.

References

• Microsoft Learn — Azure Resource Manager overview
• Microsoft Learn — Understand the structure and syntax of ARM templates
• Microsoft Learn — Define the order for deploying resources (dependsOn + implicit dependencies)
• Microsoft Learn — Deployment modes (Incremental and Complete)
• Microsoft Learn — ARM deployment what-if operation
• Microsoft Learn — Track asynchronous Azure operations
• Microsoft Learn — What is Bicep?
• Microsoft Learn — Azure deployment stacks
• Microsoft Learn — Azure Resource Manager resource provider registration
• Arthur B. Kahn, Topological sorting of large networks (1962) — the algorithm in Chapter 5
• The track's own CHEATSHEET.md and GLOSSARY.md

🛸 Hitchhiker's Guide — Phase 01: ARM & the Deployment Engine

Read this if: you've run az deployment or terraform apply a hundred times but couldn't draw what ARM actually does between "create" and "the resource exists." This is the compressed tour the WARMUP derives slowly. Skim it, then read the WARMUP, then build the engine.

0. The 30-second mental model

Every write in Azure is one idempotent PUT to a resource ID, through ARM. A deployment is ARM building a dependency graph from your template, topologically ordering it, and PUTting desired state to each resource provider — Create if absent, Modify if drifted, NoChange if identical. The whole engine is: graph → topo sort → idempotent PUT → poll till Succeeded. Build that and Bicep, Terraform, Pulumi, and CloudFormation are all the same algorithm wearing different syntax.

One sentence to tattoo: ARM doesn't run your steps — it reconciles your desired state, in dependency order, idempotently.

1. The request pipeline, one breath

client → ARM:  AUTHENTICATE (Entra JWT)  →  AUTHORIZE (RBAC: Actions−NotActions, deny wins)
             →  VALIDATE (Azure Policy: deny/modify/append/audit)  →  ROUTE (resource provider)
             →  RP does idempotent PUT, returns provisioningState (async)

Fail-closed, cheapest-check-first. Almost every "can't deploy" ticket is one of these four gates: 401 (token), 403 (RBAC / deny assignment), RequestDisallowedByPolicy, MissingSubscriptionRegistration.

2. The dependency graph (the part everyone gets wrong)

Edges come from two places:

• Explicit: "dependsOn": ["nic1"] — you typed it.
• Implicit: a property uses reference('x') or resourceId('x') → ARM adds an edge to x automatically. You never wrote dependsOn, but the order is enforced anyway.

A correct template often has almost no dependsOn because references already encode the order. Bicep makes this fully automatic via symbolic references (vnet.id). Reach for explicit dependsOn only for an ordering ARM can't infer.

3. Topo sort + cycles, the algorithm

Kahn's: emit any node whose deps are all emitted; repeat. A cycle = nodes that never reach in-degree 0 → un-deployable → ARM rejects it (the lab raises cycle: ...). Independent resources deploy in parallel in real ARM; the lab serializes ties by name purely so the plan is deterministic and testable. Complexity \(O(V+E)\).

4. Idempotency, the load-bearing property

PUT desired state →  absent: Create | drifted: Modify | identical: NoChange

Re-run the same template → all NoChange. That's why CI can re-apply on every commit and why drift reconciles. It breaks when a resource has a random name (Creates a new one every run) or when a property is immutable (Modify becomes destroy-and-recreate — that's Terraform's ForceNew, Phase 02).

5. Modes — the foot-gun

Complete = Incremental + delete-the-rest. The production-deletion story: deploy a template that's missing a resource, in Complete mode → ARM deletes the resource. By design, not a bug. Defense: stay Incremental, always what-if first, lock precious resources, use deployment stacks.

6. The numbers & names to tattoo on your arm

7. az one-liners you'll actually type

# Dry-run first — read the Deletes before you ship anything
az deployment group what-if -g rg -f main.bicep -p @params.json

# Deploy (Incremental is default; be explicit when it matters)
az deployment group create -g rg -f main.bicep --mode Incremental

# Complete mode — only with a what-if you've read and a reason
az deployment group create -g rg -f main.bicep --mode Complete

# Find the resource that's blocking a stuck/failed deploy
az deployment operation group list -g rg -n <deploymentName> \
  --query "[?properties.provisioningState!='Succeeded'].{res:properties.targetResource.id, state:properties.provisioningState, err:properties.statusMessage}"

# Bicep is just a transpiler — see the ARM JSON it emits
az bicep build -f main.bicep            # → main.json

# Deployment stack with drift protection (deny out-of-band delete/edit)
az stack group create -g rg --name app -f main.bicep \
  --deny-settings-mode denyWriteAndDelete --action-on-unmanage deleteResources

# Register a resource provider before first use of its types
az provider register --namespace Microsoft.ContainerService

8. War story shapes you'll relive

• "Complete mode deleted prod." → a template missing a hand-created DB, deployed Complete. ARM did exactly what it was told. Fix: Incremental + what-if + locks; never Complete blind.
• "Deploy is stuck forever." → one resource's provisioningState is non-terminal (quota, a Failed dependency blocking dependents). Pull deployment operations; find that resource.
• "It deployed in the wrong order / flaky." → a real ordering not captured by a reference; add the missing dependsOn. Or the reverse: over-dependsOn serialized a parallel deploy and made it slow.
• "It worked locally, 403 in CI." → the pipeline's identity lacks the RBAC Action, or a deny assignment from a stack/managed-app is overriding the allow. Deny wins.
• "Re-running created duplicates." → a uniqueString(newGuid())-style random name broke idempotency. Desired state must be stable to converge.

9. Vocabulary that signals you've held the pager

• Idempotent PUT — desired state, not a command; re-run converges to NoChange.
• Implicit dependency — the edge reference()/resourceId() creates without a dependsOn.
• provisioningState — the async truth; "PUT returned" ≠ "resource ready."
• Incremental vs Complete — additive vs "RG = template exactly (deletes the rest)."
• what-if — the read-only diff; the PR review and the Delete-catcher.
• Resource provider (RP) — the service that owns a type; must be registered.
• Deployment stack / deny-settings — managed deletion + control-plane drift protection.
• Transpile — Bicep → ARM JSON; same engine underneath.

10. Beginner mistakes that mark you in interviews

1. Thinking array position in resources is the deploy order (it's the graph).
2. Adding dependsOn everywhere instead of letting references imply edges.
3. Not knowing Incremental is the default — or what Complete deletes.
4. Saying "the deploy is done" when the PUT returned but provisioningState isn't terminal.
5. Calling a dry run "what-if" while mutating state — a what-if never mutates.
6. Treating idempotency as magic and getting bitten by a random resource name.
7. Confusing "Bicep vs ARM" (front-end vs engine — same thing) with "Terraform vs ARM" (genuinely different engines).

11. How this phase pays off later

• The request pipeline is the spine of P03 (the JWT/auth gate), P04 (RBAC + Policy gates), and P05 (inheritance down the MG tree).
• The dependency graph + topo sort + idempotency is exactly Terraform's plan/apply engine (P02) — you'll add ForceNew and a state file to the same algorithm.
• Idempotent re-apply is what makes the CI/CD pipeline (P08) safe to run on every commit.
• Modes, what-if, and deny-settings are how you protect a shared landing zone (P05) from drift and accidental deletion.

Now read the WARMUP slowly, then build the engine. After this, no IaC tool is a black box.

👨🏻 Brother Talk — Phase 01, Off the Record

No slides, no STAR method. This is me, your brother, telling you the things people only say after the second coffee. Read it once now, and again the week before your interview.

Listen. Almost everyone who's "good at Azure" can write a Bicep file and run a deploy. What almost nobody can do — and what instantly separates a senior from a principal in an interview — is explain what happens between the moment you hit Enter and the moment the resource exists. That gap, the inside of the deployment engine, is this whole phase. Once you can narrate it, ARM stops being a vending machine you put YAML into and becomes a system you operate.

Here's the thing that took me too long to internalize: ARM is not running your script. There is no script. You handed it a description of the world you want, and it figured out the order, did the diffs, and converged reality to match. The mental shift from "I'm telling the computer the steps" to "I'm declaring the end state and the engine reconciles" is the entire shift from imperative ops to infrastructure-as-code. Get that in your bones and you understand Terraform, Kubernetes, Bicep, and every reconciler ever built, because they're all the same loop: observe actual, compare to desired, converge.

Now let me give you the things that actually bite people.

Respect Complete mode like a loaded gun. I'm not being dramatic. The single most expensive Azure mistake I've watched happen — twice — is someone deploying in --mode Complete with a template that was missing a resource, and ARM deleting a production database to make the resource group match the file. And here's the part that matters for you: when it happened, the junior said "ARM has a bug, it deleted my DB." The principal said "no — Complete mode does exactly that, it's documented, and the real failure is that we deployed Complete without a what-if." The grown-up move isn't to fear the tool; it's to know its contract so precisely that you reach for it deliberately and never by accident. Default to Incremental. What-if before anything destructive. Lock the precious stuff. That's not caution, that's competence.

Idempotency is the most beautiful idea in this phase, and it's worth loving. The fact that you can run the same deploy a thousand times and the world doesn't drift — that the hundredth run is all NoChange — is why you can put infrastructure in CI and re-apply it on every commit without holding your breath. The whole modern practice of "infrastructure is code, reviewed in PRs, applied automatically" stands on that one property. When you build the lab and watch the second deploy come back all-NoChange, sit with it for a second. That green is the foundation everything else is built on. And learn where it cracks — a random resource name, an immutable property — because that's the bug that pages you at 2 a.m., when a deploy that should've been a no-op quietly recreated something.

The implicit-dependency thing is your party trick. Most people genuinely believe ARM deploys in the order you wrote the resources, or only in the order you put dependsOn. When you can look at a template and say "the NIC will deploy after the subnet even though there's no dependsOn, because the subnet.id reference creates an implicit edge" — you've shown you understand the engine, not just the syntax. That's a small thing that lands huge in an interview, because it proves you've been under the hood. Build the dependency extractor in the lab by hand and it'll be permanent knowledge, not a fact you memorized.

Don't skip the async part because the spinner hides it. az shows you a nice progress bar and then says "done," and it's tempting to think the PUT is the whole story. It isn't. Under that spinner is a poll loop checking provisioningState until it's Succeeded. The reason this matters: the day a deploy is stuck, the only useful question is "which resource's provisioningState is stuck, and why" — and if you've never thought about the async model, you'll just stare at the spinner feeling helpless. The principal pulls the deployment operations, finds the one resource that's Failed or blocked on quota or a policy modify, and fixes that. Calm in an incident comes from knowing what's actually happening, not from hoping the bar fills up.

The honest truth about this phase: the algorithm is not hard. Topological sort is a first-year data-structures exercise. Idempotency is one comparison. What makes this principal-level isn't the code — it's the judgment wrapped around it: knowing that Complete is a blast-radius decision, that what-if is a safety culture and not a command, that idempotency is a property you can break, that "deployed" means "polled to Succeeded." The lab is easy to pass and impossible to fully appreciate until you've connected each function to a real incident it would've prevented. So do that connecting. For every function you write, ask: what 2 a.m. page does this represent?

And here's the career angle, brother to brother. This phase is the keystone of all of IaC, which is half of this job. Terraform (next phase) is this same engine with a state file bolted on. Kubernetes is this same reconcile-to-desired loop. CloudFormation, Pulumi, Crossplane — same idea. If you truly own the ARM deployment engine — graph, idempotency, modes, async — you don't learn five more IaC tools, you recognize five more skins on the machine you already built. That's the leverage. That's why we start here. The people who plateau learn each tool fresh; the people who go principal learn the engine once and map everything to it.

Go build the engine. Watch the re-deploy come back all-NoChange. Then come find me in Phase 02 — that's where we put state and plan and ForceNew on top of this, and you'll see Terraform was never magic either.

— your brother 👨🏻

Lab 01 — ARM Deployment Engine

Phase: 01 — Azure Resource Manager & the Deployment Engine | Difficulty: ⭐⭐⭐☆☆ | Time: 4–6 hours

Every single write in Azure — a storage account, a subnet, a role assignment — is one idempotent PUT that ARM routes through a pipeline: authenticate, authorize, validate, then resolve a dependency graph and deploy resources in topological order. This lab builds that engine: the implicit/explicit dependency resolver, a deterministic topological sort with cycle detection, the idempotent Create/Modify/NoChange decision, the Complete-mode delete foot-gun, and the what-if dry run. Strip away the network, the JWT, and the resource providers, and what's left is a graph algorithm — the one ARM, Bicep, Terraform, and CloudFormation all share.

What you build

• extract_dependencies(resource) — the union of explicit dependsOn and the implicit dependencies parsed out of reference('NAME') / resourceId('NAME') calls hiding in (possibly nested) property strings. This is the step everyone gets wrong by hand.
• build_dependency_graph(resources) — name → {dependencies}, raising on a dependency to a resource that isn't in the template (ARM's InvalidTemplate) and on duplicate names.
• topological_order(graph) — deterministic Kahn's algorithm, tie-breaking ready nodes by name so the same template always produces the same plan; raises ValueError("cycle: ...") on a circular dependency.
• deploy(resources, state, mode) — process in topo order; idempotent PUT → Create / Modify / NoChange; Complete mode also Deletes un-templated resources. Returns the ordered actions and the converged state, without mutating the caller's state.
• what_if(resources, state, mode) — the side-effect-free dry run: the same classification as deploy, returned as a list of Change objects, mutating nothing.

Key concepts

Files

Run

pip install -r requirements.txt
pytest test_lab.py -v                      # against your lab.py
LAB_MODULE=solution pytest test_lab.py -v  # against the reference
python solution.py                         # worked example

Success criteria

• All 25 tests pass against your implementation.
• You can explain why test_topo_order_is_deterministic_tie_break_by_name matters — real ARM parallelises independent resources; we serialise ties by name only so the plan is reproducible and testable. The determinism is a lab affordance, not how ARM schedules.
• You can explain why test_redeploy_is_idempotent_all_no_change is the whole reason IaC works: a PUT of desired state that already matches is a safe no-op.
• You can explain, in one breath, why test_complete_mode_deletes_extras is the most dangerous line in the suite — and when you'd ever want Complete mode.
• Given a 5-resource template on paper, you can draw its dependency graph and write the deploy order in under two minutes.

How this maps to real Azure

What the miniature leaves out (and why it's fine): real ARM also authenticates the Entra JWT, authorizes via RBAC + deny assignments, runs Azure Policy (which can deny or modify the request), registers resource providers, calls a different REST endpoint per api-version, and handles long-running async operations by returning 202 Accepted with an Azure-AsyncOperation URL the client polls. None of that changes the graph algorithm — which is exactly the part interviewers probe and incidents turn on.

Extensions (build these in your own subscription)

• Async operations: give each resource a provisioningState that walks Accepted → Running → Succeeded|Failed over N "ticks," and make deploy poll until terminal — now you've modeled the 202/Azure-AsyncOperation loop.
• Parallel waves: instead of a flat topo list, emit waves (all nodes whose deps are satisfied deploy together) — the real ARM scheduler. Assert that independent resources land in the same wave.
• Bicep front-end: parse a tiny Bicep-like syntax (resource sa 'type' = { ... }) with symbolic references (sa.id) into your resource-dict format, proving Bicep is just a transpiler to the same graph.
• Deployment stack / deny-settings: track which resources a "stack" manages; on the next deploy, Delete the ones dropped from the stack (managed deletion) and reject an out-of-band Modify to a denySettings: denyDelete resource (drift protection).
• Wire it to a real RG: az deployment group what-if -g rg -f main.bicep and diff your predicted plan against Azure's — they should classify the same resources the same way.

Interview / resume

• Talking points: "Walk me through what happens when I run az deployment group create." / "How does ARM know to deploy the VNet before the NIC?" / "What's the difference between Incremental and Complete mode, and which is the default?" / "Why is an ARM PUT idempotent and why does that matter for CI?"
• Resume bullet: Built a dependency-graph deployment engine (implicit/explicit reference resolution, deterministic topological scheduling with cycle detection, idempotent PUT convergence, and a non-mutating what-if diff) modeling the Azure Resource Manager control plane.

Phase 02 — Terraform, CDKTF & IaC State Internals

Difficulty: ⭐⭐⭐☆☆ (the algorithm) → ⭐⭐⭐⭐⭐ (the judgment about state, replace, and locking) Estimated Time: 1 week (12–18 hours) Prerequisites: Phase 01 (the ARM deployment engine — the dependency graph, idempotent PUT, and what-if). This phase is the stateful sibling of that stateless engine, and the contrast is the whole lesson. No live Azure subscription required for the lab.

Why This Phase Exists

Phase 01 taught you the single most important sentence about ARM: every write is an idempotent PUT of desired state, and ARM derives the diff by asking the resource provider what's actually there. ARM is stateless — there is no file recording what it believes exists, so there is nothing to drift, nothing to lock, nothing to lose.

Terraform makes the opposite bet, and that one architectural decision is the reason it behaves the way it does — including every way it surprises people. Terraform keeps a state file: a recorded map from your configuration's addresses (azurerm_storage_account.main) to the real resource IDs and attributes it created. It trusts that file as its model of reality. The entire tool is the diff between the world you wrote (config) and the world it remembers (state), turned into a plan and executed by apply.

That design buys Terraform things ARM doesn't have — a fast plan that doesn't have to read every resource, a portable engine that works across 4,000+ providers, a single graph that spans Azure and DNS and GitHub and Datadog. And it costs Terraform things ARM never worries about:

• State can lie. Someone edits a resource in the portal; Terraform's state still says the old value. That gap is drift, and it's why terraform plan refreshes before it diffs — and why a plan can show changes you didn't write.
• State can be lost or corrupted, and losing it means Terraform forgets it owns your production estate — the most feared tfstate incident there is.
• State must be shared and locked. Two engineers running apply against the same state at once corrupt it, so remote state takes a lock (an Azure Storage blob lease) for the duration of an apply.

And one decision dwarfs all the others in blast radius: the difference between update in place and replace (destroy-then-create). A change to a mutable attribute is a quiet ~. A change to an immutable (ForceNew) attribute — a storage account's location, a VM's certain SKUs, a database's name — is a -/+: Terraform will destroy the real resource and build a new one, with whatever downtime and data loss that implies. The principal skill is reading a plan and catching the -/+ before you type yes.

So this phase makes you build the plan/apply engine: the per-resource diff, the ForceNew replace decision, the dependency-graph ordering (and its reversal for deletes), and the drift detector. Once you've implemented it, Terraform stops being magic, CDKTF reveals itself as "the same engine with a real language on the front," and you can debug a confusing plan by reasoning about which of the three artifacts is wrong.

What "Principal-Level" Means Here

A senior engineer writes a working Terraform module and runs apply. A principal understands the engine well enough to:

• Read a plan as a code review and flag every -/+ (replace) as a potential outage, every unexpected ~ as drift or a provider default, and every - as a deletion to justify — before approving.
• Predict replace vs update from the provider schema: know which attributes are ForceNew and never let a "small change" silently recreate a database.
• Reason about state as a separate, fragile artifact — where it lives (remote backend), how it's locked (blob lease), how it drifts, how it's recovered, and why terraform import and state mv/rm exist.
• Explain Terraform vs ARM/Bicep at the model level: state-file diff vs stateless desired-state PUT, and when each model's failure mode bites.
• Operate CDKTF knowing it changes the authoring layer (a real language, loops, abstractions) and not the engine — it synthesizes to Terraform JSON and the same plan/apply runs.

Concepts

• The three artifacts. Configuration = the desired world (.tf / .tf.json / synthesized CDKTF). State = the recorded reality (terraform.tfstate: address → real ID + attributes). Plan = the diff of config against state, per resource, in dependency order. apply executes the plan and writes a new state. Hold these three apart and almost every Terraform mystery dissolves.
• The plan algorithm. For each resource, compare config to state and emit exactly one of: create (in config, not in state, +), update-in-place (differs, only mutable attrs, ~), replace (differs, an immutable attr moved, -/+), delete (in state, not in config, -), no-op (identical). This five-way classification is Terraform.
• ForceNew / immutability → replace. Provider schemas mark some attributes ForceNew: true because the underlying API cannot change them in place (you can't move a storage account to another region). Changing one forces destroy-then-create. The hardest, most valuable skill in this phase is recognising immutable attributes and the blast radius of recreating the resource.
• Replace ordering. Default is destroy-before-create (downtime while the new one builds). The create_before_destroy lifecycle flag inverts it (new one up first, then old torn down) — essential for zero-downtime replaces, but it requires non-colliding names/addresses.
• The dependency DAG. Terraform builds a graph from references (${azurerm_storage_account.main.id} is an edge) plus explicit depends_on. It applies in topological order (a resource after the things it needs) and destroys in reverse topological order (a dependent before the thing it depends on).
• Drift detection. Real ≠ state because of out-of-band changes (the portal, another tool, a human at 2 a.m.). terraform plan refreshes — reads each resource through the provider — then diffs the refreshed reality against config, so drift shows up as changes you didn't write. State is a claim, continually re-checked, not ground truth.
• Remote state + locking. State on a developer's laptop doesn't scale to a team. The Azure Storage backend keeps tfstate in a blob and takes a blob lease lock for each apply, so concurrent applies can't corrupt it. This is the backbone of team Terraform.
• CDKTF. Author infrastructure in TypeScript/Python/Go/Java/C#, cdktf synth to Terraform JSON (cdk.tf.json), then run the same plan/apply engine. CDKTF buys you a real language (loops, conditionals, classes, tests) on top of an unchanged core.
• Terraform vs ARM/Bicep (the P01 contrast). ARM/Bicep: stateless, idempotent desired-state PUT, Azure-only, no state file, drift is recomputed each deploy. Terraform: a state file it diffs against, multi-cloud/multi-provider, a plan you review, but a fragile artifact you must store, lock, and protect.

Labs

Lab 01 — Terraform State Engine (flagship, implemented)

→ Lab folder: lab-01-terraform-state-engine/

Integrated-Scenario Suggestions (carried through the whole track)

The phases compound. Keep these in mind as you build — each plugs the state engine into a larger platform:

1. Team remote-state bootstrap — a storage account + container holding tfstate with blob-lease locking, the first thing any real Terraform org provisions (often itself created out-of-band or by a bootstrap module). The lock is the difference between a team that can run apply safely and one that corrupts state weekly. (→ P05, P12)
2. The dangerous replace — a config change that looks trivial (rename a storage account, move a region, change a SKU) but is a ForceNew → the plan shows -/+ and the apply destroys a production resource. Catching this in PR review is the whole skill. (→ P06, P11)
3. Drift reconciliation — a portal edit (someone bumped a tier in an incident) creates drift; the next plan shows changes nobody wrote. Decide: re-apply (config wins, revert the edit), import/update config (reality wins), or ignore_changes (accept the edit). This is the daily reality of shared estates. (→ P04, P05)
4. CDKTF for a landing zone — express a repeated pattern (N spokes, each a VNet + subnet + NSG) as a CDKTF loop in Python, synth to JSON, and watch the same engine plan it — the case where a real language beats hand-written HCL. (→ P05, P06)
5. Terraform vs Bicep on the same estate — model one resource group both ways and compare the failure modes: Bicep's stateless PUT (no state to lose, Azure-only) vs Terraform's state diff (multi-cloud, reviewable plan, fragile state). The ADR that chooses between them is a Round-5 architecture question. (→ P01, P15)

Guides in This Phase

• HITCHHIKERS-GUIDE.md — the 30-minute orientation; read first
• WARMUP.md — the full primer; read slowly
• BROTHER-TALK.md — the candid version

Key Takeaways

• Terraform is the diff between config and state. Three artifacts — config (desired), state (recorded reality), plan (the diff) — and apply executes the plan. Hold them apart and the tool stops surprising you.
• ForceNew is the load-bearing concept. Update is a ~; replace is a -/+ that destroys real infrastructure. Reading a plan is reading for the -/+.
• Order matters and reverses for deletes. Apply in topological order; destroy in reverse topological order. The graph drives both.
• State is a fragile, separate artifact. It drifts (real ≠ state), it must be stored remotely, and it must be locked (Azure blob lease) so a team can share it. Most Terraform horror stories are state-management failures, not config bugs.
• CDKTF changes the authoring layer, not the engine. Synthesize to Terraform JSON; the same plan/apply runs. A real language buys abstraction, not a different model.
• Stateful (Terraform) vs stateless (ARM/Bicep) is a design tradeoff, not a quality ranking. State buys a portable, reviewable, multi-cloud diff; it costs you a fragile artifact to store, lock, and protect.

Deliverables Checklist

• Lab 01 implemented; all 28 tests pass against solution.py and your lab.py
• You can name Terraform's three artifacts and say exactly what apply does
• Given a diff and a provider schema, you can call update vs replace correctly and explain the blast radius of a replace
• You can explain why deletes run in reverse dependency order
• You can define drift, explain how plan's refresh surfaces it, and list the three ways to reconcile it
• You can explain remote state + blob-lease locking and why a team needs it
• You can contrast Terraform's state model with ARM/Bicep's stateless PUT (P01)

Warmup Guide — Terraform, CDKTF & IaC State Internals

Zero-to-principal primer for Phase 02. Phase 01 built ARM's stateless deployment engine. This phase builds its stateful sibling: Terraform, whose entire behavior — the good and the terrifying — follows from one decision, to keep a state file. We go from "what is a state file" to "why does my plan show a -/+ that's about to delete my database," from first principles to the mechanism under the hood, with the diagrams, tables, and small code you need to defend every line in an interview and at 2 a.m.

Table of Contents

• Chapter 1: Why Terraform Has State (and ARM Doesn't)
• Chapter 2: The Three Artifacts — Config, State, Plan
• Chapter 3: The Plan Algorithm — The Five-Way Diff
• Chapter 4: ForceNew — Update vs Replace, the Load-Bearing Concept
• Chapter 5: The Dependency DAG — Order, and Its Reversal
• Chapter 6: Remote State and Locking
• Chapter 7: Drift — When State Lies
• Chapter 8: CDKTF — A Real Language, the Same Engine
• Chapter 9: Terraform vs ARM / Bicep — The Stateful/Stateless Tradeoff
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Why Terraform Has State (and ARM Doesn't)

From zero. "Infrastructure as Code" means you describe the infrastructure you want in text, and a tool makes the cloud match it. There are two ways a tool can figure out what to change to make reality match your text.

1. Ask reality every time (stateless). This is ARM/Bicep (Phase 01). Each deploy is a PUT of desired state; ARM hands it to the resource provider, which compares your desired state to what it actually has and converges. There is no file recording what ARM "thinks" exists — the resource provider is the source of truth, queried fresh each time. Nothing to lose, nothing to drift, nothing to lock.
2. Remember what you made (stateful). This is Terraform. When Terraform creates a resource, it writes an entry into a state file — a JSON document mapping your config's address (azurerm_storage_account.main) to the real resource's ID and attributes. Next time, instead of asking the cloud about everything, it diffs your config against this remembered state.

Why would anyone choose the harder, riskier stateful model? Because it buys things the stateless model can't:

• One graph across many clouds. ARM only knows Azure. Terraform's state can hold an Azure VNet, an AWS bucket, a Cloudflare DNS record, a GitHub repo, and a Datadog monitor in one dependency graph with one plan. The state file is what makes that universal — every provider records into the same format.
• A fast, reviewable plan. Because Terraform remembers, plan can show you the diff without a full deploy, and you review it like a PR before anything changes.
• Tracking resources the cloud can't uniquely identify by config alone — Terraform's state remembers which real resource corresponds to which config block, even when the config is a loop producing ten near-identical resources.

The cost is the rest of this phase: state can be lost, stale (drift), or contended (two people applying at once). Every famous Terraform horror story is one of those three. A principal doesn't fear state; they understand it as a separate, fragile artifact with its own lifecycle, and they protect it deliberately.

Production significance. "Where does your state live and how is it locked?" is the fastest way to tell a Terraform team that's been burned from one that hasn't. The burned team has remote state in an Azure Storage account with locking, restricted access, and versioning. The other team has terraform.tfstate on someone's laptop and a story coming.

Common misconception. "State is just a cache; I can delete it and re-run." No — if you delete state, Terraform forgets it owns your infrastructure. The next plan sees an empty state and a full config and says "create everything" — against resources that already exist, which then collide. Losing state is the worst non-data Terraform incident there is.

Chapter 2: The Three Artifacts — Config, State, Plan

From zero. Almost every confusing thing Terraform does becomes obvious the moment you hold these three apart:

The relationship, as a diagram:

          you write                    Terraform remembers
        ┌──────────────┐             ┌──────────────────────┐
        │   CONFIG     │             │       STATE          │
        │ (desired)    │             │ (recorded reality)   │
        └──────┬───────┘             └──────────┬───────────┘
               │                                │
               └────────────  diff  ────────────┘
                              │
                        ┌─────▼──────┐
                        │   PLAN     │   create / update / replace / delete / no-op
                        └─────┬──────┘
                              │ apply
                        ┌─────▼──────┐
                        │ NEW STATE  │   (and real resources changed)
                        └────────────┘

terraform plan computes the diff and shows it. terraform apply executes the plan and writes the new state. That's the whole loop. In the lab, config is a list of Resource, state is a dict (address → attributes), and plan() produces the diff that apply() executes into a new state dict.

Under the hood — what a real tfstate entry looks like:

{
  "resources": [{
    "type": "azurerm_storage_account",
    "name": "main",
    "instances": [{
      "attributes": {
        "id": "/subscriptions/…/storageAccounts/stappdata",
        "name": "stappdata",
        "location": "eastus",
        "account_tier": "Standard"
      }
    }]
  }]
}

The address azurerm_storage_account.main is type.name. The attributes block is the recorded reality the next plan diffs against. Our lab's state["azurerm_storage_account.main"] = {"name": …, "location": …, "account_tier": …} is exactly this, with the envelope stripped away.

Common misconception. "The plan and the apply are the same operation." They are not. plan is read-only (modulo the refresh, Ch. 7); apply mutates. Best practice is to plan -out=tfplan and apply tfplan so the thing you reviewed is exactly the thing that runs — otherwise the world can change between plan and apply and you apply a different diff than you approved.

Chapter 3: The Plan Algorithm — The Five-Way Diff

From zero. The plan is computed one resource at a time. For each resource Terraform compares the config version against the state version and emits exactly one action:

That five-way classification is the plan engine. Everything else — ordering, drift, rendering — wraps around it.

Under the hood — the algorithm in pseudocode (and exactly what plan_resource / plan implement in the lab):

for each address in (config ∪ state):
    desired = config.get(address)        # None ⇒ removed from config
    current = state.get(address)         # None ⇒ never created
    if desired and not current:  → create
    elif current and not desired: → delete
    elif desired == current:      → no-op
    else:
        changed = diff(desired, current)        # attr → (old, new)
        if any(attr is ForceNew for attr in changed): → replace
        else:                                          → update

The leaf primitive is the per-attribute diff (diff_attributes in the lab): walk the union of keys; a key in both with different values is a change (old, new); a key only in config is an add (None, new); a key only in state is a removal (old, None). The whole resource decision rides on whether that change set touches a ForceNew attribute — Chapter 4.

Production significance. Reading a plan is this algorithm in your head. When you see ~ tags, you know "in both, mutable attr changed." When you see -/+ name, your stomach should drop: an immutable attr moved, a real resource is about to be destroyed.

Common misconception. "No-op means Terraform did nothing useful." No-op means converged — the most valuable plan in production is the all-no-op plan, the proof that reality matches config. Re-running a converged config is all no-op, which is exactly Terraform's idempotency (the lab's test_plan_idempotent_when_state_matches).

Chapter 4: ForceNew — Update vs Replace, the Load-Bearing Concept

From zero. Some attributes of a cloud resource can be changed in place; others cannot. You can change a storage account's account_tier (Standard → Premium) — it's a mutable property and the Azure API supports an update. You cannot change its location — a storage account lives in one region forever; to "move" it you must delete it and create a new one somewhere else. The provider schema encodes this: each attribute is marked ForceNew: true (immutable) or not.

When a change touches a ForceNew attribute, an in-place update is physically impossible, so Terraform plans a replace — shown as -/+ — which means destroy the existing real resource, then create a new one. The blast radius is everything that resource held: the data in the storage account, the IP address of the public IP, the identity of the database.

This is the single most important skill in the phase. A senior runs apply and trusts the plan. A principal reads the plan for -/+ lines and asks "is this replace acceptable, and is there data behind it?" — because the difference between ~ and -/+ on the same attribute name is the difference between a tweak and an outage.

Under the hood — why the boundary is exact. The lab's plan_resource makes the call:

delta = diff_attributes(desired, current)   # {attr: (old, new)}
if not delta:                  return "no-op"
if any(a in force_new for a in delta): return "replace"   # one immutable attr is enough
return "update"

One immutable attribute in the change set is enough to turn the entire resource into a replace, even if nine other changed attributes are mutable. And the same diff is an update if force_new is empty. That boundary is the lab's test_plan_resource_force_new_boundary, and it's exactly the question an interviewer probes: "this plan shows a replace — which attribute caused it?"

Replace ordering — destroy-before-create vs create-before-destroy. A replace is two operations, and their order is a downtime decision:

• Default: destroy-then-create. Terraform destroys the old resource, then creates the new one. There is a window where the resource doesn't exist — downtime, and if the create fails you're left with nothing.
• create_before_destroy = true (a lifecycle block): create the new resource first, cut traffic over, then destroy the old. Zero (or near-zero) downtime. The catch: the new and old resources coexist briefly, so any attribute that must be globally unique (a name, a fixed IP) will collide unless you let it be generated or use name_prefix. Many a create_before_destroy has failed on a hard-coded name.

default (destroy-then-create):   [destroy old] ──► [create new]      ← outage window
create_before_destroy:           [create new] ──► [cut over] ──► [destroy old]   ← no gap

Production significance. "Change the VM size" can be an update on some SKUs and a -/+ on others; "rename a resource" is almost always a -/+; "tighten a name to satisfy a new policy" can quietly destroy a stateful resource. The plan tells you which — if you read it. The famous incident shape is approving a plan whose -/+ on a database you skimmed past.

Common misconception. "Terraform updated my resource, so my data is fine." Only if it was a ~. A -/+ destroyed and recreated it — the data is gone unless it lived elsewhere (a managed disk you detached, a backup you restored). Update and replace are not two flavors of the same thing; one preserves the resource, the other annihilates it.

Chapter 5: The Dependency DAG — Order, and Its Reversal

From zero. Resources depend on each other: a subnet needs its VNet, a NIC needs its subnet, a storage container needs its storage account. Terraform must create them in an order that respects those dependencies, and — the part everyone forgets — destroy them in the opposite order.

Where the edges come from. Two sources, exactly as in ARM (Phase 01):

• Implicit (references). Writing subnet_id = azurerm_subnet.main.id in a config makes that resource depend on azurerm_subnet.main — Terraform parses the reference and adds the edge. This is the common case and the reason you rarely write dependencies by hand.
• Explicit (depends_on). A depends_on = [azurerm_role_assignment.x] block for ordering that isn't expressed through a reference (e.g. "this must exist after that for a reason the attributes don't capture").

Under the hood — topological order, and why it reverses for deletes. Terraform builds a directed acyclic graph (DAG) and walks it:

   rg.main  ──►  sa.main  ──►  container.data         (depends-on edges)

   CREATE / UPDATE order (topological):   rg.main → sa.main → container.data
   DELETE order (REVERSE topological):    container.data → sa.main → rg.main

Why reverse for deletes? Because you cannot delete a resource something else still depends on. You can't delete a storage account while a container still lives in it; you can't delete a VNet while a subnet still sits in it. So Terraform tears down dependents first, walking the graph backwards. The lab encodes exactly this: creates/updates/replaces come out in topological order, deletes in reversed(topological_order) — that's test_plan_deletes_orphaned_state_in_reverse_order.

For determinism (so the lab is testable and a plan is reproducible) the topo sort uses Kahn's algorithm with a name tie-break: among resources that are all currently ready, emit the smallest address. Real Terraform parallelises independent resources; we serialise ties by name so the same config always yields the same plan.

The math. A valid plan order is any topological sort of the DAG ( G = (V, E) ), where an edge ( a \to b ) means "(a) depends on (b), so (b) before (a)." A cycle in (G) makes a topological order impossible — Terraform errors with Cycle: …. The number of valid orders can be large; the tie-break collapses them to one canonical order.

Common misconception. "I'll just add depends_on everywhere to be safe." Over-using depends_on serialises resources that could run in parallel (slower applies) and hides the real reference graph. Prefer implicit dependencies (use the attribute reference); reach for depends_on only when there's genuinely no reference to express the ordering.

Chapter 6: Remote State and Locking

From zero. If state lives as terraform.tfstate on your laptop, you are the only person who can apply, and if your laptop dies the state dies with it. A team needs remote state: the file lives in shared storage everyone can reach. On Azure that's the AzureRM backend — a blob in a storage account:

terraform {
  backend "azurerm" {
    resource_group_name  = "rg-tfstate"
    storage_account_name = "sttfstate"
    container_name        = "tfstate"
    key                   = "prod.terraform.tfstate"
  }
}

The problem remote state introduces: concurrency. Now two engineers can both run apply against the same state at once. Engineer A reads state, computes a plan, starts creating resources, and writes state back — while Engineer B, working from the old state, does the same. Their writes interleave; the state ends up describing neither reality. State is corrupted.

Under the hood — the lock. Before apply (and other state-mutating operations), Terraform acquires a lock. With the AzureRM backend the lock is an Azure Storage blob lease on the state blob:

apply #1:  acquire blob lease  ── lease held ──►  read state → plan → mutate → write state → release lease
apply #2:  acquire blob lease  ✗  "Error acquiring the state lock"  (lease already held)

A blob lease is a short-lived exclusive claim on the blob; while A holds it, B's lock acquisition fails fast with a clear error instead of silently corrupting state. When A finishes (or is interrupted), the lease is released (or expires), and B can proceed. The lab doesn't simulate the lease — it's offline and single-threaded — but you must be able to explain it, because "how do you stop two people corrupting state?" is a stock interview question and the answer is "the backend's lock; on Azure, a blob lease."

Production significance. A locked, remote, versioned state backend is table stakes for team Terraform. The trio you want: remote (shared, durable), locked (no concurrent corruption), versioned + access-controlled (recover a bad apply, restrict who can read secrets that land in state). Note that last point: state often contains secrets (a generated password, a connection string) in plaintext — which is why the state blob must be encrypted at rest and access-restricted like the secret store it effectively is.

Common misconception. "terraform force-unlock fixes a stuck lock, so locking is no big deal." force-unlock is a loaded gun: if you force-unlock while another apply is actually running, you re-introduce the exact corruption the lock prevents. Use it only when you're certain the lock is orphaned (a crashed run), never to "get past" a teammate's live apply.

Chapter 7: Drift — When State Lies

From zero. State is Terraform's memory of what it built, and memory goes stale. Someone opens the Azure portal during an incident and bumps a storage account's tier. Another tool changes a tag. A human deletes a resource by hand. Now the real world differs from what state records — that gap is drift, and "real ≠ state" is the one-line definition to memorise.

Why it matters. Terraform plans by diffing config against state. If state is stale, the plan is computed against a lie — it might miss a change, or propose to "fix" something that's actually correct. So before diffing, terraform plan does a refresh.

Under the hood — the refresh. terraform plan (and apply) runs in two phases:

  1. REFRESH:  for each resource in state, read its real attributes through the provider
               (a GET against the Azure API) → update an in-memory copy of state to match reality
  2. DIFF:     compare CONFIG against the refreshed reality → produce the plan

So drift surfaces in the plan as changes you didn't write: you didn't touch the tier in your .tf, but the plan shows ~ account_tier "Premium" -> "Standard" because someone bumped it in the portal and refresh caught it. The lab's detect_drift(state, real) is a pure model of the refresh-and-compare: it returns {address: {attr: (state_val, real_val)}} for every recorded resource whose real attributes diverge — and an empty result when state matches reality (test_detect_drift_none_when_in_sync).

Reconciling drift — three choices, and they're a decision:

A principal doesn't reflexively re-apply; they ask why the drift exists and choose. The worst outcome is a blind re-apply that reverts an emergency fix someone made in the portal at 3 a.m. for a good reason.

Production significance. Drift is the daily reality of any estate humans can also touch. The mature answer is to reduce the surface: restrict portal write access, run a scheduled plan that alerts on drift, and treat unexpected drift as an incident signal, not noise.

Common misconception. "Terraform manages everything, so there's no drift." Drift happens the moment anything outside Terraform can change a resource — a human, another pipeline, an Azure-managed mutation, an autoscaler. The question is never "will there be drift" but "do I detect it and have I decided how to reconcile each kind."

Chapter 8: CDKTF — A Real Language, the Same Engine

From zero. HCL (Terraform's config language) is declarative and pleasant for static infrastructure, but it's awkward when you want real programming: complex loops, conditionals, shared abstractions, unit tests, types. CDKTF (Cloud Development Kit for Terraform) lets you author infrastructure in a real language — TypeScript, Python, Go, Java, C# — using classes and loops and functions.

The crucial insight: CDKTF changes the authoring layer, not the engine. Here is the whole pipeline:

  your Python/TS program  ──cdktf synth──►  cdk.tf.json (Terraform JSON)  ──►  the SAME plan/apply engine

cdktf synth synthesizes — it runs your program, which builds an object tree, and emits Terraform JSON (cdk.tf.json), which is just Terraform configuration in JSON instead of HCL. From there it's the identical engine you built in the lab: config → diff against state → plan → apply, same ForceNew rules, same DAG, same state file, same locking. CDKTF does not have its own state model or its own diff algorithm. It is a front-end that produces config.

# CDKTF (Python): a loop that HCL would express clumsily
from cdktf import App, TerraformStack
from cdktf_cdktf_provider_azurerm.resource_group import ResourceGroup

class Spokes(TerraformStack):
    def __init__(self, scope, id):
        super().__init__(scope, id)
        for region in ["eastus", "westus", "northeurope"]:
            ResourceGroup(self, f"rg-{region}", name=f"rg-spoke-{region}", location=region)

App().synth()   # → writes cdk.tf.json that `terraform plan` consumes

Production significance. Reach for CDKTF when the generation of config is the hard part — N near-identical spokes, config driven by data, abstractions you want to unit-test, teams who'd rather write Python than HCL. Stay with HCL when the infrastructure is static and the team values the simplicity and the enormous HCL ecosystem. Either way, the thing you learn in this lab — the plan/apply engine — is what runs underneath both.

Common misconception. "CDKTF is a different tool from Terraform." It's the same Terraform engine with a programmatic front-end; debugging a CDKTF plan means reading the synthesized cdk.tf.json and reasoning about it exactly as you would HCL. If you understand the engine, you understand CDKTF for free.

Chapter 9: Terraform vs ARM / Bicep — The Stateful/Stateless Tradeoff

From zero. Phase 01 built ARM's engine; this phase builds Terraform's. They solve the same problem — make the cloud match your code — with opposite bets, and a principal can defend the choice between them with this table, not with taste:

The honest framing. Neither is "better." ARM/Bicep is stateless, Azure-native, and has no state file to lose — but it's Azure-only and you don't get a portable plan that also spans your DNS provider and your SaaS. Terraform gives you one reviewable plan across every provider on earth — at the cost of a fragile state artifact you must store, lock, version, and protect. The ADR that chooses between them weighs exactly those: "do we need multi-cloud / a unified graph?" (→ Terraform) vs "are we all-in on Azure and want to avoid state management entirely?" (→ Bicep). Many orgs run both: Bicep for Azure-native landing zones, Terraform where the graph crosses clouds.

Common misconception. "Terraform is more powerful because it has a plan and Bicep doesn't." Bicep has what-if — the same idea. The real difference is the state model, and it's a tradeoff (portability + a unified graph vs a fragile artifact), not a power ranking.

Lab Walkthrough Guidance

Lab 01 — Terraform State Engine, suggested order (each maps to a chapter):

1. Resource + diff_attributes (Ch. 2–3). The model (address = type.name, attributes dict) and the leaf comparison: union of keys → (old, new) for changed/added/removed, omit identical. Validate both sides are dicts.
2. plan_resource (Ch. 3–4). The four-way decision. current is None → create; empty diff → no-op; any changed attr in force_new → replace; else update. The ForceNew boundary is the heart of the lab — test it both ways (force_new empty vs not) on the same diff.
3. plan (Ch. 3, 5). Classify every config resource (look up its type's force_new set), mark addresses in state-but-not-config as delete, then order: creates/ updates/replaces in topological order, deletes in reverse. Don't emit no-op actions but count them in the summary. Reject duplicate addresses.
4. apply (Ch. 2). Execute the plan into a new state dict — create/update/replace write the desired attributes, delete removes the address — and never mutate the input (copy first; the test asserts the input is byte-for-byte unchanged).
5. detect_drift (Ch. 7). For each address in both state and real, diff recorded vs actual; include only diverging addresses; empty result when in sync.

Run it red, make it green: pytest test_lab.py -v (28 tests). Then LAB_MODULE=solution pytest test_lab.py -v to confirm the reference agrees, and python solution.py to watch a worked plan (update vs replace) and a drift example print.

Success Criteria

You are ready for Phase 03 when you can, from memory:

1. Name Terraform's three artifacts (config, state, plan) and say exactly what apply does — and why ARM has no state file.
2. Run the five-way diff in your head: given a config attr, a state attr, and a ForceNew flag, call create/update/replace/delete/no-op correctly.
3. Explain the ForceNew boundary — why one immutable attribute turns the whole resource into a -/+, and what a -/+ destroys.
4. Explain create-before-destroy vs destroy-before-create and the name-collision catch.
5. Explain why creates go in topological order and deletes in reverse, with the subnet/NIC example.
6. Explain remote state, the AzureRM backend, and the blob-lease lock — and why force-unlock is dangerous.
7. Define drift (real ≠ state), describe plan's refresh phase, and list the three ways to reconcile it.
8. Explain that CDKTF synthesizes to Terraform JSON and runs the same engine.
9. Contrast Terraform's state model with ARM/Bicep's stateless PUT as a tradeoff.

Interview Q&A

Q: What are Terraform's three artifacts, and what does apply actually do? Configuration is the desired world I write; state (terraform.tfstate) is the recorded reality Terraform remembers it built — an address-to-attributes map; the plan is the diff of config against state, per resource, in dependency order. terraform plan computes and shows that diff (after a refresh); terraform apply executes the plan against the real cloud and writes the new state. The whole tool is "reconcile config and state," and almost every confusing behavior is explained by which of the three is stale or wrong.

Q: My plan shows -/+ on a resource. What does that mean and what do you do? -/+ is a replace: a change touched a ForceNew (immutable) attribute, so Terraform can't update in place — it will destroy the real resource and create a new one. First I find which attribute caused it (a region, a name, a SKU that can't change in place). Then I ask the load-bearing question: is there state or identity behind this resource? If it's a stateless resource, fine. If it's a database, a storage account with data, or a public IP something depends on, a blind apply is an outage — I'd reconsider the change, or use create_before_destroy for zero-downtime where names don't collide, or move the data first. The skill is catching the -/+ in review before anyone types yes.

Q: When is a change an update and when a replace? It's an update (~) when every changed attribute is mutable — the provider's API can modify it in place. It's a replace (-/+) the instant any one changed attribute is marked ForceNew in the provider schema, because that attribute is immutable on the real resource (you can't move a storage account's region). One immutable attribute in a sea of mutable ones still forces a full replace. That boundary is exactly what the lab's plan_resource decides.

Q: Why does Terraform destroy resources in reverse dependency order? Because you can't delete a resource something else still depends on. A storage container lives inside a storage account; a subnet inside a VNet; a NIC references a subnet. To tear down cleanly Terraform walks the dependency graph backwards — dependents first, their dependencies last — so nothing is deleted while something still needs it. Creates and updates go the forward (topological) way for the same reason: a thing is built only after what it depends on exists.

Q: What is drift and how does terraform plan surface it? Drift is "real ≠ state" — the real resource diverged from what state records, because something changed it outside Terraform (a portal edit, another tool, a human). plan runs a refresh first: it reads each managed resource's real attributes through the provider and updates an in-memory copy of state to match reality, then diffs config against that refreshed reality. So drift appears in the plan as changes I didn't write. I don't blindly re-apply — I decide: re-apply if config should win, update config/import if the change was intentional, or ignore_changes if something else legitimately owns the attribute.

Q: How do you stop two engineers corrupting shared state? Remote state with locking. On Azure the backend is an Azure Storage blob, and before any state-mutating operation Terraform acquires a blob lease on the state blob — an exclusive short-lived claim. A second concurrent apply fails fast with "Error acquiring the state lock" instead of interleaving writes and corrupting state. I'd also version the blob (recover a bad apply) and restrict access, because state often holds secrets in plaintext. force-unlock exists for orphaned locks but is dangerous — using it against a live apply re-creates the corruption the lock prevents.

Q: What does CDKTF change about Terraform, and what does it leave the same? CDKTF changes the authoring layer: I write infrastructure in a real language (Python, TypeScript, …) with loops, conditionals, classes, and tests, then cdktf synth to Terraform JSON (cdk.tf.json). It leaves the engine completely unchanged — the synthesized JSON feeds the exact same plan/apply, the same ForceNew replace rules, the same DAG, the same state file and locking. CDKTF is a config generator, not a new model. So everything in this lab is what runs under CDKTF too.

Q: Terraform or Bicep for a new Azure platform? It's a tradeoff, not a ranking. If we're all-in on Azure and want to avoid managing a state file at all, Bicep's stateless PUT model (with what-if for the dry-run) is simpler — no state to store, lock, lose, or leak secrets through. If our graph spans clouds or SaaS, or we want one reviewable plan across everything, Terraform's state model earns its keep despite the fragility. Many orgs run both: Bicep for Azure-native landing zones, Terraform where the dependency graph crosses provider boundaries. I'd write the ADR around "do we need a unified multi-provider graph?" and "what's our appetite for managing state?"

References

• HashiCorp — Terraform: State (purpose, remote state, locking): https://developer.hashicorp.com/terraform/language/state
• HashiCorp — Resources: behavior & lifecycle (create_before_destroy, prevent_destroy, ignore_changes): https://developer.hashicorp.com/terraform/language/resources/behavior
• HashiCorp — Plugin SDK: schema behaviors (ForceNew and how providers declare immutability): https://developer.hashicorp.com/terraform/plugin/sdkv2/schemas/schema-behaviors
• HashiCorp — Backends: azurerm (Azure Storage state backend + blob-lease locking): https://developer.hashicorp.com/terraform/language/settings/backends/azurerm
• HashiCorp — Manage resource drift (refresh, -refresh-only, reconciliation): https://developer.hashicorp.com/terraform/tutorials/state/resource-drift
• HashiCorp — CDK for Terraform (synthesize to Terraform JSON, the same engine): https://developer.hashicorp.com/terraform/cdktf
• Terraform Registry — AzureRM provider (per-resource schema; which attributes are ForceNew): https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
• Microsoft Learn — Store Terraform state in Azure Storage: https://learn.microsoft.com/azure/developer/terraform/store-state-in-azure-storage
• Microsoft Learn — Bicep vs Terraform / what-if (the stateless contrast): https://learn.microsoft.com/azure/azure-resource-manager/bicep/deploy-what-if
• The track's own CHEATSHEET.md and GLOSSARY.md.

🛸 Hitchhiker's Guide — Phase 02: Terraform, CDKTF & IaC State Internals

Read this if: you can write a Terraform module and run apply, but a -/+ in the plan doesn't make your stomach drop yet, and "where does your state live and how is it locked" isn't a question you can answer in one sentence. This is the compressed tour the WARMUP derives slowly. Skim it, then read the WARMUP, then build the lab.

0. The 30-second mental model

Terraform is the diff between the world you wrote and the world it remembers. Three artifacts: config (desired), state (recorded reality, terraform.tfstate), plan (the diff). apply executes the plan and writes new state. ARM (P01) is stateless — it re-derives reality every deploy. Terraform is stateful — it trusts a file. Everything good (a portable, reviewable, multi-cloud plan) and everything scary (state loss, drift, the -/+ that deletes your database) follows from that one decision.

One sentence to tattoo: read the plan for -/+ — that's a real resource about to be destroyed.

1. The plan symbols, one breath

 +    create        in config, not in state
 ~    update         differs, only MUTABLE attrs changed     (in place, safe-ish)
-/+   replace        differs, a ForceNew (IMMUTABLE) attr     (DESTROY then create ← danger)
 -    delete         in state, not in config
(none) no-op         identical                                (= converged = good)

Plan: N to add, M to change, K to destroy is just these counted. The whole job of reading a plan is hunting the -/+ and the - you didn't intend.

2. The numbers and facts to tattoo on your arm

3. The dials you'll turn forever

update ⟷ replace            (mutable attr = ~ ; immutable = -/+ destroys real infra)
destroy-then-create ⟷ create-before-destroy   (downtime vs name collisions)
implicit deps ⟷ depends_on  (prefer references; depends_on serialises & hides the graph)
config wins ⟷ reality wins ⟷ ignore_changes   (the three drift reconciliations)
local state ⟷ remote+locked  (laptop = solo & fragile; blob+lease = team-safe)
HCL ⟷ CDKTF                 (static & simple vs generated config in a real language)
Terraform ⟷ Bicep           (multi-cloud + reviewable plan vs no state file to lose)

The interview signal: when they say "I want to rename this storage account," can you say "that's a ForceNew — it'll be a -/+, which destroys it; is there data behind it?"

4. The terraform / az one-liners you'll actually type

# bootstrap remote state (the first thing a real team does)
az group create -n rg-tfstate -l eastus
az storage account create -n sttfstate -g rg-tfstate --sku Standard_LRS --encryption-services blob
az storage container create -n tfstate --account-name sttfstate

# the core loop
terraform init                              # download providers, configure the backend
terraform plan -out=tfplan                  # compute the diff; SAVE it so apply runs exactly it
terraform apply tfplan                       # execute the saved plan (takes the blob-lease lock)

# read the plan for danger
terraform plan | grep -E '(-/+|# .* will be (destroyed|replaced))'   # find the replaces/deletes

# drift
terraform plan -refresh-only                # show drift WITHOUT proposing config changes
terraform apply -refresh-only               # accept reality into state (no infra change)

# state surgery (know these exist; use with care)
terraform state list                         # what state thinks it owns
terraform state show azurerm_storage_account.main
terraform import azurerm_resource_group.main /subscriptions/…/resourceGroups/rg-app  # adopt an existing resource
terraform state mv  <old> <new>             # rename without destroy/create
terraform state rm  <addr>                  # forget a resource (does NOT delete it)
terraform force-unlock <LOCK_ID>            # ⚠️ only for an ORPHANED lock, never a live apply

# CDKTF
cdktf synth                                  # run your program → cdk.tf.json
cdktf plan ; cdktf deploy                    # same engine, programmatic front-end

5. War story shapes you'll relive

• "The apply deleted our database." → the plan had a -/+ on it (a ForceNew attr — a rename, a region, a SKU) and someone skimmed past it and typed yes. Fix is reading the plan: every -/+ is a destroy. Often the cause is "tightened the name to satisfy a new naming policy" — and the name is immutable.
• "Two of us applied at once and state is garbage." → no locking, or someone force-unlocked a live run. Fix is remote state + blob-lease locking, and never force-unlocking a running apply.
• "The plan wants to change things I never touched." → drift. Someone edited the resource in the portal; refresh caught it. Decide: revert (re-apply), keep (update config / import), or ignore_changes.
• "We lost the state file." → state on a laptop, no remote backend, no versioning. Now Terraform thinks it owns nothing and wants to create resources that already exist. Hours of terraform import. This is why remote + versioned state is non-negotiable.
• "create_before_destroy still caused an outage." → the resource had a hard-coded unique name, so the new one collided with the old. Use name_prefix / let it be generated.

6. Vocabulary that signals you've held the pager

• State — Terraform's recorded reality; an address→attributes map it trusts.
• ForceNew / replace (-/+) — an immutable attribute changed → destroy-then-create.
• create_before_destroy — the lifecycle flag that makes a replace zero-downtime.
• Drift — real ≠ state; the out-of-band change plan's refresh surfaces.
• Backend — where state lives (AzureRM = a storage blob) and how it's locked.
• Blob lease — the Azure lock that prevents concurrent state corruption.
• import / state mv / state rm — adopt / rename / forget without touching infra.
• Refresh — the read-real-resources phase before the diff.
• Synthesize (cdktf synth) — turn a CDKTF program into Terraform JSON.
• prevent_destroy — the lifecycle guardrail that fails the plan on a delete/replace of a protected resource — the seatbelt for stateful resources.

7. Beginner mistakes that mark you in interviews

1. Not knowing the three artifacts — talking about apply without distinguishing config, state, and plan.
2. Treating ~ and -/+ as the same ("it'll just change it") — a replace destroys the resource.
3. Saying "Terraform handles drift automatically" — it detects drift via refresh; you decide how to reconcile.
4. Not having an answer to "where does state live and how is it locked."
5. Forgetting deletes go in reverse dependency order.
6. depends_on everywhere — serialises parallelisable resources and hides the real graph.
7. Thinking CDKTF is a different engine — it synthesizes to Terraform JSON and runs the same plan/apply.
8. Forgetting state holds secrets in plaintext — so it needs encryption + access control like the secret store it effectively is.
9. force-unlock as a routine fix — it re-creates the corruption locking prevents.

8. How this phase pays off later

• The plan/apply engine (the lab) is the engine under every IaC tool — CDKTF, and the same shape as ARM/Bicep (P01) and CloudFormation/Pulumi.
• The ForceNew replace instinct protects every stateful resource you'll ever manage — databases (P00 data layer), public IPs and subnets (P06), Key Vaults (P12).
• Remote state + locking is the same shared-mutable-state-with-a-lock pattern you'll see in Service Bus sessions (P10) and distributed coordination everywhere.
• Drift reconciliation is the daily reality of governed estates (P04 RBAC, P05 landing zones) where humans and policy can also mutate resources.
• The stateful vs stateless tradeoff is a Round-5 architecture ADR (P15).

Now read the WARMUP slowly, build the state engine, and never type yes on a plan you haven't read for -/+ again.

👨🏻 Brother Talk — Phase 02, Off the Record

No slides, no certification objectives. This is me, your brother, telling you what I wish someone had told me before I learned it the expensive way. Read it once now, and again the day you're about to run an apply against production.

Listen. Almost everyone who "knows Terraform" knows it the way you know how to drive without knowing what's under the hood: you can get from A to B, and you're completely lost the moment something goes wrong. They can write a module, run apply, and feel productive. And then one day the plan says -/+ next to a database, they don't notice, they type yes, and the database is gone. Not down. Gone. Recreated empty.

I'm not telling you that to scare you. I'm telling you because the entire difference between the person who causes that outage and the person who catches it in review is one habit: they read the plan. That's it. That's the whole separation between senior and principal in IaC. Not memorizing more resource types. Reading the plan like it's a loaded document, because it is.

So let me give you the things that actually matter, brother to brother.

The state file is not a config file. It's a load-bearing wall. Here's the mental shift that took me too long: config is just text you can rewrite anytime. State is Terraform's memory of what it owns in the real world. Lose it and Terraform doesn't lose your config — it loses the knowledge that your production estate belongs to it. The next plan looks at an empty memory and a full config and cheerfully proposes to create everything you already have. I've watched a team spend two days terraform import-ing a production environment back into a state file someone deleted "to start clean." Treat state like you'd treat the master key to the building: remote, locked, versioned, access-controlled, and never on a laptop.

Fall in love with the -/+. Everyone's eyes glaze over reading a plan — it's long, it's green, it scrolls. Train yourself to do the opposite of glaze: scan for -/+ and - first, every single time, before you read anything else. Those two symbols are the only ones that can hurt you. + creates, ~ tweaks — fine. -/+ destroys and recreates; - destroys. The moment you make "grep the plan for -/+" a reflex, you've internalized the one thing that makes you safe to give production access to. I literally pipe my plans through a grep for it. It's not paranoia; it's the job.

Immutability is a fact about the world, not a Terraform quirk. You can't move a house to a different city; you build a new one and tear the old one down. A storage account's region, a resource's name, certain VM SKUs — same thing. ForceNew isn't Terraform being annoying; it's Terraform being honest that the underlying cloud API physically cannot change that attribute in place. So when a "small rename" turns into a -/+, don't fight it — understand it. The question is never "why is Terraform recreating this?" It's "is there data or identity behind this resource that recreation destroys?" If yes, you have a real engineering problem (move the data, use create-before-destroy, or don't make the change) — not a Terraform problem.

Drift is not Terraform's failure; it's the world being the world. People get weirdly moralistic about drift — "someone touched the portal, that's against the rules." Sure, reduce the surface, restrict portal access, alert on drift. But also: sometimes the person who edited the resource in the portal at 3 a.m. was right — they were stopping an incident and didn't have time for a PR. The principal move isn't to reflexively re-apply and revert their fix. It's to ask why the drift exists and choose: revert it, adopt it into config, or ignore it. A blind terraform apply that undoes someone's emergency fix is its own incident. Drift is a conversation, not a violation.

Locking is the cheapest insurance you'll ever buy, and force-unlock is a loaded gun. The whole reason remote state takes a blob lease is so two people can't apply at once and shred the state. It works silently and you never think about it — until someone's apply crashes and leaves an orphaned lock, and the "fix" everyone reaches for is force-unlock. Here's the trap: if you force-unlock while someone's apply is actually still running, you just re-created the exact corruption the lock was preventing. Use it only when you're certain the lock is orphaned. When in doubt, find the human, don't unlock.

CDKTF won't save you from understanding the engine — and that's good news. Some people reach for CDKTF hoping a real language will let them skip learning how Terraform actually works. It won't. CDKTF is just a fancier way to write the config; it synthesizes to Terraform JSON and runs the identical plan/apply, the identical state file, the identical -/+ rules. Which means the thing you're learning in this lab — the engine — is exactly what makes you good at CDKTF for free. Learn the engine once, and HCL, CDKTF, and honestly the shape of ARM and Pulumi and CloudFormation all become front-ends you already understand. That's the leverage of going under the hood: you stop learning tools and start recognizing the same machine wearing different clothes.

The honest truth about this phase's difficulty: the algorithm is not hard. A five-way diff and a topological sort — you'll write it in an afternoon. What's hard is the judgment: knowing that a -/+ on a stateless thing is fine and on a database is a catastrophe, knowing when drift should be reverted versus adopted, knowing that state is the fragile thing to protect and not the config. That judgment is what the interview is actually testing when they ask "walk me through what happens when you run apply." They're not checking if you can recite the steps. They're checking if you respect the state file and fear the -/+ — because that respect is the difference between someone you can trust with production and someone whose first bad day takes the company down with it.

One more thing, and it's the one that compounds. Write down which resources can never be replaced, and protect them. Put prevent_destroy on your stateful crown jewels — the production database, the state storage account itself, the Key Vault. It's a lifecycle flag that makes the plan fail if anything would destroy them. It feels like overhead until the day it turns a 2 a.m. catastrophe into a clean error message that says "no." That's the whole game in this role: turning the irreversible into the merely annoying, on purpose, before you need it.

Go build the state engine. Read every plan for the -/+ from now on. Then come find me in Phase 03, where we make identity click — OAuth, OIDC, JWTs — the stuff everyone name-drops and quietly can't draw on a whiteboard.

— your brother 👨🏻

Lab 01 — Terraform State Engine

Phase: 02 — Terraform, CDKTF & IaC State Internals | Difficulty: ⭐⭐⭐☆☆ | Time: 3–4 hours

ARM (Phase 01) is stateless: every PUT re-derives reality from the resource provider, so there is no file to drift. Terraform is stateful: it remembers what it created in a terraform.tfstate file, and the entire tool is the diff between the world you wrote (config) and the world it remembers (state). This lab builds that engine — the per-resource diff, the ForceNew "replace not update" decision that destroys real infrastructure, the dependency ordering that flips for deletes, and the drift detector that catches the portal edit nobody told Terraform about.

What you build

A miniature Terraform plan/apply engine over three artifacts — config (desired), state (recorded reality), plan (the diff):

• Resource — address ("azurerm_storage_account.main" = type.name, the state key), attributes (a flat dict), and the set of ForceNew attribute names.
• diff_attributes(desired, current) — the leaf comparison: attribute → (old, new) for every changed / added / removed key.
• plan_resource(desired, current, force_new) — the four-way decision: create (no state) · no-op (identical) · update (only mutable attrs moved) · replace (a ForceNew attr moved → destroy-then-create).
• plan(config, state, force_new_by_type, deps) — the whole diff: classify every resource, mark orphaned state for delete, and order it — creates/updates/replaces in topological order, deletes in reverse topological order. Returns a Plan with .actions and a .summary count.
• apply(plan, config, state) — execute the plan, return the new state without mutating the input (the property terraform apply depends on to be re-runnable).
• detect_drift(state, real) — address → {attr: (state_val, real_val)} for resources whose real attributes diverge from recorded state: the out-of-band change terraform plan surfaces after its refresh.

Key concepts

Files

Run

pip install -r requirements.txt
pytest test_lab.py -v                      # against your lab.py
LAB_MODULE=solution pytest test_lab.py -v  # against the reference
python solution.py                         # the worked example

Success criteria

• All 28 tests pass against your implementation.
• You can explain why test_plan_resource_force_new_boundary is the crux of the lab: the same diff is an update with an empty force_new and a replace the instant one changed attribute is immutable — and a replace destroys real infrastructure.
• You can explain why test_plan_deletes_orphaned_state_in_reverse_order reverses the order: you cannot delete a subnet a NIC still sits in, so dependents die first.
• You can explain why apply must not mutate its input state (a partially-applied run must leave the on-disk state recoverable; tests assert it stays untouched).
• You can read a terraform plan and instantly flag every -/+ (replace) line as a potential outage before you type yes.

How this maps to real Azure (and real Terraform)

Two things the miniature deliberately omits but you must know:

• Locking. Real remote state takes a lease on the Azure Storage blob for the duration of an apply; a second apply gets Error acquiring the state lock. Without it, two concurrent applies corrupt state. (WARMUP Ch. 6.)
• create_before_destroy. A replace is destroy-then-create by default, which means downtime; the create_before_destroy lifecycle flag inverts the order to create the new resource first — but only works when names/addresses don't collide. (WARMUP Ch. 4.)

Extensions (for your own subscription / deeper study)

• Add replace ordering. Make plan emit a replace as an explicit (address, "destroy") + (address, "create") pair, and support a per-resource create_before_destroy flag that flips their order. Test that a CBD replace creates before it destroys.
• Add terraform import. A function that takes an unmanaged real resource and writes it into state (address → attributes) so the next plan is no-op instead of create — the real tool's answer to "ClickOps already made this."
• Render the plan. Emit the + / ~ / -/+ / - symbols and a colored diff per attribute so your output reads like real terraform plan — then diff it against actual Terraform on a throwaway resource group.
• Author it in CDKTF. Re-express the worked-example config in CDKTF (Python), cdktf synth it to cdk.tf.json, and confirm the synthesized JSON feeds the same plan/apply engine — the whole point of CDKTF.

Phase 03 — Microsoft Entra ID: OAuth2 / OIDC / JWT

Difficulty: ⭐⭐⭐☆☆ (the protocol) → ⭐⭐⭐⭐⭐ (the judgment about grants, claims, and the validation order) Estimated Time: 1 week (12–18 hours) Prerequisites: Phase 00 (control plane vs data plane, identity is the perimeter) and Phase 01 (the ARM request pipeline — this phase builds the authenticate gate that pipeline starts with). Comfortable reading JSON. No live Entra tenant required for the lab.

Why This Phase Exists

Phase 01 taught the single most important sentence about the Azure control plane: every write is one idempotent PUT through ARM, and ARM's first gate is authenticate the caller's Entra token. This phase opens that gate and builds the machine inside it — because identity is the perimeter, and in a cloud with no network edge to hide behind, the token is the security boundary. Get the token wrong and nothing else you built matters.

Here is the thing almost nobody can actually do in an interview, even people who've wired up "Login with Microsoft" a dozen times: draw the authorization-code + PKCE flow from memory, say exactly what PKCE defends against, and then list — in order — every check a resource API must run on the JWT it receives. They can name OAuth2. They cannot tell you why a public SPA must use PKCE but a daemon must not, why the access token's aud is your API but the ID token's aud is the client, what the difference between scp and roles signals about who is calling, or which single skipped check turns your service into a confused deputy that honors a token minted for somebody else.

Those are not academic distinctions. "We validated the signature but not the aud" is a real breach shape: service A accepts a perfectly-signed, perfectly-current token that Entra issued for service B, and now an attacker with legitimate access to B is acting as A. "alg: none" is a real CVE family. "The clock skew rejected every token for ten minutes" is a real outage. This phase makes you build the token engine — the base64url codec, the PKCE proof, the authorization server's two grants, and above all the validator whose check order is the security — so that OAuth2 stops being three letters you nod at and becomes a sequence of if statements you can defend line by line. This is the keystone for P04 (RBAC, which runs after this gate), P09 (the API gateway that runs this validation on every request), and P12 (Managed Identity, which is just this flow with the secret removed).

What "Principal-Level" Means Here

A senior engineer wires an app to Entra with an SDK and it works. A principal understands the protocol well enough to:

• Pick the right grant on sight — authorization-code+PKCE for a user in a browser/native app, client-credentials for a daemon with no user, on-behalf-of for a middle tier passing the user's identity downstream, device-code for a screen with no keyboard — and explain why each alternative is wrong for the case.
• Read a JWT like a sentence — name what every claim means (iss, aud, exp, nbf, scp, roles, appid/azp, oid/tid), tell an access token from an ID token by its aud, and tell a delegated call from an app-only call by scp-vs-roles.
• Recite and defend the validation order — structure/alg → signature → iss → aud → nbf/exp (with skew) → scope/role — and explain why each step is where it is and what breaks if you reorder or skip one (the confused deputy, the alg:none bypass, the timing side-channel a non-constant-time compare leaks).
• Explain PKCE from first principles — code_challenge = BASE64URL(SHA256(verifier)), why the one-way hash means an attacker who intercepts the auth code still can't redeem it, and why this replaced the implicit flow entirely for public clients.
• Reason about Conditional Access as a policy engine over signals (user, device, location, risk) that gates token issuance — the layer that turns "valid credentials" into "valid credentials, from a compliant device, in an expected place, with MFA."

Concepts

• The Entra ID model — a tenant is an isolated directory (a GUID); it holds users and groups; an app registration is the global definition of an application (its client id, redirect URIs, secrets/certs, exposed scopes, app roles), and a service principal is that app's local identity in a tenant (what role assignments and consent attach to). One app registration → one service principal per tenant it's used in. Get this app-registration-vs-service-principal distinction cold; it confuses everyone.
• The v2.0 endpoints & discovery — https://login.microsoftonline.com/{tenant}/oauth2/v2.0/{authorize,token}, and the self-describing OpenID Connect discovery document at …/{tenant}/v2.0/.well-known/openid-configuration, which publishes the issuer, the jwks_uri, and every endpoint so SDKs configure themselves. JWKS is the issuer's published set of public signing keys; the token header's kid selects which one.
• OAuth2 grants — and exactly when each is correct. Authorization code (+ PKCE) for an interactive user (the only right answer for SPAs and native/mobile apps). Client credentials for a daemon/service with no user (app-only, authenticates with its own secret or cert). On-behalf-of (OBO) for a middle-tier API that must call a downstream API as the original user. Device code for input- constrained devices (TVs, CLIs, IoT). Choosing wrong is a design smell an interviewer pounces on.
• Access token vs ID token vs refresh token. The access token authorizes a call to a resource — its aud is the API, and it carries scp/roles; treat it as opaque and never decode it as a client. The ID token (OIDC) authenticates the user to the client — its aud is the client, and it carries sub/name/preferred_username. The refresh token buys new access tokens without re-prompting; it never leaves the client.
• OIDC: authentication vs authorization. OAuth2 is authorization ("what may this caller do"); OIDC layers authentication on top ("who is this user") via the ID token and the userinfo/discovery machinery. Authentication answers who; authorization answers what. A principal never conflates them — and the lab's validator separates them (signature+iss+ aud+time = authentication of the token; scp/roles = authorization).
• PKCE (RFC 7636). A public client can't keep a secret, so it can't prove at the token endpoint that it started the login. PKCE fixes it without a secret: send code_challenge = BASE64URL(SHA256(code_verifier)) (method S256) to /authorize, then present the raw code_verifier at /token; the server re-hashes and compares. An attacker who intercepts the code can't redeem it — they never saw the verifier.
• JWT structure & the validation ORDER. base64url(header).base64url(payload).base64url(signature). Validate, failing closed, in this order: kid → JWKS key → signature → iss → aud → nbf/exp (with clock skew) → authorization claim. scp (space-delimited) is delegated authority (a user delegated it); roles is app authority (the app itself was granted it). Skip aud and you've built the confused deputy.
• Conditional Access (framing). A policy engine: signals (user/group, device compliance, location/IP, sign-in risk, app) → a decision (grant / block / require MFA / require a compliant device). It gates token issuance at /authorize — the modern "Zero Trust" control plane on top of OAuth2.

Labs

Lab 01 — OAuth2 / OIDC / JWT Engine (flagship, implemented)

→ Lab folder: lab-01-oauth-oidc-engine/

Integrated-Scenario Suggestions (carried through the whole track)

The phases compound. Keep these in mind as you build — each plugs the token engine into a larger system:

1. SPA → API → downstream API — a browser app uses auth-code+PKCE to call your API (access token, aud = your API, scp carries the user's scopes); your API uses on-behalf-of to call Microsoft Graph as the user. Three audiences, two grants, one user identity preserved end-to-end. (→ P09)
2. Daemon ingestion job — a nightly service uses client-credentials (no user) to write to a storage account; its token carries roles, and the storage RP authorizes via RBAC on the service principal. The grant choice is the identity model. (→ P04, P10)
3. API gateway authorization — API Management (or your API) validates every inbound JWT — kid→JWKS→signature→iss→aud→exp→scp/roles — then routes; an invalid aud is a 401, a missing scope is a 403. This lab's validate_jwt is that policy. (→ P09)
4. CI/CD without stored secrets — a pipeline presents its OIDC token to Entra and gets an Azure access token via workload-identity federation (matching issuer/subject/ audience) — the same JWT validation, applied to a federated credential, with nothing to leak. (→ P08)
5. Managed Identity — a Function asks IMDS for a token and gets an Entra access token with no secret at all; downstream APIs validate it exactly like any other Entra JWT. This phase is the protocol; P12 removes the credential. (→ P12)

Guides in This Phase

• HITCHHIKERS-GUIDE.md — the 30-minute orientation; read first
• WARMUP.md — the full primer; read slowly
• BROTHER-TALK.md — the candid version

Key Takeaways

• Identity is the perimeter, and the JWT is the boundary. In a cloud with no network edge, the token is what stands between your service and the world — so the validator that checks it is the most security-critical code you'll write.
• The grant encodes the identity model. Auth-code+PKCE = a user is present; client-creds = an app acting as itself; OBO = a middle tier acting as the user; device-code = no browser. Choosing the grant is deciding who is calling and how they prove it.
• aud tells access token from ID token; scp-vs-roles tells delegated from app-only. Two distinctions that, once internalized, make every token legible at a glance.
• The validation order is the security. Verify the signature before you read any claim (it's attacker JSON until then); check aud always (or you're a confused deputy); pin the alg (or you accept alg:none); compare in constant time; allow clock skew. Order is not style — reorder it and you have a vulnerability.
• PKCE made public clients safe without secrets, which is the same idea that makes Managed Identity (P12) and OIDC federation (P08) the modern, secretless default: prove possession, don't store a password.

Deliverables Checklist

• Lab 01 implemented; all 32 tests pass against solution.py and your lab.py
• You can draw the authorization-code + PKCE flow from memory and say what PKCE defends against
• You can list the JWT validation order and explain why each step is where it is
• You can explain access token vs ID token (by aud) and scp vs roles (by grant)
• You can explain the confused-deputy attack and the single check that prevents it
• You can pick the correct grant for a SPA, a daemon, a middle tier, and a CLI — and say why the others are wrong
• You can explain app registration vs service principal in one breath

Warmup Guide — Microsoft Entra ID: OAuth2 / OIDC / JWT

Zero-to-principal primer for Phase 03: the Entra identity model (tenant, app registration vs service principal, the v2.0 endpoints, discovery, JWKS), the OAuth2 grants and exactly when each is correct, OIDC and the authentication-vs-authorization line, PKCE from first principles, the JWT wire format and the validation order that is the security, and the Conditional Access framing. Every concept goes from what it is to why it exists to the mechanism under the hood (diagrams, tables, code) to production significance to the misconceptions that get people owned.

Table of Contents

• Chapter 1: Why Identity Is the Perimeter
• Chapter 2: The Entra ID Model — Tenant, App Registration, Service Principal
• Chapter 3: The Endpoints, Discovery, and JWKS
• Chapter 4: OAuth2 — The Grants and When Each Is Correct
• Chapter 5: Access Token vs ID Token vs Refresh Token
• Chapter 6: OIDC — Authentication vs Authorization
• Chapter 7: PKCE — The Proof Key, From First Principles
• Chapter 8: The JWT Wire Format
• Chapter 9: The Validation Order Is the Security
• Chapter 10: scp vs roles, and the Confused Deputy
• Chapter 11: Conditional Access — Signals to Decision
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Why Identity Is the Perimeter

From zero. For decades, security meant a network perimeter: a firewall, a DMZ, a VPN — "inside is trusted, outside is not." That model dies in the cloud. Your services run on shared infrastructure, talk to managed APIs over the public internet, and scale across regions; there is no wall to stand behind. So the industry moved the boundary from where you are to who you are: identity is the new perimeter, and every single request must carry a verifiable proof of identity that the receiver checks before doing anything.

What it is. In Azure that proof is a JSON Web Token (JWT) minted by Microsoft Entra ID (formerly Azure AD). Every ARM write (Phase 01's pipeline starts by authenticating this token), every Microsoft Graph call, and every API you protect receives a bearer token and must decide: is this real, is it for me, is it current, and does it permit this action?

Why it exists / production significance. Because the alternatives are worse. Shared API keys leak and never expire; IP allow-lists break the moment anything moves; passwords on the wire are a disaster. A short-lived, cryptographically-signed, audience-scoped token solves all three: it expires (so a leak is time-boxed), it's signed (so it can't be forged), and it names its intended audience (so it can't be replayed elsewhere — if you check). The entire rest of this phase is the machinery that issues and validates that token.

Misconception. "We're behind a VNet / private endpoint, so identity is less important." No — defense in depth means the token is still the boundary inside the network too; lateral movement after one foothold is exactly what a stolen-but-unchecked token enables. The network is a layer; identity is the perimeter.

Chapter 2: The Entra ID Model — Tenant, App Registration, Service Principal

From zero. Before any tokens, you need the nouns. Three of them trip up nearly everyone.

• A tenant is an isolated instance of the Entra directory — one organization's identity universe, identified by a tenant ID (a GUID). Your users, groups, and apps live in a tenant. The token issuer (iss) is scoped to a tenant; a token from tenant A is not a token from tenant B, full stop.
• Users and groups are the human identities and their collections (groups are how you assign access at scale instead of per-user).
• An app registration is the global definition of an application: its client id (appId, a GUID), its redirect URIs, its secrets/certificates (for confidential clients), the scopes it exposes (if it's an API) and app roles it defines, and its Application ID URI (e.g. api://contoso-api). It lives in the tenant where the app was registered (its home tenant).
• A service principal is the app's local identity in a tenant — the concrete object that role assignments, consent, and Conditional Access attach to. When an app is used in a tenant, a service principal for it exists there.

Under the hood — the relationship. The cleanest analogy is class vs instance:

So a multi-tenant app has one app registration (its global definition) and N service principals (one in each tenant that consented to it). You assign an RBAC role (Phase 04) to the service principal, not the registration.

App registration "contoso-api"  (home tenant: contoso)
   client_id = 11111111-...
   exposes scope  Files.Read
   defines  app role  Invoices.Process.All
        │  (used in two tenants)
        ├── service principal in tenant CONTOSO  ← RBAC + consent attach here
        └── service principal in tenant FABRIKAM ← and here, independently

Production significance. "It worked in my tenant but 403s in the customer's" is almost always a missing service principal or consent in the customer tenant, not a code bug. "Where do I grant the daemon access to the storage account?" — to its service principal, by object id.

Misconception. "App registration and service principal are the same thing / are interchangeable." They are not. The registration is the definition; the service principal is the identity-in-a-tenant. The portal blurs them, which is why so few engineers can explain the difference — and why it's a great interview filter.

Chapter 3: The Endpoints, Discovery, and JWKS

From zero. Every tenant exposes a fixed set of OAuth2/OIDC endpoints under https://login.microsoftonline.com/{tenant}/. The two you'll touch constantly are the v2.0 endpoints:

• …/{tenant}/oauth2/v2.0/authorize — the front-channel: a browser redirect where the user authenticates and (in the auth-code flow) a one-time code comes back.
• …/{tenant}/oauth2/v2.0/token — the back-channel: a server-to-server POST that exchanges a grant (a code, a secret, an assertion) for tokens.

What discovery is. Rather than hard-code those URLs and the signing keys, a client reads the OpenID Connect discovery document:

GET https://login.microsoftonline.com/{tenant}/v2.0/.well-known/openid-configuration
→ {
    "issuer":                 "https://login.microsoftonline.com/{tid}/v2.0",
    "authorization_endpoint": ".../oauth2/v2.0/authorize",
    "token_endpoint":         ".../oauth2/v2.0/token",
    "jwks_uri":               "https://login.microsoftonline.com/{tid}/discovery/v2.0/keys",
    "id_token_signing_alg_values_supported": ["RS256"],
    ...
  }

Every Entra SDK fetches this on startup so it never hard-codes an endpoint or a key. The issuer here is the exact string you'll later require a token's iss to equal.

What JWKS is. The jwks_uri returns a JSON Web Key Set — the issuer's public signing keys:

{ "keys": [
  { "kid": "abc123", "kty": "RSA", "use": "sig", "n": "<base64url-modulus>", "e": "AQAB" },
  { "kid": "def456", "kty": "RSA", "use": "sig", "n": "...",                  "e": "AQAB" }
]}

Under the hood — how kid ties it together. A signed token's header names a kid (key id). To verify, you (1) read the unverified header, (2) find the JWKS key whose kid matches, (3) verify the signature with that public key. Entra rotates these keys, which is why you fetch JWKS by kid and cache with refresh — never pin one key forever.

In this phase's lab we sign with HS256 (a shared symmetric secret) instead of Entra's RS256 (asymmetric) purely to stay offline and deterministic — no keypair, no network. The kid→key→signature shape is identical; only the signature primitive differs. The first extension swaps HS256 for RS256+JWKS and you've modeled real Entra exactly.

Misconception. "I'll just download the public key once." Keys rotate; a validator that caches one key forever starts rejecting every token the day Entra rolls the key. Cache by kid, refresh on an unknown kid.

Chapter 4: OAuth2 — The Grants and When Each Is Correct

From zero. OAuth2 (RFC 6749) is an authorization framework: it defines how a client obtains an access token to call a resource on behalf of (or as) some principal, without that resource ever seeing a password. The variations are called grants (or flows), and picking the right one is the first design decision — and a favorite interview probe, because the wrong grant is a security smell.

The four that matter in Azure:

Under the hood — the canonical one (authorization code + PKCE):

 user        client (SPA/native)            Entra /authorize        Entra /token        resource API
  │                 │                              │                      │                   │
  │ click "Sign in" │                              │                      │                   │
  │────────────────►│  redirect with               │                      │                   │
  │                 │  client_id, redirect_uri,    │                      │                   │
  │                 │  scope, code_challenge ──────►│  (user authenticates,│                   │
  │                 │                              │   Conditional Access) │                   │
  │                 │◄──── redirect back with ─────│                      │                   │
  │                 │      one-time `code`         │                      │                   │
  │                 │  POST code + code_verifier ─────────────────────────►│ verify verifier   │
  │                 │                              │                      │ hashes to challenge│
  │                 │◄──── access_token + id_token (+ refresh_token) ──────│                   │
  │                 │  call API with Bearer access_token ──────────────────────────────────────►│
  │                 │                              │                      │   validate JWT ────►│

Why each exists / when it's wrong.

• Authorization code + PKCE is the only correct flow for a user in a browser or native app. The older implicit flow (token straight from /authorize) is dead — it leaked tokens in URLs and browser history. PKCE made auth-code safe even for clients that can't keep a secret (Chapter 7), so it replaced implicit entirely.
• Client credentials is for no user. Using it where a user is present means you've lost the user's identity (everything becomes "the app did it"), which breaks auditing and least-privilege. Using auth-code for a daemon means there's no human to log in — it can't work.
• On-behalf-of is the only way to call a downstream API as the user from a middle tier. The wrong move — having the middle tier use client-credentials — silently elevates every downstream call to the app's permissions instead of the user's.
• Device code exists because some clients have no browser to redirect; using it elsewhere is just a worse UX.

Production significance. The grant is the identity model. When someone says "the API calls Graph," your first question is "as the user (OBO) or as the app (client-creds)?" — the answer changes the permissions, the audit trail, and the blast radius.

Chapter 5: Access Token vs ID Token vs Refresh Token

From zero. A flow can hand back up to three tokens, and conflating them is a top-three OAuth2 mistake.

Under the hood — the aud is the tell. The same login produces an access token whose aud is the API and an ID token whose aud is the client. That's not a detail — it's the whole point:

• The client must never use the access token's contents to make decisions; to the client it's an opaque string to forward. Decoding and trusting an access token in the front-end is a classic bug.
• The API must never accept an ID token in place of an access token — the ID token's aud is the client, not the API, so a correct validator (Chapter 9) rejects it on the aud check. (This is a real attack: present the ID token to the API and hope it doesn't check aud.)

Production significance. "Login works but the API rejects my token" is often the front-end sending the ID token (its aud is the client) where the API wants the access token (aud = the API). The aud mismatch is the validator doing its job.

Misconception. "A token is a token." No — three tokens, three audiences, three readers. Read the aud and you know instantly which one you're holding and whether it's for you.

Chapter 6: OIDC — Authentication vs Authorization

From zero. OAuth2 alone is about authorization — "what may this caller do." It was not designed to tell a client who the user is; people bolted that on insecurely for years. OpenID Connect (OIDC) is the standard layer that fixes it: it adds the ID token (a JWT proving the user's identity to the client), the discovery document (Chapter 3), and a userinfo endpoint.

The one sentence to carry forever:

Authentication answers who you are. Authorization answers what you may do. OIDC does the first; OAuth2 does the second; they ride together but are different questions.

Under the hood — where the line falls in your validator. This phase's validate_jwt literally splits along it:

AUTHENTICATION of the token  (is it real, current, for me, from whom I trust):
    structure / alg  →  signature  →  iss  →  aud  →  nbf/exp
AUTHORIZATION                  (may the caller do this specific thing):
    required_scopes ⊆ scp   and/or   required_roles ⊆ roles

A 401 Unauthorized is an authentication failure (bad/absent/expired/wrong-audience token). A 403 Forbidden is an authorization failure (a valid token that lacks the scope or role). Returning the wrong code leaks information and confuses clients; the distinction maps exactly onto the two halves above.

Production significance. Interviewers love "what's the difference between authn and authz?" — and they love it more when you say "and here's where the line falls in a JWT validator," because that proves you've implemented the distinction, not just defined it.

Misconception. "OIDC replaces OAuth2." No — OIDC is built on OAuth2; you use both at once. The ID token (OIDC) authenticates the user to the client; the access token (OAuth2) authorizes the call to the API.

Chapter 7: PKCE — The Proof Key, From First Principles

From zero. A confidential client (a server) can keep a secret, so at the token endpoint it can prove "I'm the app that started this login" by presenting that secret. A public client — a SPA whose code ships to the browser, a mobile app on a user's phone — cannot keep a secret; anything you embed is extractable. So how does a public client prove, at /token, that it (and not an attacker who intercepted the code) started the flow?

What PKCE is. Proof Key for Code Exchange (RFC 7636) solves it with a one-time, per- login proof and no stored secret:

1. The client invents a high-entropy random code_verifier (43–128 chars).
2. It computes code_challenge = BASE64URL(SHA256(code_verifier)) and sends only the challenge to /authorize (method S256).
3. At /token, it presents the raw code_verifier.
4. The server re-computes BASE64URL(SHA256(verifier)) and checks it equals the stored challenge.

Under the hood — why it defeats interception. SHA-256 is a one-way function. Write the challenge as

$$ \text{challenge} = \mathrm{BASE64URL}\big(\mathrm{SHA256}(\text{verifier})\big) $$

An attacker who intercepts the redirect sees the challenge but not the verifier, and cannot invert the hash to recover it. An attacker who intercepts the code (the prize in an auth-code interception attack) still can't redeem it at /token, because they can't supply the matching verifier. The flow is bound to whoever generated the verifier.

attacker sees:   challenge  ✗ (one-way)→ verifier        # can't go backward
attacker steals: the code   ✗ (no verifier)→ tokens      # code alone is useless
only the real client knows:  verifier  →  redeems code   # because it made it

S256 is the method you want. plain (where challenge == verifier, no hash) exists only for clients that genuinely can't SHA-256, and it's a downgrade — an interceptor of the challenge would have the verifier — so a production server should reject plain.

Production significance. PKCE is now mandatory for public clients and recommended for all clients (OAuth 2.1 folds it in). It is the reason the implicit flow is dead. The deeper lesson — prove possession instead of storing a password — is the same idea behind Managed Identity (P12) and OIDC federation (P08): the modern secretless default.

Misconception. "PKCE replaces the client secret." For public clients there was no usable secret to replace — PKCE is what makes secret-less safe. For confidential clients you use both (PKCE and the secret). And PKCE protects the code, not the token; you still validate the token (Chapter 9).

Chapter 8: The JWT Wire Format

From zero. A JWT is three base64url segments joined by dots:

        base64url(header)   .   base64url(payload)   .   base64url(signature)
        eyJhbGciOiJI...     .   eyJpc3MiOiJod...     .   dBjftJeZ4CV...

• header — JSON like {"alg":"HS256","typ":"JWT","kid":"abc123"}: the signature algorithm and the key id.
• payload — the claims JSON (the data): iss, aud, exp, scp, etc.
• signature — SIGN(key, base64url(header) + "." + base64url(payload)).

Crucial mechanical detail. The signature is computed over the encoded segments — the exact ASCII bytes header_b64 + "." + payload_b64 — not over the raw JSON. To verify you re-sign those same bytes and compare. A common bug is to JSON-parse, re-serialize, and re-sign — but JSON re-serialization can change byte order or spacing and the signature won't match. Verify the bytes that were signed.

base64url, not base64. JOSE uses URL-safe base64 (+→-, /→_) with the = padding stripped. Decoders must re-add padding to a multiple of four before decoding. Using standard base64 or leaving padding on is the "works locally, 401s in Entra" classic.

The claims you must know cold:

Misconception. "A JWT is encrypted, so it's safe to put secrets in it." A signed JWT (JWS) is signed, not encrypted — anyone can base64url-decode and read the payload (decode_jwt_unverified does exactly that). Never put a secret in a JWT payload. The signature gives integrity, not confidentiality.

Chapter 9: The Validation Order Is the Security

This is the chapter the whole lab exists for. A resource API that receives a bearer token must answer, in order, failing closed (any miss → reject):

1. structure / alg  parse 3 segments; header.alg == the algorithm I expect
                    (reject `alg: none` and algorithm-substitution)
2. signature        recompute and constant-time-compare (hmac.compare_digest)
   ───────────────  ↑ EVERYTHING ABOVE THIS LINE IS UNTRUSTED ATTACKER JSON ↑
3. iss              == my expected issuer (the tenant I trust)
4. aud              my API ∈ the token's audience(s)   ← skip → CONFUSED DEPUTY
5. nbf              nbf ≤ now + leeway     (token already valid, modulo skew)
6. exp              exp >  now − leeway    (token not expired, modulo skew)
7. authorization    required_scopes ⊆ scp   and/or   required_roles ⊆ roles

Why this order — under the hood.

• Signature first, before any claim. Until the signature verifies, the payload is just bytes an attacker controls. Reading iss or exp from an unverified token and acting on it is trusting attacker input. So the cryptographic gate comes before every claim check. Everything below the line in the diagram is only safe to read because the signature passed.
• Pin the alg. If you trust whatever alg the header claims, an attacker sets alg: "none" and ships an unsigned token — and a naive verifier "verifies" an empty signature and passes it. (There's a sibling attack: an RS256 verifier tricked into treating the RSA public key as an HS256 secret.) Defense: accept only the algorithm you expect. The lab's validator rejects anything but HS256; a real one pins RS256.
• Constant-time signature compare. Comparing the signature with a normal == that short-circuits on the first differing byte leaks, via timing, how many leading bytes matched — enough to forge a signature byte-by-byte over many requests. Use hmac.compare_digest, which compares in time independent of where the mismatch is.
• aud is not optional. See Chapter 10 — skipping it is the confused-deputy vulnerability.
• nbf/exp with leeway. Clocks drift. Entra allows ±5 minutes (300 s) of skew; a validator with zero tolerance rejects legitimate tokens whenever an NTP server hiccups. The boundary is the off-by-one interviewers probe: with leeway $L$ and current time $t$, a token is expired iff $exp \le t - L$ and not-yet-valid iff $nbf > t + L$.
• Authorization last. A token can be perfectly authentic and still not permit this action. Scope/role checks are authorization (a 403), not authentication (a 401), so they come after the token is proven real.

Mechanism, in code (the heart of the lab):

# 2. signature — constant-time, after pinning alg to HS256
signing_input = f"{seg0}.{seg1}".encode("ascii")
expected = hmac.new(key, signing_input, hashlib.sha256).digest()
if not hmac.compare_digest(expected, b64url_decode(seg2)):
    raise JWTValidationError("signature: HMAC verification failed")
# ── below here the payload is trusted ──
# 4. aud — string OR list on both sides; membership, not equality
if token_auds.isdisjoint(wanted_auds):
    raise JWTValidationError("aud: ...")     # confused-deputy guard
# 6. exp — strict, with leeway
if exp is not None and exp <= now - leeway:
    raise JWTValidationError("exp: token expired")

Production significance. Most real-world JWT breaches are a missing or reordered check: no aud (confused deputy), no alg pin (alg:none), a non-constant-time compare (timing), or zero skew (self-inflicted outage). The order is not style — it's the security.

Chapter 10: scp vs roles, and the Confused Deputy

From zero — the two authorization claims. After a token is proven authentic, how it authorizes depends on which grant minted it:

• scp (scope) — a space-delimited string like "Files.Read User.Read". It means delegated permission: a user is present and delegated these scopes to the client. Produced by the authorization-code flow.
• roles — a JSON array like ["Invoices.Process.All"]. It means application permission: no user — the app itself was granted these roles. Produced by the client-credentials flow.

So scp vs roles is how a validator tells "a user delegated this" from "an app is acting as itself" — which changes what you log, what you audit, and how much you trust the call. The lab's two grants produce exactly these two shapes (and the tests assert the client-credentials token has roles and no scp).

The confused deputy — what it is. A deputy is a program with more authority than its caller (your API can read the database; the caller can't directly). A confused deputy is tricked into misusing its authority on behalf of someone who shouldn't have it. With JWTs the attack is concrete:

Entra issues a valid token  aud = api://SERVICE-B   to a caller legitimately using B.
The caller replays that exact token to your  SERVICE-A.
If A verifies the signature, iss, and exp — but NOT aud — A accepts it.
→ A token minted for B now acts on A. A is the confused deputy.

The token is genuinely signed by Entra and not expired — every check passes except the one that asks "was this token minted for ME?" That one check — aud contains my API — is the entire defense. It is why Chapter 9 puts aud in the required path and the cheat sheet says never trust a token you didn't validate aud on.

Production significance. Microservice meshes are full of services that share an issuer and a signing key; the only thing keeping service A from honoring service B's tokens is the aud check. Skip it once and any service can impersonate any other.

Misconception. "If the signature is valid the token is safe." Validity proves it came from Entra unmodified — it says nothing about who it was for. Authenticity without audience is the confused deputy.

Chapter 11: Conditional Access — Signals to Decision

From zero. Everything so far validates a token once it exists. Conditional Access (CA) is the policy engine that decides whether to issue one — the "Zero Trust" control plane that runs at /authorize before the token is minted.

Under the hood — signals to a decision. A CA policy is, in effect, IF signals THEN control:

sign-in attempt
   │ collect signals: who, which app, where, what device, how risky
   ▼
evaluate CA policies (all matching policies combine; block wins)
   │
   ├── all satisfied ────────────► issue token (the flows in Ch. 4)
   ├── needs MFA ───────────────► challenge, then issue
   └── blocked ─────────────────► no token

Production significance. CA is how "valid password" becomes "valid password, from a compliant laptop, in an expected country, with MFA, for a non-risky sign-in." It's the layer that contains stolen credentials: the password alone no longer gets a token. You don't build CA in this lab, but a principal frames token issuance as conditional, not automatic.

Misconception. "MFA and Conditional Access are the same." MFA is one possible control; CA is the engine that decides when to require MFA (or block, or demand a compliant device) based on signals.

Lab Walkthrough Guidance

Lab 01 — OAuth2 / OIDC / JWT Engine, suggested order:

1. b64url_encode / b64url_decode (Ch. 8) — URL-safe base64, padding stripped on encode and re-added on decode. Test the round-trip and that no = survives.
2. pkce_challenge(verifier, method) (Ch. 7) — S256 = b64url(sha256(verifier)), plain returns the verifier, unknown method raises. The test recomputes the expected challenge with hashlib.sha256 (a known-answer vector — it equals the RFC 7636 Appendix B value E9Melhoa2OwvFrEMTJguCHaoeK1t8URWbuGJSstw-cM for the standard's verifier).
3. make_jwt / decode_jwt_unverified (Ch. 8) — sign HS256 over the encoded segments (sorted-key JSON so it's deterministic); decode without verifying. Round-trip and determinism tests.
4. validate_jwt (Ch. 9–10) — the seven ordered checks, failing closed and naming the failed check. This is where the negative tests live: bad signature, tampered payload, alg:none, wrong iss, wrong aud, expired, not-yet-valid, missing scope, missing role, and the leeway boundaries.
5. AuthorizationServer.authorize (Ch. 4, 7) — validate client/redirect/scope, store a PKCE-bound one-time code, return it.
6. token_authorization_code (Ch. 4, 5, 7) — pop the code (single-use, even on failure), re-derive the challenge from the verifier and constant-time-compare, then sign the access token (scp, user, aud = API) and the ID token (aud = client). Test wrong-verifier rejection and single-use replay.
7. token_client_credentials (Ch. 4, 10) — constant-time secret check, then a token with roles and no scp, no ID token. Test the wrong-secret rejection.

Run it red, make it green: pytest test_lab.py -v, then LAB_MODULE=solution pytest test_lab.py -v, then python solution.py for the worked flow.

Success Criteria

You are ready for Phase 04 when you can, from memory:

1. Explain app registration vs service principal (blueprint vs instance-in-a-tenant) and say which one you grant RBAC to.
2. Draw the authorization-code + PKCE flow and state precisely what PKCE defends against and why the hash makes it work.
3. Pick the correct grant for a SPA, a daemon, a middle tier, and a CLI — and say why each alternative is wrong.
4. Tell an access token from an ID token by its aud, and a delegated call from an app-only call by scp-vs-roles.
5. Recite the JWT validation order and justify each position — why signature precedes every claim, why aud is mandatory, why alg is pinned, why the compare is constant-time, why there's clock-skew leeway.
6. Define the confused deputy and name the single check that prevents it.
7. Frame Conditional Access as signals → decision and place it at token issuance.

Interview Q&A

Q: Walk me through the authorization-code + PKCE flow. What does PKCE defend against? The client redirects the user to /authorize with its client_id, redirect_uri, requested scope, and a code_challenge = BASE64URL(SHA256(code_verifier)). The user authenticates (Conditional Access runs here), and Entra redirects back with a one-time code. The client then POSTs that code plus the raw code_verifier to /token; Entra re-hashes the verifier, checks it equals the stored challenge, and returns the access + ID (+ refresh) tokens. PKCE defends against authorization-code interception: an attacker who steals the code can't redeem it because they never saw the verifier and can't invert the SHA-256 of the challenge. It's what makes auth-code safe for public clients that can't keep a secret — which is why the implicit flow is dead.

Q: Exactly what does a resource API check on an incoming JWT, and in what order? Failing closed, in order: (1) structure and that alg is the algorithm I expect — reject alg:none; (2) the signature, constant-time, against the issuer's key picked by kid — and nothing below here is trustworthy until this passes; (3) iss equals the tenant I trust; (4) aud contains my API — skip this and I'm a confused deputy; (5) nbf <= now + leeway; (6) exp > now - leeway, with ~300 s skew; (7) authorization — the required scp (delegated) or roles (app) is present. The order matters: I never read a claim I haven't verified the signature on, and authorization (a 403) only happens after the token is proven authentic (or it's a 401).

Q: What's the difference between scp and roles? scp is a space-delimited string of delegated scopes — a user is present and delegated those permissions to the app; it comes from the authorization-code flow. roles is an array of app roles — no user, the application itself was granted them; it comes from the client-credentials flow. So the validator reads scp vs roles to know whether a human delegated this action or an app is acting as itself — which changes auditing and trust.

Q: What's the confused-deputy attack and which single check prevents it? Entra mints a valid, signed, unexpired token whose aud is service B. An attacker replays it to service A. If A checks the signature, iss, and exp but not aud, A accepts a token that was never meant for it — A is the confused deputy, acting on B's authorization. The single defense is the aud check: require that my API is in the token's audience. Authenticity proves Entra issued it; only aud proves it was issued for me.

Q: Access token vs ID token — what's each for and who's the audience? The access token authorizes a call to a resource; its aud is the API, it carries scp/roles, and only the API should read it (it's opaque to the client). The ID token (OIDC) authenticates the user to the client; its aud is the client, it carries sub/name/preferred_username, and the client reads it to know who logged in. A correct API rejects an ID token on the aud check, because the ID token isn't for the API.

Q: Why HS256 in the lab but RS256 in real Entra — does it change the validation? HS256 is symmetric (one shared secret signs and verifies); we use it to stay offline and deterministic with no keypair. RS256 is asymmetric — Entra signs with a private key you never see and you verify with the public key from the tenant's JWKS, selected by the token's kid. The only thing that changes is the signature primitive and key management (fetch + cache JWKS by kid, handle rotation). The order, the claims (iss/aud/exp/nbf/scp/roles), and the confused-deputy and alg:none defenses are identical.

Q: App registration vs service principal? The app registration is the global definition of an app (client id, redirect URIs, secrets, exposed scopes, app roles) — one, in its home tenant. The service principal is the app's local identity in a tenant — the object that RBAC role assignments, admin consent, and Conditional Access attach to; there's one per tenant the app is used in. Blueprint vs instance. You grant access to the service principal, not the registration.

References

• RFC 6749 — The OAuth 2.0 Authorization Framework (the grants: authorization code, client credentials, refresh) — https://www.rfc-editor.org/rfc/rfc6749
• RFC 7636 — Proof Key for Code Exchange (PKCE) (code_verifier/code_challenge, S256, the Appendix B test vector) — https://www.rfc-editor.org/rfc/rfc7636
• RFC 7519 — JSON Web Token (JWT) (claims: iss/aud/exp/nbf/sub) — https://www.rfc-editor.org/rfc/rfc7519
• RFC 7515 — JSON Web Signature (JWS) (the base64url-segments wire format and signing input) — https://www.rfc-editor.org/rfc/rfc7515
• OpenID Connect Core 1.0 — the ID token and authentication layer — https://openid.net/specs/openid-connect-core-1_0.html
• Microsoft identity platform — the v2.0 endpoints, the supported flows, access-token and ID-token claims, and validation guidance — https://learn.microsoft.com/entra/identity-platform/ (see Authentication flows and app scenarios, Access tokens, ID tokens, and Validating tokens).
• Conditional Access — What is Conditional Access? — https://learn.microsoft.com/entra/identity/conditional-access/overview
• The track's own CHEATSHEET.md and GLOSSARY.md.

🛸 Hitchhiker's Guide — Phase 03: Entra ID, OAuth2 / OIDC / JWT

Read this if: you've clicked "Login with Microsoft" and wired up an SDK, but couldn't draw the auth-code + PKCE flow or list — in order — what an API checks on the JWT it receives. This is the compressed tour the WARMUP derives slowly. Skim it, read the WARMUP, then build the engine.

0. The 30-second mental model

Identity is the perimeter; the JWT is the boundary. Entra ID mints a short-lived, signed, audience-scoped token; your service validates it on every request before doing anything. A login is: client → /authorize (user signs in, get a one-time code) → /token (swap code

• PKCE proof for tokens) → call the API with Bearer <access_token> → the API validates the JWT. The whole security of it is one ordered checklist: signature first, then iss, then aud, then time, then scope/role — fail closed on any miss.

One sentence to tattoo: a valid signature proves the token is real; only the aud check proves it was minted for YOU — skip aud and you're a confused deputy.

1. Pick the grant in one breath

user in a browser / native app   →  authorization code + PKCE   (access + ID + refresh)
daemon, cron, service-to-service →  client credentials          (access w/ `roles`, no user)
middle-tier API calling an API   →  on-behalf-of (OBO)          (downstream token AS the user)
TV / CLI / IoT (no browser)      →  device code                 (type a code on your phone)

The grant is the identity model. "API calls Graph" → ask: as the user (OBO) or as the app (client-creds)? Different permissions, different audit trail.

2. The token claims to memorize

Access token: aud = the API, carries scp/roles. ID token: aud = the client, proves who the user is. Refresh token: opaque, goes back to Entra only.

3. The validation order (fail closed on any miss)

1. parse header → pick key by `kid` from JWKS; alg == what you expect (no `alg:none`)
2. verify signature        (constant-time compare — hmac.compare_digest)
   ─────────────────────── everything above is untrusted attacker JSON ───────────────────────
3. iss  == expected issuer (the tenant)
4. aud  contains your API  ← the confused-deputy guard; never skip
5. nbf ≤ now ≤ exp         (apply ±clock-skew, default 300 s)
6. authorize: scp (delegated) or roles (app) contains the required value

401 = authentication failed (1–5: bad/absent/expired/wrong-audience). 403 = authorization failed (6: valid token, missing scope/role). Don't swap the codes.

4. PKCE in one picture

verifier  ──SHA256──► challenge        client sends only `challenge` to /authorize
   (secret to client)                  client sends raw `verifier`   to /token
challenge = BASE64URL(SHA256(verifier))   # method S256 (the only one to use)

Attacker steals the code but never the verifier, and can't invert the hash → can't redeem it. PKCE protects the code; you still validate the token. plain (no hash) is a downgrade — reject it.

5. The numbers & names to tattoo on your arm

6. az / curl one-liners you'll actually type

# Read the discovery document (endpoints + issuer + jwks_uri) for a tenant
curl -s https://login.microsoftonline.com/<tenant>/v2.0/.well-known/openid-configuration | jq .

# Fetch the tenant's public signing keys (JWKS)
curl -s https://login.microsoftonline.com/<tenant>/discovery/v2.0/keys | jq '.keys[].kid'

# Get an access token as yourself (then paste it on https://jwt.ms to decode)
az account get-access-token --resource api://<your-api-app-id-uri> --query accessToken -o tsv

# Client-credentials (daemon) token by hand — note grant_type and .default scope
curl -s -X POST https://login.microsoftonline.com/<tenant>/oauth2/v2.0/token \
  -d grant_type=client_credentials \
  -d client_id=<app-id> -d client_secret=<secret> \
  -d scope=api://<your-api>/.default | jq '.access_token'

# Register an app + service principal; add an app role / API permission
az ad app create --display-name my-api
az ad sp create --id <appId>                       # the service principal (grant RBAC to THIS)
az ad app permission add --id <clientAppId> --api <resourceAppId> --api-permissions <scopeId>=Scope

# Inspect a token's claims locally (jq the middle segment)
echo "$TOKEN" | cut -d. -f2 | tr '_-' '/+' | base64 -d 2>/dev/null | jq .

7. War story shapes you'll relive

• "Valid token, wrong service." → a token minted for service B accepted by service A; A never checked aud. The confused deputy. Add the aud check; it's the whole defense.
• "Login works, the API 401s." → the front-end is sending the ID token (aud = the client) where the API wants the access token (aud = the API). The validator is right.
• "Every token rejected for 10 minutes." → zero clock-skew tolerance + an NTP hiccup, or a rotated JWKS key the validator cached forever. Add ±300 s leeway; cache JWKS by kid and refresh on an unknown kid.
• "Someone forged an admin token." → alg:none accepted (no algorithm pin) or a non-constant-time signature compare leaked the bytes. Pin the alg; use hmac.compare_digest.
• "It works in our tenant, 403s in the customer's." → no service principal / no admin consent in the customer tenant. The app registration isn't the identity; the SP is.
• "The daemon can't act as the user." → it used client-credentials (app roles), so every downstream call is "the app," not the user. It needed on-behalf-of.

8. Vocabulary that signals you've held the pager

• Confused deputy — a valid token for B accepted by A because A skipped aud.
• alg:none attack — stripping the signature by claiming "no algorithm"; pin the alg.
• PKCE — challenge = BASE64URL(SHA256(verifier)); secret-less proof for public clients.
• scp vs roles — delegated (user present) vs app (daemon, no user).
• App registration vs service principal — blueprint vs instance-in-a-tenant; grant to the SP.
• JWKS / kid — the issuer's public keys; the header's kid selects which one.
• On-behalf-of (OBO) — a middle tier calling downstream as the user.
• Discovery — /.well-known/openid-configuration; how SDKs self-configure.
• Bearer token — "whoever holds it can use it" → short-lived, audience-scoped, over TLS.
• Authn vs authz — who you are (401) vs what you may do (403).

9. Beginner mistakes that mark you in interviews

1. Saying "OAuth" when you mean OIDC (or thinking OIDC replaces OAuth — it's built on it).
2. Picking client-credentials where a user is present (you lost the user's identity).
3. Using the implicit flow (it's dead) instead of auth-code + PKCE for a SPA.
4. Decoding/trusting the access token in the front-end (it's opaque to the client).
5. Validating the signature but not aud (the confused deputy) — or not pinning alg.
6. Comparing the signature with == (timing side channel) instead of hmac.compare_digest.
7. Zero clock-skew tolerance, or caching one JWKS key forever (self-inflicted outage).
8. Confusing app registration with service principal — and not knowing which gets the RBAC.
9. Calling a signed JWT "encrypted" and putting a secret in the payload (it's readable).

10. How this phase pays off later

• The validation order is exactly what the API gateway runs on every request (P09).
• Identity is the perimeter is the first gate of the ARM request pipeline (P01) and the input to RBAC evaluation (P04, which runs after this authenticates the caller).
• PKCE's "prove possession, don't store a secret" is the same idea behind OIDC workload-identity federation (P08, secretless CI) and Managed Identity (P12, the token from IMDS with no credential).
• scp/roles + aud is how every secured Azure resource decides who may touch it.

Now read the WARMUP slowly, then build the engine. After this, no token is a black box — and "is this token safe to trust?" becomes a checklist you can recite under pressure.

👨🏻 Brother Talk — Phase 03, Off the Record

No slides, no STAR method. This is me, your brother, telling you the things people only say after the second coffee. Read it once now, and again the week before your interview.

Listen. Almost everyone who's "done auth" has wired an app to Entra with an SDK, watched the login pop up, and moved on. What almost nobody can do — and what instantly separates a senior from a principal in a security or platform interview — is draw the token flow from memory and then list, in order, every check a service runs on the JWT it receives, and say what breaks if you skip or reorder one. That gap — the inside of the token machine — is this whole phase. Once you can narrate it, OAuth2 stops being three letters you nod at and becomes a sequence of if statements you can defend line by line. And those if statements are, no exaggeration, the most security-critical code you will ever write. Get them wrong and the firewall, the VNet, the encryption — none of it matters, because the attacker walked in the front door holding a valid- looking badge.

Here's the thing that took me too long to internalize: the SDK hides the part that matters. acquireToken() and [Authorize] are beautiful until the day they're not — until a token is getting rejected for a reason you can't see, or worse, accepted for a reason you didn't think about. The engineers who get owned are the ones who treated auth as a library call. The ones who go principal treated it as a protocol they could implement by hand. That's why we build the validator in this lab from hashlib and hmac — not because you'll ship HS256 (you won't, real Entra is RS256), but because once you've written the seven ordered checks, you own them. They're not magic anymore. You can debug them at 2 a.m. and reason about them in a design review.

Now let me give you the things that actually bite people.

The aud check is the one. Love it, never skip it. I want you to overweight this, because the industry under-weights it and it's where the real breaches live. A token can be perfectly signed by Entra, perfectly current, from exactly the tenant you trust — and still be a weapon against you, because it was minted for a different service and someone replayed it at yours. The signature proves it's real. Only aud proves it's for you. That's the confused deputy, and it is shockingly common in microservice meshes where everyone shares an issuer and a key. When you can say in an interview "I verify the signature and I require my API in the aud, because authenticity isn't audience" — that one sentence tells them you've actually thought about token security instead of trusting a middleware default. Tattoo it: valid is not the same as for-me.

Respect the order like it's load-bearing, because it is. Signature before you read any claim — because until the signature verifies, that payload is just bytes an attacker handed you, and reading exp or iss from it and acting on it is trusting the attacker. Pin the algorithm, or someone sets alg:none and ships you an unsigned token your naive verifier happily "verifies." Compare the signature in constant time, or you leak it byte-by-byte over a thousand requests. Allow clock skew, or you take yourself down the next time an NTP server sneezes. None of these is style. Each one is a real attack or a real outage with a name. The order is the security, and the principal move is being able to justify every position in it, not just recite the list.

Understand authn vs authz so deeply it's boring. Who you are versus what you may do. A 401 versus a 403. The first half of your validator (signature, iss, aud, time) authenticates the token; the second half (scp/roles) authorizes the action. People who blur this return the wrong status codes, leak whether a resource exists, and build systems where "logged in" gets confused with "allowed." When an interviewer asks the tired "what's the difference between authentication and authorization" — most candidates recite a textbook line. You say "and here's exactly where the line falls inside a JWT validator," and point at the two halves. That's the difference between knowing the definition and having built the distinction.

The grant choice is a judgment call, and they're testing your judgment. When someone says "the service calls Graph," the junior says "use client credentials" because that's the snippet they remember. The principal says "as the app or as the user? Because if a user kicked this off and we use client-creds, we just elevated every downstream call to the app's permissions and erased the user from the audit trail — we want on-behalf-of." Same with SPAs: if you reach for the implicit flow, you've told them you stopped learning around 2018. Auth-code + PKCE, every time, for anything with a user and a browser. The grant is the identity model — picking it is the actual engineering, and picking wrong is the smell they're sniffing for.

The secretless thing is the future, and this phase is its foundation. PKCE's whole trick is prove you possess the verifier, don't store a password. Sit with that, because it's the same idea that runs the rest of your career: Managed Identity (P12) is a token from IMDS with no secret at all; OIDC workload-identity federation (P08) is your CI pipeline trading its own OIDC token for an Azure token with nothing stored to leak. The arc of cloud security is "stop storing secrets, start proving possession," and PKCE is where you first see the idea cleanly. The engineer who internalizes that stops asking "where do I put the client secret" and starts asking "why do I have one at all" — and that question is worth a level.

The honest truth about this phase: the code is not hard. HMAC is a library call, base64url is twenty minutes, the flow is a handful of HTTP redirects. What makes this principal-level is that it's unforgiving — security code that's 95% right is 0% secure, and the missing 5% is always a check you skipped or mis-ordered. So don't rush the validator. For every line, ask: what attack does this stop, and what happens if it's not here? The bad signature, the wrong aud, the alg:none, the expired token, the missing scope — every negative test in the lab is a real breach or a real outage someone lived through. Connect each one to its incident and the knowledge becomes permanent instead of memorized.

And here's the career angle, brother to brother. Identity is the perimeter, which means the people who truly understand it are trusted with the perimeter. Every other phase in this track — RBAC, networking, secrets, the API gateway, the whole platform — sits behind the gate you build here. If you own the token, you own the conversation about who gets in, and that's the conversation principals get pulled into. The people who plateau treat auth as a solved library problem; the people who go principal can open the box, find the missing aud check in a code review, and explain in one calm sentence why it would've been a breach. Be that person. It starts with the validator in this lab — write it by hand, break it on purpose, and watch it fail closed.

Go build the engine. Forge a token with the wrong aud and watch your validator reject it. Then come find me in Phase 04 — that's where the token you just validated meets RBAC, and "who are you" becomes "what, exactly, are you allowed to touch."

— your brother 👨🏻

Lab 01 — OAuth2 / OIDC / JWT Engine

Phase: 03 — Microsoft Entra ID: OAuth2 / OIDC / JWT | Difficulty: ⭐⭐⭐☆☆ | Time: 4–6 hours

Every authenticated call into Azure — and into the APIs you build on it — carries a JWT that Microsoft Entra ID minted, and the only thing standing between your service and a forged or replayed token is a validator that runs its checks in the right order and fails closed. This lab builds that machine: the base64url codec every token field uses, the PKCE proof that lets a secret-less public client log a user in safely, an authorization server that runs the auth-code+PKCE and client-credentials grants offline, and a JWT validator that checks signature → iss → aud → time → scope/role in exactly the order that prevents the confused-deputy attack. Strip away TLS and the RS256 keypair, and what's left is HMAC, SHA-256, and a careful sequence of if statements — the part interviewers probe and breaches turn on.

What you build

• b64url_encode / b64url_decode — URL-safe base64 without padding (RFC 7515). Every JWT segment and the PKCE code_challenge are encoded this way; using standard base64 or leaving padding on is the classic "works in my test, 401s in Entra" bug.
• pkce_challenge(verifier, method) — S256 = BASE64URL(SHA256(verifier)), plus the plain downgrade. This is the one-way function that lets a public client prove, at the token endpoint, that it's the app that started the login — without ever holding a secret.
• make_jwt / decode_jwt_unverified — sign a JWT with HS256 over base64url(header).base64url(payload) (deterministic), and split/decode one without trusting it (to read the kid before you verify).
• validate_jwt(...) — the heart of the lab: verify the signature with a constant-time HMAC compare, then iss, then aud (string or list), then nbf/exp with leeway, then required_scopes ⊆ scp and/or required_roles ⊆ roles — raising a ValueError that names which check failed.
• AuthorizationServer — Entra's /authorize and /token, offline: authorize(...) mints a one-time PKCE-bound code; token_authorization_code(...) exchanges code + verifier for access + ID tokens (only if the verifier hashes to the stored challenge); token_client_credentials(...) issues an app-only token carrying roles (no user, no ID token) after a constant-time secret check.

Key concepts

Files

Run

pip install -r requirements.txt
pytest test_lab.py -v                      # against your lab.py
LAB_MODULE=solution pytest test_lab.py -v  # against the reference
python solution.py                         # worked example

Success criteria

• All 32 tests pass against your implementation.
• You can recite the validation order and explain why signature must come before any claim read (an unverified payload is attacker-controlled JSON), and why aud must be checked at all (the confused deputy).
• You can explain why test_pkce_s256_known_answer_vector recomputes the challenge with hashlib.sha256 in the test — and why the solution's example prints E9Melhoa2OwvFrEMTJguCHaoeK1t8URWbuGJSstw-cM (it's the RFC 7636 Appendix B test vector, a free correctness check that your PKCE matches the standard).
• You can explain why test_validate_rejects_alg_none_attack exists and what it would mean for a validator to accept alg: none.
• You can explain why test_authcode_is_single_use matters: PKCE stops an attacker who intercepts the code, and single-use stops one who replays it.
• You can state, without looking, the difference between scp and roles and which grant produces each.

How this maps to real Azure (Microsoft Entra ID)

What the miniature leaves out (and why it's fine): real Entra signs RS256 with a private key you never see and rotates its JWKS, so production validators fetch and cache the JWKS by kid; it runs Conditional Access (device, location, risk, MFA) at /authorize; it issues refresh tokens and supports the on-behalf-of and device-code grants; and the whole thing is over TLS. None of that changes the validation order or the claims — which is exactly the part interviewers probe and incidents turn on. We use HS256 (symmetric) and an injected clock purely to stay offline and deterministic; swap the signature primitive for RSA-verify-against-JWKS and the logic is the real thing.

Extensions (build these in your own tenant)

• RS256 + JWKS: replace HS256 with an RSA keypair; publish a JWKS document ({"keys":[{"kid":..., "n":..., "e":...}]}); have validate_jwt select the key by the token's kid and verify with the public key. Now you've modeled real Entra exactly.
• Discovery: serve a /.well-known/openid-configuration JSON with issuer, jwks_uri, authorization_endpoint, token_endpoint — and have the client discover endpoints instead of hard-coding them (what every Entra SDK does on startup).
• On-behalf-of (OBO): add a grant where a middle-tier API exchanges an incoming user access token (audience = the API) for a downstream token (audience = a second API), preserving the user identity — the flow that powers API-calls-API with the user's context.
• Device code: add the device_authorization grant (return a user_code + verification_uri, poll /token until the user approves) — the flow for TVs, CLIs, and IoT with no browser.
• Refresh tokens: issue and redeem a refresh token; enforce single-use rotation and a longer expiry than the access token.
• Wire it to a real tenant: az ad app create, grab a token with az account get-access-token, decode it on jwt.ms, and check your validate_jwt accepts it (with RS256/JWKS) and rejects it with the wrong aud.

Interview / resume

• Talking points: "Walk me through the authorization-code + PKCE flow and what PKCE defends against." / "Exactly what does a resource API check on an incoming JWT, and in what order?" / "What's the difference between scp and roles, and which grant produces each?" / "What's the confused-deputy attack and which single check prevents it?" / "Access token vs ID token — what's each for and who's the aud?"
• Resume bullet: Built an OAuth2/OIDC authorization-server and JWT-validation engine (authorization-code + PKCE and client-credentials grants, constant-time signature verification, ordered iss/aud/exp/nbf and scp/roles claim checks with clock skew) modeling Microsoft Entra ID token issuance and resource-API validation.

Phase 04 — RBAC & Azure Policy Evaluation

Difficulty: ⭐⭐⭐☆☆ (the matchers) → ⭐⭐⭐⭐⭐ (the judgment about least privilege, deny, and effect choice) Estimated Time: 1 week (12–18 hours) Prerequisites: Phase 00 (control plane vs data plane, the resource-ID shape and its scope levels), Phase 03 (Entra ID — the token that arrives carries the principal whose access we evaluate here). No live Azure subscription required for the lab.

Why This Phase Exists

Phase 03 ended with a validated JWT: ARM now knows who is calling (the oid/sub claim) and that the token is genuine. This phase answers the very next question — the one the entire security posture of an Azure estate turns on: is this caller allowed to do this, to this resource?

There are two gates, in series, and almost nobody can describe both precisely:

1. Azure RBAC — authorization. Given the principal, the action, and the resource's scope, gather every role assignment that applies, compute the effective permission set, and let any deny assignment win. This is the gate behind AuthorizationFailed.
2. Azure Policy — governance. Even if RBAC says yes, Policy can deny the write (or append/modify/audit it) based on the shape of the resource — its SKU, region, encryption setting, tags. This is the gate behind RequestDisallowedByPolicy.

Here is the thing that separates a senior from a principal in an interview: a senior can assign the Owner role; a principal can explain why that Owner still can't read a blob (control plane vs data plane), why a Contributor can't grant access to anyone (NotActions on Microsoft.Authorization/*), why a deny assignment from a Deployment Stack overrides a subscription-Owner grant, and why a policy with effect: deny blocks a deployment that RBAC happily authorized. These are not trivia — they are the four shapes of the support ticket "I have permission but it still fails."

So this phase makes you build both evaluators — the wildcard Actions/NotActions matcher, scope containment, the deny-wins decision, and the Policy condition-tree evaluator. Once you've written the algorithm, "effective permissions" stops being a portal blade you squint at and becomes a function you can run in your head during an incident. This is the keystone for Phase 05 (landing zones are RBAC + Policy inherited down a management-group tree at scale) and underpins every later phase that touches identity.

What "Principal-Level" Means Here

A senior assigns roles and writes policies. A principal understands the evaluation machinery well enough to:

• Predict an authorization decision from the assignments without the portal — by computing union(Actions) − union(NotActions), checking scope containment, and applying deny-wins, in that order.
• Debug the four "I have permission but it fails" tickets: (a) control-plane grant but data-plane action (Owner reading a blob), (b) a NotActions subtraction (Contributor granting access), (c) a deny assignment overriding the grant, (d) RBAC allows but Policy denies the resource shape.
• Design least privilege: pick the narrowest built-in role that covers the action, reach for a custom role only when no built-in fits, and prefer scope narrowing (assign at the RG, not the subscription) over broader roles.
• Choose the right Policy effect and know when it runs: deny/append/modify at the request, audit/auditIfNotExists/deployIfNotExists on the compliance scan — and why deployIfNotExists needs a managed identity.
• Reason about blast radius of an assignment: a role at a management group reaches every subscription, RG, and resource beneath it; a deny at an RG stops there.

Concepts

• The RBAC model — a role assignment is the triple (principal, role definition, scope). RBAC is allow-based (there is no "deny by role"), additive (assignments union together), and inherited down the hierarchy management group → subscription → resource group → resource. You grant by adding an assignment and revoke by removing one (or by a deny assignment).
• Role-definition internals — Actions (allowed control-plane operations), NotActions (subtracted from this role's own Actions), DataActions / NotDataActions (the same pair for the data plane), and AssignableScopes (where the role may be assigned). Effective perms = union(Actions) − union(NotActions). Wildcards: * matches any run of characters including / (so Microsoft.Storage/*/read spans segments).
• Control plane vs data plane — Actions manage the resource (create/configure a storage account); DataActions govern the data inside it (read a blob). They are separate sets, which is why Owner (Actions=["*"]) cannot read a blob — it has no DataActions; you add Storage Blob Data Reader. The same split exists for Key Vault, Service Bus, Cosmos DB, and more.
• Deny assignments — an explicit block on actions at a scope that overrides role assignments. Deny always wins, and it is evaluated first. You cannot create deny assignments with a role; they come from Blueprints, Deployment Stacks (denySettings), and managed applications — which is exactly why they're the system's last word for protecting platform-managed resources.
• Scope hierarchy & inheritance — a scope is an ARM resource ID (or a prefix). An assignment at a higher scope applies to all descendants. A resource is "in scope of" an assignment iff its ID is the scope or sits below it by whole path segments — the reason …/resourceGroups/rg contains …/rg/providers/… but not …/rg2.
• Azure Policy — a policyRule of if (a condition tree) and then.effect. Leaf conditions test a field/alias (equals/notEquals/in/exists/like); logical nodes combine them (allOf/anyOf/not). Effects: disabled, audit, auditIfNotExists, append, modify, deployIfNotExists, deny — and deny dominates. deny/append/modify run at create/update; audit* and deployIfNotExists run on the compliance scan.
• Aliases — Policy can't read raw JSON paths; it reads aliases that map a friendly name to the resource property at a given API version (e.g. Microsoft.Storage/storageAccounts/supportsHttpsTrafficOnly). The lab models the dotted lookup; the extension wires real aliases.

Labs

Lab 01 — RBAC & Azure Policy Evaluation Engine (flagship, implemented)

→ Lab folder: lab-01-rbac-policy-engine/

Integrated-Scenario Suggestions (carried through the whole track)

These compound into Phase 05 (landing zones) and the Phase 15 capstone. Keep them in mind as you build — each plugs the evaluator into a larger governance story:

1. Least-privilege CI/CD identity — a deployment service principal that is Contributor on one RG (not subscription Owner), plus the narrowest data-plane roles for the secrets/blobs it touches. The exercise is removing every action it never uses and proving the deploy still works. (→ P08, P12)
2. Break-glass and deny protection — a Deployment-Stack denySettings deny assignment that blocks delete on production resources even for Owner, with a documented break-glass account that is not in the deny scope. (→ P05)
3. Policy-as-code guardrails — an initiative (policy set) that denies non-HTTPS storage, denies public network access, appends required tags, and audits resources missing a cost-center tag — assigned at a management group so every subscription inherits it. (→ P05)
4. Data-plane vs control-plane separation of duties — platform admins get Owner to manage a Key Vault but not read its secrets; app identities get Key Vault Secrets User (a data role) but cannot change the vault's firewall. (→ P12)
5. Effective-access audit — given a principal and a resource, compute the effective permissions across all inherited assignments and deny assignments — the offline version of the portal's "Check access" / az role assignment list --include-inherited. (→ P13)

Guides in This Phase

• HITCHHIKERS-GUIDE.md — the 30-minute orientation; read first
• WARMUP.md — the full primer; read slowly
• BROTHER-TALK.md — the candid version

Key Takeaways

• Authorization is two gates in series. RBAC (may this identity?) then Policy (is this resource shape allowed?). The four "I have permission but it fails" tickets each map to one specific mechanism — name it and you've solved it.
• A role assignment is (principal, role, scope) — additive, inherited down the hierarchy, and allow-based. You revoke by removing or by a deny assignment, never by a "deny role" (there is none).
• Effective perms = union(Actions) − union(NotActions), and the data plane is a separate set (DataActions). Owner ≠ Storage Blob Data Reader.
• Deny always wins and is evaluated first. A deny assignment from a stack/blueprint overrides even subscription Owner — that's its whole purpose.
• Pick the narrowest role at the tightest scope. Least privilege is the default; a custom role is a last resort, and broad-scope assignments are a blast-radius decision.
• Choose the Policy effect deliberately — deny/append/modify shape the request; audit*/deployIfNotExists report and remediate on the scan; deny dominates them all.

Deliverables Checklist

• Lab 01 implemented; all 43 tests pass against solution.py and your lab.py
• You can state the RBAC evaluation order (deny first, then allow, default deny) from memory and explain why deny is evaluated first
• You can explain why Owner can't read a blob and which role fixes it
• You can name the four built-in roles (Owner/Contributor/Reader/User Access Administrator) and the one operation Contributor lacks
• You can list the seven Policy effects, order them by precedence, and say which run at request time vs the compliance scan
• Given assignments + a resource ID, you can compute the decision on paper

Warmup Guide — RBAC & Azure Policy Evaluation

Zero-to-principal primer for Phase 04: the two authorization gates every Azure write passes through. We start from "what is a permission" and end at the exact evaluation order ARM runs — role assignments, NotActions subtraction, the control-plane / data-plane split, deny assignments, scope inheritance, and the Azure Policy condition engine and its effects. By the end you can answer "why does this fail even though I'm Owner?" from first principles, and you will have built both evaluators.

Table of Contents

• Chapter 1: The Two Gates — Authorization vs Governance
• Chapter 2: Actions and the Wildcard Matcher
• Chapter 3: Role Definitions — Actions, NotActions, and the Plane Split
• Chapter 4: Scope, Hierarchy, and Inheritance
• Chapter 5: Deny Assignments and the Evaluation Order
• Chapter 6: Azure Policy — The Governance Gate
• Chapter 7: Effects and Their Precedence
• Chapter 8: Least Privilege as an Engineering Discipline
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: The Two Gates — Authorization vs Governance

From zero. When a client calls Azure to create or change a resource, the request is a single idempotent PUT (or PATCH/DELETE) to a resource ID — you built the parser for that ID in Phase 00. Before any resource provider acts on it, Azure Resource Manager (ARM) runs the request through a pipeline:

client ──JWT──▶ Entra (authenticate: is the token genuine? — Phase 03)
            │
            ▼
          ARM ──▶ RBAC        (authorize:  may THIS principal do THIS action HERE?)
            │  ──▶ Azure Policy (govern:     is the resulting RESOURCE allowed to exist?)
            │
            ▼
   resource provider (Microsoft.Storage, …) performs the idempotent PUT

What it is. Two distinct gates, in series, both deny-biased:

• RBAC (Role-Based Access Control) is authorization: it decides whether the identity is permitted to perform the action at the scope. Failure looks like AuthorizationFailed (HTTP 403).
• Azure Policy is governance: even when RBAC says yes, Policy can block or mutate the write based on the shape of the resource — its region, SKU, tags, encryption flag. A blocking failure looks like RequestDisallowedByPolicy (HTTP 403, different code).

Why it exists. They answer different questions and are owned by different people. RBAC is "who is on the team and what may they touch" (identity/security). Policy is "what is the organization allowed to build" (governance/compliance). Conflating them is the root of most "I have permission but it still fails" tickets: you check RBAC, see Owner, and miss that Policy denied the non-compliant resource.

Production significance. The single most useful debugging move in an Azure incident is to read the error code: AuthorizationFailed sends you to RBAC (assignments, scope, NotActions); RequestDisallowedByPolicy sends you to Policy (which definition, which effect, which field). They are never the same investigation.

Common misconception. "Policy is just RBAC for resource properties." No — Policy never looks at who is calling; it looks at what is being created. A principal with full rights is still stopped by a deny policy, and a principal with no rights never reaches Policy because RBAC stopped them first.

Chapter 2: Actions and the Wildcard Matcher

From zero. Every operation in Azure has a name — a slash-delimited string:

Microsoft.Storage/storageAccounts/read
Microsoft.Storage/storageAccounts/write
Microsoft.Storage/storageAccounts/listKeys/action
Microsoft.Compute/virtualMachines/read

The shape is {ResourceProvider}/{resourceType}/{operation} (sometimes deeper, with nested types and an /action suffix for POST-style operations).

What it is. A role grants patterns over these strings, and the one wildcard is *. The critical semantic: * matches any run of characters, including the / separator, so it can span one or more path segments and match within a segment: