Phase 12 — Secrets, Key Vault & Managed Identity

Difficulty: ⭐⭐⭐☆☆ (the mechanisms) → ⭐⭐⭐⭐⭐ (the judgment about secret-zero, blast radius, and rotation) Estimated Time: 1 week (12–18 hours) Prerequisites: Phase 03 (Entra — OAuth2 / OIDC / JWT; a managed-identity token is a JWT, and IMDS is just another token endpoint), Phase 04 (RBAC — Key Vault's recommended authorization model is the same Actions/DataActions evaluator, and reading a secret is a data-plane action), and Phase 00 (control plane vs data plane; quantify the tradeoff). No Azure subscription required for the lab.

Why This Phase Exists

Every breach post-mortem you will ever read has the same root cause near the top: a secret in the wrong place. A connection string in appsettings.json committed to git. An access key in an environment variable that leaked through a crash dump. A service principal secret that expired at 3 a.m. and nobody knew where it was configured. The JD asks for an engineer who can "integrate serverless applications with Azure security services, including Azure Key Vault, Managed Identities, and Microsoft Entra ID." That sentence is the whole game of this phase, and it is really one principle stated three ways:

The most secure secret is the one you never store.

This phase makes that principle mechanical. There are three mechanisms, and a principal must own all three down to the byte:

1. Managed Identity — the answer to "how does my code prove who it is without a credential in the code?" Azure manages an Entra service principal for the resource; the code asks the Instance Metadata Service (IMDS) at the non-routable link-local address 169.254.169.254 for an access token, scoped to a target audience (the resource it wants to call), and the SDK caches that token until near expiry. There is no secret to rotate, leak, or expire-by-surprise, because there is no secret. A senior says "use a managed identity." A principal can tell you why the IP is non-routable (so only code on the resource can reach it), why the token is audience-scoped (so a leaked vault token can't read storage), and system-assigned vs user-assigned (lifecycle-bound to one resource vs standalone and attachable to many).

2. Key Vault — for the secrets you can't eliminate (a third-party API key, a database password you don't control, a TLS certificate), a hardened, audited, RBAC-gated, versioned store. The recurring Azure distinction lives here in its sharpest form: control plane (manage the vault — Owner, Contributor) is a different system from data plane (read the secret's value — Key Vault Secrets User). An Owner who can delete the entire vault still gets a 403 reading a secret unless they hold a data-plane role. And the vault is recoverable by design: soft-delete makes a fat-fingered delete undoable for a retention window, and purge protection means not even a compromised admin can make a deletion permanent on demand.

3. Envelope encryption — the mechanism behind every "encryption at rest with your own key" feature in Azure (Storage, SQL, Disk, Cosmos with customer-managed keys). You do not encrypt terabytes of data directly with a Key Vault key. You encrypt the data with a fast symmetric data encryption key (DEK), then wrap (encrypt) the small DEK with a key encryption key (KEK) that lives in the vault/HSM and never leaves it. You store (ciphertext, wrapped_DEK) together. The payoff is rotation: to rotate the KEK you unwrap the DEK with the old key and rewrap it with the new — you touch a few bytes, not the petabyte. Key versioning means old wrapped DEKs still unwrap against the old key version, so rotation is non-breaking. This is the single most elegant idea in cloud security, and this phase makes you build it so it stops being a diagram and becomes a mechanism you can defend line by line.

What "Principal-Level" Means Here

A senior engineer uses Key Vault and managed identities correctly. A principal understands them well enough to:

• Eliminate secret-zero. They can take any "the app needs to read a database / a storage account / another API" requirement and produce a design with no stored credential: a managed identity for the Azure-native hops, Key Vault (read via the managed identity) only for the genuinely external secrets, and federation for cross-cloud / CI. They know the one irreducible secret ("secret-zero") and have made it a managed identity, not a string.
• Reason about the token flow from first principles. Given "my function calls Key Vault in a tight loop and IMDS is throttling," they know it's a caching bug, not a scale problem — the SDK should serve a cached token until near expiry. Given "the call returns a token but Storage rejects it," they know it's an audience mismatch. They can draw the code → IMDS (169.254.169.254) → Entra → token (aud=vault) → vault path from memory.
• Pick system- vs user-assigned on sight. System-assigned when the identity's lifecycle is the resource's and nothing else needs it; user-assigned when several resources share one identity (so you grant the RBAC role once), or when the identity must exist before the resource (to pre-assign roles and break a chicken-and-egg deploy ordering problem).
• Choose the authorization model and the rotation story. Azure RBAC over legacy access policies (RBAC is auditable, deny-aware, and scoped through the same MG/Sub/RG hierarchy as everything else). And a rotation design: short-lived where possible, Key Vault references so apps never hold the secret, event-driven rotation (SecretNearExpiry → Event Grid → a rotation function), and envelope encryption so the key rotates without re-encrypting the data.
• Treat the vault as a blast-radius unit. They know a single vault is a failure domain and a throttling unit; they segment vaults by environment/blast-radius, enable soft-delete + purge protection on anything that matters, and never make the vault the single point of failure for a hot path (cache the secret, fail open/closed deliberately).

Concepts

• Managed Identity = an Azure-managed Entra service principal. The platform creates and rotates the credential; your code never sees it. System-assigned: created with the resource, deleted with it, exactly one per resource, identity = the resource's lifecycle. User-assigned: a standalone Azure resource (Microsoft.ManagedIdentity/userAssigned Identities) you attach to one or many resources and select by client_id.
• IMDS (Instance Metadata Service). A read-only REST endpoint at the link-local, non-routable address 169.254.169.254, reachable only from the resource itself, that among other things issues OAuth2 access tokens for the resource's managed identity at /metadata/identity/oauth2/token?resource=<audience>. The credential SDK (DefaultAzureCredential / ManagedIdentityCredential) calls it and caches the token per (identity, audience) until shortly before expiry.
• The audience (resource/scope). A token is minted for one target service — its aud claim. https://vault.azure.net for Key Vault, https://storage.azure.com for Storage, https://management.azure.com for ARM. A token is rejected by any service that is not its audience; reusing the wrong-audience token is a classic silent 401.
• Workload identity federation (no secret at all). Beyond IMDS: AKS pods and GitHub Actions can present an external OIDC token, and Entra exchanges it for an access token via a federated credential trust — no client secret, no IMDS. The same "no stored secret" principle, extended across trust boundaries (you built the OIDC half in P03/P08).
• Key Vault objects: secrets, keys, certificates — all versioned. A secret is an opaque value (string ≤ 25 KB). A key is an asymmetric/symmetric key you use without exporting (encrypt, decrypt, wrapKey, unwrapKey, sign, verify). A certificate bundles a key + an X.509 cert with a lifecycle (issuance, auto-rotation, CA policy). Every object is versioned: each write creates an immutable version with its own id/URL; omitting the version returns current.
• Two authorization models: RBAC (recommended) vs access policies (legacy). Azure RBAC uses the same role-assignment + inheritance engine as the rest of Azure (Key Vault Secrets User, Secrets Officer, Administrator, Crypto User, …), is auditable and deny-aware, and is scoped at vault or even individual-secret level. Access policies are a per-vault list of (principal → permitted operations) — simpler but not inheritable, not deny-aware, and harder to audit. New vaults should use RBAC.
• Control plane vs data plane (the sharpest example in Azure). Managing the vault resource (create, delete, configure firewall, set the authorization model) is control plane via ARM/RBAC. Reading a secret's value, wrapping a key, signing — data plane, gated by data-plane roles. Owner (control plane) does not grant secret read (data plane). Confusing the two is the #1 Key Vault 403.
• Soft-delete + purge protection. Deletes are soft (always on now): the object/vault is recoverable for a retention window (7–90 days, default 90). Purge protection, when enabled (irreversibly), means a soft-deleted object cannot be purged before retention elapses — not by an admin, not by an attacker. It's the guarantee that deletion cannot be weaponized.
• Envelope encryption. Encrypt data with a DEK; wrap the DEK with a KEK in the vault/HSM; store (ciphertext, wrapped_DEK). Rotate the KEK by unwrapping + rewrapping the DEK only — O(#DEKs), never O(bytes). Key versioning keeps old wrapped DEKs decryptable. This is how Azure Storage/SQL/Disk customer-managed-key encryption works.
• Managed HSM / FIPS framing. Standard Key Vault is multi-tenant, FIPS 140-2 Level 2. Azure Key Vault Managed HSM is a single-tenant, fully customer-controlled, FIPS 140-2 Level 3 HSM pool for regulated workloads and the highest key-isolation requirements.

Labs

Lab 01 — Key Vault & Managed Identity: Token Acquisition + Envelope Encryption (flagship, implemented)

→ Lab folder: lab-01-keyvault-identity/

Integrated-Scenario Suggestions (carried through the whole track)

The phases compound. Keep these in mind as you build — each wires secrets and identity into the larger platform:

1. Zero-secret serverless app. A Function App (P11) with a user-assigned managed identity reads a third-party API key from Key Vault via a Key Vault reference (@Microsoft.KeyVault(SecretUri=...)) injected at startup, calls Storage with the same identity (no key), and is fronted by API Management (P09) validating the caller's JWT (P03). The only "credential" anywhere is the managed identity. (→ P11, P09, P03)
2. Customer-managed-key encryption at rest. A Storage account / SQL DB uses a CMK in Key Vault: the service holds a wrapped DEK and unwraps it via the vault using its managed identity. You model the rotation — a new KEK version, rewrap the DEK, data untouched — and the revocation path (disable the key → the data is cryptographically unreadable). (→ P00 control/data plane)
3. Event-driven secret rotation. Key Vault emits a SecretNearExpiry event to Event Grid (P10); a Durable Function (P11) rotates the credential at the source (e.g. regenerates a storage key), writes the new version to the vault, and verifies before flipping traffic — all authenticated by managed identity, no human in the loop. (→ P10, P11)
4. CI/CD with no stored secret. A GitHub Actions / Azure DevOps pipeline (P08) authenticates to Azure via workload identity federation (OIDC → Entra, no client secret) and reads deployment secrets from Key Vault scoped to the pipeline's identity. The federated subject (repo:org/name:ref:...) is the trust. (→ P08, P03)
5. Regulated workload on Managed HSM. A workload requiring FIPS 140-2 Level 3 uses Azure Key Vault Managed HSM for its KEKs, with RBAC local roles, purge protection mandatory, and key material that never leaves the single-tenant HSM — verified by attestation. You reason about the cost/isolation tradeoff vs standard Key Vault. (→ P05 governance)

Guides in This Phase

• HITCHHIKERS-GUIDE.md — the 30-minute orientation; read first
• WARMUP.md — the full primer; read slowly
• BROTHER-TALK.md — the candid version

Key Takeaways

• The most secure secret is the one you never store. A Managed Identity removes secret-zero for Azure-native hops: code asks IMDS (169.254.169.254) for an audience-scoped, cached token with no credential in the code. System-assigned = tied to one resource; user-assigned = standalone and shareable. Federation extends "no secret" across trust boundaries.
• Control plane ≠ data plane — Key Vault is the sharpest example. Managing the vault is control plane (Owner); reading a secret's value is data plane (Key Vault Secrets User). An Owner who can delete the vault still can't read a secret without a data-plane role. Use Azure RBAC over legacy access policies.
• Envelope encryption makes key rotation cheap. Encrypt data with a DEK, wrap the DEK with a KEK in the vault/HSM, store both. Rotating the KEK rewraps the DEK only — O(#DEKs), never O(bytes) — and key versioning keeps old data decryptable. This is the engine behind every CMK feature in Azure.
• Deletes are recoverable on purpose. Soft-delete makes a fat-finger undoable for a retention window; purge protection means deletion can't be made permanent on demand — not by an admin, not by an attacker. Enable both on anything that matters.
• The vault is a blast-radius and throttling unit. Segment by environment, cache the secret on the hot path, and never let the vault be the single point of failure for a request you can't afford to fail.

Deliverables Checklist

• Lab 01 implemented; all 29 tests pass against solution.py and your lab.py
• You can draw the managed-identity token flow (code → IMDS → token aud=vault → vault) from memory and explain the cache and the audience
• You can state when to choose system- vs user-assigned, with a concrete reason for each
• You can explain why an Owner of a Key Vault gets a 403 reading a secret
• You can explain envelope encryption and why KEK rotation does not re-encrypt the data
• You can explain soft-delete vs purge, and exactly what purge protection guarantees