🛸 Hitchhiker's Guide — Phase 12: Secrets, Key Vault & Managed Identity

Read this if: you've used Key Vault from the portal but "how does a managed identity actually get a token?" and "why does my Owner get a 403?" still feel fuzzy. This is the compressed tour the WARMUP derives slowly. Skim it, then read the WARMUP, then build the lab.

0. The 30-second mental model

The most secure secret is the one you never store. A Managed Identity removes the stored credential for Azure-native hops — your code asks IMDS (169.254.169.254) for a short-lived, audience-scoped, cached token, no secret in the code. For the secrets you can't eliminate, Key Vault is the one hardened, versioned, RBAC-gated store — and reading a secret is a data-plane action, so Owner (control plane) gets a 403. For encryption, envelope encryption wraps a data key (DEK) with a vault key (KEK), so rotating the key costs a few bytes, not the petabyte. One sentence to tattoo: identity replaces the credential; the vault holds the irreducible secret; envelope encryption protects the data.

1. The token flow, one breath

your code ──get_token("vault")──▶ IMDS 169.254.169.254 ──▶ Entra ──▶ JWT {aud, exp, oid}
   │                                  (link-local, non-routable: only this box can ask)
   └── Bearer <token> ──▶ https://vault.azure.net/secrets/db-pwd ──▶ 200 {value} | 403

The SDK caches the token per (identity, audience) and refetches only near expiry. The audience (aud) is load-bearing: a vault.azure.net token is rejected by storage.azure.com.

2. System- vs user-assigned (an interview staple)

Default-junior answer: "system-assigned, it's simpler." Principal answer: "user-assigned when 30 functions need the same role — one assignment, not thirty — or when IaC ordering needs the identity first."

3. Control plane vs data plane (the #1 Key Vault 403)

CONTROL PLANE (ARM/RBAC)        DATA PLANE (vault endpoint)
create/delete/configure vault   READ a secret's value, wrap/unwrap/sign
Owner, Contributor              Key Vault Secrets User, Crypto User

Owner can delete the whole vault but can't read one secret. Reading a value is a DataAction. Burn this in — it shows up identically in Storage (Owner ≠ Blob Data Reader).

4. The numbers you'll actually use

5. Envelope encryption in one diagram

data ──enc(DEK)──▶ ciphertext   ┐  store both
DEK  ──wrap(KEK)─▶ wrapped_DEK   ┘  (KEK never leaves the vault/HSM)

rotate KEK:  DEK = unwrap(old_KEK, wrapped) ; wrapped' = wrap(new_KEK, DEK)
             ▲ the DATA ciphertext is NEVER touched — that's the whole point
revoke:      disable the KEK → wrapped_DEK can't be unwrapped → data unreadable (crypto-shred)

6. az / SDK one-liners

# user-assigned identity, attach to a function app, grant data-plane read (RBAC)
az identity create -g rg -n app-uai
az functionapp identity assign -g rg -n myfunc --identities <uai-resource-id>
az role assignment create --assignee <uai-client-id> \
  --role "Key Vault Secrets User" --scope <vault-resource-id>

# vault with the security defaults ON
az keyvault create -g rg -n myvault --enable-rbac-authorization true \
  --enable-purge-protection true --retention-days 90

# secrets are versioned; soft-delete is recoverable
az keyvault secret set     --vault-name myvault -n db-pwd --value 'hunter2'
az keyvault secret delete  --vault-name myvault -n db-pwd          # soft-delete
az keyvault secret recover --vault-name myvault -n db-pwd          # undo
az keyvault secret purge   --vault-name myvault -n db-pwd          # REFUSED if purge-protection + in retention

# key rotation (new version) — the wrapped DEKs rewrap; data untouched
az keyvault key rotate     --vault-name myvault -n my-kek

# no stored secret anywhere — DefaultAzureCredential uses the managed identity via IMDS
from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient
cred = DefaultAzureCredential()                       # caches the token
kv = SecretClient("https://myvault.vault.azure.net", cred)
pwd = kv.get_secret("db-pwd").value                   # needs Key Vault Secrets User

# Key Vault reference in app settings — platform resolves it via the app's MI
app_settings = { DB_PWD = "@Microsoft.KeyVault(SecretUri=https://myvault.vault.azure.net/secrets/db-pwd/)" }

7. War story shapes you'll relive

• "It returns a token but the call 401s." → wrong audience. You asked IMDS for vault.azure.net and called Storage. A token is scoped to one resource. (Lab: test_token_audience_scoping.)
• "My Owner gets 403 reading the secret." → control vs data plane. Grant a data-plane role. Not a bug; the model.
• "IMDS is throttling my function." → you're fetching a token every call. It's a caching bug — the SDK should serve a cached token until near expiry, not a scale problem.
• "We rotated the encryption key and re-encrypted 40 TB over a weekend." → you didn't use envelope encryption. Rotation should rewrap the DEK (milliseconds), not the data.
• "An admin deleted the prod vault and we lost the keys." → soft-delete + purge protection would have made it recoverable and un-purgeable. Enable both.
• "The client secret expired at midnight and prod went down." → that's exactly what a managed identity eliminates. There's no secret to expire.

8. Vocabulary that signals you've held the pager

• Secret-zero — the one irreducible credential; make it a managed identity, not a string.
• IMDS — the non-routable 169.254.169.254 token endpoint; possession of access is the credential.
• Audience / aud — the one service a token is valid for.
• DEK / KEK / wrap / unwrap — data key, key-encryption key, and the envelope.
• Rewrap-on-rotate — KEK rotation rewraps the DEK only; data untouched.
• Control plane vs data plane — manage the vault vs read the secret; different roles.
• Soft-delete / purge protection — recoverable delete; un-purgeable until retention.
• Workload identity federation — OIDC → Entra token, no secret at all (CI, AKS).
• Crypto-shredding — revoke the KEK and the data is unreadable without touching it.

9. Beginner mistakes that mark you in interviews

1. Saying "managed identity = no token" (there is a token — a JWT; what's gone is the stored secret).
2. Defaulting to system-assigned and never reaching for user-assigned's RBAC-grant economy.
3. Granting Owner/Contributor and expecting to read secrets (data plane needs its own role).
4. Re-encrypting data to rotate a key (the whole point of envelope encryption is you don't).
5. Putting a secret in an env var / config / git instead of Key Vault + a reference.
6. Shipping a vault without soft-delete + purge protection on anything that matters.
7. Forgetting the audience and reusing one token across services.

10. How this phase pays off later

• The token flow (P03 JWT + this) is how every Azure-to-Azure call authenticates.
• Control vs data plane (P00) is sharpest here and reused everywhere.
• RBAC (P04) is Key Vault's recommended authorization model — same engine.
• Envelope encryption is the engine behind CMK for Storage/SQL/Disk/Cosmos.
• The secret-zero habit makes your Functions (P11), CI/CD (P08), and event-driven (P10) designs secure by construction — no stored credentials anywhere.

Now read the WARMUP slowly, then build the lab. The token cache, the rewrap-on-rotate, and the purge-protection block are the three mechanisms an interviewer will actually push on.