🛸 Hitchhiker's Guide — Phase 14: Reliability, HA & Operational Troubleshooting

Read this if: you can quote a 99.9% SLA but couldn't tell me what it buys (43 minutes a month), whether a four-hop chain can even deliver it, or why a retry without jitter and a budget is a loaded gun. This is the compressed tour the WARMUP derives slowly. Skim it, read the WARMUP, then build the engine.

0. The 30-second mental model

Reliability is arithmetic you do before the design review, plus a method for the 2 a.m. incident. Serial hops multiply (∏ aᵢ — more hops, lower SLA, below every hop); redundant replicas combine the inverse (1 − ∏(1 − aᵢ) — more replicas, higher SLA); N+k is a binomial. Make retries safe with backoff + jitter + a budget (and idempotency as the precondition), honor Retry-After, and put a circuit breaker in front of a flaky dependency. When it breaks, read the signature (429? 403? DNS?) and walk the decision tree to a cause. One sentence to tattoo: availability is a product across the path, downtime is a budget you spend, and a retry is a sharp tool with three safety catches.

1. SLA composition, one breath

serial chain (all must be up):     A = ∏ aᵢ                  → BELOW every hop
redundant replicas (any up):       A = 1 − ∏(1 − aᵢ)         → ABOVE the best replica
N+k (≥ n of n+k up):               A = Σ C(N,i)·pⁱ·(1−p)^(N−i), i = n..N   (binomial)
downtime budget:                   (1 − A) × window
composite chain of redundant hops: collapse each hop with redundant, then multiply

The principal move: compose the chain, find the weakest single-zone hop, make it zone- redundant, recompute. Adding a nine to the weakest hop moves the composite the most.

2. The downtime budget table (memorize it)

Front Door 0.9995 × APIM 0.9995 × Function 0.9995 × SQL 0.9999 ≈ 0.9984 → ~69 min/month, below every hop. Three 0.999 in series ≈ 0.997 (barely 2.5 nines).

3. The numbers to tattoo on your arm

4. The retry safety catches (all three, or don't retry)

backoff   → space retries (min(cap, base·2^n))          [don't hammer a recovering dep]
jitter    → uniform(0, backoff)                          [de-sync the thundering herd]
budget    → cap total retries, fail fast when exhausted  [don't amplify into a retry storm]
+ idempotency (the precondition)                         [retry twice == once, or corruption]
+ honor Retry-After on 429/503                           [the server told you; obey it]

Backoff alone keeps the herd synchronized. Jitter breaks the sync. The budget bounds the amplification. Skip any and your retries cause the outage (or duplicate the side effect).

5. The circuit breaker in one picture

CLOSED   pass requests, count consecutive failures
   │ failure_threshold consecutive failures
   ▼
OPEN     reject immediately (FAIL FAST) — protects dep AND caller
   │ reset_timeout elapses → allow ONE probe
   ▼
HALF_OPEN  one trial request:  success → CLOSED (recovered)
                               failure → OPEN (still broken, restart timer)

A breaker is the opposite of a retry: retry = "try again" (transient blip); breaker = "STOP trying for a while" (sustained outage). Pair it with a timeout and a fallback (cache / dead- letter / default).

6. The triage decision tree (signature → cause → first action)

symptom: name resolution failed  → DNS/Private Endpoint: Private DNS zone, VNet link, A record → PE NIC (not public IP)
symptom: timeout                 → dependency slow: circuit-break, check health/SLA + pool, ensure backoff+jitter
symptom: connection refused      → connectivity: NSG/firewall, Private Endpoint, listener/port, effective routes
429  → throttling: honor Retry-After, back off + jitter, spread load, check limits/quota
403  → authz: RBAC role + SCOPE, DATA-PLANE role (Blob Data Reader ≠ Reader), Key Vault, deny assignment, Conditional Access
401  → authn: token iss/aud/exp/scp, Managed Identity acquisition, clock skew
404  → routing: URL/path/resource-id/API-version, or a Private Endpoint resolving you to the wrong instance
503  → dependency unavailable: circuit-break, fail fast, check health/SLA + composite SLA, shed load
502/504 → gateway/upstream: backend health probes, pool, upstream timeouts

Symptom beats status: a DNS error is a network problem regardless of any HTTP status after it.

7. az one-liners you'll actually type

# Which zones does a region expose? (place zonal resources deliberately)
az vm list-skus --location eastus --query "[?contains(locationInfo[0].zones, '1')].name" -o tsv | head

# Create a ZONE-REDUNDANT public IP + Standard LB (survives a datacenter failure)
az network public-ip create -g rg -n pip --sku Standard --zone 1 2 3
az network lb create -g rg -n lb --sku Standard   # Standard LB is zone-redundant by default

# Zone-redundant App Service / Function plan
az appservice plan create -g rg -n plan --sku P1V3 --zone-redundant --number-of-workers 3

# Zone-redundant Azure SQL
az sql db update -g rg -s server -n db --zone-redundant true

# A region's PAIR (your DR target)
az account list-locations --query "[?name=='eastus'].metadata.pairedRegion[0].name" -o tsv

# Service Health / Resource Health during an incident
az rest --method get --url "https://management.azure.com/subscriptions/<sub>/providers/Microsoft.ResourceHealth/availabilityStatuses?api-version=2022-10-01"

# Tail a Front Door / APIM health probe + see 429/503 in Log Analytics (KQL, P13)
#   AzureDiagnostics | where httpStatusCode_d in (429,503) | summarize count() by bin(TimeGenerated,5m)

# Diagnose a Private Endpoint DNS problem
nslookup mystorage.blob.core.windows.net    # should resolve to a 10.x PRIVATE IP, not public
az network private-dns zone list -g rg -o table
az network private-dns link vnet list -g rg -z privatelink.blob.core.windows.net -o table

8. War story shapes you'll relive

• "All our components are 99.9% but we keep missing the SLA." → serial composition makes the system worse than any component. Four 99.9% hops ≈ 99.6%. Make the weakest hop redundant; stop pretending the system inherits a single hop's number.
• "A 2-second blip became a 20-minute outage." → a retry storm. Every client retried with no jitter and no budget; the synchronized retries kept the dependency overloaded. Add full jitter
  • a retry budget + honor Retry-After.
• "We're 'redundant' but a single zone outage took us down." → both replicas were in the same availability zone. Redundancy without independent failure domains is theater. Spread across zones.
• "The Function can see the storage account but gets 403 reading blobs." → control-plane role (Reader) without a data-plane role (Storage Blob Data Reader). The P00 control/data plane split is the whole answer.
• "Name resolution failed right after we added a Private Endpoint." → the Private DNS zone isn't linked to the VNet, or the A record points at the public IP. The PE gave it a private IP; DNS still hands out the public one.
• "Half-open let the whole fleet through and re-killed the dependency." → your breaker allowed more than one probe in half-open. It must allow exactly one trial request.

9. Vocabulary that signals you've held the pager

• Availability zone — a physically separate datacenter (own power/cooling/network) in a region.
• Zone-redundant vs zonal — the platform spreads replicas for you vs you pin to one zone.
• Region pair — two regions with sequential updates + recovery priority; your DR target.
• Composite / serial SLA — ∏ aᵢ; the system is below every hop.
• Redundant SLA — 1 − ∏(1 − aᵢ); the group is down only if all replicas are down.
• N+k / binomial survival — need at least n of n+k nodes; the spare count is k.
• Downtime / error budget — (1 − A) × window; a budget you spend, not a wall.
• Exponential backoff + full jitter — uniform(0, min(cap, base·2^n)).
• Retry budget — a cap on total retries so they can't amplify an outage.
• Retry storm / thundering herd — synchronized retries that re-create the overload.
• Retry-After — the server telling you exactly how long to back off (honor it).
• Idempotency — twice == once; the precondition for any safe retry.
• Circuit breaker — closed→open→half-open; fail fast, then probe for recovery.
• Throttling / backpressure — the system saying "slow down"; obey, don't push.
• Failure signature — status + symptom; the input to the triage decision tree.

10. Beginner mistakes that mark you in interviews

1. Quoting a 99.9% SLA without composing it across the path (the system is below every hop).
2. Saying "redundant" without independent failure domains (same-zone replicas aren't redundant).
3. Retrying with no jitter (thundering herd) or no budget (retry storm), or retrying a non- idempotent mutation (double effect).
4. Ignoring Retry-After and fighting the throttle on your own schedule.
5. Confusing a circuit breaker with a retry (breaker says stop, retry says again).
6. Letting more than one probe through in half-open (re-stampede on recovery).
7. Reading a 403 as authentication (it's authz) — debugging the wrong layer.
8. Forgetting the control-plane vs data-plane role split behind a Storage 403.
9. Calling a Private-Endpoint DNS failure a "Storage outage" instead of a Private DNS zone link.
10. Treating the error budget as a wall to defend instead of a budget to spend (no deploys, no velocity, paralysis).

11. How this phase pays off later

• Phase 15 (capstone) — the combined security/SLA/cost scorer uses this SLA composition math directly; the platform you score is the Front Door → APIM → Function → DB chain.
• Phase 13 (Monitor/KQL) — the downtime/error-budget math here is the foundation for burn- rate alert rules; you see the 429/503/DNS signatures in Log Analytics.
• Phase 09 (APIM) and Phase 10 (Service Bus) — 429 rate limiting and message throttling are where Retry-After, backoff, and dead-letter (the breaker's fallback) actually live.
• Phase 06 (networking) — half the failure signatures (DNS, Private Endpoint, connection refused) are network problems this phase teaches you to triage.
• Phase 00 (control/data plane) — the 403-data-plane-role and the "quantify the tradeoff" habit are this phase applied under a live incident.

Now read the WARMUP slowly, then build the resiliency engine. After this, "make it reliable" stops being a vibe and becomes a number you can compute and a tree you can walk.