Phase 14 — Reliability, HA & Operational Troubleshooting

Difficulty: ⭐⭐⭐☆☆ (the arithmetic) → ⭐⭐⭐⭐⭐ (the judgment under a live incident) Estimated Time: 1 week (12–18 hours) Prerequisites: Phase 00 (control plane vs data plane; quantify the tradeoff), Phase 06 (networking — Private Endpoints and DNS are half the failure signatures), Phase 09 (API Management — 429 / rate limits / Retry-After), Phase 10 (Service Bus / Event Grid retry and dead-letter), and Phase 13 (Azure Monitor / KQL — how you see the failure signature in the first place). No Azure subscription required for the lab.

Why This Phase Exists

Every prior phase built a mechanism. This phase builds the discipline that keeps all of them up — and the method that gets them back up at 2 a.m. The JD names this directly:

"Provide troubleshooting and operational support for Azure cloud services and infrastructure." … "Monitor, troubleshoot, and optimize cloud infrastructure to ensure high availability, security, and performance."

A senior engineer says "we have a 99.9% SLA." A principal can tell you what that buys you (43.2 minutes of downtime a month), whether a four-hop chain can even deliver it (Front Door → APIM → Function → DB multiplies to something below every hop), and what it would cost to add a nine (zone-redundant replicas, an N+1 spare, the binomial math behind it). Reliability at this level is arithmetic you do before the design review, not a hope you express after the incident.

And when it does break — because it will — a principal does not flail. They read the failure signature (429? 403? a DNS error? a dependency timeout?), walk a decision tree to a likely cause, and reach for the right resilience primitive: a retry with exponential backoff and jitter under a retry budget (so the retries don't amplify the outage), a circuit breaker (so you stop hammering a dead dependency), or honoring the server's Retry-After (so a throttled client stops fighting the throttle). This phase makes you build those primitives so they stop being words in a Well-Architected checklist and become mechanisms you can explain — and defend — line by line.

What "Principal-Level" Means Here

A senior engineer uses a retry library and quotes an SLA. A principal understands the reliability stack well enough to:

• Compose an SLA in their head. Serial hops multiply (∏ aᵢ — more hops, lower SLA); redundant replicas combine as 1 − ∏(1 − aᵢ); an N+k cluster's availability is a binomial survival probability. They can take "Front Door → APIM → Function → SQL, each at three to four nines" and tell you the composite and the monthly downtime budget — with numbers, before anyone builds it.
• Spread for survival, not just speed. They know an availability zone is a physically separate datacenter (independent power, cooling, network) within a region, that zone- redundant services place replicas across zones so a datacenter fire is a non-event, and that paired regions give you a DR target with sequential update and recovery priority. "Make it HA" becomes a concrete placement decision.
• Make retries safe. They never retry without idempotency, never retry without jitter (the thundering herd that re-creates the outage), and never retry without a budget (the retry storm that turns a blip into an incident). They honor 429 / Retry-After because the server is telling them exactly what to do.
• Run a circuit breaker from first principles. Closed → Open (fail fast after a threshold) → Half-Open (one probe after a reset timeout) → Closed (recovered) or → Open (still broken). They can explain why each state exists and what failure it prevents.
• Triage by signature. They have a decision tree from failure signature to likely cause: 429 → throttling, 403 → authz/RBAC/data-plane role, a DNS error → Private DNS / Private Endpoint, 503 → dependency unavailable → circuit-break. Triage is a method, not a vibe.

Concepts

• Availability zones & region pairs. A zone is a physically separate datacenter (its own power, cooling, and network) inside a region; a region has three or more. Zone- redundant services replicate across zones automatically; zonal services pin to one zone (you place the redundancy). Paired regions are two regions in the same geography with sequential platform updates, recovery priority, and (for some services) built-in geo- replication — your DR target. Spreading replicas across zones is what makes a single- datacenter failure survivable.
• SLA composition math. A serial dependency chain is only as available as the product of its hops: A_chain = ∏ aᵢ — so every hop you add lowers the composite, and the result is below the worst hop. Redundant components combine the other way: A_redundant = 1 − ∏(1 − aᵢ) — the group is down only if all replicas are down, so redundancy drives availability up. N+k redundancy (need at least n of n+k nodes) is a binomial survival probability. From availability you get the downtime budget: (1 − A) × window.
• Retry with exponential backoff + jitter, under a budget. Backoff spaces retries (min(cap, base · 2^attempt)); full jitter randomizes the wait (uniform(0, backoff)) so a fleet that failed together doesn't retry in lockstep; a retry budget caps total retries so they can't amplify an outage. Honoring Retry-After (on 429 / 503) obeys the server's explicit instruction. Idempotency is the precondition that makes any of this safe to repeat.
• Circuit breaker. A state machine — Closed (pass, count failures), Open (fail fast after the failure threshold), Half-Open (one probe after the reset timeout) — that stops a client from hammering a dependency that's already down, and gives it room to recover.
• Throttling, backpressure, and the triage method. Throttling (429) and backpressure (downstream slowness flowing upstream as flow control) are the system telling you to slow down. The troubleshooting method turns a failure signature (HTTP status + symptom) into a likely cause and a first action via a decision tree — the muscle memory that turns a page into a plan.

Labs

Lab 01 — Resiliency Engine (flagship, implemented)

→ Lab folder: lab-01-resiliency-engine/

Integrated-Scenario Suggestions (carried through the whole track)

The phases compound. Keep these in mind as you build — each wires reliability into the larger platform:

1. The composite-SLA design review. Take the platform you've built across the track — Front Door → API Management (P09) → Function (P11) → SQL/Cosmos — and compute its serial composite SLA, its monthly downtime budget, and where adding a nine is cheapest (usually the single-zone hop). Then make the front door and database zone-redundant and recompute. This is the arithmetic behind a real availability target. (→ P09, P11, P00 cost models)
2. The retry-storm post-mortem. A dependency blips; every client retries; the synchronized retries (no jitter, no budget) turn a 2-second blip into a 20-minute outage. Reconstruct it, then fix it with full jitter + a retry budget + honoring Retry-After from the throttled service (P09 rate limiting, P10 Service Bus throttling). (→ P09, P10)
3. The circuit-breaker in front of a flaky dependency. A Function calls a downstream API that periodically dies. Without a breaker, every invocation waits for a timeout and piles up; with one, it fails fast while Open and probes once on Half-Open. Size the failure_threshold and reset_timeout, and wire it to the dead-letter path (P10) for the messages it sheds. (→ P10, P11)
4. The DNS / Private Endpoint incident. "Name resolution failed" after a Private Endpoint was added — the Private DNS zone isn't linked to the VNet, or the A record points at the public IP. Walk the diagnose tree, fix the zone link (P06), and add a synthetic DNS- resolution probe so the next one alarms before customers notice (P13). (→ P06, P13)
5. The error-budget burn alert. From the SLO (e.g. 99.9% → 43.2 min/month) compute the error budget, then write a KQL alert (P13) that fires when the budget burns too fast (multi-window burn rate). This is SRE error-budget thinking made concrete on top of the downtime math from this lab. (→ P13, P00 SLO framing)

Guides in This Phase

• HITCHHIKERS-GUIDE.md — the 30-minute orientation; read first
• WARMUP.md — the full primer; read slowly
• BROTHER-TALK.md — the candid version

Key Takeaways

• Serial chains multiply; redundancy combines the inverse. A_serial = ∏ aᵢ (more hops, lower SLA — below every hop); A_redundant = 1 − ∏(1 − aᵢ) (more replicas, higher SLA). N+k is a binomial survival probability. From availability comes the downtime budget (1 − A) × window. This is arithmetic you do before the review, not after the incident.
• Availability zones make a datacenter failure a non-event — if you spread replicas across them. Zone-redundant services do it for you; zonal services make you place it. Paired regions are your DR target. "Make it HA" is a placement decision with a number behind it.
• A retry without jitter and a budget is a loaded gun. Backoff spaces retries, jitter de-synchronizes the herd, a budget caps the amplification, Retry-After obeys the server, and idempotency is what makes any of it safe. Get one wrong and your retries cause the outage.
• A circuit breaker stops you hammering a corpse. Closed (count) → Open (fail fast) → Half-Open (probe) → Closed/Open. Each state prevents a specific failure: the breaker buys the dependency room to recover and your callers a fast failure instead of a pile-up.
• Triage is a method, not a vibe. A failure signature (status + symptom) maps to a likely cause and a first action: 429 → throttle/back-off, 403 → authz/RBAC/data-plane role, DNS error → Private DNS/Endpoint, 503 → dependency-unavailable/circuit-break. The decision tree is what keeps you calm at 2 a.m.

Deliverables Checklist

• Lab 01 implemented; all 40 tests pass against solution.py and your lab.py
• You can compose a four-hop serial SLA and state its monthly downtime budget from memory
• You can explain why two 0.99 replicas give 0.9999 and three give 0.999999
• You can compute an N+1 cluster's availability with the binomial (2-of-3 @ 0.99)
• You can state the three reasons a retry needs backoff, jitter, and a budget — and why idempotency is the precondition
• You can draw the circuit-breaker state machine and name what each state prevents
• You can take a failure signature (429/403/503/DNS) and give the likely cause and first action without looking it up