Warmup Guide — Reliability, HA & Operational Troubleshooting

Zero-to-principal primer for Phase 14: what availability actually is and how you compose it across a real Azure chain (serial hops multiply, redundant replicas combine the inverse, N+k is a binomial), where availability zones and paired regions make a datacenter or region failure survivable, how to make retries safe with exponential backoff + jitter under a retry budget while honoring Retry-After, how a circuit breaker stops you hammering a dead dependency, and the troubleshooting method that turns a failure signature into a likely cause. Every concept goes from what it is to why it exists to the mechanism under the hood (diagrams, tables, math, code) to production significance to the misconceptions that page people at 2 a.m.

Table of Contents

Chapter 1: What Availability Actually Means

From zero. Availability is the probability that a system is up and serving correct responses when you ask it. We write it as a number in [0, 1] (or a percentage), and we casually count its nines: 0.99 is "two nines," 0.999 is "three nines," 0.9999 is "four nines."

Why it exists / what it buys. An availability is only meaningful as a downtime budget over a window. The arithmetic is trivial and you must be able to do it instantly:

$$ \text{downtime} = (1 - A) \times \text{window} $$

Over a 30-day month (43200 minutes), the table you should have memorized:

AvailabilityNinesDowntime / monthDowntime / year
0.99two432 min (7.2 h)~3.65 days
0.999three43.2 min~8.76 h
0.9999four4.32 min~52.6 min
0.99999five0.43 min (~26 s)~5.26 min

Production significance. An SLA (Service Level Agreement) is the promise with a financial credit attached; an SLO (objective) is your internal target, usually stricter; an SLI (indicator) is the measurement (success rate, latency). The principal habit: when someone says "three nines," you immediately translate to "43 minutes a month — is that acceptable for this workload, and can the architecture even deliver it?" (Chapter 3 shows why a chain often can't.)

Misconception. "We have a 99.9% SLA" is not a property of your system — it's a property of each component's SLA composed across the path a request takes. A 99.9% database behind a 99.9% gateway behind a 99.9% function is not a 99.9% system (Chapter 3). And note the error budget framing: 43.2 minutes/month is a budget you spend on deploys, experiments, and incidents — not a failure to avoid at all costs. Spending it deliberately (SRE thinking) is healthier than guarding it religiously.

Chapter 2: Availability Zones and Region Pairs

From zero. Azure's physical hierarchy, bottom-up: a datacenter is a building full of racks. An availability zone (AZ) is one or more datacenters within a region that have independent power, cooling, and physical network — so a fire, a flooded transformer, or a network-gear failure in one zone does not take down another. A region is a set of datacenters in a metro area, and an enabled region has three or more zones. A region pair is two regions in the same geography (e.g. East US ↔ West US) with special platform behavior.

Geography (e.g. United States)
└── Region: East US  ──────paired with──────►  Region: West US (DR target)
    ├── Availability Zone 1   (datacenter(s): own power/cooling/network)
    ├── Availability Zone 2   (independent failure domain)
    └── Availability Zone 3
        a zone failure = ONE datacenter down; the other zones keep serving

Why it exists / the mechanism. Zones are independent failure domains inside a region with low-latency links between them (typically < 2 ms round trip), so you get fault isolation and synchronous replication. Two deployment styles:

  • Zone-redundant — the service spreads replicas across zones for you (e.g. zone- redundant Storage, zone-redundant SQL, a Standard Load Balancer, zone-redundant App Gateway / Front Door). A zone outage is automatically absorbed.
  • Zonal — you pin a resource to a specific zone (e.g. a VM in Zone 2). To get redundancy you place instances in different zones yourself and load-balance across them.

Region pairs add: sequential platform updates (Azure won't update both halves of a pair at once), recovery priority (a broad outage restores at least one region of each pair first), data residency within the geography, and for some services built-in geo-replication (e.g. GRS Storage replicates to the pair). The pair is your disaster-recovery target — a region dying (rare but real) is what region-pair DR is for, where zones are for the common datacenter failure.

Production significance. "Make it highly available" decomposes into a placement decision: single zone (cheapest, a datacenter failure is an outage) → zone-redundant (survives a datacenter failure, near-zero added latency, the default for production) → multi-region (survives a region failure, costs you cross-region latency, async replication, and a failover runbook). Each step up the ladder adds a nine and a cost; you pick the rung the SLO requires.

Misconception. "Multi-region first." No — zone redundancy is the high-leverage default: it removes the most common failure (a single datacenter) for almost no latency cost and little complexity. Multi-region is for region-level failures and strict RTO/RPO requirements, and it brings real cost: cross-region replication lag, conflict handling, and a failover process you must test. Most workloads need zone redundancy, not a second region.

Chapter 3: SLA Composition — Serial Chains Multiply

This is the chapter that separates "we have a 99.9% SLA" from understanding what your system can actually deliver.

From zero. A serial dependency chain is a path where every hop must be up for the request to succeed: Front Door → API Management → Function → Database. If the hops fail independently, the composite availability is the product:

$$ A_{\text{serial}} = \prod_{i=1}^{n} a_i $$

The mechanism / the consequence. Multiplying numbers below 1 always gives a smaller number, so every hop you add lowers the composite SLA, and the result is below the worst hop. Worked from the lab's worked example:

Front Door 0.9995 × APIM 0.9995 × Function 0.9995 × SQL 0.9999  =  0.998401
   → composite ≈ 99.84%, BELOW every hop
   → monthly downtime budget ≈ 69 minutes (vs 4.3 min for the best single hop)

Three "three nines" hops in series:

$$ 0.999 \times 0.999 \times 0.999 \approx 0.997002999 $$

— so a chain of three-nines components is barely two-and-a-half nines. The lab's serial_sla is exactly this product, and test_serial_sla_multiplies_and_is_below_every_hop pins the 0.997002999.

Production significance. This is the reliability arithmetic for a design review. Before anyone builds the Front Door → APIM → Function → DB platform, you can say "the composite is 99.84%, that's ~69 minutes a month — if the SLO is 99.9% (43 min), we're already over budget on paper, so we need redundancy on the weakest hop." Which hop? The one whose availability is lowest and which isn't already redundant — usually the single-region/single-zone component. Adding a nine to the weakest hop moves the composite the most.

Misconception. "All the components are 99.9%, so the system is 99.9%." No — serial composition makes the system worse than any component. The only way the composite equals a single hop's SLA is a one-hop system. The other misconception: that the hops are independent. If two hops share a failure domain (same zone, same dependency, same control plane), their failures correlate and the simple product over-estimates availability — correlated failure is why a "redundant" pair in the same zone isn't real redundancy (Chapter 4).

Chapter 4: Redundancy and N+k — Combining the Inverse

From zero. Redundancy is the opposite of a serial chain: instead of "all must be up," it's "at least one must be up." Put n replicas behind a load balancer; the group is down only if every replica is down. Since the replicas fail independently, the unavailabilities multiply:

$$ A_{\text{redundant}} = 1 - \prod_{i=1}^{n} (1 - a_i) $$

The mechanism / the magic. Each replica's unavailability (1 − a) is a small number; multiplying small numbers makes them tiny; one minus tiny is close to one. Redundancy drives availability up, fast:

one replica  @ 0.99             → 0.99        (two nines)
two replicas @ 0.99 each        → 1 − 0.01²   = 0.9999      (FOUR nines)
three        @ 0.99 each        → 1 − 0.01³   = 0.999999    (SIX nines)

Each independent replica roughly doubles your nines. The lab's redundant_sla is exactly 1 − ∏(1 − aᵢ), and test_redundant_sla_raises_availability pins the 0.9999 and 0.999999.

N+k redundancy — the binomial. Real clusters need at least n of n+k nodes up to serve (a quorum, a capacity floor). With each node independently up with probability p, the cluster availability is the binomial survival probability:

$$ A_{N+k} = \sum_{i=n}^{N} \binom{N}{i}, p^{i} (1-p)^{N-i} \quad\text{where } N = n+k $$

The spare count k is total − required (N+1 = one spare). Worked: need 2 of 3 nodes, each 0.9:

$$ \binom{3}{2}(0.9)^2(0.1) + \binom{3}{3}(0.9)^3 = 3(0.81)(0.1) + 0.729 = 0.243 + 0.729 = 0.972 $$

The lab's n_plus_k_sla sums these binomial terms, and test_n_plus_k_binomial_known_value pins the 0.972. Two boundary identities the lab also proves: need-all (n = N) is just the serial product pⁿ, and need-one (n = 1) is exactly redundant_sla of N replicas.

Composite chains with redundant hops. Real systems are serial chains of redundant hops: a zone-redundant front door (replicas) → a single APIM → a zone-redundant database (replicas). You collapse each redundant hop with redundant_sla first, then multiply the hops serially — which is precisely the lab's composite_sla.

composite( [0.99, 0.99],  apim 0.999,  function 0.999,  [0.999, 0.999] )
   step 1: collapse redundant hops → 0.9999,  0.999,  0.999,  0.999999
   step 2: serial product of the collapsed hops → the composite SLA

Production significance. This is how you buy a nine: find the weakest serial hop, make it redundant, recompute. The cost ladder is real — a second replica doubles that hop's bill, and correlated failure (same zone) silently breaks the independence the math assumes — so you place the replicas in different availability zones (Chapter 2) to keep the failures independent.

Misconception. "Two replicas means it's always up." No — it means it's down only when both are down and the failures were independent. Put both in the same zone and a zone outage takes both; the 1 − ∏(1 − a) formula silently assumed an independence you didn't have. Redundancy without independent failure domains is theater.

Chapter 5: Retry, Exponential Backoff, and Jitter

From zero. A transient failure (a blip, a brief throttle, a momentary network drop) is one that succeeds if you simply try again. A retry is trying again. The naive version — retry immediately, in a tight loop — is dangerous: it hammers a struggling dependency exactly when it's least able to cope, and it does so from every client at once.

Exponential backoff — the mechanism. Space the retries out, doubling the wait each time:

$$ \text{delay}(n) = \min\big(\text{cap},; \text{base} \cdot 2^{n}\big) $$

where n is the 0-indexed attempt. The doubling gives a recovering dependency progressively more room; the cap bounds the wait so a long outage doesn't push retries hours apart. The lab's backoff_delay is exactly this, and test_backoff_doubles_then_caps walks 1 → 2 → 4 → 8 → 10(capped) → 10:

base=1s, cap=10s:
  attempt 0:  1s     attempt 3:  8s
  attempt 1:  2s     attempt 4: 16s → capped to 10s
  attempt 2:  4s     attempt 5+: stays 10s

Jitter — why pure backoff isn't enough. Picture a thousand clients that all hit the same dependency, which blips. They all fail at the same instant, so they all retry at exactly base later, then 2·base later — a synchronized thundering herd that re-creates the overload on every retry wave. Jitter randomizes the delay to de-synchronize them. The robust form is full jitter:

$$ \text{jitter}(n) = \text{uniform}\big(0,; \text{delay}(n)\big) $$

— a uniform random wait between 0 and the backoff ceiling, so the herd smears across the whole interval instead of stacking on the boundary. The lab's full_jitter is exactly this, bounded 0 ≤ jitter ≤ delay(n), and — critically — it takes an injected rng (defaulting to a seeded random.Random(0)) so the behavior is deterministic and testable. test_full_jitter_within_bounds_seeded checks the bounds across many draws with a seeded rng.

Why a seeded, injected rng? Reliability code that calls un-seeded random is untestable — you can't assert anything about its output. Injecting the rng (and defaulting to a seeded one) is the pattern that lets you prove the bounds deterministically. Never put un-seeded randomness in logic you need to test.

Production significance. AWS's "Exponential Backoff and Jitter" paper showed full jitter both reduces total work and clears a backlog faster than backoff alone — counterintuitively, more randomness is more stable. Every Azure SDK's retry policy uses backoff + jitter; you should be able to explain why both knobs exist.

Misconception. "Backoff is enough." No — backoff alone keeps the herd synchronized (they just all wait the same exponentially-growing time). Jitter is what breaks the synchronization that turns a blip into a sustained outage. And the other misconception (Chapter 6): that you can retry forever — without a budget, retries amplify the very outage they're responding to.

Chapter 6: Retry Budgets, Idempotency, and Retry-After

From zero — the retry storm. Here's the failure mode backoff + jitter doesn't fix: a dependency is degraded, so every request fails, so every client retries (with backoff and jitter, like a good citizen) — but now the dependency is receiving its normal load plus all the retries. The extra load keeps it degraded, which causes more failures, which causes more retries. The retries have amplified the outage. This is a retry storm, and it's why a brief blip becomes a long incident.

The retry budget — the mechanism. Cap the total retries allowed in a window, as a budget of tokens. Each retry spends a token; when the budget is exhausted, clients stop retrying and fail fast. The budget bounds the amplification: retries can add at most the budget's worth of extra load, no matter how bad things get.

RetryBudget(cap=3):
  retry → record() → used=1  (remaining 2)   can_retry()=True
  retry → record() → used=2  (remaining 1)   can_retry()=True
  retry → record() → used=3  (remaining 0)   can_retry()=True
  retry → can_retry()=False  → STOP. Fail fast. Do not amplify the outage.

The lab's RetryBudget is exactly this token tracker (can_retry, record, remaining), and test_retry_budget_exhaustion proves record() raises once the budget is spent. (Real systems often express the budget as a ratio — "retries may be at most 10% of requests" — but the token model is the kernel.)

Idempotency — the precondition. A retry is only safe if doing the operation twice has the same effect as doing it once — that is idempotency. PUT a resource: idempotent (ARM relies on this, P01). Append a charge to an account: not idempotent — retrying double- charges. Before you retry anything that mutates state, you need idempotency, usually via an idempotency key (dedup the duplicate) or a naturally-idempotent operation (upsert, set-to- value). This is the same exactly-once-effect idea from P10's Service Bus duplicate detection: at-least-once delivery + idempotent processing = exactly-once effect. No idempotency → retries corrupt state → don't retry.

Retry-After — let the server tell you. When a server throttles you it returns 429 Too Many Requests (or sometimes 503) with a Retry-After header — the server telling you exactly how long to wait. Honoring it is strictly better than guessing with backoff, because the server knows when its quota window resets. The rule: if Retry-After is present, honor it; otherwise fall back to client-side full-jitter backoff. The lab's retry_after_seconds implements exactly this — a case-insensitive header lookup that, when present, returns the header's seconds, and otherwise returns full_jitter(...). test_retry_after_honors_header and test_retry_after_falls_back_to_jitter pin both paths.

Production significance. A throttled client that ignores Retry-After and retries on its own schedule fights the throttle, stays throttled, and wastes the dependency's capacity. Azure services (Storage, Cosmos DB, ARM, APIM, Entra) return 429 + Retry-After under throttling; honoring it is the difference between a brief slowdown and a sustained 429 storm.

Misconception. "Retries make things more reliable." Only bounded, idempotent, jittered retries do. Unbounded retries make things less reliable by amplifying outages; non- idempotent retries make things incorrect by duplicating effects. Retry is a sharp tool with three safety catches — budget, jitter, idempotency — and you need all three.

Chapter 7: The Circuit Breaker State Machine

From zero. Even with backoff, jitter, and a budget, there's a failure mode left: a dependency that's fully down for a while. Every request still tries it, waits for a timeout (slow!), fails, and the failures pile up — consuming threads, connections, and latency budget on a dependency that cannot succeed right now. A circuit breaker detects this and fails fast: it stops sending requests to a dead dependency until there's reason to believe it recovered.

The mechanism — three states. It's a small state machine (the electrical-breaker analogy: trip on overload, reset after a cooldown):

          failure_threshold consecutive failures
   ┌────────┐ ───────────────────────────────────► ┌──────┐
   │ CLOSED │                                       │ OPEN │
   │ (pass, │ ◄─────────────────────────────────────│(fail │
   │ count) │            on_success in HALF_OPEN     │ fast)│
   └────────┘                                        └──────┘
        ▲                                                │
        │ probe succeeds                                 │ reset_timeout elapses
        │                                                ▼
        │                                          ┌───────────┐
        └──────────────────────────────────────── │ HALF_OPEN │
                  probe fails → back to OPEN        │  (1 probe)│
                                                    └───────────┘
  • Closed (normal): requests pass; consecutive failures are counted. At failure_threshold consecutive failures, trip → Open. (A success resets the count — the threshold is about a sustained failure, not occasional blips.)
  • Open (tripped): requests are rejected immediately — fail fast, don't even try. This protects the dependency (no load) and the caller (no waiting on timeouts). After reset_timeout elapses, the next request is allowed as a probe and the state moves to Half-Open.
  • Half-Open (probing): exactly one trial request is allowed through. If it succeeds, the dependency is back → Closed. If it fails, it's still broken → Open (and the reset timer restarts). Half-Open is what prevents a stampede the instant the timeout expires: one probe decides, not the whole fleet.

The lab's CircuitBreaker implements this precisely with an injected now (so it's deterministic — no wall clock). The tests walk every edge: closed → open on threshold, open → half_open exactly at the reset boundary, half_open → closed on a successful probe, and half_open → open (with the timer restarting) on a failed probe.

Production significance. A breaker turns "the dependency is down and now we're down because we're all blocked waiting on it" into "the dependency is down, we fail fast, shed to a fallback or dead-letter, and probe for recovery." It's the bulkhead that keeps one sick dependency from sinking the caller — combine it with a timeout (so a slow call doesn't hang) and a fallback (cache, default, dead-letter) and you have the resilience triad.

Misconception. "A breaker retries for me." No — a breaker is the opposite of a retry: a retry says "try again," a breaker says "stop trying for a while." They're complementary — retry handles transient failures (a blip), the breaker handles sustained ones (a dependency that's down) — but conflating them is a classic error. Another: that Half-Open lets everyone through. It lets exactly one probe through; the whole point is to test recovery without re-stampeding.

Chapter 8: Throttling, Backpressure, and the Triage Method

From zero. Throttling is a server deliberately rejecting requests (with 429) to protect itself from overload — it's a feature, not a bug. Backpressure is the more general idea: when a downstream stage can't keep up, that slowness propagates upstream as a signal to slow down (a full queue, a blocked write, a 429). Both are the system telling you "too much, slow down" — and the right response is to obey (back off, honor Retry-After, shed load), not to push harder.

The triage method — from signature to cause. When something breaks, the principal doesn't guess; they read the failure signature — the HTTP status and the symptom — and walk a decision tree to a likely cause and a first action. This is the muscle memory the lab's diagnose encodes. The common Azure signatures:

SignatureLikely causeFirst action
429throttling / rate limit / quotahonor Retry-After, back off + jitter, spread load, check service & subscription limits
403authz — RBAC / data-plane role / Key Vault / denycheck the role and scope, the data-plane role (Storage Blob Data Reader, not just Reader), Key Vault access policy/RBAC, deny assignments, Conditional Access
401authn — bad/expired/missing tokencheck iss/aud/exp/scp claims, the Managed Identity token acquisition, clock skew
404not found / routing / wrong instancecheck URL/path, resource id, API version, whether a Private Endpoint resolves you to the wrong instance
503dependency unavailablecircuit-break, fail fast, check its health/SLA and your composite SLA, shed load
502 / 504gateway / upstreamcheck backend health probes, the pool, upstream timeouts (Front Door/APIM couldn't reach a healthy backend)
symptom: name resolution failedDNS / Private Endpointcheck the Private DNS zone, the zone-to-VNet link, and that the A record points at the Private Endpoint NIC, not the public IP
symptom: timeoutdependency slow/stuckcircuit-break, check dependency health/SLA and connection pool, ensure retries use backoff + jitter
symptom: connection refusedconnectivityNSG/firewall, Private Endpoint, listener/port, effective routes

The ordering matters. The lab checks symptom first — a DNS error is a network problem regardless of any HTTP status that follows it, so {"http_status": 503, "symptom": "name resolution failed"} diagnoses as DNS, not as a 503. test_diagnose_symptom_beats_status pins exactly this precedence.

Production significance. Two of these are the most common Azure incidents in this curriculum's world:

  1. The 403 from a Function calling Storage. The Function has Reader on the storage account (a control-plane role) but not a data-plane role (Storage Blob Data Reader), so it can see the resource but not the bytes. Control plane vs data plane (P00) is the whole answer — and diagnose({"http_status": 403}) points you straight at the data-plane role.
  2. The DNS error after a Private Endpoint. Adding a Private Endpoint (P06) changes name resolution: the resource's public FQDN must now resolve to the private IP via a Private DNS zone linked to the VNet. If the zone isn't linked, or the A record is wrong, you get "name resolution failed" — and diagnose sends you to the Private DNS zone link.

Misconception. "Throttling means scale up." Sometimes — but often it means you're hitting a quota and the fix is to honor Retry-After, spread load, or request a limit increase, not to throw more clients at a service that's already saying "too many." And "a 403 is an authentication problem" — no, 403 is authz (you're authenticated but not permitted); 401 is authn. Mixing them up sends you debugging the wrong layer.

Lab Walkthrough Guidance

Lab 01 — Resiliency Engine, suggested order (matches the file top-to-bottom):

  1. SLA math (Ch. 3–4) — _validate_avail (reject outside [0,1] and NaN), then serial_sla (the product ∏), redundant_sla (1 − ∏(1 − a)), composite_sla (collapse each list-valued hop with redundant_sla, then serial-multiply), downtime_minutes_per_month ((1 − A) × minutes), and n_plus_k_sla (sum the binomial terms with math.comb). Pin the exact numbers: three 0.999 ≈ 0.997002999; two 0.99 redundant = 0.9999; 2-of-3 @ 0.9 = 0.972; three nines = 43.2 min.
  2. Backoff & jitter (Ch. 5) — backoff_delay = min(cap, base · 2^attempt); full_jitter = rng.uniform(0, backoff_delay(...)) with the injected, seeded rng (test the bounds deterministically). Then RetryBudget (can_retry/record/remaining, raise on overspend) and retry_after_seconds (case-insensitive Retry-After → honor it; else full_jitter).
  3. Circuit breaker (Ch. 7) — the CircuitBreaker state machine with injected now: allow_request (Closed→pass, Open→reject until reset_timeout then →Half-Open + allow the probe, Half-Open→pass), on_success (clear failures, →Closed), on_failure (Half-Open→Open immediately; Closed→count, trip at threshold). Walk every transition including reopen-on- half-open-failure with a restarted timer.
  4. Triage tree (Ch. 8) — diagnose: symptom-first (DNS, timeout, connection refused, TLS), then status codes (429/403/401/404/503/502/504/other-5xx), then the unknown fallback. Validate the input is a mapping.

Run it red, make it green: pytest test_lab.py -v, then LAB_MODULE=solution pytest test_lab.py -v (40 tests), then python solution.py for the worked SLA chain, redundancy, N+k, the backoff/jitter sequence, the budget exhausting, the breaker tripping and recovering, and a few diagnose calls.

Success Criteria

You are ready for Phase 15 (the capstone) when you can, from memory:

  1. State the downtime budget for two/three/four nines over a month (432 min / 43.2 min / 4.32 min) and explain it as an error budget you spend, not a wall you defend.
  2. Define an availability zone vs a region pair, and say when you reach for zone redundancy vs multi-region — with the cost each adds.
  3. Compose a serial chain's SLA (∏ aᵢ), explain why it's below every hop, and identify which hop to make redundant first.
  4. Compose redundant replicas (1 − ∏(1 − a)), show two 0.99 → 0.9999, and explain why same- zone replicas break the independence the formula assumes.
  5. Compute an N+1 cluster's availability with the binomial (2-of-3 @ 0.99) and state the need- all and need-one boundary identities.
  6. State the three safety catches on a retry — backoff, jitter, budget — and why idempotency is the precondition, and when to honor Retry-After.
  7. Draw the circuit-breaker state machine and name what each of the three states prevents.
  8. Take a failure signature (429/403/503/DNS) and give the likely cause and first action, and explain why symptom beats status.

Interview Q&A

Q: Compose the SLA of Front Door → APIM → Function → SQL, each ~99.95–99.99%, and tell me the monthly downtime. It's a serial chain — every hop must be up — so the composite is the product of the hops: roughly 0.9995 × 0.9995 × 0.9995 × 0.9999 ≈ 0.9984. That's below every hop (the "more hops, lower SLA" law), about 99.84%, or ~69 minutes of downtime a month. If the target is 99.9% (43 min), the chain is already over budget on paper, so I'd make the weakest single- zone hop zone-redundant — redundant replicas combine as 1 − ∏(1 − a), which drives that hop's availability up and lifts the whole product. Adding a nine to the weakest hop moves the composite the most.

Q: Why does redundancy raise availability so fast — two 0.99 replicas to four nines? Because a redundant group is down only when every replica is down, and (if failures are independent) the unavailabilities multiply: 1 − (1 − 0.99)(1 − 0.99) = 1 − 0.01² = 0.9999. Each independent replica roughly doubles the nines. The catch is independent — if both replicas sit in the same availability zone, a zone outage takes both and the formula's independence assumption is false. So I place replicas across different zones to keep the failures independent; redundancy without independent failure domains is theater.

Q: A retry — what makes it safe, and how can it make things worse? Three safety catches plus a precondition. Backoff (min(cap, base · 2^attempt)) spaces retries so I don't hammer a recovering dependency. Jitter (uniform(0, backoff)) de- synchronizes a fleet that failed together — without it they retry in lockstep and re-create the overload (thundering herd). A retry budget caps total retries so they can't amplify the outage into a retry storm. And the precondition is idempotency — retrying a non-idempotent mutation double-applies it; I need an idempotency key or a naturally-idempotent operation first. Get any of these wrong and the retries either cause the outage or corrupt state. And if the server sends Retry-After (on a 429), I honor it instead of guessing — it knows when its quota resets.

Q: Walk me through a circuit breaker's three states and what each prevents. Closed is normal: requests pass and I count consecutive failures; at the failure threshold I trip to Open. Open rejects requests immediately — fail fast — which prevents both piling up on timeouts on my side and hammering a dependency that's already down; after a reset timeout I allow one probe and move to Half-Open. Half-Open lets exactly one trial request through: success → Closed (recovered), failure → Open (still broken, restart the timer). The single probe is what prevents the whole fleet stampeding the instant the timeout expires. A breaker is the opposite of a retry — retry handles a transient blip, the breaker handles a sustained outage — and I pair it with a timeout and a fallback.

Q: A Function returns 403 calling Blob Storage. What's your first check? 403 is authz, not authn — the call is authenticated but not permitted. The classic cause is a control-plane vs data-plane role mismatch: the Function's identity has Reader on the storage account (control plane — it can see the resource) but not a data-plane role like Storage Blob Data Reader (it can't read the bytes). So I check the data-plane RBAC assignment first, then the scope, then any deny assignment or Conditional Access. If it were a 429 I'd think throttling and honor Retry-After; a 503, dependency-unavailable and circuit- break; a "name resolution failed", Private DNS / Private Endpoint.

Q: After adding a Private Endpoint, clients get "name resolution failed." Why? A Private Endpoint gives the resource a private IP in your VNet, but clients still use its public FQDN — which by default resolves to the public IP. The fix is a Private DNS zone (e.g. privatelink.blob.core.windows.net) with an A record pointing the FQDN at the Private Endpoint's NIC, linked to the VNet so its resolver uses it. "Name resolution failed" almost always means the zone isn't linked, the link's missing the VNet, or the A record is wrong/ pointing at the public IP. That's why my triage checks the symptom (DNS) before any HTTP status — it's a network problem regardless of what status follows.

Q: How do you turn an SLO into an alert? From the SLO I get the error budget: 99.9% over a month is (1 − 0.999) × 43200 ≈ 43.2 minutes. I don't alert on every failed request — I alert on the burn rate of the budget (a multi-window, multi-burn-rate alert: page if I'd exhaust the month's budget in an hour at the current rate, ticket if it's a slow burn). That's SRE error-budget thinking on top of the downtime arithmetic — and it's why the downtime math in this lab is the foundation for the alert rule in P13.

References