Lab 01 — Resiliency Engine

Phase: 14 — Reliability, HA & Operational Troubleshooting | Difficulty: ⭐⭐⭐☆☆ | Time: 3–5 hours

"We have a 99.9% SLA" is a sentence most engineers say and few can defend. Pull the cover off and reliability is two things: arithmetic you do before the design review, and a method you run during the incident. The arithmetic — serial hops multiply (∏ aᵢ, so a four-hop chain lands below every hop), redundant replicas combine the inverse (1 − ∏(1 − aᵢ), so two 0.99 replicas make 0.9999), and N+k is a binomial survival probability — tells you whether your design can even deliver the number. The method — make retries safe with backoff + jitter under a retry budget, honor Retry-After, put a circuit breaker in front of a flaky dependency, and walk a decision tree from failure signature to likely cause — is what keeps you calm at 2 a.m. This lab builds all of it: the SLA composer, the retry primitives, the breaker state machine, and the triage tree.

What you build

• SLA composition — serial_sla(avails) (the product ∏ aᵢ), redundant_sla(avails) (1 − ∏(1 − aᵢ)), composite_sla(components) where each hop may be a single availability or a nested set of redundant replicas (collapse with redundant_sla, then serial- multiply), downtime_minutes_per_month(avail) ((1 − A) × 43200), and n_plus_k_sla(p, n_required, total_nodes) — the binomial probability that at least n_required of total_nodes are up. All validate availabilities into [0, 1].
• Retry backoff & jitter — backoff_delay(attempt, base, cap) = min(cap, base · 2^attempt); full_jitter(attempt, base, cap, rng) = rng.uniform(0, backoff_delay(...)) with an injected, seeded rng (so the bounds are testable deterministically); a token-style RetryBudget(cap) (can_retry() / record() / remaining()) that stops retries when the budget is exhausted; and retry_after_seconds(headers, ...) that honors a Retry-After header when present and falls back to full-jitter backoff otherwise.
• The circuit breaker — CircuitBreaker(failure_threshold, reset_timeout) with states closed / open / half_open, driven by on_success(), on_failure(now), and allow_request(now) — all with injected now so the machine is deterministic. Closed → Open after failure_threshold consecutive failures; Open fails fast until reset_timeout elapses, then → Half-Open (one probe); a Half-Open success → Closed, a failure → Open.
• The triage decision tree — diagnose(signature) maps a failure signature (an http_status and/or a symptom) to a likely cause and a first action: 429 → throttling, 403 → authz/RBAC/data-plane role, 503 → dependency-unavailable/circuit-break, a DNS symptom → Private DNS/Endpoint, and more — with symptom checked before status.

Key concepts

Files

Run

pip install -r requirements.txt
pytest test_lab.py -v                      # against your lab.py
LAB_MODULE=solution pytest test_lab.py -v  # against the reference
python solution.py                         # worked example

Success criteria

• All 40 tests pass against your implementation.
• You can explain why test_serial_sla_multiplies_and_is_below_every_hop asserts 0.997002999 for three 0.999 hops — and why the composite is below every hop (the "more hops, lower SLA" law that breaks "all components are 99.9% so the system is 99.9%").
• You can explain why test_redundant_sla_raises_availability gives 0.9999 for two 0.99 replicas — and why that only holds if the replicas fail independently (different zones).
• You can explain why test_n_plus_k_binomial_known_value is 0.972 for 2-of-3 @ 0.9, and why need-all reduces to the serial product and need-one to the redundant SLA.
• You can explain why test_full_jitter_within_bounds_seeded uses a seeded rng — reliability logic that calls un-seeded random is untestable; the bound 0 ≤ jitter ≤ backoff is the whole point and you can only assert it deterministically.
• You can explain why test_retry_budget_exhaustion is the line that stops a retry storm: once the budget is spent, you fail fast instead of amplifying the outage.
• You can explain why test_breaker_half_open_failure_reopens matters — Half-Open allows exactly one probe, and a failed probe re-opens with the timer restarted (no fleet stampede on recovery).
• You can explain why test_diagnose_symptom_beats_status exists — a DNS error is a network problem regardless of any HTTP status that follows it.

How this maps to real Azure (Well-Architected Reliability + the resilience patterns)

What the miniature leaves out (and why it's fine): real reliability engineering also involves correlated failures (shared zones, control planes, dependencies) that the simple independence-assuming products over-estimate, load shedding and admission control, health probes and automatic failover, chaos testing, RTO/RPO and failover runbooks, and burn-rate alert windows. None of that changes the kernels this lab isolates — (1) availability composes as a product across a serial path and an inverse-product across redundancy, (2) a safe retry is backoff + jitter + a budget + idempotency + honoring Retry-After, (3) a circuit breaker is a three-state machine that fails fast and probes, and (4) triage is a decision tree from signature to cause. Those are exactly what interviewers probe and incidents turn on.

Extensions (build these in your own subscription)

• Correlated failure: extend composite_sla to take a correlation factor between replicas (e.g. same-zone replicas share a failure), so the math stops assuming independence — and watch how much a shared zone costs you in real availability.
• Burn-rate alerting: from downtime_minutes_per_month, compute a multi-window error-budget burn rate and emit a KQL alert rule (P13) that pages on a fast burn and tickets on a slow one — SRE-style multi-burn-rate alerting on top of this lab's downtime math.
• Decorrelated jitter: add the AWS "decorrelated jitter" variant (sleep = min(cap, uniform(base, prev·3))) alongside full jitter and compare their bounds and spread with a seeded rng.
• Ratio-based retry budget: replace the token budget with a ratio budget ("retries may be at most 10% of requests in the last minute"), the form most production systems actually use.
• A real composite SLA: take the platform you've built across the track, look up each service's published SLA, and compute the real composite — then make the weakest hop zone-redundant (az ... --zone-redundant) and recompute the number you'd put in the design doc.
• Wire diagnose to Azure Monitor (P13): feed real AzureDiagnostics rows (status codes + exception messages) through diagnose to auto-suggest a first action in an alert's annotation.

Interview / resume

• Talking points: "Compose the SLA of a four-hop chain and tell me the monthly downtime." / "Why must a retry have jitter and a budget — and why is idempotency the precondition?" / "Walk me through the circuit breaker's three states and what each prevents." / "You get a 403 from a Function calling Storage — what do you check first, and why isn't it authentication?"
• Resume bullet: Built a reliability engine — SLA composition (serial-product, redundant inverse-product, and N+k binomial survival, with monthly downtime budgets), retry with exponential backoff + full jitter under a token retry budget, Retry-After honoring, a circuit-breaker state machine (closed/open/half-open with injected time), and a failure-signature triage decision tree — modeling the Well-Architected Reliability pillar and the cloud resilience patterns.