Lab 02 — Chaos & Reconciliation Hardening

Phase: 10 — Reliability & Debugging | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–6 hours Language: Python 3 (stdlib) | Hardware: none

Concept primer: ../WARMUP.md Ch. 5–6, ../HITCHHIKERS-GUIDE.md §5–§6.

Run

python3 solution.py     # baseline -> transient chaos -> outage+recovery -> permanent outage

0. The mission

Prove (don't just claim) that reliability patterns matter: build a chaos source that injects transient failures and outages into a dependency (a flaky BMC), and show a hardened reconcile loop (retry + backoff + jitter + circuit breaker + idempotency) protects the system where a naive loop creates a retry storm.

1. The lesson the numbers tell

  • Transient failures: both converge — retry handles transient faults.
  • Outage + recovery: both converge after recovery, but the naive loop hammers the dead BMC every tick (a retry storm), while the hardened loop's breaker opens and fast-fails, hitting the dead dependency far fewer times.
  • Permanent outage: the decisive case — the naive loop hammers the dead BMC 60×; the hardened loop hits it only ~14× (breaker open, periodic half-open probes). When a dependency is truly down, a retry storm prevents its recovery and can cascade the failure; the breaker is what stops that.

The breaker transitions (->open, open->half-open, ->closed) show the state machine that makes this work — and it's the same self-healing idea as the Phase 06 operator, hardened.

2. The patterns in play

  • Idempotency: converging = setting desired state, so retries can't corrupt it.
  • Retry + backoff + jitter (capped): absorbs transient faults without a thundering herd.
  • Circuit breaker: fail fast when a dependency is clearly down; protect it and the caller; probe for recovery.

3. Extensions

  1. Add a bulkhead (a per-dependency concurrency limit) and show one slow dependency can't starve the others.
  2. Inject a slow dependency (latency, not failure) and add a timeout; show an unbounded wait hangs the loop without one.
  3. Add an SLO and measure the error budget burned under each chaos scenario (Phase 07).
  4. Make the chaos a game day: a scripted sequence of faults the on-call must diagnose using the breaker transitions and metrics.

4. What this lab proves

You can reason about and demonstrate resilience — the difference between code that limps along and code that fails safe and recovers. "What's a circuit breaker and why do you need one with retries?" becomes a runnable proof (the 60× vs 14× retry-storm contrast), and these patterns harden the Phase 13 capstone's control loop. Chaos engineering — finding the unhappy paths before production does — is the proactive half of the JD's reliability mandate.