Lab 02 — Fleet Canary & Rollback

Phase: 11 — CI/CD & Testing | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–5 hours Language: Python 3 (stdlib) | Hardware: none

Concept primer: ../WARMUP.md Ch. 6, ../HITCHHIKERS-GUIDE.md §4.

Run

python3 solution.py     # healthy rollout to 100%; bad version caught at canary + rolled back

0. The mission

Ship a new version to a fleet without an incident: canary → progressive waves, each gated on health, with automatic image-pinned rollback and failure-domain-aware waves. This is the unifying rollout discipline — identical for an agent, a config, firmware (Phase 08), and a k8s upgrade (Phase 06).

1. What the run proves

  • Healthy version reaches 100% through waves (canary 1 → 1% → 10% → rest), each promoted only after passing the health gate.
  • Bad version is caught at the canary (1 node) and rolled back + halted — the self-test asserts zero nodes end up on the bad version and only the canary was ever touched. A bad release at 1% is an annoyance; at 100% it's an outage.
  • A version that regresses slightly later (not in the canary) is still caught in an early wave, so the fleet is never fully exposed.

2. The mechanics

  • plan_waves: canary=1, then 1%, 10%, rest — widening confidence.
  • The gate: deploy a wave → bake → health-check → promote or roll back to the pinned prior version and halt.
  • Failure-domain awareness (Phase 01): waves are spread so a bad release can't down a whole PDU/ToR at once.

3. Extensions

  1. Error-budget halting (Phase 07): halt if the rollout burns the budget too fast, not just on hard failures.
  2. Bake time per wave (a real clock) and soak before promotion.
  3. Integrate with the Phase 08 firmware orchestrator (each node's "deploy" is a verified A/B firmware update) and Phase 06 drain (cordon/drain before reboot).
  4. Blue/green variant: stand up the new version beside the old and switch, for instant rollback.

4. What this lab proves

You can ship to a fleet safely — the "how do you roll out to 10,000 nodes?" answer as a runnable controller: canary, health gates, auto-rollback, domain-aware waves. Combined with Lab 01's test pyramid, you own the JD's "build, test, and deployment pipelines," and you can articulate it as the same discipline behind firmware and k8s rollouts. Feeds the Phase 13 capstone and system-design/03.