Lab 01 — Firmware Update Orchestrator (staged + rollback)

Phase: 08 — Firmware & RAS | Difficulty: ⭐⭐⭐⭐☆ | Time: 5–7 hours Language: Python 3 (stdlib) | Hardware: none

Concept primer: ../WARMUP.md Ch. 2–5, ../HITCHHIKERS-GUIDE.md §2–§5.

Run

python3 solution.py     # happy path, bad-signature reject, failed-activation rollback, anti-rollback

0. The mission

Make the most dangerous operation in the role — flashing firmware on a remote node — safe: verify before touching hardware, stage to the inactive A/B bank, activate, verify after, and roll back to the previous bank on failure. Plus signature verification and anti-rollback.

1. The four behaviors (read the state machine)

  1. Happy path v10→v11: stage to inactive bank → activate → verify → commit (advance the anti-rollback floor).
  2. Tampered image: HMAC signature check fails → rejected before any flash (the device is never touched).
  3. Bad new firmware: passes signature but fails post-activation health → automatic rollback to the previous bank (v10 restored). This is why A/B banks exist and why you verify after activation, not just before.
  4. Downgrade: a v9 image below the v10 floor → anti-rollback reject (no downgrading past a security fix).

2. Why each piece matters

  • Verify before AND after: a flash can succeed yet the firmware misbehave; only post- activation health (sensors/links — Phase 04) confirms real success.
  • A/B banks: a bad flash never destroys the working firmware → recovery is a pointer switch, not a truck roll. Critical for the BMC (don't lose your OOB lifeline).
  • Anti-rollback + A/B together: you roll back to the other good bank, never to an old vulnerable version — resolving the security-vs-recoverability tension.

3. Extensions

  1. Drive a real Redfish UpdateService (SimpleUpdate) against the Phase 03 mock + poll the Task to completion.
  2. Add fleet canary + drain-first (Phases 06/11): 1 node → 1% → rest with auto halt-and- rollback on SLO regress.
  3. Model power-loss mid-flash: crash between stage and activate, then prove resume leaves a bootable bank.
  4. Real signing: replace HMAC with RSA/ECDSA signature verification.

4. What this lab proves

You can build firmware-update software that won't brick a fleet — verify, A/B, rollback, anti- rollback — and explain every safety decision. "How do you update firmware on 1,000 nodes safely" becomes a demonstration. This orchestrator + the Phase 06 drain flow + Phase 11 canary = the fleet firmware system (system-design/03).