System Design 03 — Fleet Firmware Update System

"Update firmware across 10,000 nodes without bricking the fleet." Phases: 08 (firmware/RAS), 06 (drain), 11 (canary/rollback), 07 (gates), 01 (failure domains), 12 (EBOM).

1. Drive the requirements

  • Scope: which components (BIOS/BMC/NIC/accelerator/PSU/switch)? Each independently versioned.
  • Constraints: can't reboot nodes running tenant work without draining; can't brick the BMC (lose OOB); some components have A/B banks, some don't; air-gapped sites need local artifact mirrors.
  • SLOs: rollout duration, max nodes-down at once, zero fleet-wide bricks.

2. Architecture

 [Firmware repo + signing] --signed images--> [Update controller] -- desired baseline (EBOM, P12)
        |                                            |
        |                                   plans WAVES (canary 1 -> 1% -> 10% -> rest)
        |                                   failure-domain-aware (P01), dependency-ordered
        v                                            v
  [artifact mirror per site] <-- pull --   per-node: drain (P06) -> Redfish UpdateService/Task (P08)
                                            -> A/B stage+activate -> verify after -> commit/ROLLBACK
                                            -> health/SLO gate (P07) -> promote or HALT+rollback

3. The hard decisions (and the tradeoffs)

  • Per-node safe update (P08): verify (checksum+signature+version+anti-rollback) → flash inactive A/B bank → activate → verify after (version + health/PCIe — P04) → commit or roll back. Tradeoff: A/B needs hardware support; where absent, the risk and caution rise (stage to a recovery path).
  • Canary + progressive, failure-domain-aware (P11/P01): 1 node → 1% → 10% → rest, each gated on health/SLOs (P07), auto-rollback to the pinned prior bank on regression, never a whole PDU/ToR at once. Tradeoff: slower rollout vs. bounded blast radius. A bad image at 1% is an annoyance; at 100% an outage.
  • Drain-first (P06): cordon/drain tenant work before any reboot. Tradeoff: capacity dip during the window; respect PodDisruptionBudgets.
  • Dependency ordering (P08): e.g., BMC before BIOS if the new BIOS needs new BMC support; encode dependencies, don't flash blindly.
  • BMC update specially (P08): the riskiest target (brick it → lose OOB). Extra-conservative canary, recovery path validated, A/B mandatory.
  • Air-gap (P06/P09): local signed-artifact mirrors per site; no internet dependency mid-rollout.

4. Observability & control

  • Track firmware_version{component,version} across the fleet (P07) → rollout progress + drift are visible; version-skew alerts.
  • Each wave's gate reads health/SLOs and RAS (P08) — a spike in crashes/AER halts the rollout.
  • A kill-switch: halt + roll back the whole rollout on a fleet-level regression.

5. Failure modes & operations

  • Bad firmware → caught at the 1% canary; auto-rollback to the prior A/B bank; rollout halts; RCA (P10) → tighten the canary criteria + add the failure signature to pre-promotion checks + expand the HIL/conformance matrix (P11) + feed the defect to the ODM (P12).
  • Power loss mid-flash (P08) → A/B ensures a bootable bank survives; resume safely.
  • A node that won't come back → quarantine, OOB recovery (virtual media — P04), or RMA.
  • Partial fleet on a new version → backward-compatible interfaces (P11) so mixed versions interoperate during the (possibly long) rollout.

6. Honest limits

The conservative path makes large rollouts slow (days/weeks for the whole fleet) — that's the deliberate tradeoff for "zero fleet-wide bricks." I'd measure rollout duration vs. blast radius and tune wave sizes/bake times, and invest in the A/B + recovery path for components that lack it.