🛸 Hitchhiker's Guide — Phase 08: Firmware, Crash Dumps & RAS
Read this if: "firmware updates and crash-dump parsing" sounds scary. It should — this is where you can brick a node or save a fleet. Compressed map of safe updates and reading hardware pain.
0. The 30-second mental model
Two jobs: update firmware without bricking (A/B banks + verify-before-and-after + rollback,
canaried across the fleet) and read RAS (PCIe AER, CPU MCE, SEL, accelerator codes) to a
FRU-level diagnosis so you replace one DIMM, not a whole node. The mindset: a firmware
rollout is a deployment with the blast radius of physical hardware — you can't rollout undo a
bricked BMC.
1. The firmware zoo
BIOS/UEFI · BMC · NIC/DPU · accelerator · PSU · PCIe switch · CPLD · drives. Each is independently versioned; the correct set = the baseline / EBOM (Phase 12). The BMC is the riskiest target — brick it and you lose your OOB lifeline.
2. The safe update state machine
download → VERIFY (checksum+signature+version+anti-rollback) [reject here, before touching HW]
→ stage to INACTIVE A/B bank [active firmware still running]
→ activate (switch pointer / reboot)
→ VERIFY AFTER (new version + healthy: sensors/links)
→ commit OR ROLLBACK to previous bank
Verify before and after. A/B means a bad flash never destroys the good firmware. Idempotent + resumable (Phase 05).
3. Redfish update = a Task
POST UpdateService (SimpleUpdate / multipart) → 202 + Task → poll TaskState/PercentComplete. Activation usually needs a reboot. Don't block on a synchronous response.
4. Signing & anti-rollback
Signed images (verify vs trusted key) tie into secure boot (Phase 09). Anti-rollback blocks downgrading past a security fix. Tension (trapped on bad new version) → resolved by A/B (roll back to the other good bank, not an old vulnerable one) + vendor recovery path.
5. Fleet rollout
drain-first (Phase 06) for reboots · dependency order (BMC before BIOS) · canary 1→1%→10%→
rest with auto halt-and-rollback on SLO/health regress (Phase 07) · don't update a whole
failure domain at once (Phase 01) · track firmware_version{component,version} as a metric.
6. RAS sources (reading hardware pain)
| source | what | key split |
|---|---|---|
| PCIe AER | fabric errors per BDF | correctable (rate=pre-failure) vs uncorrectable (fatal/non-fatal) |
| CPU MCE/MCA | memory/cache/interconnect | correctable ECC vs uncorrectable; bank→subsystem, addr→DIMM |
| ACPI/APEI | firmware→OS error report; BERT = last-crash cause | post-reboot diagnosis |
| SEL | BMC event log | PSU/thermal/DIMM events |
| accelerator (Xid) | GPU/accel faults | ECC / off-the-bus / thermal / app |
7. Decode to a FRU + an action
Parser job: decode each record → map to a FRU (which DIMM/device/PSU) → severity → predictive flag (rising correctable rate) → action. Decision tree:
single correctable / benign → log-only
rising correctable rate → monitor → proactive FRU replacement (avoid outage)
degrading-but-up → cordon + drain (Phase 06)
uncorrectable / persistent fault → RMA the FRU + re-validate (Phase 01) → return to service
8. Remote diagnosis toolkit (host is dead — all OOB)
SoL console · SEL + BERT (last-crash cause after reboot) · kdump vmcore (crash/drgn) ·
AER/MCE via the parser · Phase 07 telemetry correlation · Phase 01 failure-domain check.
9. The "done this before" tells
"A/B banks, right?" · "Did you verify after activation, not just before?" · "Is it canaried and drained first?" · "Correctable rate or a one-off?" · "Which FRU does that MCE bank point to?" · "Replace the DIMM, not the node."
10. How this phase pays off later
Safe updates → the Phase 06 cordon/drain flow + Phase 11 canary/rollback. RAS verdicts →
Phase 10 RCA and Phase 12 runbooks/RMA. Firmware baseline → Phase 01 commissioning +
Phase 12 EBOM. The fleet update system is system-design/03.