Lab 02 — Crash-Dump & RAS Log Parser

Phase: 08 — Firmware & RAS | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–6 hours Language: Python 3 (stdlib) | Hardware: none

Concept primer: ../WARMUP.md Ch. 6–8, ../HITCHHIKERS-GUIDE.md §6–§8.

Run

python3 solution.py     # parse mixed AER/MCE/SEL/Xid logs -> FRU-level diagnoses + actions

0. The mission

Turn the hardware's cryptic error telemetry into a decision: parse PCIe AER, CPU MCE, BMC SEL, and accelerator (Xid-style) records, decode each to a meaning, map it to a FRU, assign severity, detect predictive patterns (rising correctable rate), and recommend an action — log-only / monitor / cordon / RMA the FRU. This is "crash dump parsing / remote diagnostics" from the JD.

1. The decoders to study

  • AER: correctable = noise individually (log-only); uncorrectable fatal (MalformedTLP/ LinkError) → RMA, non-fatal → cordon. BDF identifies the device.
  • MCE: the bank maps to a subsystem; bank 7 (memory controller) + address → a specific DIMM. Correctable ECC = log; uncorrectable = RMA.
  • SEL/Xid: PSU input-lost → monitor; a fatal Xid (GPU off the bus) → RMA.

2. The key insight: rate, not count

predictive_escalation is the senior move: one correctable error is noise, but 60 corrected ECC errors on DIMM B1 is a degrading part → escalate to a scheduled replacement (RMA) before it throws an uncorrectable (a crash). Same lesson as Phase 07 alerting (alert on rates), applied to hardware — and it's how you turn unplanned outages into planned maintenance.

3. The output

A per-record diagnosis list (critical/warning surfaced), rate-based predictive escalations, and a summary verdict (how many FRUs to RMA / monitor / log). That's the difference between "RMA the whole node" and "replace DIMM B1 and the accelerator at the next window."

4. Extensions

  1. Parse real dmesg (AER lines), mcelog/ras-mc-ctl --error-count, and ipmitool sel output.
  2. Correlate with Phase 07 telemetry (did the DIMM errors rise with temperature? → thermal root cause).
  3. Feed verdicts into the Phase 10 RCA workflow and auto-open an RMA ticket (Phase 12).
  4. Add BERT (last-crash cause after reboot) parsing for diagnosing random reboots.

5. What this lab proves

You can do remote hardware diagnosis — the JD's most-repeated theme — turning a flood of RAS records into a precise FRU-level verdict and the right action. "A node crashes randomly in production — find the root cause remotely" becomes a methodical walk (SEL/BERT → AER/MCE decode → predictive flag → RMA the DIMM), no site visit required. Feeds Phase 10 and the capstone.