Runbooks — Rack Management Platform

Complete, filled on-call runbooks (no placeholders), executable by anyone at 3 a.m. Format per ../RUNBOOKS-AND-OPS.md. Each is linked from its Phase 07 alert's runbook annotation.

Runbook A: Fleet firmware rollout (BMC obmc-1.7.2 → 1.8.0)

  • Trigger: a planned rollout ticket, or firmware_version{component="bmc"} drift alert.
  • Goal: every node at the EBOM BMC baseline obmc-1.8.0, zero fleet-wide bricks.
  • Owner / escalation: Rack SW on-call → Platform lead → Pegatron FW contact (#pegatron-fw).
  • Risk: BMC is the riskiest target (brick → lose OOB); A/B + recovery path required (ADR-003).

Procedure

  1. Pre-checks: confirm artifact signed by Platform key and version ≥ anti-rollback floor (obmc-1.7.2); confirm A/B supported on target SKUs; confirm site artifact mirror is populated (air-gap). rackctl fw verify --image obmc-1.8.0.bin.
  2. Canary (1 node): pick a node in a non-critical failure domain, drain it (kubectl drain <node> --ignore-daemonsets), update via the orchestrator (rackctl fw update --node <n> --image obmc-1.8.0), watch the Task to Completed, verify post-activation health (sensors nominal, PCIe at full width, attestation re-passes), uncordon. Bake 30 min.
  3. Gate: check the canary's health/SLO panel — zero new crashes/AER, BMC reachable, attest OK. If regressed → STOP, rollback (step R), open an incident.
  4. Progressive waves: 1% → 10% → 50% → 100%, each wave failure-domain-aware (never a whole PDU/ToR at once), drain-first, gated as in step 3. Auto-halt + rollback on any wave regression.
  5. Verify fleet: firmware_version{component="bmc"} shows 100% at 1.8.0; no version-skew alerts.

(R) Rollback: rackctl fw rollback --node <n> switches the BMC to the previous A/B bank (the prior 1.7.x) and reboots; confirm version + health. Halt the rollout; RCA (Phase 10).

Post: update node history; if a new failure mode appeared, write the RCA + add a pre-promotion check (Phase 11) + feed the defect to Pegatron (EBOM finding).

Runbook B: Node fails to POST after provisioning

  • Trigger: provisioning stuck pre-OS, or NodeNotReady with the host unreachable in-band.
  • Goal: node POSTs and resumes provisioning, or is quarantined with a FRU verdict.
  • Owner / escalation: Rack SW on-call → HW bring-up → Pegatron.
  • Risk: single node; power actions are physical (dry-run/confirm).

Procedure

  1. Scope (Phase 01): rackctl status --rack <rack> — are peers on the same PDU/ToR/CDU also down? If many → shared-resource incident (different runbook); this is for a single node.
  2. Reach the BMC: redfishtool -r <bmc> Systems get → confirm PowerState. BMC unreachable → mgmt-network/BMC issue → escalate.
  3. Why it didn't POST: read SEL + BERT for the last-boot error (ipmitool -I lanplus -H <bmc> -U <u> -P <p> sel list); watch live: ipmitool ... sol activate.
  4. Physical signals (Phase 04): inlet/accel temps, PSU health, PCIe link width; CDU ΔT/leak — if leak, STOP → CDU-leak safety runbook.
  5. Reset once, watched: redfishtool -r <bmc> Systems reset ForceRestart; watch SoL for the POST code.
  6. Decide (Phase 08 tree):
    • POSTs → resume provisioning; record the transient.
    • RAS → a FRU (DIMM/accel/PSU) → cordon, schedule RMA, re-validate after swap.
    • Firmware ≠ EBOM → re-flash to baseline (Runbook A path), retry.
  7. Quarantine if unresolved: mark QUARANTINED (Phase 05 gate), file a ticket with SEL/BERT/RAS evidence; do not hand to tenants.

Post: update history; new failure mode → RCA (Phase 10) + an earlier-catching alert/test.

Runbook C: Coolant leak detected (safety-critical)

  • Trigger: CDU leak=1 event (Phase 04/07) — highest priority.
  • Goal: protect hardware and people; minimize damage.
  • Owner / escalation: Rack SW on-call → Facilities/DC ops immediately (parallel, not serial).
  • Risk: liquid near live electronics — hardware loss + hazard.

Procedure (fail-safe, act first)

  1. Confirm + page facilities in parallel; do not wait for confirmation to start step 2.
  2. Reduce heat load: cap power on the affected rack's nodes (rackctl power cap --rack <r> --watts <low>) and/or drain + graceful-shutdown non-critical nodes (Phase 06) to cut the thermal source.
  3. Per policy/interlock: trigger the rack's emergency coolant action (drain/valve) if the platform owns it; otherwise hand to facilities.
  4. Do NOT power-on anything in the affected rack until facilities clears it (interlock).
  5. Capture evidence: CDU telemetry (flow/pressure/ΔT trend), timestamps, affected components.

Post: incident + blameless RCA; verify the leak interlock and the alert path worked; restore service only after facilities sign-off + re-validation (Phase 01 commissioning).