Runbooks & Operational Procedures

Deliverable 3 of Phase 12. A worked runbook, a runbook template, and the manual→self- healing progression (WARMUP Ch. 5). A runbook must be executable by someone other than the author, at 3 a.m., safely.

Part A — Worked runbook (example)

Runbook: Node fails to POST after provisioning

Trigger: provisioning state machine (Phase 05) reports a node stuck pre-OS, or alert NodeNotReady fires (Phase 07); the host is unreachable in-band.
Goal: get the node to POST and continue provisioning, or quarantine it with a clear FRU verdict.
Owner / escalation: Rack SW on-call → HW bring-up → ODM (Pegatron) contact [link].
Risk / blast radius: single node; power actions are physical — use dry-run/confirm (Phase 04).

Steps (OOB — the host is down, so everything is via the BMC):

Confirm scope (Phase 01): are peers on the same PDU/ToR/CDU also down? If many → escalate as a shared-resource incident; this runbook is for a single node. rackctl status --rack <rack> (Phase 02 CLI).
Reach the BMC (Phase 03): redfishtool -r <bmc> Systems get → confirm PowerState. If the BMC itself is unreachable → mgmt-network/BMC issue → escalate (different runbook).
Read why it didn't POST: SEL + BERT for last-boot error (Phase 08); watch live via SoL: ipmitool -I lanplus -H <bmc> ... sol activate.
Check the obvious physical signals (Phase 04): inlet/accel temps, PSU health, PCIe — a thermal trip or a PSU fault blocks POST. CDU ΔT/leak? (If leak → STOP, safety runbook.)
Reset once, watched: redfishtool ... ComputerSystem.Reset ForceRestart and watch SoL for the POST code / error. (Dry-run/confirm guard — Phase 04.)
Decide (Phase 08 decision tree):
- POSTs now → resume provisioning (Phase 05); record the transient in the node history.
- RAS points at a FRU (DIMM/accelerator/PSU) → cordon (Phase 06), schedule RMA (Phase 08), re-validate after replacement (Phase 01).
- Firmware mismatch vs EBOM (Phase 12) → re-flash to baseline (Phase 08 orchestrator), retry.
Quarantine if unresolved: mark the node QUARANTINED (Phase 05 gate), file a ticket with the SEL/BERT/RAS evidence, and do not hand it to tenants.
Close: update the node history; if this was a new failure mode, write/extend the RCA (Phase 10) and add an alert/test (Phase 11) so it's caught earlier next time.

Rollback / safety: every power action is reversible (power the node back on); never proceed past a CDU leak signal; if uncertain, quarantine rather than risk tenant exposure.

Part B — Runbook template (copy per procedure)

# Runbook: <task or incident>
- Trigger: <the exact alert/symptom that brings someone here>  (Phase 07)
- Goal: <the outcome that means "done">
- Owner / escalation: <on-call → next tier → partner contact + link>
- Risk / blast radius: <scope; destructive steps; pre-checks>  (Phase 01/04)
Steps:
  1. <pre-check / confirm scope>
  2. <command> → <expected output> ; if <X> then <Y>
  ...
Rollback / safety: <how to undo; when to STOP and escalate>
Links: dashboard <...>, ADR <...>, related runbooks <...>
Post: <update history; if new, write RCA (Phase 10) + add a test/alert (Phase 11)>

Part C — The maturity progression (Staff drives this)

manual heroics  →  RUNBOOK (repeatable by anyone)  →  AUTOMATED (a script/tool runs it)
                                                    →  SELF-HEALING (the operator reconciles it, Phase 06)

Every RCA (Phase 10) with a manual fix → leave a runbook.
Every repeated runbook → automate it (a rackctl subcommand, Phase 02).
Every automatable recovery → fold into the operator's reconcile loop (Phase 06) so it self-heals.

This progression is how a distributed, follow-the-sun on-call scales without burning people out — and it's a concrete way a Staff engineer multiplies the team (WARMUP Ch. 7).

How to use this deliverable

Write one real runbook (the "node won't POST" above, or "fleet firmware rollout" — Phase 08/11) using Part B.
Link it from the relevant Phase 07 alert's runbook annotation.
Identify one runbook to automate and one to fold into the operator (Part C).