System Design 05 — Bare-Metal Provisioning Service

"Design a service to onboard racks from bare metal to running tenant workloads." Phases: 05 (provisioning/CM), 01 (commissioning/model), 12 (EBOM), 06 (orchestrator handoff), 09 (attestation).

1. Drive the requirements

  • Scale & cadence: how many racks, how often? (Continuous onboarding — you commission racks forever, not once.)
  • Inputs: a rack arrives from the ODM (Pegatron) as bare metal; you have the EBOM (P12).
  • Outputs: every node provisioned to baseline, validated, attested, joined to the orchestrator.
  • Constraints: ~3% of nodes fail somewhere mid-flow; multi-site; some air-gapped.
  • SLOs: time-to-ready per rack; % nodes auto-onboarded without human touch.

2. Architecture

 [Provisioning API: desired rack state] --> [Per-node STATE MACHINE controller (P05), persisted]
        |                                          | discover -> EBOM reconcile (P12) -> firmware
        |                                          |   baseline (P08) -> attest (P09) -> OS install
        |                                          |   -> configure (CM, P05) -> validate -> join (P06)
        v                                          v
 [Smart proxies per site]: DHCP/TFTP/HTTP(iPXE) + installer + Redfish/IPMI to BMCs (P03/P04)
        |  (control central, device traffic local — scales multi-site)
        v
   [racks: BMCs power/PXE/virtual-media; nodes install OS]

3. The hard decisions (and the tradeoffs)

  • Persisted per-node state machine, not a script (P05): every step idempotent + resumable, so a node failing at "configure" resumes there, not at "reinstall the rack." Tradeoff: more to build vs. a 3%-failure-rate being a non-event at scale.
  • Validation gates that quarantine (P05/P01): a node advances to ready only after passing checks (EBOM reconciliation, firmware baseline, PCIe link width, sensors, attestation); failures are quarantined for RCA (P10), not shipped to tenants. Tradeoff: stricter gates → more quarantines, but no bad nodes reach tenants.
  • Discovery + EBOM reconciliation (P01/P12): a discovery image inventories hardware and reconciles against the EBOM agreed with the ODM; mismatches are findings raised to the partner. Tradeoff: upfront EBOM discipline vs. "works in the lab, not in Riyadh."
  • Image-based + cloud-init vs. full CM: a thin golden image + cloud-init for determinism and easy reimage-as-rollback, then CM (Ansible) for the last mile + drift correction. Tradeoff: image size/ rebuilds vs. per-node config flexibility — blend them.
  • Smart proxies (P05): DHCP/TFTP/HTTP/installer near the hardware, control logic central. Tradeoff: more components vs. multi-site scale and WAN resilience.
  • Boot security (P09): signed PXE/iPXE chain (Secure Boot) + attestation gate before tenant trust. Tradeoff: more setup vs. provable trust (required for sovereign sites).

4. Observability & control

  • Every state transition emits structured telemetry (P07) → watch hundreds of nodes onboard, SLO the time-to-ready, and triage the stuck/quarantined ones.
  • Declarative API (desired rack state) so it composes with Terraform/IaC (P05) and the orchestrator.

5. Failure modes & operations

  • Node fails mid-flow → resumes from its last good state; no rack-wide restart.
  • EBOM mismatch (missing accelerator / firmware drift) → quarantine + a finding to the ODM (P12).
  • Won't POST → the Phase 12 runbook (OOB: SEL/BERT, RAS, reset, RMA if a FRU fault).
  • Air-gapped site → local mirrors of images/artifacts; no internet dependency.
  • Continuous onboarding → it's a pipeline, run forever, observable and resumable — the Staff value is making it repeatable and automated, not a one-off heroics.

6. Honest limits / what I'd measure

Time-to-ready and the auto-onboard rate (% with no human touch) are the headline SLOs; the quarantine rate and its top causes tell me where to harden (EBOM gaps with the ODM, a flaky firmware step). I'd start strict (quarantine aggressively) and loosen gates only as confidence and conformance (P11) grow.