Lab 01 — PXE/iPXE Bare-Metal Provisioning Flow

Phase: 05 — Provisioning & Config Mgmt | Difficulty: ⭐⭐⭐⭐☆ | Time: 5–7 hours Language: Python 3 (state machine) + real iPXE/dnsmasq/kickstart | Hardware: none (QEMU for the optional real install)

Concept primer: ../WARMUP.md Ch. 2–3, Ch. 8, ../HITCHHIKERS-GUIDE.md §1–§3, §7.

Run

python3 solution.py        # the provisioning state machine: clean run, resume, quarantine

Files

FileWhat it is
solution.pythe orchestrator — a resumable per-node provisioning state machine
boot.ipxethe real iPXE script served over HTTP (kernel/initrd/autoinstall)
dnsmasq.confa real DHCP+TFTP config (options 66/67, BIOS/UEFI arch detection, iPXE chainload)
ks.cfga real kickstart for unattended OS install + handoff to config management

0. The mission

Build the software the JD's "rack provisioning" means — not a script, a resumable state machine that takes bare metal → ready, with a validation gate that quarantines bad nodes — and pair it with the real boot artifacts (boot.ipxe, dnsmasq.conf, ks.cfg) it drives. The boot chain and the orchestrator are the two halves of provisioning.

1. What the state machine demonstrates

  • Resumability: Node B hits an injected failure at OS_INSTALLED and resumes from FIRMWARE_BASELINED — it never re-runs the earlier steps (the self-test asserts INVENTORIED happened exactly once). At 500 nodes, this is the difference between a 3% failure rate being a non-event and being a re-provision-the-whole-rack disaster.
  • Idempotency: re-provisioning an already-READY node is a no-op.
  • Validation gate: Node C's discovered inventory (3 accelerators) mismatches the EBOM (4) → it's QUARANTINED with a reason, not shipped to tenants. This is Phase 01 commissioning enforced in code.

2. The boot artifacts (read them)

dnsmasq.conf shows the two-stage PXE trick (PXE → iPXE binary → HTTP iPXE script) and arch detection; boot.ipxe shows HTTP boot with a serial console so the BMC's SoL can watch (Phase 04); ks.cfg shows unattended install ending with an SSH-key drop and a callback to the orchestrator ("os-installed") — the seam between the boot chain and the state machine.

3. Extensions (do them)

  1. A real PXE install in QEMU. Install dnsmasq, grab ipxe.efi/undionly.kpxe, serve the artifacts over a tiny HTTP server, and boot a diskless QEMU VM (-boot n, -netdev). Watch it DHCP → chainload iPXE → install Ubuntu unattended.
  2. UEFI HTTP Boot: skip TFTP entirely — DHCP hands a URL, the firmware fetches over HTTP.
  3. EBOM reconciliation: wire the INVENTORIED step to the Phase 01 inventory model and produce a real declared-vs-discovered diff.
  4. Persist state to disk/DB: make the state machine truly crash-safe (process dies → restart → resumes), the way a production provisioner stores per-node state.
  5. Secure Boot: make the boot chain use signed shim→GRUB→kernel and discuss what breaks.

4. What this lab proves

You can design the provisioning control plane, not just paste a kickstart. "Onboard 500 nodes where 3% fail mid-flow" becomes a confident answer — resumable state machine, idempotent steps, validation gates, EBOM reconciliation — and you can show the real iPXE/DHCP/kickstart that the control plane drives. This is the seed of system-design/05 and the Phase 13 capstone.