Phase 05 — Provisioning & Configuration Management
Difficulty: ⭐⭐⭐⭐☆ | Estimated Time: 2 weeks Roles supported: Rack Management SWE (Senior/Staff), Infra/Platform SWE, SRE Hardware needed: none for the core labs (simulators + real config files); a VM/QEMU for the optional PXE extension
Why This Phase Exists
A rack arrives from the ODM as bare metal. Getting it from "powered on" to "running tenant workloads" is provisioning, and the JD asks for it directly: "Implement, test, and maintain software solutions for rack provisioning" and "Integrate rack management software with orchestration platforms (e.g., Kubernetes, Terraform, Foreman)" plus "Familiarity with Foreman, Ansible, Chef, and Puppet for provisioning and configuration."
This phase covers the two halves of turning hardware into a usable node:
- Bare-metal provisioning — the PXE/iPXE boot chain that installs an OS onto a diskless machine over the network (DHCP → TFTP/HTTP → bootloader → kernel → automated installer), orchestrated by tools like Foreman.
- Configuration management — bringing a freshly-installed node to a declared desired state (packages, drivers, network, services) idempotently, with Ansible/Chef/ Puppet, plus Terraform for infrastructure-as-code.
The unifying idea is declared vs discovered state and convergence (Phase 01 Ch. 10): you declare what a rack should be; the system makes reality match, repeatably. You'll build a provisioning state machine and a config-convergence engine, and write the real iPXE/ kickstart/Ansible/Terraform artifacts.
Concepts
Bare-metal provisioning
- The boot chain: PXE vs iPXE; DHCP options (
next-server/66,filename/67, UEFI vs BIOS, arch detection); TFTP vs HTTP boot; the bootloader (pxelinux/iPXE/GRUB); kernel + initrd - Automated install: kickstart (RHEL), preseed/autoinstall (Debian/Ubuntu), cloud-init (first-boot config); unattended OS install
- UEFI HTTP boot and the interaction with Secure Boot (Phase 09) — signed shim/bootloader
- The discovery/inventory step (a "discovery image" that reports hardware before install)
- Image-based (golden image / immutable) vs config-based provisioning; tradeoffs
Foreman (the provisioning lifecycle manager)
- Host groups, provisioning templates, the discovery plugin, smart proxies (DHCP/TFTP/DNS), Katello for content
- Where Foreman sits: it orchestrates DHCP/TFTP/installer and tracks host lifecycle
Configuration management
- Declarative desired-state & idempotency (the central concept — apply twice = same result)
- Ansible: agentless, push, SSH, YAML playbooks, modules, inventory, roles; idempotent modules
- Chef (agent, pull, Ruby DSL, resources/recipes/cookbooks) and Puppet (agent, pull, declarative manifests, catalog compilation) — the model and when each fits
- Convergence, drift detection, and remediation
- Secrets in CM (Ansible Vault, etc.); ordering vs declarative dependencies
Terraform / Infrastructure-as-Code
- Providers, resources, state,
plan/apply, drift, modules; the declarative graph - Bare-metal/Foreman/Redfish providers; what IaC means for physical racks
- Idempotency and the plan-before-apply discipline
The rack onboarding flow (the synthesis)
- discover → inventory/EBOM reconcile → firmware baseline (Phase 08) → OS install → configure → validate → join orchestrator (Phase 06) → ready
- Idempotent, resumable, observable; the state machine and its failure handling
Labs
Lab 01 — PXE/iPXE Bare-Metal Provisioning Flow
| Field | Value |
|---|---|
| Goal | Build a provisioning state machine that takes a node from discovered → ready (DHCP/boot → installer → configure → validate), plus the real iPXE/DHCP/kickstart artifacts it would drive. |
| Concepts | The PXE/iPXE boot chain, DHCP options, automated install, the provisioning state machine, idempotency/resumability, validation gates. |
| Steps | 1) python3 solution.py — runs the state machine, including a failure+resume. 2) Read the boot-chain steps and the kickstart/iPXE example files. 3) Inject a failure and watch it resume from the last good state. 4) Extensions below (real QEMU PXE). |
| Stack | Python 3 stdlib (state machine) + real ipxe/kickstart/dnsmasq config examples |
| Output | A resumable provisioning state machine + ready-to-use boot artifacts. |
| How to Test | Asserts verify state transitions, that a failure resumes from the right state, and that validation gates block "ready" on bad config. |
| Talking Points | What DHCP option 67 does; BIOS vs UEFI PXE; why provisioning must be resumable/idempotent; the discovery step; secure boot's effect on PXE. |
| Resume Bullet | "Built a resumable bare-metal provisioning state machine (discover→install→configure→validate→ready) with iPXE/kickstart automation and validation gates." |
| Extensions | Run a real PXE install in QEMU with dnsmasq + an iPXE script; add UEFI HTTP boot; reconcile discovered inventory against the EBOM (Phase 01). |
Lab 02 — Ansible + Terraform Rack Onboarding
| Field | Value |
|---|---|
| Goal | Build a config-convergence engine that proves idempotency (apply twice = no-op, drift gets corrected), plus a real Ansible playbook and Terraform/Redfish example for node config. |
| Concepts | Declarative desired state, idempotency, convergence, drift detection/remediation, Ansible modules/roles, Terraform plan/apply/state. |
| Steps | 1) python3 solution.py — converge a node, re-run (no-op), introduce drift, re-converge. 2) Read the Ansible playbook + Terraform example. 3) Extensions below (run real Ansible). |
| Stack | Python 3 stdlib (convergence engine) + real ansible playbook + terraform HCL examples |
| Output | A convergence engine demonstrating idempotency + drift remediation, with real CM/IaC artifacts. |
| How to Test | Asserts verify idempotency (second apply changes nothing), drift detection, and remediation back to desired state. |
| Talking Points | Why idempotency is the core CM idea; Ansible (push/agentless) vs Chef/Puppet (pull/agent); Terraform state & drift; declared-vs-discovered as the same idea as a k8s reconcile loop. |
| Resume Bullet | "Built a configuration-convergence engine demonstrating idempotency and drift remediation; authored Ansible roles and Terraform modules for rack node onboarding." |
| Extensions | Run the playbook against a container/VM with real ansible; add Ansible Vault for secrets; write a Terraform redfish provider config; detect drift on a schedule. |
Deliverables Checklist
- You can draw the PXE/iPXE boot chain end to end (DHCP → TFTP/HTTP → bootloader → kernel → installer)
- You can explain DHCP options 66/67 and BIOS vs UEFI PXE
- Provisioning state machine is resumable: a mid-flow failure resumes from the last good state
- You can explain idempotency and show a convergence that's a no-op on the second run
- Convergence engine detects and remediates drift
- You can compare Ansible vs Chef vs Puppet (push/pull, agent/agentless, language) and pick one
- You can explain Terraform state, plan/apply, and drift
- You can connect declared-vs-discovered here to the Phase 06 reconcile loop
Interview Relevance
- "Walk me through what happens when a diskless server PXE-boots."
- "How do you install an OS on 500 servers without touching them?"
- "What is idempotency and why does it matter in configuration management?"
- "Ansible vs Puppet vs Chef — when would you choose each?"
- "What does Terraform's state file do, and what is drift?"
- "Your provisioning fails halfway on node 237 of 500. What happens?" (resumable/idempotent)
- "How do you onboard a brand-new rack from bare metal to running workloads?"