Warmup Guide — Provisioning & Configuration Management

Zero-to-expert primer for Phase 05. You need no prior ops background: this guide starts at "how does a diskless machine get an OS" and ends with you designing a resumable rack onboarding pipeline using PXE/Foreman + Ansible/Terraform, grounded in one idea: declare the desired state, then converge reality to it.

Table of Contents

• Chapter 1: The Provisioning Problem
• Chapter 2: The PXE/iPXE Boot Chain
• Chapter 3: Automated OS Install
• Chapter 4: Foreman and Lifecycle Management
• Chapter 5: Configuration Management & Idempotency
• Chapter 6: Ansible vs Chef vs Puppet
• Chapter 7: Terraform and Infrastructure-as-Code
• Chapter 8: The Rack Onboarding Pipeline
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: The Provisioning Problem

A new rack's nodes have no operating system — bare metal with empty (or wiped) disks. You cannot SSH to a machine with no OS. You cannot insert a USB stick into 500 servers in a data center hall in Riyadh from your desk. So provisioning must be remote, networked, and automated, and it builds on the OOB tools you already have (Phase 04: virtual media is one path; network boot is the scalable one).

The job: take N bare-metal nodes and, with zero physical touches, end with each running a known OS, configured to spec, validated, and joined to the orchestrator — repeatably (you do this for every rack, forever) and resumably (failures are normal at scale; node 237 of 500 will fail mid-install, and the pipeline must recover, not restart everything).

Two broad strategies (know the tradeoff):

• Image-based / immutable: lay down a complete golden disk image, identical everywhere. Fast, deterministic, easy rollback (reimage); but images are large and any change means a rebuild. Favored for fleets at scale and immutable-infra shops.
• Config-based: install a base OS, then converge it to desired state with a CM tool. Flexible, smaller artifacts, drift-correctable; but more moving parts and slower per node.

Real fleets blend them: a thin golden image + cloud-init/CM for the last mile.

Chapter 2: The PXE/iPXE Boot Chain

PXE (Preboot eXecution Environment) is firmware in the NIC that can boot a machine over the network. The chain, step by step (memorize it — it's the #1 interview question here):

1. Power on → NIC PXE firmware runs (you select "network boot" in BIOS/UEFI, or it's the default boot order — and the BMC can force it once, Phase 04).
2. DHCP: the NIC broadcasts a DHCP request. The DHCP server replies with an IP and two special options: option 66 / next-server (the TFTP/boot server address) and option 67 / filename (the boot file to fetch). UEFI vs BIOS clients send a client- arch tag so the server hands the right bootloader (this arch-detection is a common gotcha).
3. TFTP (or HTTP): the client downloads the bootloader named in option 67 — classic pxelinux.0 (BIOS) or a UEFI binary, or (better) iPXE.
4. iPXE (the modern upgrade): a richer bootloader that can boot over HTTP/HTTPS (faster, reliable, firewall-friendly vs TFTP), run scripts, and make decisions (chainloading). You often PXE-boot a tiny loader that then chainloads iPXE, which fetches an iPXE script over HTTP telling it what kernel/installer to load.
5. Kernel + initrd: the bootloader downloads the installer kernel and initial ramdisk and boots them.
6. Installer runs with an automated answer file (Chapter 3).

TFTP vs HTTP: TFTP is simple but slow and UDP-fragile; iPXE-over-HTTP is the modern default for speed and reliability at scale. UEFI HTTP Boot removes PXE/TFTP entirely on newer firmware — DHCP hands a URL, the firmware fetches over HTTP directly.

Secure Boot interaction (Phase 09): with Secure Boot on, the firmware will only run a signed bootloader (the Microsoft-signed shim → signed GRUB → signed kernel chain). Your PXE/iPXE artifacts must be signed, or provisioning fails at boot — a real gotcha when enabling Secure Boot on a previously-working PXE setup.

Chapter 3: Automated OS Install

The installer must run unattended — no human clicking "Next." Each OS family has an answer-file format:

• Kickstart (RHEL/Rocky/Alma): a ks.cfg specifying disk partitioning, packages, users, network, and %post scripts. The kernel cmdline points at it (inst.ks=http://.../ks.cfg).
• preseed (Debian) / autoinstall (Ubuntu Subiquity, YAML): the same idea, different syntax.
• cloud-init: not an installer but a first-boot configurator — when a node boots an OS image, cloud-init applies user-data (users, SSH keys, packages, run-commands, write files). Ubiquitous for cloud and increasingly bare metal (with the right datasource).

The pattern: the boot chain delivers kernel+initrd+answer-file; the installer partitions, installs the base OS, drops an SSH key / joins CM, reboots into the installed system. From there, configuration management (Chapter 5) takes over.

Idempotency starts here: a good answer file + post-script should be safe to re-run, and the provisioning orchestrator must track which node reached which step so a failure resumes rather than reinstalls the whole rack (Lab 01).

Chapter 4: Foreman and Lifecycle Management

Doing the above by hand-editing DHCP/TFTP configs doesn't scale. Foreman is the open- source tool that orchestrates it and tracks host lifecycle (the JD names it). What it gives you:

• Smart proxies: Foreman drives DHCP, TFTP, DNS, and the installer via proxy services near the hardware (so the control plane stays central, the services stay local — a recurring rack-software pattern).
• Host groups & provisioning templates: declare "this class of node gets this OS, this partitioning, this kickstart" once; apply to many.
• The discovery plugin: unprovisioned nodes PXE-boot a discovery image that inventories the hardware and registers it with Foreman before you install — the software form of Phase 01 commissioning (discover → reconcile against EBOM → provision).
• Katello (add-on): content/repo and lifecycle-environment management (dev/test/prod content views) — relevant to controlled OS/package rollouts.

Foreman is one option; hyperscalers often build bespoke equivalents (MaaS, Tinkerbell, or in-house). The concepts — discovery, templated provisioning, smart proxies, lifecycle states — transfer regardless of tool, which is what an interviewer probes.

Chapter 5: Configuration Management & Idempotency

Once the OS is installed, the node still needs packages, drivers (accelerator/NIC firmware hooks, Phase 08), kernel params, network config, users, services, and security hardening. Configuration management (CM) brings it to a declared desired state.

Idempotency is the central idea: you declare the end state ("package X present, service Y running, file Z with these contents"), not the steps, and applying the configuration any number of times converges to that state and is a no-op once converged. This is what makes CM safe to run continuously and what lets it correct drift (someone changed a config by hand → the next run puts it back). Lab 02 proves it: apply → no-op on re-apply → drift → re-converge.

Contrast with imperative scripts: a shell script that runs useradd bob fails the second time (bob exists); the idempotent CM resource "user bob present" succeeds every time. That difference is why CM exists.

Convergence + drift: the system continuously (or on-demand) compares declared vs actual (declared-vs-discovered again — Phase 01 Ch. 10) and remediates the difference. This is exactly the reconcile loop a Kubernetes operator runs (Phase 06) — CM is the same control pattern applied to OS config.

Chapter 6: Ansible vs Chef vs Puppet

The JD lists all three; know the model and the tradeoffs (a guaranteed question).

How to choose (the senior answer): Ansible when you want agentless simplicity, ad-hoc runs, and orchestration of multi-step flows (often the rack-provisioning default); Puppet when you want a strong declarative model enforced continuously by agents at large scale; Chef when the team wants programmatic flexibility. Many shops standardize on Ansible for its agentless model and pair it with image-based provisioning. The key insight to convey: they're all idempotent declarative convergence engines — the architecture (push vs pull, agent vs agentless) is the real differentiator, not the capability.

Chapter 7: Terraform and Infrastructure-as-Code

Terraform declares infrastructure as code: you write the desired set of resources (in HCL), and Terraform makes reality match. Core concepts:

• Providers: plugins for a platform (AWS, a bare-metal/Foreman provider, a Redfish provider that configures BMCs, a Kubernetes provider). Providers are the bridge to the real world.
• State: Terraform records what it created in a state file — the map between your declared resources and real-world IDs. State is the source of truth for what Terraform manages; it must be stored safely (remote backend, locked) and never hand-edited.
• plan then apply: plan shows the diff between declared and actual (create/change/ destroy) before changing anything — the discipline that prevents surprise destruction. apply executes it.
• Drift: if reality changes outside Terraform, the next plan shows the drift; you re-apply to reconcile (declared-vs-discovered, again).

For physical racks, Terraform's role is configuring the infrastructure layer — BMC settings via the Redfish provider, network config, Foreman host definitions, then the Kubernetes/RKE2 cluster (Phase 06). It's the same declarative-convergence idea as CM, at the infra level.

CM vs IaC, briefly: IaC (Terraform) provisions/declares infrastructure (machines, networks, BMC config); CM (Ansible/Puppet/Chef) configures inside the OS. They compose: Terraform stands up the node and its infra; Ansible configures the OS; the orchestrator (Phase 06) schedules workloads.

Chapter 8: The Rack Onboarding Pipeline

Synthesize everything into the pipeline you build (Lab 01) — the software the JD's "rack provisioning" really means:

[discover] → [inventory/EBOM reconcile] → [firmware baseline] → [OS install] →
[configure] → [validate] → [join orchestrator] → [READY]
   P05/P04        P01/P12                  P08          P05         P05/P02     P06

Design properties that separate a real pipeline from a script:

• State machine, persisted: each node has a state; transitions are recorded so the pipeline is resumable after a crash or a node failure (node 237 resumes at its last good state, not from zero). Lab 01 implements this.
• Idempotent steps: re-running a step is safe; partial progress doesn't corrupt.
• Validation gates: a node doesn't advance to "ready" until it passes checks (firmware at baseline, sensors nominal, network reachable, accelerators enumerated at full PCIe width — Phase 04). A bad node is quarantined for RCA (Phase 10), not shipped to tenants.
• Observable: every transition emits structured logs/metrics (Phase 07) so you can watch 500 nodes onboard and spot the stuck ones.
• Reconciled against intent: the discovered hardware is checked against the EBOM (Phase 01/12); discrepancies are findings, not silent acceptance.

This pipeline is the seed of the Phase 13 capstone, and "design a bare-metal provisioning service" is a core system-design prompt (see system-design/05).

Lab Walkthrough Guidance

Order: Lab 01 (provisioning state machine) → Lab 02 (config convergence). Provisioning gets the OS on; CM brings it to spec — the natural sequence.

Lab 01 (PXE/iPXE flow):

1. python3 solution.py — watch a node move discover→…→ready, then a run where a step fails and the pipeline resumes from the last good state.
2. Read the real iPXE script, the dnsmasq/DHCP options, and the kickstart file — the artifacts the state machine would drive.
3. Extension: do a real PXE install in QEMU with dnsmasq + iPXE (instructions in the README).

Lab 02 (Ansible + Terraform):

1. python3 solution.py — converge a node; re-run (a no-op — idempotency); introduce drift; re-converge (remediation).
2. Read the real Ansible playbook and Terraform example.
3. Extension: run the playbook against a container/VM with real ansible.

Phase capstone question: "Design the software to onboard a 500-node fleet from bare metal, where ~3% of nodes will fail somewhere mid-flow. What are the must-have properties?" (Answer: a persisted per-node state machine that's resumable + idempotent steps + validation gates that quarantine bad nodes + EBOM reconciliation + full observability — not a big sequential script.)

Success Criteria

You're done with this phase when — without notes:

• You can recite the PXE/iPXE boot chain and explain DHCP options 66/67 and arch detection (Ch. 2)
• You can explain TFTP vs HTTP boot and Secure Boot's effect on PXE (Ch. 2)
• You can describe automated install (kickstart/cloud-init) and where CM takes over (Ch. 3)
• You can explain what Foreman/discovery/smart-proxies give you (Ch. 4)
• You can define idempotency and demonstrate a no-op re-apply and drift remediation (Ch. 5)
• You can compare Ansible/Puppet/Chef by architecture and choose one with reasons (Ch. 6)
• You can explain Terraform state, plan/apply, and drift (Ch. 7)
• You can design a resumable, validated, observable rack onboarding pipeline (Ch. 8)

Interview Q&A

Q1: Walk me through what happens when a diskless server PXE-boots. A: On power-on the NIC's PXE firmware runs and broadcasts a DHCP request. The DHCP server replies with an IP plus option 66 (next-server, the boot server) and option 67 (filename, the boot file), using the client's arch tag to pick BIOS vs UEFI. The client fetches that bootloader over TFTP (or HTTP) — typically iPXE — which then fetches an iPXE script over HTTP that tells it which installer kernel and initrd to load. It boots the kernel with a cmdline pointing at an automated answer file (kickstart/autoinstall), the installer partitions and lays down the base OS unattended, drops an SSH key or joins config management, and reboots into the installed system, where CM converges it to desired state. With Secure Boot on, the bootloader chain (shim→GRUB→kernel) must be signed or it won't boot.

Q2: How do you install an OS on 500 servers without touching them, and what happens when some fail? A: Network provisioning driven by a state machine. Each node PXE/iPXE-boots and runs an automated installer (kickstart/cloud-init) orchestrated by something like Foreman (DHCP/TFTP/ templates) or an in-house equivalent. The critical design point is failure: at 500 nodes, some will fail mid-flow, so I don't run a big sequential script — I run a persisted per-node state machine with idempotent steps, so a node that fails at "configure" resumes at "configure," not at "reinstall the rack." Bad nodes hit a validation gate (firmware baseline, sensors, PCIe link width, network) and get quarantined for RCA rather than shipped to tenants, while the rest proceed. Everything emits structured telemetry so I can watch the fleet onboard and triage the stuck ones.

Q3: What is idempotency, and why is it the core of configuration management? A: Idempotency means applying the same configuration any number of times converges to the same end state and is a no-op once you're there. CM is built on it because you declare the desired state ("package present, service running, file with these contents"), not imperative steps, so the tool can safely run continuously, correct drift (someone hand-edits a config → the next run restores it), and be re-run after a partial failure without corruption. The classic contrast: a script doing useradd bob fails the second time, but the idempotent resource "user bob present" succeeds every time. It's the same control pattern as a Kubernetes reconcile loop — declared vs discovered, continuously converged.

Q4: Ansible vs Puppet vs Chef — when would you choose each? A: They're all idempotent declarative convergence engines; the real difference is architecture. Ansible is agentless and push-based over SSH with YAML playbooks — fastest to adopt, great for orchestration and ad-hoc runs, often the rack-provisioning default. Puppet is agent-based and pull-based with a strong declarative DSL and a master compiling catalogs — excellent for continuously enforced state at large scale. Chef is agent/pull with a Ruby DSL — flexible and programmer-friendly. I'd pick Ansible when I want no agents and simple orchestration (and pair it with image-based provisioning), Puppet when I want agents continuously enforcing declared state across a big fleet, and Chef when the team values programmatic flexibility. The capability is similar; push-vs-pull and agent-vs-agentless drive the decision.

Q5: What does Terraform's state file do, and what is drift? A: The state file is Terraform's record of what it created — the mapping between your declared HCL resources and real-world resource IDs. It's how Terraform knows what it manages and computes the diff on the next run. Drift is when reality changes outside Terraform (someone edits a BMC setting by hand); the next plan detects the difference between declared and actual, and apply reconciles it. The disciplines around state are: store it in a locked remote backend (never local for teams), never hand-edit it, and always plan before apply so you see create/change/destroy before it happens. It's declared-vs-discovered convergence at the infrastructure layer.

Q6 (Staff): Design the provisioning system for continuously onboarding new racks at fleet scale. A: I'd build a control plane around a persisted state machine per node with the pipeline discover → EBOM reconcile → firmware baseline → OS install → configure → validate → join orchestrator → ready. Properties: every step idempotent and resumable (crash-safe); smart proxies near the hardware for DHCP/TFTP/HTTP/installer while the control logic stays central (so it scales across sites — relevant to a globally distributed team); a discovery image that inventories hardware and reconciles against the EBOM agreed with the ODM, raising discrepancies as findings; firmware brought to a known baseline (Phase 08) before install; image-based install (a thin golden image + cloud-init) for determinism and easy reimage-as- rollback; CM (Ansible) for the last-mile config and continuous drift correction; validation gates that quarantine nodes failing health/PCIe/firmware checks; and end-to-end observability (Phase 07) so I can watch hundreds of nodes onboard and SLO the time-to-ready. I'd expose it as a declarative API (desired rack state) so it composes with Terraform/IaC and the orchestrator. The whole thing is a convergence loop — declared rack vs discovered rack — which is the same pattern as the Phase 06 operator and is what makes it testable and maintainable.

References

• iPXE — open network bootloader, scripting, HTTP boot — https://ipxe.org/
• PXE specification & DHCP options (66/67, client-arch) — RFC 2131/4578; SYSLINUX/pxelinux docs
• Kickstart (RHEL), Ubuntu autoinstall (Subiquity), cloud-init — https://cloudinit.readthedocs.io/
• Foreman & the Discovery plugin; Katello — https://theforeman.org/ , https://theforeman.org/plugins/foreman_discovery/
• Ansible docs (modules, roles, idempotency, Vault) — https://docs.ansible.com/
• Puppet & Chef overviews (declarative resources, pull model) — https://www.puppet.com/docs , https://docs.chef.io/
• Terraform — providers, state, plan/apply; the Redfish provider — https://developer.hashicorp.com/terraform
• Tinkerbell / Canonical MAAS — alternative bare-metal provisioning stacks — https://tinkerbell.org/ , https://maas.io/
• Cross-track: Phase 06 (Kubernetes/RKE2) — where provisioning hands off