Interview Bank 03 — Provisioning, Orchestration & Observability
Phases 05–07. PXE/Foreman/Ansible/Terraform, Kubernetes/RKE2/operators, Prometheus/Grafana/tracing.
Q1. Walk me through what happens when a diskless server PXE-boots.
NIC PXE firmware → DHCP returns an IP + option 66 (boot server) + option 67 (boot file), using the client-arch tag for BIOS vs UEFI → fetch the bootloader (iPXE) over TFTP/HTTP → iPXE fetches an HTTP script → loads installer kernel+initrd → unattended install via kickstart/autoinstall → reboot into the OS where config management converges it. With Secure Boot on, the shim→GRUB→kernel chain must be signed. (P05 Ch. 2; Q1.)
Q2. How do you onboard 500 nodes where ~3% fail mid-flow?
A persisted per-node state machine, not a sequential script: discover → EBOM reconcile → firmware baseline → OS install → configure → validate → join orchestrator → ready, with every step idempotent and resumable (node 237 resumes at its last good state) and a validation gate that quarantines bad nodes (failed health/PCIe/firmware/attestation) instead of shipping them. Smart proxies near the hardware; full observability so I can watch the fleet onboard. (P05 Ch. 8; Q2.)
Q3. What is idempotency and why is it central to config management?
Applying the same configuration any number of times converges to the same end state and is a no-op once
there. CM declares the desired state (package present, service running) not steps, so it's safe to
run continuously, corrects drift, and survives partial failures. useradd bob breaks the second time;
"user bob present" always succeeds. It's the same control loop as a Kubernetes operator. (P05 Ch. 5;
Q3.)
Q4. Ansible vs Puppet vs Chef — when each?
All idempotent declarative convergence engines; architecture differs. Ansible: agentless, push, SSH, YAML — fast to adopt, great for orchestration/ad-hoc (often the provisioning default). Puppet: agent, pull, strong declarative DSL — continuous enforcement at scale. Chef: agent, pull, Ruby — programmatic flexibility. Choose by push-vs-pull / agent-vs-agentless, not capability. (P05 Ch. 6; Q4.)
Q5. Terraform state and drift?
State is Terraform's map of declared HCL ↔ real resource IDs — how it computes the diff. Drift = reality
changed outside Terraform; the next plan shows it and apply reconciles. Disciplines: locked remote
state (never local for teams), never hand-edit it, always plan before apply (see create/change/
destroy). Declared-vs-discovered at the infra layer. (P05 Ch. 7; Q5.)
Q6. What happens when you kubectl apply a Deployment?
API server validates + writes to etcd → Deployment controller makes a ReplicaSet → ReplicaSet controller makes Pods → scheduler binds each to a node (filter by capacity/taints/affinity, then score) → that node's kubelet pulls the image and starts the container → controllers keep reconciling. Nothing imperative — every component watches state and converges it. (P06 Ch. 2; Q1.)
Q7. Explain the operator/reconcile loop. Why level-triggered?
A CRD (e.g., RackNode) + a controller running observe → diff(spec vs status/world) → act idempotently
→ update status → requeue. Level-triggered = reacts to current state, not the change event, so a
missed/duplicated event isn't fatal — the next reconcile sees reality and fixes it (self-healing).
Edge-triggered breaks on a missed event. Plus finalizers for delete-time cleanup. (P06 Ch. 4; Q2.)
Q8. How does a GPU/accelerator get scheduled to a pod, and why is a pod Pending?
A device-plugin DaemonSet advertises accelerators to the kubelet as an extended resource (e.g.,
qualcomm.com/ai-accelerator: 4); pods request it; the scheduler treats it as capacity; the kubelet
allocates the device. Pending = the filter phase found no node: insufficient resources anywhere, an
unmatched nodeSelector/affinity, or an untolerated taint. (P06 Ch. 5–6; Q4.)
Q9. What is RKE2 and why use it over upstream k8s?
A hardened, CIS-benchmarked, FIPS-capable Kubernetes distro with a simple single-binary, embedded-etcd, air-gappable install — exactly what sovereign/regulated data centers (e.g., in the Kingdom) need. It's still Kubernetes, so operators/DaemonSets run unchanged. (P06 Ch. 7; Q5.)
Q10. Design a telemetry pipeline for a rack fleet with Redfish/IPMI/SNMP devices.
Unified exporters near the hardware (Phase 03 driver abstraction, poll-on-own-schedule + cache, consistent metric/label scheme) → Prometheus scrapes them (federate/remote-write to Mimir/Thanos at scale) → Grafana fleet→rack→node dashboards (USE/RED) → Alertmanager on SLO symptoms with runbook links; logs to Loki, traces via OTel, correlated by IDs. Two disciplines: strict cardinality control and alert on symptoms, not causes. (P07 Ch. 4–9; Q1.)
Q11. Counter vs gauge vs histogram, and the cardinality trap?
Counter (only up → query rate()): PCIe AER errors. Gauge (up/down): temperature, power, CDU ΔT.
Histogram (distribution → p99): provisioning step duration. Cardinality = product of label values;
never put unbounded values (IDs/timestamps/messages) in labels → OOM. Labels = bounded dimensions
(rack/node/sensor); detail goes in logs/traces. (P07 Ch. 3, 6; Q3–4.)
Q12. What would you alert on for an AI rack, and how avoid alert fatigue?
Symptoms that matter, with for: + severity + a runbook: node/exporter down, rack over power budget,
sustained accelerator overheat/throttle, CDU leak/ΔT anomaly, SLO breaches. Avoid fatigue by not
paging on causes — a single correctable PCIe error is a dashboard, not a page; alert on a rate or
sustained condition. Every page must be actionable. (P07 Ch. 9; Q5.)
Quick-fire drills
- PXE: DHCP opt 66/67 → iPXE over HTTP → kickstart/autoinstall.
- Idempotent + resumable + validation-gated = a real provisioner.
- Reconcile loop = level-triggered, idempotent, requeue, finalizers.
- Pod Pending = capacity / taint / affinity (the filter phase).
- RKE2 = hardened + air-gappable k8s for sovereign deploys.
- Pull metrics; counter→rate, gauge→value, histogram→p99; bound cardinality.
- Alert on symptoms (SLOs), not causes; runbook on every page.
- metrics → traces → logs (correlate by ID) is the debug loop.