Warmup Guide — Capstone: Composing a Rack Management Platform

The capstone is about composition: the prior phases each gave you a part; here you assemble them into one coherent, testable, operable system — and learn what changes when one rack becomes ten thousand. This guide frames the architecture, the seams, and the integration pitfalls.

Table of Contents

Chapter 1: From Parts to a Platform

You've built: an inventory model (Phase 01), production code patterns (Phase 02), protocol clients behind a driver abstraction (Phase 03), hardware interfacing (Phase 04), a provisioning state machine (Phase 05), a reconcile loop (Phase 06), a telemetry exporter (Phase 07), firmware orchestration (Phase 08), tenancy/secure config (Phase 09), reliability patterns (Phase 10), and a test/release pipeline (Phase 11). The capstone is the integration test of you: can you compose these into a system where the parts reinforce rather than fight each other?

The recurring idea that makes composition work — and that this whole curriculum has been teaching — is declared vs discovered, converged through one control loop, over one data model, behind stable seams. Get those three right (one model, one reconcile loop, clean seams) and the parts snap together; get them wrong and you have a pile of scripts.

Chapter 2: The Architecture and Its Seams

The platform is a control plane over a driver abstraction:

  • One data model (Phase 01): every component, relationship, state, and the declared baseline (EBOM) live in one inventory graph. Everything reads/writes it; it's the spine.
  • One control loop (Phase 06): a reconcile loop drives each node from its current state toward desired (provisioned, at firmware baseline, healthy, correctly configured), idempotently and resumably. Provisioning (Phase 05) and firmware (Phase 08) are actions the loop takes, not separate orchestrators.
  • One device abstraction (Phase 03): DeviceDriver (Redfish-first, IPMI/SNMP fallback) hides protocol/vendor variance; a FakeDriver makes the whole platform testable without hardware (Phase 02/11). This is the most important seam.
  • Cross-cutting: telemetry (Phase 07) observes the model and the drivers; tenancy/security (Phase 09) constrains configuration; the CLI (Phase 02) is the operator interface; reliability patterns (Phase 10) wrap every external call.

The seams (interfaces) are where you spend design care: the driver boundary (testability + portability), the model (the contract every part shares), and the reconcile API (declared state in, convergence out). Good seams are why you can swap a real BMC for a fake, add a vendor, or test a failure path without rewiring the system.

Chapter 3: The Lifecycle, End to End

The platform expresses the rack lifecycle (Phase 01 Ch. 9) as states the reconcile loop drives:

DISCOVERED → INVENTORIED(+EBOM reconcile, P12) → FIRMWARE_BASELINED(P08) → ATTESTED(P09)
   → OS_INSTALLED(P05) → CONFIGURED(tenancy/secure, P09) → VALIDATED(gate) → READY
   → (operate: telemetry P07, drift-correct P06) → (maintain: firmware/RMA P08/P10) → DECOMMISSION

Key properties carried through from earlier phases: idempotent + resumable (a crash mid-flow resumes), validation-gated (bad nodes quarantined, not shipped), observable (every transition emits telemetry), and reconciled against intent (discovered vs the EBOM). The capstone makes these concrete in one program.

Chapter 4: Failure Handling Across the System

A platform is judged by how it fails, not how it runs on the happy path. The integration must handle (drawing on Phase 10):

  • A flaky BMC: the driver retries with backoff + a circuit breaker; the reconcile loop requeues; the node doesn't get stuck or hammer the BMC.
  • A node that fails provisioning/validation: quarantined with a reason (EBOM mismatch, failed attestation, RAS fault), not handed to tenants; an alert + a runbook (Phase 12).
  • A bad firmware: A/B rollback (Phase 08); fleet rollout caught at canary (Phase 11).
  • A correlated failure (PDU/ToR/CDU): blast-radius reasoning (Phase 01) turns an alarm storm into one root cause; placement respected failure domains.
  • The control plane itself: idempotent, restartable; state persisted; no single in-flight operation corrupts on restart (graceful shutdown, Phase 02).

Demonstrating these failure paths — not just the happy path — is what separates a real platform from a demo, and it's exactly what an interviewer probes.

Chapter 5: What Changes at Fleet Scale

The mini-platform manages one rack; the design doc must answer "what about 10,000 nodes?" — the bridge to the system-design/ chapters:

  • Telemetry (Phase 07): one Prometheus won't scrape 10k nodes × dozens of metrics — federate/ remote-write to Mimir/Thanos; ruthless cardinality control. (system-design/02)
  • Control plane: shard/partition by rack/site; the reconcile loop is per-node and parallel; smart proxies near hardware (Phase 05) so the control plane stays central but the device traffic stays local. (system-design/01)
  • Provisioning/firmware: progressive rollout, failure-domain-aware waves, canary (Phase 08/11). (system-design/03)
  • Tenancy: isolation at scale across network/compute/accelerator; attestation gating (Phase 09). (system-design/04)
  • Operations: SLOs, error budgets, on-call across sites, runbooks (Phase 07/12).

The honest move in the design doc is to build the single-rack version cleanly and state precisely what you'd change at scale — showing you understand both.

Chapter 6: Shipping It — Design Doc, Runbook, Retro

The capstone isn't done when the code runs; it's done when it's shipped like an engineer ships (Phase 12):

  • Design doc: problem, architecture + the seams, the data model, the key ADRs (driver abstraction, A/B firmware, operator lifecycle), failure handling, and the fleet-scale section.
  • Runbook (Phase 12): one real operation end to end (commission a rack / firmware rollout / node RMA) — trigger, steps, risk bounds, rollback.
  • Release cycle (Phase 11): tests (the pyramid), a canary rollout, and a retrospective — what worked, what you'd change, the RCA-style learnings. The retro is where you demonstrate the Phase 10 reflective discipline on your own work.

Capstone Walkthrough Guidance

  1. Run the provided mini-platform: python3 capstone-mini-rack-manager/solution.py — it composes inventory + driver + reconcile + provisioning + firmware + telemetry + tenancy into one end-to-end demo (discover → ready → operate → a fault → recovery), with self-tests.
  2. Read it as an integration of the prior labs — find each phase's pattern in it and the seams between them.
  3. Then extend it into your capstone: wire in a real Redfish emulator (Phase 03), add the canary rollout (Phase 11), and write the design doc + runbook + retro (Phase 12).
  4. The interview deliverable is the running platform plus the three documents.

The capstone question to be able to answer cold: "Walk me through the rack-management platform you built — the architecture, the seams, how it handles a flaky BMC and a bad firmware, and what changes at 10,000 nodes." If you can give that tour from your own code and design doc, you're ready.

Success Criteria

  • The mini-platform runs end to end with no hardware and passes its tests
  • One model + one reconcile loop + one driver abstraction underpin it (no protocol leaks)
  • It demonstrates failure paths (flaky BMC, quarantine, firmware rollback), not just happy path
  • You wrote a design doc with the seams, key ADRs, and a fleet-scale section
  • You wrote a runbook and a retrospective (Phase 12)
  • You can give the 5-minute platform tour and the "what changes at scale" answer

Interview Q&A

Q1: Walk me through the rack-management platform you built. A: It's a control plane over a device abstraction. One inventory model (components, relationships, state, the EBOM baseline) is the spine; one reconcile loop drives each node from discovered → provisioned → at-firmware-baseline → attested → configured → validated → ready, and keeps it there by correcting drift — idempotently and resumably. All hardware access goes through a DeviceDriver abstraction (Redfish-first, IPMI/SNMP fallback) so protocol/vendor variance is contained and I can run the whole thing against fakes/emulators with no hardware. Around that: provisioning is a validation-gated state machine that quarantines bad nodes (EBOM mismatch, failed attestation, RAS faults); firmware updates are A/B with rollback; telemetry is exposed in Prometheus format with alerts; tenancy enforces isolation invariants; and it's operated via a structured-logged CLI and covered by a test pyramid. I ship it with a design doc, a runbook, and a retrospective. The three load-bearing ideas are one model, one reconcile loop, and clean seams.

Q2: Where are the seams and why do they matter? A: The most important is the DeviceDriver boundary — it gives portability (add a vendor/protocol behind it without touching callers) and testability (inject a FakeDriver to exercise every failure path with no hardware), which is what makes the whole platform CI-able (Phase 11). The second is the inventory model — the single contract every part reads/writes, so telemetry, provisioning, and firmware all speak the same nouns. The third is the reconcile API — declared state in, convergence out — which lets provisioning and firmware be actions the loop takes rather than competing orchestrators. Good seams are why I can swap a real BMC for a fake, add a tenant, or test a bad- firmware rollback in isolation.

Q3: How does it handle a flaky BMC, a bad node, and a bad firmware? A: A flaky BMC: the driver wraps calls in retry+backoff+jitter and a circuit breaker (Phase 10), and the reconcile loop requeues rather than getting stuck or hammering it. A bad node: the provisioning validation gate quarantines it with a reason (EBOM mismatch / failed attestation / RAS fault — Phases 08/09/12) and raises an alert with a runbook, instead of handing it to tenants. A bad firmware: the orchestrator verifies before and after and rolls back to the previous A/B bank on failure (Phase 08), and at fleet scale a rollout is caught at the 1% canary and auto-rolled-back (Phase 11). I'd demo each of those failure paths, because how it fails is the real test of a platform.

Q4: What changes when this manages 10,000 nodes instead of one rack? A: The single-rack design stays, but several things scale out. Telemetry: one Prometheus can't scrape 10k nodes × dozens of metrics, so I federate/remote-write to Mimir/Thanos and get strict about cardinality (system-design/02). Control plane: shard by rack/site, run the per-node reconcile loops in parallel, and use smart proxies near the hardware so the control plane stays central while device traffic stays local (system-design/01). Provisioning and firmware become progressive, failure- domain-aware, canaried rollouts (system-design/03). Tenancy isolation and attestation scale across network/compute/accelerator (system-design/04). And operations become SLO-driven with cross-site on-call and runbooks. The honest answer is: build the one-rack version cleanly, and know exactly what you'd change at scale — which is what the design doc states.

References

  • The prior phases' READMEs/WARMUPs — the capstone is their composition
  • The Datacenter as a Computer (Barroso et al.) — warehouse-scale systems thinking
  • Kubernetes operator pattern (the reconcile model) — https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
  • This track's system-design/ — the fleet-scale designs the capstone points to
  • Cross-track capstones: Head of SWE — GPU, Phase 12 capstone