Capstone — Mini Rack Management Platform

Phase: 13 — Capstone | Difficulty: ⭐⭐⭐⭐⭐ | Time: 2 weeks (the integration + docs) Language: Python 3 (stdlib) | Hardware: none (composes the prior phases' fakes/emulators)

Concept primer: ../WARMUP.md, ../HITCHHIKERS-GUIDE.md.

Run

python3 solution.py     # discover -> reconcile to READY -> quarantine -> telemetry -> drift-correct

0. The mission

Compose the whole curriculum into one runnable platform and the three Staff documents that ship it. solution.py is the integrated reference; your capstone extends it and adds the design doc, runbook, and retrospective.

1. Find every phase in the code

PhaseIn the platform
01 modelNode/Rack inventory graph + domain (failure domain)
03 driversFakeDriver behind the device seam (swap for a real Redfish emulator)
06 reconcileRackManager.reconcile_node/all — declared→discovered convergence, idempotent
05 provisioningthe state machine DISCOVERED→…→READY with a validation gate
12 EBOM gatefirmware verify against baseline_firmware; quarantine on mismatch
09 attestation + tenancyattest() gate; tenancy_violations() invariant
08 firmwareset-then-verify + drift correction (re-flash on out-of-band change)
10 reliabilityresilient() retry wrapper around the flaky driver
07 telemetrymetrics() Prometheus exposition (node_ready, node_temp)

2. The failure paths it demonstrates (the point)

  • Bad node (node-3 fails attestation) → QUARANTINED with a reason, not shipped.
  • Flaky BMC (node-4) → resilient() retries recover it → it still reaches READY.
  • Firmware drift (node-0 downgraded out-of-band) → the reconcile loop re-flashes to baseline and re-converges to READY (the self-healing loop).
  • Tenancy invariant → a reserved-plane misassignment is caught.

The self-tests assert all of these — demonstrating how it fails, which is what makes it a platform rather than a demo.

3. Build YOUR capstone (extend this)

  1. Real drivers: replace FakeDriver with the Phase 03 Redfish client against the DMTF mockup server (and keep the fake for tests — the seam pays off).
  2. Real provisioning + firmware: wire in the Phase 05 state machine and the Phase 08 A/B orchestrator (with rollback) as the loop's actions.
  3. Real telemetry: serve metrics() over HTTP (Phase 07 exporter) and add the alert rules.
  4. Canary release: ship a platform change via the Phase 11 canary controller.
  5. The three documents (Phase 12):
    • Design doc: architecture + the seams + key ADRs (driver abstraction, A/B firmware, operator lifecycle) + failure handling + a fleet-scale section (point at system-design/).
    • Runbook: one real op end to end (commission a rack / firmware rollout / node RMA).
    • Retrospective: what worked, what you'd change, RCA-style learnings on your own build.

4. What this proves

You can build and operate a rack-management platform end to end — the JD, composed — and ship it like a Staff engineer. "Walk me through a rack-management system you built" becomes a confident 5-minute tour of your own running code and design doc, including the failure paths and the what-changes-at-scale answer. This is the portfolio behind every other interview answer.