System Design 01 — Rack Management Control Plane at Fleet Scale

"Design the software that manages a fleet of AI accelerator racks." Phases: 01 (model), 03 (drivers), 06 (reconcile), 13 (capstone), 07/08/09 (operate/maintain/secure).

1. Drive the requirements

  • Scale: how many racks/nodes/sites? (Assume 10,000 nodes across several sites — multi-site is in the JD.)
  • Hardware: single-vendor or heterogeneous (Redfish + IPMI + SNMP gear)? → driver abstraction.
  • Workloads: tenant AI training/serving → orchestration integration (k8s/RKE2).
  • Sovereignty: air-gapped/regulated sites? → RKE2, attestation, on-prem control plane.
  • SLOs: time-to-ready for a new rack; node availability; control-plane availability.

2. Architecture

                         +------------------ Control Plane (per region/site) ------------------+
 operators/CLI (P02) --> | API (desired state) | Inventory model (P01) | Reconcile loops (P06) |
                         | Provisioning (P05) | Firmware (P08) | Tenancy/secure cfg (P09)        |
                         | Telemetry/alerting (P07)                                              |
                         +-----------------------------------------------------------------------+
                                 |                         |                         |
                         +--------------+          +--------------+          +--------------+
                         | Smart proxy  |   ...    | Smart proxy  |          | Smart proxy  |  (near hardware)
                         | (DHCP/TFTP,  |          | DeviceDriver |          | exporter     |
                         |  Redfish/IPMI/SNMP to BMC/PDU/CDU/switch — P03/P04)               |
                         +--------------+          +--------------+          +--------------+
                                 | management VLAN (isolated, P09)
                          [ racks: BMCs, PDUs, CDU, switches, nodes ]

3. The hard decisions (and the tradeoffs)

  • One model + one reconcile loop (P01/P06): every node is an object with a desired state; per-node reconcile loops converge it (provision, firmware-baseline, attest, configure, validate). Tradeoff: a control loop is more to build than imperative scripts, but it's self-healing, idempotent, and resumable — essential at 10k nodes.
  • The driver abstraction (P03): Redfish-first with IPMI/SNMP fallback behind one DeviceDriver. Tradeoff: an abstraction tax + risk of a lowest-common-denominator API (mitigate with capability flags/OEM passthrough), vs. protocol quirks leaking everywhere.
  • Central control plane + smart proxies (P05): control logic central; DHCP/TFTP/device traffic via proxies near the hardware. Tradeoff: more moving parts, but it scales across sites and keeps the management plane reachable locally during a WAN blip — critical for multi-site.
  • Sharding/partitioning: shard the control plane by rack/site; reconcile loops are per-node and parallel; a regional control plane per site (sovereign/air-gap friendly). Tradeoff: cross-shard views need aggregation.
  • State store: a consistent store (etcd-like) for desired state + a time-series store for telemetry. Keep declared intent (EBOM) separate from discovered reality and reconcile (P01/P12).

4. Failure modes & operations

  • Flaky BMC (P03/P10): driver retries + circuit breaker; the loop requeues — no stuck nodes, no retry storms.
  • Bad node (P05/P08/P09): validation gate quarantines it (EBOM mismatch / failed attestation / RAS fault) — never handed to tenants; alert + runbook (P12).
  • Correlated failure (P01): a PDU/ToR/CDU loss → blast-radius reasoning turns an alarm storm into one root cause; placement respected failure domains.
  • Control-plane restart: idempotent + persisted state → no in-flight corruption (graceful shutdown, P02); the reconcile loop simply re-converges.
  • Fleet operations: firmware/k8s upgrades as gated canary rollouts (P08/P11); cordon/drain (P06); observability + SLOs + cross-site on-call (P07/P12).

5. What changes at scale (the Staff answer)

  • Telemetry federates (→ design 02); provisioning/firmware become progressive domain-aware rollouts (→ designs 03); tenancy/attestation scale (→ design 04). The single-rack design (the capstone) stays; you shard it, add proxies, and make every fleet action gated/reversible.
  • Honest limits: cross-site consistency, the cardinality budget, and the size of the HIL/conformance matrix (P11) are the things I'd measure and bound first.