« All Roles

Software Engineer — Rack Management (Senior / Staff) — Complete Apprenticeship Curriculum

Target Roles:

• Senior / Staff Software Engineer — Rack & Data Center Management Software (Qualcomm, NVIDIA, AMD, Microsoft Azure, Meta, Google, ODMs)
• Infrastructure Software Engineer — AI accelerator racks, bare-metal fleets, BMC/Redfish control planes
• Platform Engineer — provisioning, lifecycle, and observability for rack-scale AI systems
• Data Center Software Engineer — out-of-band management, firmware orchestration, telemetry pipelines
• SRE / Reliability Engineer — close-to-hardware infrastructure software in production

Duration: 26–34 weeks core — extendable to 12 months for full Staff-level readiness Goal: Build the rare profile this JD demands — the ability to write high-quality, secure, close-to-hardware infrastructure software that provisions, monitors, orchestrates, and maintains rack-scale AI accelerator systems; to speak the management protocols (Redfish/IPMI/SNMP) and hardware (BMC/PDU/CDU/PCIe) fluently; to integrate with Kubernetes/Terraform/Foreman and ship Prometheus/Grafana telemetry; and to debug and root-cause complex failures in deployed production data centers — then to lead that work and mentor others at the Staff level.

Why This Curriculum Exists

The "Rack Management Software Engineer" role at a hyperscale AI accelerator program is deceptively broad. The JD reads like five jobs:

1. Systems software engineering — production Python, C++, and shell that runs close to hardware: daemons, agents, drivers-adjacent code, written to be secure, testable, and maintainable.
2. Hardware/firmware interfacing — talking to BMCs, PDUs, CDUs, and PCIe switches over Redfish, IPMI, and SNMP; firmware updates; crash-dump parsing; remote diagnostics; RAS.
3. Infrastructure automation — bare-metal provisioning (Foreman/PXE), configuration management (Ansible/Chef/Puppet), infrastructure-as-code (Terraform), and orchestration (Kubernetes/RKE2).
4. Observability & operations — unified telemetry pipelines (Prometheus/Grafana), logging, metrics, tracing, runbooks, and SLOs across heterogeneous rack components.
5. Production debugging & ownership — root-cause analysis of complex issues in deployed systems, secure multi-tenant operation (secure boot, VLAN separation, QoS), and — at Staff — driving implementation, reviewing designs, and mentoring.

Most engineers know one or two of these. This curriculum is built backward from the JD so that every "required" and "preferred" line maps to phases and hands-on labs. The whole thing runs on a single laptop — every BMC, PDU, CDU, PCIe switch, and firmware flow is reproduced with open emulators and simulators, so you learn the mechanism without a data center.

Going beyond the JD (because interviews and the job do)

Real interviews and real on-call shifts go past the bullet list. This track deliberately adds:

• RAS depth — PCIe AER, MCE/MCA, ACPI/APEI, Xid-style error decoding (Phase 08) — because "crash-dump parsing" in practice means decoding hardware error logs.
• Systematic debugging & RCA as a first-class discipline with gdb/strace/perf/eBPF and written postmortems (Phase 10) — the JD's most-repeated theme.
• Reliability patterns — reconciliation loops, idempotency, circuit breakers, chaos/fault injection (Phase 10).
• Hardware-in-the-loop testing and emulator-based CI so infra code is actually testable (Phase 11).
• Staff-level leadership — design reviews, ADRs, EBOM, protocol specs, mentorship, and distributed-team communication (Phase 12).
• Fleet-scale system design — what changes when one rack becomes ten thousand (the entire system-design/ section).

What You Will Build

By the end you will have produced, with your own hands, a portfolio that is the job:

• A rack data model + inventory engine (Python) representing compute trays, accelerators, NICs, switches, PDUs, CDUs, and BMCs — with a power & thermal budget calculator that catches over-subscription before it trips a breaker (Phase 01)
• A C++ sensor-polling daemon (RAII, error handling, unit tests, systemd-ready) and a production-grade Python rack CLI (typed, packaged, retried, structured-logged) — the JD's language matrix, done right (Phase 02)
• A Redfish client that walks Chassis→Power→Thermal, performs resets, and subscribes to events against a real Redfish emulator — plus SNMP poller/trap receiver and IPMI sensor reader (Phase 03)
• A BMC interaction toolkit (power-cycle, sensor read, virtual media) and a PDU/CDU telemetry simulator + PCIe topology mapper that turns lspci into a fabric graph (Phase 04)
• A bare-metal provisioning flow (PXE/iPXE → kickstart → Ansible) and a Foreman/Terraform onboarding state machine that takes a rack from "powered on" to "ready" (Phase 05)
• A Kubernetes operator (a RackNode CRD with a reconciliation loop) and an accelerator device-plugin + scheduling walkthrough on RKE2 (Phase 06)
• A Redfish→Prometheus exporter, PromQL alert rules, and a Grafana dashboard for a rack — plus OpenTelemetry tracing across a provisioning workflow (Phase 07)
• A firmware-update orchestrator with staging, signature verification, A/B rollback — and a crash-dump / RAS log parser that classifies PCIe AER and MCE errors into an actionable diagnostics report (Phase 08)
• A VLAN + QoS tenant-isolation lab (network namespaces, tc) and a secure-boot / measured-boot + attestation simulator (TPM PCRs, signed firmware) (Phase 09)
• A debugging dojo: a planted bug in a rack daemon you find with gdb/strace/eBPF and write up — plus a chaos/fault-injection harness that hardens a reconciliation loop (Phase 10)
• A CI pipeline with emulated hardware (Redfish mock in CI, full test pyramid) and a fleet canary + rollback release system (Phase 11)
• A complete Staff deliverable pack — a protocol spec, a software EBOM, an ADR, a design-review checklist, a runbook, and a mentorship plan (Phase 12)
• A capstone: mini rack-management platform — discovery/inventory + Redfish/IPMI/SNMP drivers behind an abstraction + provisioning state machine + telemetry + firmware orchestration + secure multi-tenant config, with a design doc and a release cycle (Phase 13)

Folder Structure

Software Engineer Rack Management - Senior and Staff Engineer level/
├── README.md                                   ← You are here
├── jd.md                                       ← Target role profile
├── phase-01-rack-scale-systems-foundations/    ← What an AI rack IS: trays, accel, BMC, PDU, CDU, power/thermal
├── phase-02-production-systems-programming/     ← Python, C++, shell, Node.js — secure, testable, maintainable
├── phase-03-management-protocols-redfish-ipmi-snmp/ ← The out-of-band control plane: Redfish, IPMI, SNMP
├── phase-04-hardware-interfacing-bmc-pdu-cdu-pcie/   ← BMC, PDU, CDU (liquid cooling), PCIe switch fabric
├── phase-05-provisioning-config-management/    ← PXE/Foreman, Ansible/Chef/Puppet, Terraform — bare metal to ready
├── phase-06-orchestration-kubernetes-rke2/     ← K8s/RKE2, the operator pattern, device plugins, scheduling
├── phase-07-telemetry-observability/           ← Prometheus, Grafana, logging, metrics, tracing, unified pipelines
├── phase-08-firmware-diagnostics-ras/          ← Firmware update, crash dumps, RAS (PCIe AER/MCE/APEI), remote diag
├── phase-09-networking-secureboot-multitenancy/← Mgmt vs data plane, VLAN isolation, QoS, secure/measured boot, TPM
├── phase-10-reliability-debugging-rca/         ← Systematic debugging, gdb/strace/perf/eBPF, RCA, chaos, reconciliation
├── phase-11-cicd-testing-release-engineering/  ← Test pyramid + HIL, emulated-hardware CI, canary, rollback
├── phase-12-collaboration-ebom-leadership/     ← EBOM, protocol docs, ADRs, design reviews, Staff mentorship
├── phase-13-capstone-rack-management-platform/ ← Integrate everything into a mini rack manager + release cycle
├── interview-prep/                             ← Concepts, coding, protocols, debugging, behavioral/Staff
└── system-design/                              ← Control plane, telemetry, firmware-at-fleet, tenancy, provisioning

26-Week Core Schedule

Phase Structure

Every phase follows a consistent format:

Labs deliberately span the JD's language matrix: Python (clients, exporters, simulators, operators, parsers), C++ (the sensor daemon, close-to-hardware code), shell (provisioning glue, CI), and Node.js (a telemetry/tooling service — the JD's "plus"). Where the JD names a tool (Redfish, IPMI, SNMP, Foreman, Ansible, Terraform, Kubernetes, RKE2, Prometheus, Grafana), you use a real, open implementation of it.

No data center? You can still do 100% of this track

Every lab runs on a single Linux/macOS laptop using open emulators:

1. Redfish — DMTF's Redfish-Mockup-Server and the OpenBMC sushy-emulator give you a real Redfish service.
2. IPMI/BMC — a software BMC simulator (or ipmi_sim from OpenIPMI) plus a provided mock.
3. SNMP — snmpsim serves real MIBs; a local trap receiver completes the loop.
4. Kubernetes/RKE2 — kind/k3d locally; RKE2 differences are explained in the WARMUP.
5. Provisioning — PXE/iPXE flows run against local VMs/QEMU; the rest is simulated with the same data model.
6. PDU/CDU/PCIe — provided simulators emit realistic telemetry; lspci topology mapping uses your own machine.

The WARMUPs always explain what changes at real rack and fleet scale so you're not surprised on the job.

Prerequisites

• Solid Python; working C or C++ (you can read pointers, structs, headers, and a Makefile)
• Comfort in a Linux shell (you'll write scripts and use systemctl, journalctl, ip, lspci, gdb, strace)
• Basic networking literacy (IP, VLAN, DNS/DHCP) and willingness to go deeper — Phase 09 starts from fundamentals
• Git, Docker basics; a Linux VM or WSL2 if you're on Windows
• No prior BMC/Redfish/firmware experience required — Phases 01 and 03 start at the hardware and the protocol

Related Tracks in This Repo

• Head of Software Engineering — GPU — the silicon/CUDA/driver side of the same racks; its Phases 04/06 pair with our 04/06
• Security Engineer — deeper offensive/defensive security; pairs with our Phase 09 (secure boot, tenancy)
• AI Specialist — inference/accelerator workloads that run on the racks you manage
• LLM Inference Engineer — what the accelerators actually run, for context on workload behavior