Phase 04 — Hardware Interfacing: BMC, PDU, CDU, PCIe Switches
Difficulty: ⭐⭐⭐⭐☆ | Estimated Time: 2 weeks
Roles supported: Rack Management SWE (Senior/Staff), Firmware/Platform SWE, HW-bringup SWE
Hardware needed: none — simulators + your own machine's lspci
Why This Phase Exists
Phase 03 gave you the protocols; this phase gives you the devices those protocols talk to. The JD names them explicitly: "Interface with rack hardware (BMC, PDU, CDU, PCIe switches)." Each is a distinct managed component with its own model, failure modes, and control surface, and a rack manager must understand all four:
- BMC — the management brain of every node (power, sensors, console, virtual media, firmware).
- PDU — power distribution and metering; the source of per-outlet/per-phase power truth.
- CDU — the coolant distribution unit of a liquid-cooled rack; flow/ΔT/pressure/leak (Phase 01 Ch. 5).
- PCIe switches — the fabric wiring accelerators, NICs, and CPUs together; topology, lanes, hot-plug, errors.
"Understanding of server hardware, networking, and storage systems" (a JD requirement) is
exactly this layer. You'll learn what each device is, how to read and control it, what
breaks, and how to model it — then build the tools: a BMC interaction toolkit, a PDU/CDU
telemetry simulator, and a PCIe topology mapper that turns your own machine's lspci
into a fabric graph.
Concepts
BMC / OpenBMC (deeper than Phase 01)
- The BMC SoC (e.g., ASPeed AST2600), its RTOS/Linux, standby power, the host↔BMC interfaces (KCS, LPC/eSPI, PCIe VDM, NC-SI for shared NIC)
- OpenBMC architecture: D-Bus,
phosphor-*services, the sensor/Entity model,obmcmapping to Redfish/IPMI - Capabilities: power/reset control, sensor aggregation, SoL, KVM/iKVM, virtual media (mount ISO), SEL, firmware (host BIOS + BMC self-update)
- The BMC as a security boundary (Phase 09): it can do anything; protect it accordingly
PDU (Power Distribution Unit)
- Types: basic, metered, switched, metered-and-switched; per-outlet vs per-phase metering
- Busbar vs PDU power topologies (OCP); 3-phase balance; the metering data model
- Reading draw, switching outlets, alarms; protocols (SNMP common, Redfish PDU schema emerging)
- Why per-outlet metering is the ground truth for the Phase 01 budget calculator
CDU (Coolant Distribution Unit) & liquid cooling telemetry
- The secondary (rack) loop vs primary (facility) loop; heat exchanger; pumps; manifolds; cold plates
- Telemetry: supply/return temperature, ΔT, flow (L/min), pressure, pump speed, leak detection
- Failure modes: pump failure, fouling (ΔT rises at constant load), blockage, leak → emergency action
- Protocols: Modbus/SNMP/Redfish; the safety-critical nature of leak handling
PCIe switches & fabric
- PCIe basics: lanes, links, generations (Gen4/5/6 bandwidth), the tree topology (root complex → switches → endpoints)
- PCIe switches (e.g., Broadcom/Microchip): fanout, lane allocation, hot-plug, peer-to-peer (P2P) for GPU-direct
- Reading topology: BDF (bus:device.function),
lspci, sysfs (/sys/bus/pci/devices), capabilities, link speed/width - Errors: AER (Advanced Error Reporting), correctable vs uncorrectable, link down/degraded (a degraded x16→x4 link is a silent perf killer) — feeds Phase 08 RAS
- The scale-up fabric (NVSwitch-class) vs PCIe — where each lives
Cross-cutting
- The device-driver abstraction extended to all four device classes (Phase 03 Ch. 8)
- Modeling them in the Phase 01 inventory graph; relating telemetry to components
- Safety: power and cooling actions are physical and irreversible-ish — confirmation, interlocks, dry-run
Labs
Lab 01 — BMC Power Control, Sensors & Virtual Media
| Field | Value |
|---|---|
| Goal | Build a BMC interaction toolkit (over the Phase 03 Redfish/IPMI patterns) that reads sensors, controls power safely (with dry-run + confirmation), and drives virtual-media for remote reinstall. |
| Concepts | OpenBMC capabilities, power/reset semantics, SoL/KVM/virtual-media, the BMC as a security boundary, safe mutating operations. |
| Steps | 1) python3 solution.py — runs against the bundled BMC mock. 2) Read the power state machine and the virtual-media flow. 3) Note the dry-run/confirm guard on every mutating call. 4) Extensions below. |
| Stack | Python 3 stdlib (BMC mock) |
| Output | A toolkit that reads sensors, power-cycles safely, and mounts/ejects virtual media — the operations behind "remote reinstall." |
| How to Test | Asserts verify the power state machine, the dry-run guard (no mutation), and the virtual-media mount/eject lifecycle. |
| Talking Points | Why the BMC works when the host is dead; virtual media → remote OS install; why every power op needs a guard; BMC as attack surface. |
| Resume Bullet | "Built a BMC interaction toolkit (Redfish/IPMI) for safe power control, sensor aggregation, and virtual-media-driven remote reinstall, with dry-run/confirmation guards on all mutating operations." |
| Extensions | Add SoL capture; add a power-on→POST-watch flow; wire it to the real OpenBMC sushy-emulator. |
Lab 02 — PDU/CDU Telemetry Simulator + PCIe Topology Mapper
| Field | Value |
|---|---|
| Goal | Two tools: a PDU/CDU telemetry simulator (draw, ΔT, flow, leak) feeding the Phase 01 budget logic, and a PCIe topology mapper that parses lspci-style output into a fabric graph and flags degraded links. |
| Concepts | PDU metering, CDU thermal telemetry & leak handling, PCIe BDF/topology, link speed/width, degraded-link detection, AER. |
| Steps | 1) python3 solution.py. 2) Read the simulated PDU/CDU telemetry + the leak/ΔT alarms. 3) See the PCIe tree built from sample lspci output, with a degraded x16→x4 link flagged. 4) Run it on your own machine's lspci. 5) Extensions below. |
| Stack | Python 3 stdlib; optional real lspci -mm -nn on Linux |
| Output | Live-ish PDU/CDU telemetry with alarms + a PCIe topology graph with degraded-link detection. |
| How to Test | Asserts verify leak/ΔT alarm thresholds, PDU draw aggregation, and degraded-link detection in the parsed topology. |
| Talking Points | Why ΔT rising at constant load predicts a cooling failure; leak handling as safety-critical; how a silent x16→x4 link halves bandwidth; BDF addressing. |
| Resume Bullet | "Built a PDU/CDU telemetry simulator with leak and ΔT alarming and a PCIe topology mapper that parses lspci into a fabric graph and detects degraded links (x16→x4) and AER errors." |
| Extensions | Parse /sys/bus/pci/devices for link state; correlate AER counts; emit Prometheus metrics (Phase 07); feed draw into the Phase 01 calculator as live data. |
Deliverables Checklist
- You can list OpenBMC capabilities and explain the host↔BMC interfaces
- BMC toolkit reads sensors and controls power with a dry-run/confirmation guard
- You can describe the virtual-media remote-reinstall flow end to end
- PDU simulator aggregates per-outlet draw into a rack total (feeds Phase 01)
- CDU simulator alarms on leak and on ΔT rising at constant load
-
PCIe mapper builds a topology graph from
lspciand flags a degraded link - You can explain why a degraded x16→x4 link is a silent performance killer
- You can relate each device to the Phase 01 inventory model and Phase 08 RAS
Interview Relevance
- "What is a BMC made of, and how does the host talk to it?"
- "How would you reinstall an OS on a server you can't physically touch?" (virtual media)
- "A rack's cooling is degrading — what telemetry tells you before it fails?" (ΔT, flow, pump)
- "How do you handle a coolant leak signal in software?" (safety-critical, emergency action)
- "Walk me through reading PCIe topology on Linux. What's a BDF?"
- "An accelerator is slow but 'healthy' — how could PCIe be the cause?" (degraded link width/speed)
- "How do you make a power-off operation safe to expose in a tool?"