Warmup Guide — Hardware Interfacing: BMC, PDU, CDU, PCIe Switches

Zero-to-expert primer for Phase 04. Phase 03 taught the protocols; this teaches the devices — what each one is, how to read and control it, how it fails, and how to model it. By the end you can reason about a node's BMC, its power and cooling, and its PCIe fabric the way a hardware-fluent software engineer must.

Table of Contents

Chapter 1: The BMC, Under the Hood

The BMC (Phase 01 Ch. 3) is a real computer; now look inside. A typical server BMC is an ASPeed AST2500/2600 — an ARM SoC with its own RAM, flash, Ethernet MAC, and a pile of hardware interfaces to the host. It runs on standby power (alive whenever the rack has any power, even with the host off) and boots its own firmware (often OpenBMC, a Linux distribution) independently of the host.

How the BMC senses and controls the host — the interfaces matter because they're where bugs and security boundaries live:

  • KCS / BT (LPC or eSPI): the legacy in-band channel the host OS uses to talk IPMI to the BMC locally.
  • eSPI / LPC: the bus carrying low-pin-count peripherals, TPM, and host↔BMC messaging.
  • PECI: the Intel interface the BMC uses to read CPU temperature/telemetry directly.
  • I²C / PMBus: how the BMC reads voltage regulators, PSUs, temperature sensors, and the FRU EEPROMs (the parts inventory — Phase 01 Ch. 10).
  • NC-SI: lets the BMC share the host's NIC for management traffic (vs a dedicated mgmt port) — a shared-NIC design has security implications (Phase 09).
  • PCIe VDM / MCTP: modern in-band management messaging (e.g., to NICs, accelerators).

The key idea: the BMC aggregates every low-level sensor and control into one management endpoint, then exposes them via Redfish/IPMI (Phase 03). When you GET a temperature over Redfish, the BMC read it over PECI/I²C and translated it. Understanding this chain is what lets you debug "the sensor reads garbage" (Phase 10) — is it the sensor, the I²C bus, the BMC firmware, or the Redfish mapping?

Chapter 2: OpenBMC — How the Management Stack Is Built

OpenBMC (a Linux Foundation project, used by Meta, Google, IBM, Microsoft, and many ODMs including for accelerator platforms) is worth knowing because it's the firmware your software talks to, and increasingly the one you might contribute to alongside an ODM (Phase 12).

Architecture, briefly:

  • It's embedded Linux + a set of services communicating over D-Bus (the IPC bus).
  • The phosphor-* services each own a domain: phosphor-hwmon (sensors from kernel hwmon), phosphor-state-manager (host/chassis power state), phosphor-host-ipmid (the IPMI daemon), entity-manager (discovers what hardware is present from config + EEPROMs), phosphor-logging (events/SEL).
  • bmcweb is the HTTP server that exposes Redfish (and a web UI), translating D-Bus state into the Redfish resource tree you navigated in Phase 03. So "the Redfish Thermal resource" is bmcweb rendering phosphor-hwmon's D-Bus sensor objects.

Why this matters: when a Redfish field is missing or wrong on an OpenBMC system, the chain is sensor → hwmon → D-Bus → bmcweb → Redfish JSON. Knowing the chain turns "vendor bug" into a specific, reportable defect (and sometimes a patch you send upstream). It's also why "every BMC is subtly different" (Phase 03 Ch. 9): different entity-manager configs and firmware versions render different trees.

Chapter 3: BMC Operations — Power, Console, Virtual Media

The operations your toolkit (Lab 01) performs, and their semantics:

Power control — the host has a small state machine the BMC drives:

  • On / ForceOff (yank power — abrupt) / GracefulShutdown (ask the OS to shut down — needs a cooperating OS) / ForceRestart / GracefulRestart / PowerCycle.
  • The subtlety: GracefulShutdown can fail silently if the host OS is hung (nothing to ask). Robust tooling tries graceful, waits, then escalates to force — exactly the retry/escalation pattern from Phase 02.

Console — see what the host is doing without a monitor:

  • SoL (Serial-over-LAN): the host's serial console redirected over the BMC's network — text, scriptable, the workhorse for watching boot/POST and kernel panics (Phase 08/10).
  • KVM / iKVM: full remote keyboard-video-mouse (graphical), for BIOS screens.

Virtual media — the killer OOB feature: the BMC presents a network ISO/image as if it were a physical USB/CD drive to the host. Combined with boot-device override, this is how you reinstall an OS on a server you can't touch: mount the installer ISO over Redfish, set next-boot to the virtual CD, power-cycle, and the host boots your installer. This is the backbone of bare-metal provisioning's "rescue path" (Phase 05) and a primary reason OOB exists. Lab 01 implements the mount→set-boot→cycle→eject lifecycle.

Firmware — the BMC can update the host's BIOS/UEFI/NIC firmware and self-update its own firmware (Phase 08), all over the management network.

Chapter 4: PDUs and Power Distribution

The PDU is where your power truth comes from. Types (in increasing capability):

  • Basic: dumb power strip. No telemetry. (Rare in modern racks.)
  • Metered: reports power draw (per-phase and/or per-outlet). Read-only truth for budgeting.
  • Switched: can turn individual outlets on/off remotely (e.g., hard power-cycle a wedged device whose BMC is also wedged).
  • Metered + switched: both — the common AI-rack choice.

Per-outlet vs per-phase metering: per-outlet tells you which device draws what (great for the Phase 01 budget and for attributing power to tenants); per-phase tells you balance across the 3 phases (an unbalanced load wastes capacity and can trip a phase). A good rack manager reads both.

Busbar designs (OCP): in OCP racks the power shelf feeds a DC busbar and nodes blind-mate on; metering may live in the power shelf rather than discrete PDUs. Same data (draw, per-feed), different physical form.

Protocols: SNMP is the historical standard for PDUs (Phase 03 Ch. 6 — you walk an outlet subtree); the DMTF Redfish PDU/Outlet schema is the modern path. Switching an outlet is a physical, disruptive action — guard it (Chapter 8).

Why it feeds Phase 01: the budget calculator's "measured draw" is exactly per-outlet PDU telemetry. Nameplate is a guess; the PDU is ground truth. Lab 02 simulates this and you can feed it back into the Phase 01 calculator.

Chapter 5: CDUs and Liquid-Cooling Telemetry

The CDU (Phase 01 Ch. 5) is a managed device you monitor and, in emergencies, act on.

The two loops: the secondary (rack-side) loop is clean coolant the CDU pumps through cold plates; the primary (facility) loop is chilled water. The CDU's heat exchanger moves heat between them while keeping the clean loop isolated from facility water (which may be dirty/corrosive). You manage the secondary loop's behavior.

Telemetry you read (and what each tells you):

  • Supply / return temperature and ΔT (return − supply): the workhorse signal. Heat removed ∝ flow × ΔT. ΔT rising at constant IT load is the classic early warning of reduced flow or fouling — a pre-failure signal you alert on (Lab 02).
  • Flow (L/min): drops indicate a failing/throttled pump or a blockage.
  • Pressure: out-of-range indicates a leak, blockage, or pump issue.
  • Pump speed / status: redundant pumps (N+1) — losing one is tolerable, losing the spare is not.
  • Leak detection: rope/spot sensors. Safety-critical — a leak near live electronics can destroy hardware and is a hazard. Software response is decisive: alarm at top priority, and (per policy/interlock) trigger an emergency action (reduce load via power capping, or controlled shutdown/valve action). Treat it like a fire alarm in code.

Protocols: CDUs commonly speak Modbus (industrial) and/or SNMP, with Redfish CoolingLoop/CoolingUnit schemas emerging. Your software normalizes whichever into the unified model (Phase 03 Ch. 8).

Failure modes to recognize (interview-ready): pump failure (flow→0, temps spike), fouling/scaling (ΔT creeps up over weeks), air in the loop (erratic flow/pressure), blockage (high pressure, low flow), leak (pressure drop + leak sensor). Each has a distinct telemetry signature — that's why you collect all of it.

Chapter 6: PCIe — The Fabric Inside the Node

PCIe (Peripheral Component Interconnect Express) wires the CPU, accelerators, NICs, and NVMe together inside a node. You must understand it because (a) accelerators and NICs hang off it, (b) its topology determines performance (P2P, NUMA), and (c) its errors are a major RAS source (Phase 08).

The basics:

  • A lane is a serial differential pair (TX+RX). A link bundles lanes: x1, x4, x8, x16 (the common GPU/accelerator width). More lanes = more bandwidth.
  • Generations double per-lane bandwidth: Gen4 ≈ 2 GB/s/lane, Gen5 ≈ 4 GB/s/lane, Gen6 ≈ 8 GB/s/lane. So a Gen5 x16 link ≈ 64 GB/s each way. (Recall Phase 01: PCIe is the narrow tier vs HBM/NVLink.)
  • Topology is a tree: a Root Complex (in the CPU) → PCIe switches (fan-out) → endpoints (accelerators, NICs, NVMe). Switches let one root port fan out to many devices and enable peer-to-peer (P2P) transfers (e.g., GPUDirect RDMA: NIC→GPU without bouncing through CPU memory).
  • BDF addressing: every function is Bus:Device.Function (e.g., 0000:3b:00.0), the coordinate you use to identify a device in lspci/sysfs.

PCIe switches (Broadcom/Microchip et al.) are managed devices in AI systems: they allocate lanes, support hot-plug (swap an NVMe/accelerator without reboot), and report errors. In some designs the switch fabric is reconfigurable (lanes reassigned between hosts).

Errors and AER (the RAS hook for Phase 08): PCIe Advanced Error Reporting classifies errors as correctable (recovered by the link layer — but a rising rate predicts failure) vs uncorrectable (fatal/non-fatal — data integrity at risk). A degraded link — negotiated down from x16 to x4, or Gen5 to Gen1 — is the silent killer: the device works and reports "healthy," but bandwidth collapsed, so an accelerator mysteriously runs at a fraction of speed. Detecting link width/speed below capability is a high-value diagnostic (Lab 02).

Chapter 7: Reading PCIe Topology on Linux

The concrete skills (Lab 02 makes them software):

  • lspci: lists PCIe devices. lspci -tv shows the tree; lspci -vvv shows capabilities including LnkCap (what the link can do) vs LnkSta (what it negotiated). The diagnostic gold is LnkSta width/speed < LnkCap → degraded link. lspci -mm -nn gives machine-parseable output with vendor/device IDs.
  • sysfs: /sys/bus/pci/devices/<BDF>/ exposes current_link_speed, current_link_width, max_link_speed, max_link_width, aer_dev_correctable, aer_dev_fatal, and numa_node. This is what your monitoring code reads (no shelling out).
  • NUMA: numa_node tells you which CPU socket a device is "near." Placing a workload's GPU and NIC on the same NUMA node avoids cross-socket PCIe hops — a real perf lever and a scheduling input (Phase 06).

A rack manager that maps topology can answer: "are all 8 accelerators at full x16 Gen5?", "which NIC is P2P-adjacent to which GPU?", "is this node's slow training a degraded-link problem?" — questions that otherwise take an engineer hours on-site.

Chapter 8: Modeling and Safety

Modeling: each device class slots into the Phase 01 inventory graph — BMCs manage nodes, PDUs power them, CDUs cool them, PCIe switches connect components — and their telemetry attaches to components with thresholds (Phase 03 unified Reading). The driver abstraction (Phase 03 Ch. 8) extends to all four: a DeviceDriver per class, one model above.

Safety — this phase is the first where your software causes physical, hard-to-undo effects: powering off a node mid-job, switching a PDU outlet (kills whatever's on it), acting on a leak. The disciplines:

  • Dry-run by default for mutating operations; require explicit confirmation (--yes) for destructive ones (Phase 02 Lab 02 pattern). Lab 01 enforces this.
  • Interlocks: don't allow a power-on if cooling is failed; don't switch an outlet feeding a node that's mid-firmware-update.
  • Blast-radius awareness (Phase 01 Ch. 7): "switch off outlet 3" might drop a whole shelf — show the operator the blast radius before acting.
  • Idempotency & confirmation of effect: after a power action, read back the state to confirm it happened (don't trust the request succeeded).
  • Audit: every physical action is logged with who/what/when (Phase 07/12).

The leak case is the archetype: software that can act on the physical world must be more careful, not less, than software that only moves bytes.

Lab Walkthrough Guidance

Order: Lab 01 (BMC) → Lab 02 (PDU/CDU/PCIe). The BMC is the control surface; the others are the power/cooling/fabric truth around it.

Lab 01 (BMC toolkit):

  1. python3 solution.py — runs against the bundled BMC mock; read the power state machine.
  2. Find the dry-run/confirm guard on every mutating call — this is the safety discipline.
  3. Trace the virtual-media flow: mount ISO → set next-boot → power-cycle → eject.
  4. Extensions: SoL capture; power-on→POST-watch; point at OpenBMC sushy-emulator.

Lab 02 (PDU/CDU sim + PCIe mapper):

  1. python3 solution.py — read the PDU draw aggregation and the CDU telemetry with leak/ΔT alarms.
  2. See the PCIe tree built from sample lspci output, with a degraded x16→x4 link flagged.
  3. Run it on your own machine: pipe real lspci -mm -nn in and watch it build your topology.
  4. Extensions: read /sys/bus/pci/devices link state and AER counts; feed PDU draw into the Phase 01 calculator as live data; emit Prometheus metrics.

Phase capstone question: "An accelerator node passes all health checks but trains 4× slow. Walk me from symptom to root cause using this phase's tools." (Answer: check PCIe LnkSta vs LnkCap for a degraded link / check NUMA adjacency of GPU+NIC / check CDU ΔT for thermal throttling / check PDU draw for power capping — a degraded x16→x4 link or thermal throttle is the usual culprit, and your topology mapper + telemetry find it in minutes.)

Success Criteria

You're done with this phase when — without notes:

  • You can name the BMC's host interfaces (PECI/I²C/eSPI/NC-SI) and the sensor→Redfish chain (Ch. 1–2)
  • You can describe OpenBMC's D-Bus/phosphor/bmcweb architecture at a high level (Ch. 2)
  • You can explain the virtual-media remote-reinstall flow (Ch. 3)
  • You can distinguish PDU types and per-outlet vs per-phase metering (Ch. 4)
  • You can list CDU telemetry and the signature of each cooling failure mode, incl. leak handling (Ch. 5)
  • You can explain PCIe lanes/gens/topology/BDF and why a degraded link is a silent killer (Ch. 6–7)
  • You can read LnkCap vs LnkSta and sysfs link state to find a degraded link (Ch. 7)
  • You can state the safety disciplines for physical actions (dry-run, interlocks, read-back) (Ch. 8)

Interview Q&A

Q1: What is a BMC physically, and how does it read a CPU temperature you then see in Redfish? A: The BMC is an independent ARM SoC (often an ASPeed AST2600) on the motherboard with its own RAM, flash, NIC, and standby power, running its own firmware (frequently OpenBMC). It reads the CPU temperature over PECI (the Intel telemetry interface) — or board sensors over I²C/PMBus — and exposes it through its stack: on OpenBMC, the kernel hwmon driver → phosphor-hwmon → a D-Bus sensor object → bmcweb renders it into the Redfish Thermal resource. So "GET the temperature over Redfish" is the end of a chain that starts at PECI. Knowing that chain is how I debug a wrong/missing sensor: is it the sensor, the I²C bus, the hwmon driver, the D-Bus object, or the Redfish mapping?

Q2: How do you reinstall an OS on a server you can't physically touch? A: Out-of-band via the BMC's virtual media. Over Redfish I mount the installer ISO from a network share as a virtual CD/USB on the BMC, set the next boot device to that virtual media (a one-time boot override), and power-cycle the host through the BMC. The host boots the installer; I watch progress over Serial-over-LAN, and when done I eject the media and let it boot from disk. This works even if the host has no OS or a broken one — that's the whole point of OOB. It's also the rescue path behind PXE provisioning in Phase 05.

Q3: A liquid-cooled rack's cooling is degrading. What telemetry warns you before an outage, and how should software respond to a leak? A: I watch the CDU: ΔT (return − supply) rising at constant IT load is the classic early warning of reduced flow or fouling — heat removed is ∝ flow × ΔT, so if load is flat and ΔT climbs, flow is dropping. I also watch flow (a failing pump), pressure (blockage/leak), and pump status (N+1 — losing the spare is critical). For a leak, software treats it as safety-critical, like a fire alarm: highest-priority alarm immediately, and per policy/ interlock trigger an emergency response — reduce heat load via power capping and/or a controlled shutdown or valve action — because a leak near live electronics is a hazard and can destroy hardware. The principle is that software which can act on the physical world must be more careful, with interlocks and confirmed effect, not less.

Q4: An accelerator is "healthy" but runs at a fraction of its speed. How could PCIe be the cause, and how do you find it? A: The PCIe link likely negotiated down — a degraded link, e.g., x16→x4 or Gen5→Gen1 — which collapses bandwidth while the device still enumerates and reports healthy, so it's silent. I find it by comparing LnkSta (negotiated width/speed) against LnkCap (what the link can do): in lspci -vvv or sysfs (current_link_width/current_link_speed vs max_link_width/max_link_speed). If they don't match, that's the smoking gun (often a reseat/cable/connector or a marginal lane). I'd also check NUMA adjacency — if the GPU and its NIC are on different sockets, cross-socket PCIe hops hurt — and PCIe AER correctable-error counts, since a rising rate precedes hard failures. My topology mapper automates exactly this check across all devices.

Q5: What's the difference between a metered and a switched PDU, and why do you care about per-outlet data? A: A metered PDU reports power draw (per-phase and/or per-outlet); a switched PDU can turn outlets on/off remotely; metered+switched does both, which is the AI-rack norm. I care about per-outlet metering because it's the ground truth for the power budget (Phase 01) — nameplate is a guess, the outlet is reality — and it lets me attribute power to specific devices/tenants and spot anomalies. Switching matters operationally: if a device and its BMC are both wedged, a switched outlet is my last-resort hard power-cycle. But outlet switching is physical and disruptive, so I guard it with dry-run, confirmation, blast-radius display, and read-back.

Q6 (Staff): You're integrating a new accelerator platform with an ODM. What hardware-interface issues do you anticipate and how do you de-risk them? A: I expect variance and gaps: incomplete/incorrect Redfish sensor mappings (the sensor→hwmon→D-Bus→bmcweb chain has bugs), PDU/CDU protocol differences (Modbus vs SNMP vs Redfish), PCIe topology surprises (lane allocation, degraded links under load, hot-plug quirks), and firmware-version-dependent behavior. To de-risk: agree a precise interface spec and software EBOM up front (Phase 12) — exact sensor list, Redfish schema coverage, OEM extensions, error semantics; build a conformance harness (Phase 11) that runs against their emulator/early hardware in CI so deviations are caught before deployment; instrument PCIe link state and AER from day one so degraded-link/thermal issues are visible during bring-up; and establish a tight defect-feedback loop with the ODM (Pegatron and peers), including upstreaming OpenBMC fixes where applicable. The abstraction layer (one model, per- class drivers) means their quirks stay contained while we drive them toward conformance.

References

  • OpenBMC project & architecture (D-Bus, phosphor, bmcweb) — https://github.com/openbmc/openbmc , https://github.com/openbmc/docs
  • ASPeed AST2600 BMC SoC overview — vendor datasheet / OpenBMC hardware docs
  • DMTF Redfish PDU/Outlet & CoolingUnit/CoolingLoop schemas — https://redfish.dmtf.org/schemas/
  • PCI-SIG — PCI Express Base Specification (lanes, links, AER); link speeds by generation
  • Linux PCIe: lspci(8), sysfs PCI docs (/sys/bus/pci/devices), PCIe AER kernel docs — https://docs.kernel.org/PCI/
  • Open Compute Project — Open Rack & power/busbar design — https://www.opencompute.org/
  • Modbus (for CDUs) — https://modbus.org/specs.php
  • Cross-track: Head of SWE — GPU, Phase 04 (Linux drivers/containers) — the driver side of these devices