🛸 Hitchhiker's Guide — Phase 04: Hardware Interfacing

Read this if: you can call Redfish/IPMI (Phase 03) but the devices behind them — BMC, PDU, CDU, PCIe switch — are still abstractions. This is the compressed device tour.

0. The 30-second mental model

Four device classes you manage: the BMC (control brain), the PDU (power truth), the CDU (cooling truth + a safety system), and the PCIe fabric (how everything connects and where silent perf bugs hide). Each is a managed device with telemetry and a control surface; each slots into the Phase 01 graph and the Phase 03 driver abstraction. One rule that earns respect: physical actions need guards (dry-run, interlocks, read-back).

1. BMC internals cheat sheet

Host CPU  --PECI-->   BMC (ARM SoC, standby power, OpenBMC)
DIMMs/VRM --I²C/PMBus--> BMC  --renders--> Redfish/IPMI
FRU EEPROM --I²C-->   BMC (inventory)
Host OS   --KCS/eSPI--> BMC (in-band IPMI)
Shared NIC --NC-SI-->  BMC (mgmt traffic)

OpenBMC chain for a sensor: hwmon → phosphor-hwmon → D-Bus → bmcweb → Redfish JSON. Debug a bad sensor by walking that chain.

2. BMC power state machine (don't confuse these)

On · ForceOff (yank) · GracefulShutdown (asks OS — fails silently if host hung) · ForceRestart · GracefulRestart · PowerCycle. Robust tooling: try graceful → wait → escalate to force → read back state to confirm.

3. The killer BMC feature: virtual media

Mount ISO over Redfish → set next-boot = virtual CD → power-cycle → host boots installer → watch over SoL → eject. This is how you reinstall a server you can't touch. Backbone of Phase 05 provisioning's rescue path.

4. PDU in one line

Metered = reads draw (per-outlet truth for the Phase 01 budget); Switched = toggles outlets (last-resort hard power-cycle); read per-phase too (balance). Switching = physical = guard it.

5. CDU: the telemetry and what it means

Signal	Healthy	What a bad value means
ΔT (return−supply)	stable for the load	rising at constant load → flow loss/fouling (early warning)
flow (L/min)	steady	drop → failing pump / blockage
pressure	in range	drop → leak; spike → blockage
pump status	N+1	lost spare → critical
leak sensor	dry	safety-critical: alarm + emergency action

Treat leak like a fire alarm in code: top priority, fail-safe, interlock the power-on.

6. PCIe in five facts

lane = 1 serial pair; link = x1/x4/x8/x16; Gen5 ≈ 4 GB/s/lane → Gen5 x16 ≈ 64 GB/s.
tree: Root Complex → switches → endpoints (accelerators/NICs/NVMe). Switches enable P2P (GPUDirect).
BDF = Bus:Device.Function (0000:3b:00.0) — the device coordinate.
degraded link = LnkSta < LnkCap (x16→x4 or Gen5→Gen1): device "healthy", bandwidth collapsed → the silent perf killer. Check it always.
AER: correctable (recovered, but rising rate = pre-failure) vs uncorrectable (fatal).

7. Reading PCIe on Linux (the commands)

lspci -tv                       # the tree
lspci -vvv | grep -A2 LnkSta    # negotiated vs capable width/speed
cat /sys/bus/pci/devices/<BDF>/current_link_width   # vs max_link_width
cat /sys/bus/pci/devices/<BDF>/numa_node            # GPU/NIC adjacency
cat /sys/bus/pci/devices/<BDF>/aer_dev_correctable  # error counts

Software Engineer — Rack Management (Senior / Staff)