Lab 02 — PDU/CDU Telemetry Simulator + PCIe Topology Mapper

Phase: 04 — Hardware Interfacing | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–6 hours Language: Python 3 (stdlib only) | Hardware: none (your own lspci optional)

Concept primer: ../WARMUP.md Ch. 4 (PDU), Ch. 5 (CDU), Ch. 6–7 (PCIe), ../HITCHHIKERS-GUIDE.md §4–§7.

Run

python3 solution.py                          # PDU + CDU telemetry + PCIe topology demo
lspci -mm -nn | python3 solution.py --lspci  # parse YOUR machine's PCIe devices (Linux)

0. The mission

Three tools that turn the power/cooling/fabric layer into software: a PDU meter (feeds the Phase 01 budget), a CDU monitor that alarms on the pre-failure signals (ΔT rising, low flow, lost pump) and on a safety-critical leak, and a PCIe topology mapper that finds the silent accelerator-perf killers — degraded links and AER errors.

1. The three lessons

  • PDU: per-outlet draw → rack total → compare to the 80%-derated capacity. This is the measured truth nameplate can't give you (WARMUP Ch. 4; ties to Phase 01 Lab 02).
  • CDU: ΔT = return − supply; ΔT rising at constant load → flow/fouling warning (the early signal), low flow → critical, leak → dominate-everything safety alarm (WARMUP Ch. 5). Read the three samples (healthy/degrading/leaking) and the alarms each raises.
  • PCIe: a device is degraded when LnkSta < LnkCap (negotiated width/speed below capability). The demo fabric has Accelerator 5 at x4 instead of x16 — 16 GB/s where it should be 64 — and Accelerator 2 with high AER correctable errors. Both "look healthy" but aren't (WARMUP Ch. 6–7).

2. The output to internalize

The PCIe tree prints with a <-- DEGRADED flag showing the bandwidth you're actually getting vs capable, and a <-- AER flag for error-prone links. This is the tool that turns "the accelerator is mysteriously slow" into a 30-second diagnosis.

3. Extensions

  1. Read your real machine's link state: parse /sys/bus/pci/devices/<BDF>/ current_link_width/max_link_width (and _speed) to detect degraded links on actual hardware, and aer_dev_correctable/aer_dev_fatal for errors.
  2. NUMA adjacency: read numa_node and report GPU/NIC pairs that are cross-socket (a scheduling input for Phase 06).
  3. Feed Phase 01: pipe PDU draw into the Phase 01 budget calculator as live "measured" data and turn it into a headroom monitor.
  4. Emit Prometheus: render PDU/CDU/PCIe metrics in exposition format (cdu_delta_t_celsius, pcie_link_width{bdf=...}) — the Phase 07 bridge.
  5. Build the full lspci -t parser: infer parent/child from bus numbers to construct the real tree from your machine.

4. Common pitfalls

  1. Trusting "healthy" — a degraded x16→x4 link and a thermally-throttled accelerator both report healthy. Check link state and ΔT explicitly.
  2. Treating leak like any alarm — it's safety-critical; it must dominate and (per policy) trigger emergency action.
  3. Nameplate budgeting — use measured PDU draw, not nameplate (Phase 01).
  4. Ignoring correctable AER — correctable errors are "recovered," but a rising rate predicts a hard failure (Phase 08 RAS).

5. What this lab proves

You can turn the physical power/cooling/fabric layer into actionable software signals and catch the failures that are invisible to naive health checks. "An accelerator passes health checks but is 4× slow — find it" becomes a confident walk: check link width/speed, ΔT, and power capping — the exact tools you just built. These feed Phase 07 dashboards, Phase 08 RAS, and the Phase 13 capstone.