« All Roles

Software Engineer — Rack Management (Senior / Staff) — Complete Apprenticeship Curriculum

Target Roles:

• Senior / Staff Software Engineer — Rack & Data Center Management Software (Qualcomm, NVIDIA, AMD, Microsoft Azure, Meta, Google, ODMs)
• Infrastructure Software Engineer — AI accelerator racks, bare-metal fleets, BMC/Redfish control planes
• Platform Engineer — provisioning, lifecycle, and observability for rack-scale AI systems
• Data Center Software Engineer — out-of-band management, firmware orchestration, telemetry pipelines
• SRE / Reliability Engineer — close-to-hardware infrastructure software in production

Duration: 26–34 weeks core — extendable to 12 months for full Staff-level readiness Goal: Build the rare profile this JD demands — the ability to write high-quality, secure, close-to-hardware infrastructure software that provisions, monitors, orchestrates, and maintains rack-scale AI accelerator systems; to speak the management protocols (Redfish/IPMI/SNMP) and hardware (BMC/PDU/CDU/PCIe) fluently; to integrate with Kubernetes/Terraform/Foreman and ship Prometheus/Grafana telemetry; and to debug and root-cause complex failures in deployed production data centers — then to lead that work and mentor others at the Staff level.

Why This Curriculum Exists

The "Rack Management Software Engineer" role at a hyperscale AI accelerator program is deceptively broad. The JD reads like five jobs:

1. Systems software engineering — production Python, C++, and shell that runs close to hardware: daemons, agents, drivers-adjacent code, written to be secure, testable, and maintainable.
2. Hardware/firmware interfacing — talking to BMCs, PDUs, CDUs, and PCIe switches over Redfish, IPMI, and SNMP; firmware updates; crash-dump parsing; remote diagnostics; RAS.
3. Infrastructure automation — bare-metal provisioning (Foreman/PXE), configuration management (Ansible/Chef/Puppet), infrastructure-as-code (Terraform), and orchestration (Kubernetes/RKE2).
4. Observability & operations — unified telemetry pipelines (Prometheus/Grafana), logging, metrics, tracing, runbooks, and SLOs across heterogeneous rack components.
5. Production debugging & ownership — root-cause analysis of complex issues in deployed systems, secure multi-tenant operation (secure boot, VLAN separation, QoS), and — at Staff — driving implementation, reviewing designs, and mentoring.

Most engineers know one or two of these. This curriculum is built backward from the JD so that every "required" and "preferred" line maps to phases and hands-on labs. The whole thing runs on a single laptop — every BMC, PDU, CDU, PCIe switch, and firmware flow is reproduced with open emulators and simulators, so you learn the mechanism without a data center.

Going beyond the JD (because interviews and the job do)

Real interviews and real on-call shifts go past the bullet list. This track deliberately adds:

• RAS depth — PCIe AER, MCE/MCA, ACPI/APEI, Xid-style error decoding (Phase 08) — because "crash-dump parsing" in practice means decoding hardware error logs.
• Systematic debugging & RCA as a first-class discipline with gdb/strace/perf/eBPF and written postmortems (Phase 10) — the JD's most-repeated theme.
• Reliability patterns — reconciliation loops, idempotency, circuit breakers, chaos/fault injection (Phase 10).
• Hardware-in-the-loop testing and emulator-based CI so infra code is actually testable (Phase 11).
• Staff-level leadership — design reviews, ADRs, EBOM, protocol specs, mentorship, and distributed-team communication (Phase 12).
• Fleet-scale system design — what changes when one rack becomes ten thousand (the entire system-design/ section).

What You Will Build

By the end you will have produced, with your own hands, a portfolio that is the job:

• A rack data model + inventory engine (Python) representing compute trays, accelerators, NICs, switches, PDUs, CDUs, and BMCs — with a power & thermal budget calculator that catches over-subscription before it trips a breaker (Phase 01)
• A C++ sensor-polling daemon (RAII, error handling, unit tests, systemd-ready) and a production-grade Python rack CLI (typed, packaged, retried, structured-logged) — the JD's language matrix, done right (Phase 02)
• A Redfish client that walks Chassis→Power→Thermal, performs resets, and subscribes to events against a real Redfish emulator — plus SNMP poller/trap receiver and IPMI sensor reader (Phase 03)
• A BMC interaction toolkit (power-cycle, sensor read, virtual media) and a PDU/CDU telemetry simulator + PCIe topology mapper that turns lspci into a fabric graph (Phase 04)
• A bare-metal provisioning flow (PXE/iPXE → kickstart → Ansible) and a Foreman/Terraform onboarding state machine that takes a rack from "powered on" to "ready" (Phase 05)
• A Kubernetes operator (a RackNode CRD with a reconciliation loop) and an accelerator device-plugin + scheduling walkthrough on RKE2 (Phase 06)
• A Redfish→Prometheus exporter, PromQL alert rules, and a Grafana dashboard for a rack — plus OpenTelemetry tracing across a provisioning workflow (Phase 07)
• A firmware-update orchestrator with staging, signature verification, A/B rollback — and a crash-dump / RAS log parser that classifies PCIe AER and MCE errors into an actionable diagnostics report (Phase 08)
• A VLAN + QoS tenant-isolation lab (network namespaces, tc) and a secure-boot / measured-boot + attestation simulator (TPM PCRs, signed firmware) (Phase 09)
• A debugging dojo: a planted bug in a rack daemon you find with gdb/strace/eBPF and write up — plus a chaos/fault-injection harness that hardens a reconciliation loop (Phase 10)
• A CI pipeline with emulated hardware (Redfish mock in CI, full test pyramid) and a fleet canary + rollback release system (Phase 11)
• A complete Staff deliverable pack — a protocol spec, a software EBOM, an ADR, a design-review checklist, a runbook, and a mentorship plan (Phase 12)
• A capstone: mini rack-management platform — discovery/inventory + Redfish/IPMI/SNMP drivers behind an abstraction + provisioning state machine + telemetry + firmware orchestration + secure multi-tenant config, with a design doc and a release cycle (Phase 13)

Folder Structure

Software Engineer Rack Management - Senior and Staff Engineer level/
├── README.md                                   ← You are here
├── jd.md                                       ← Target role profile
├── phase-01-rack-scale-systems-foundations/    ← What an AI rack IS: trays, accel, BMC, PDU, CDU, power/thermal
├── phase-02-production-systems-programming/     ← Python, C++, shell, Node.js — secure, testable, maintainable
├── phase-03-management-protocols-redfish-ipmi-snmp/ ← The out-of-band control plane: Redfish, IPMI, SNMP
├── phase-04-hardware-interfacing-bmc-pdu-cdu-pcie/   ← BMC, PDU, CDU (liquid cooling), PCIe switch fabric
├── phase-05-provisioning-config-management/    ← PXE/Foreman, Ansible/Chef/Puppet, Terraform — bare metal to ready
├── phase-06-orchestration-kubernetes-rke2/     ← K8s/RKE2, the operator pattern, device plugins, scheduling
├── phase-07-telemetry-observability/           ← Prometheus, Grafana, logging, metrics, tracing, unified pipelines
├── phase-08-firmware-diagnostics-ras/          ← Firmware update, crash dumps, RAS (PCIe AER/MCE/APEI), remote diag
├── phase-09-networking-secureboot-multitenancy/← Mgmt vs data plane, VLAN isolation, QoS, secure/measured boot, TPM
├── phase-10-reliability-debugging-rca/         ← Systematic debugging, gdb/strace/perf/eBPF, RCA, chaos, reconciliation
├── phase-11-cicd-testing-release-engineering/  ← Test pyramid + HIL, emulated-hardware CI, canary, rollback
├── phase-12-collaboration-ebom-leadership/     ← EBOM, protocol docs, ADRs, design reviews, Staff mentorship
├── phase-13-capstone-rack-management-platform/ ← Integrate everything into a mini rack manager + release cycle
├── interview-prep/                             ← Concepts, coding, protocols, debugging, behavioral/Staff
└── system-design/                              ← Control plane, telemetry, firmware-at-fleet, tenancy, provisioning

26-Week Core Schedule

Phase Structure

Every phase follows a consistent format:

Labs deliberately span the JD's language matrix: Python (clients, exporters, simulators, operators, parsers), C++ (the sensor daemon, close-to-hardware code), shell (provisioning glue, CI), and Node.js (a telemetry/tooling service — the JD's "plus"). Where the JD names a tool (Redfish, IPMI, SNMP, Foreman, Ansible, Terraform, Kubernetes, RKE2, Prometheus, Grafana), you use a real, open implementation of it.

No data center? You can still do 100% of this track

Every lab runs on a single Linux/macOS laptop using open emulators:

1. Redfish — DMTF's Redfish-Mockup-Server and the OpenBMC sushy-emulator give you a real Redfish service.
2. IPMI/BMC — a software BMC simulator (or ipmi_sim from OpenIPMI) plus a provided mock.
3. SNMP — snmpsim serves real MIBs; a local trap receiver completes the loop.
4. Kubernetes/RKE2 — kind/k3d locally; RKE2 differences are explained in the WARMUP.
5. Provisioning — PXE/iPXE flows run against local VMs/QEMU; the rest is simulated with the same data model.
6. PDU/CDU/PCIe — provided simulators emit realistic telemetry; lspci topology mapping uses your own machine.

The WARMUPs always explain what changes at real rack and fleet scale so you're not surprised on the job.

Prerequisites

• Solid Python; working C or C++ (you can read pointers, structs, headers, and a Makefile)
• Comfort in a Linux shell (you'll write scripts and use systemctl, journalctl, ip, lspci, gdb, strace)
• Basic networking literacy (IP, VLAN, DNS/DHCP) and willingness to go deeper — Phase 09 starts from fundamentals
• Git, Docker basics; a Linux VM or WSL2 if you're on Windows
• No prior BMC/Redfish/firmware experience required — Phases 01 and 03 start at the hardware and the protocol

Related Tracks in This Repo

• Head of Software Engineering — GPU — the silicon/CUDA/driver side of the same racks; its Phases 04/06 pair with our 04/06
• Security Engineer — deeper offensive/defensive security; pairs with our Phase 09 (secure boot, tenancy)
• AI Specialist — inference/accelerator workloads that run on the racks you manage
• LLM Inference Engineer — what the accelerators actually run, for context on workload behavior

Job Area: Engineering Group, Engineering Group > Software Engineering General Summary:

About Us

Qualcomm is growing its presence in Riyadh and is hiring Data Centre Engineers to support our expanding infrastructure across the region. As Saudi Arabia accelerates its digital transformation under Vision 2030, Qualcomm is investing in world‑class computing and data centre capabilities to power AI, cloud, and advanced connectivity at scale. This is a unique opportunity to work in a fast‑growing technology hub, supporting critical environments and helping shape the future of data centre operations in the Kingdom and beyond.

About the Role

Qualcomm is seeking Senior and Staff Software Engineers to contribute to the development and operation of rack management software for AI accelerator products deployed in large-scale data center environments. This role is hands-on and execution-focused, emphasizing high-quality implementation, debugging, testing, and operational support of infrastructure software that runs close to hardware.

You will collaborate closely with lead engineers and architects across multiple sites, participate in design discussions and reviews, and take on design-ownership roles, while focusing primarily on implementing defined designs, investigating complex issues, and ensuring reliable, secure software behaviour in production environments.

In this role, you will work on software responsible for provisioning, monitoring, orchestration, and lifecycle management of rack‑scale AI systems. You will be expected to take strong ownership of the code you deliver — from implementation and validation through deployment and debugging in real‑world data center conditions.

This position is well‑suited for a senior engineer who thrives in a distributed team model, communicates effectively across sites, and takes pride in deep technical debugging, system understanding, and operational excellence, even when architectural direction is defined elsewhere.

At the Staff level, the engineer is expected to drive implementation of the designs assigned to the local engineering team and provide guidance and mentorship to junior engineers.

Key Responsibilities will include

Implement, test, and maintain software solutions for rack provisioning, monitoring, and workload deployment Interface with rack hardware (BMC, PDU, CDU, PCIe switches) using Redfish, SNMP, and IPMI protocols. Integrate rack management software with orchestration platforms (e.g., Kubernetes, Terraform, Foreman). Collaborate with ODMs, ISVs, and internal teams to define and deliver scalable rack software solutions. Develop dashboards and unified telemetry pipelines using Prometheus and Grafana. Support firmware updates, crash dump parsing, and remote diagnostics across rack components. Ensure secure boot, VLAN separation, and QoS enforcement for multi-tenant environments. Contribute to software EBOM and protocol documentation with Pegatron and other partners. Write high‑quality, secure, and maintainable code following established coding standards and best practices. Participate in design discussions and reviews, providing implementation‑focused feedback on feasibility, performance, reliability, and maintainability. Own debugging and root‑cause analysis of complex issues across development, integration, and production environments, including deployed systems. Collaborate closely with architects, lead engineers, hardware, firmware, and platform teams to ensure correct system behavior and smooth integration. Implement observability features such as logging, metrics, and tracing to support troubleshooting and operational reliability. Contribute towards documentation, runbooks, and operational procedures related to the software you own or support. Participate in code reviews, testing efforts, and continuous improvement of build, test, and deployment pipelines. Required Qualifications

Bachelor’s or Master’s degree in Electrical or Computer engineering, Computer Science, Information Systems, or a related field and 5+ years of software engineering experience Experience in infrastructure software development Proficiency in Python, C++, and shell scripting. Experience with Redfish APIs and DCIM tools. Understanding of server hardware, networking, and storage systems. Strong written and verbal communication skills for working effectively in a multi‑site, globally distributed team. Preferred Qualifications

Experience with Node.js is a plus. Experience with Rack Management tools. Knowledge of AI workload orchestration tools (e.g., K8s, RKE2). Experience with tools like Prometheus and Grafana. Familiarity with rack-level lifecycle management. Familiarity with Foreman, Ansible, Chef, and Puppet for provisioning and configuration What's on Offer

Apart from working with great people, we offer the below:

Salary including housing & transport allowance

Stock (RSU's) and performance related bonus

16 weeks fully paid Maternity Leave

6 weeks fully paid Paternity Leave

Employee stock purchase scheme

Child Education Allowance

Relocation and immigration support (if needed)

Life and Medical Insurance

Live+ Well Reimbursement for health and recreational membership fees

Minimum Qualifications:

• Bachelor's degree in Engineering, Information Systems, Computer Science, or related field and 4+ years of Software Engineering or related work experience. OR Master's degree in Engineering, Information Systems, Computer Science, or related field and 3+ years of Software Engineering or related work experience. OR PhD in Engineering, Information Systems, Computer Science, or related field and 2+ years of Software Engineering or related work experience.

• 2+ years of work experience with Programming Language such as C, C++, Java, Python, etc. *References to a particular number of years experience are for indicative purposes only. Applications from candidates with equivalent experience will be considered, provided that the candidate can demonstrate an ability to fulfill the principal duties of the role and possesses the required competencies.

Qualcomm is an equal opportunity employer. If you are an individual with a disability and need an accommodation during the application/hiring process, rest assured that Qualcomm is committed to providing an accessible process. You may e-mail disability-accomodations@qualcomm.com or call Qualcomm's toll-free number found here. Upon request, Qualcomm will provide reasonable accommodations to support individuals with disabilities to be able participate in the hiring process. Qualcomm is also committed to making our workplace accessible for individuals with disabilities. (Keep in mind that this email address is used to provide reasonable accommodations for individuals with disabilities. We will not respond here to requests for updates on applications or resume inquiries).

Qualcomm expects its employees to abide by all applicable policies and procedures, including but not limited to security and other requirements regarding protection of Company confidential information and other confidential and/or proprietary information, to the extent those requirements are permissible under applicable law.

To all Staffing and Recruiting Agencies: Our Careers Site is only for individuals seeking a job at Qualcomm. Staffing and recruiting agencies and individuals being represented by an agency are not authorized to use this site or to submit profiles, applications or resumes, and any such submissions will be considered unsolicited. Qualcomm does not accept unsolicited resumes or applications from agencies. Please do not forward resumes to our jobs alias, Qualcomm employees or any other company location. Qualcomm is not responsible for any fees related to unsolicited resumes/applications.

If you would like more information about this role, please contact Qualcomm Careers.

https://careers.qualcomm.com/careers/job/446718277935?hl=en-US&utm_source=linkedin&domain=qualcomm.com&source=APPLICANT_SOURCE-6-2

Phase 01 — Rack-Scale AI Systems & Data Center Foundations

Difficulty: ⭐⭐☆☆☆ | Estimated Time: 2 weeks Roles supported: Rack Management SWE (Senior/Staff), Data Center SWE, Infra/Platform Engineer Hardware needed: none — both labs run on any laptop

Why This Phase Exists

You cannot write software that manages a thing you cannot picture. Before a single Redfish call or Ansible play, you must be able to draw an AI rack from memory — every box, every cable, every power and cooling path — and reason about it as a system with budgets, failure domains, and a lifecycle. This is the mental model that the rest of the curriculum populates with code.

The JD's first line is "rack management software for AI accelerator products deployed in large-scale data center environments." That sentence hides an enormous amount of physical reality: an AI rack today can draw 40–130+ kW (a traditional enterprise rack drew 5–10 kW), is often liquid-cooled, contains its own management network and dozens of independently-firmwared components, and is the unit that gets provisioned, monitored, and maintained as a whole. Get the physical model wrong and every abstraction above it leaks.

This phase builds that base — not trivia, mechanism and numbers. By the end you can explain why an AI rack needs a CDU, why management traffic rides a separate network, what a "failure domain" means when a single PDU feeds eight nodes, and what fields your inventory database absolutely must have. You will encode that understanding as two pieces of working software: a rack data model and a power/thermal budget calculator.

Concepts

• What "rack-scale" means: the rack as the unit of deployment, sale, and management for AI systems; scale-up vs scale-out
• Rack anatomy: compute trays/nodes, AI accelerators, host CPUs, NICs/DPUs, scale-up fabric (NVLink/NVSwitch-class), top-of-rack (ToR) data switches, management switch, PDUs, busbar, PSU shelves, CDU
• The management plane: BMC (baseboard management controller) as "the computer inside the computer"; out-of-band (OOB) vs in-band; why they are physically separate networks
• Power: 3-phase power, PDUs and busbars, PSU shelves/redundancy (N+1, 2N), rack power density, power budgeting and capping, the difference between nameplate, rated, and measured power
• Cooling: air vs direct-liquid-cooling (DLC), CDU (coolant distribution unit), rear-door heat exchangers, thermal budgets, ΔT, PUE; why AI density forced the shift to liquid
• Units & physical facts: rack U / OpenRack OU, dimensions, weight, BTU/hr, kW↔BTU conversion
• Reference designs: OCP (Open Compute Project) & Open Rack, NVIDIA MGX / HGX / GB200 NVL72, DGX, hyperscaler in-house designs — and where Qualcomm AI accelerators fit
• Failure domains & blast radius: power domain, cooling domain, network domain, the rack as a correlated-failure unit; redundancy vs isolation
• The rack lifecycle: design → ODM build (Pegatron and peers) → integration/L10–L11 → ship → install → commission/burn-in → provision → operate → maintain/RMA → decommission
• The ecosystem: OEM vs ODM vs ISV; the software EBOM; DCIM (data center infrastructure management) and where rack management software sits relative to it
• What rack management software must model: inventory/asset schema, identity, relationships (contains/powers/cools/connects), state, and telemetry

Labs

Lab 01 — Rack Data Model & Inventory Engine (Python)

Lab 02 — Power & Thermal Budget Calculator (Python)

Deliverables Checklist

• You can draw an AI rack from memory: trays, accelerators, fabric, ToR, mgmt switch, PDU/busbar, PSU, CDU, BMC
• You can explain OOB vs in-band and why the management network is physically separate
• Inventory engine builds a realistic rack and reports invariant violations
• You can compute a rack's power budget with the 80% breaker rule and N+1 redundancy in your head for simple cases
• Budget calculator flags over-subscription and redundancy loss with reasons
• You can name the rack lifecycle stages and where rack-management software acts in each
• You can do the kW→BTU/hr conversion and explain why AI density forced liquid cooling

Interview Relevance

• "Walk me through the components of an AI rack and how they're powered and cooled."
• "What's the difference between the management network and the data network, and why does it matter for your software?"
• "A rack has two 3-phase 30A feeds. How much IT load can you safely put on it, and why not 100%?"
• "Design the data model for a rack inventory system. What are the entities and relationships?"
• "What's a failure domain? Give an example in a rack and how your software should treat it."
• "Why are AI racks liquid-cooled when enterprise racks weren't?"

Warmup Guide — Rack-Scale AI Systems & Data Center Foundations

Zero-to-expert primer for Phase 01. You need no data-center background: this guide starts at "what is a rack unit" and ends with you sizing power and cooling for a 120 kW AI rack and explaining why your management software needs a separate network. Every later phase — Redfish, BMC, provisioning, telemetry, firmware — stands on this physical model.

Table of Contents

• Chapter 1: Why "Rack-Scale" Is the Unit That Matters
• Chapter 2: Anatomy of an AI Rack, Box by Box
• Chapter 3: The Management Plane — BMC, OOB vs In-Band
• Chapter 4: Power — From the Feed to the Chip
• Chapter 5: Cooling — Why AI Racks Went Liquid
• Chapter 6: Networks — Three of Them, Not One
• Chapter 7: Failure Domains & Blast Radius
• Chapter 8: Reference Designs — OCP, MGX, NVL72, and Where Qualcomm Fits
• Chapter 9: The Rack Lifecycle and the Ecosystem (OEM/ODM/ISV)
• Chapter 10: What Rack-Management Software Must Model
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Why "Rack-Scale" Is the Unit That Matters

Zero background: a rack is a standardized steel cabinet (19-inch wide mounting rails, historically ~42U tall, where 1U = 1.75 inches) that holds servers and networking gear. For thirty years the server was the unit of everything — you bought a server, provisioned a server, monitored a server. The rack was just furniture.

The shift: AI training and inference need many accelerators wired together with very fast links (a "scale-up" fabric like NVLink/NVSwitch). Those links only reach a few meters at full bandwidth, so the natural boundary is one rack. Inside a rack, accelerators talk at ~hundreds of GB/s to TB/s; between racks, they drop to network speeds (tens of GB/s). That bandwidth cliff makes the rack the natural unit of design, sale, deployment, and management. NVIDIA sells a GB200 NVL72 as a rack. Hyperscalers deploy and retire capacity in racks.

Two words you must separate:

• Scale-up = make one coherent machine bigger by tightly coupling accelerators (NVLink fabric inside a rack). Latency/bandwidth-sensitive; bounded by physics to ~a rack.
• Scale-out = add more independent machines connected by a network (Ethernet/InfiniBand across racks). This is the data-center-wide dimension.

Why this matters for your software: rack-management software treats the rack as a managed object with sub-components, a power/cooling budget, a single management network, and a correlated failure profile. Your inventory model (Lab 01), your provisioning unit (Phase 05), and your blast-radius reasoning (Chapter 7) all key off "the rack." When the JD says "lifecycle management of rack-scale AI systems," rack-scale is the literal scope of the object you manage.

Common misconception: "a rack is just N servers, manage each server." For AI, the rack has shared resources (power shelf, busbar, CDU, NVSwitch trays, ToR) that belong to no single server and must be managed as rack-level objects. Miss them and you can't model power, cooling, or the scale-up fabric at all.

Chapter 2: Anatomy of an AI Rack, Box by Box

Picture a modern liquid-cooled AI rack from top to bottom. Memorize this list — Lab 01 turns it into a data model.

• Compute trays / nodes (the bulk of the rack): each tray holds host CPUs, system memory, AI accelerators (GPUs / Qualcomm Cloud AI / TPUs / custom ASICs), local NVMe, and NICs. A tray is one or more "nodes" (a node = one OS instance = one BMC, usually).
• AI accelerators: the reason the rack exists. 4–8 per tray is common. Each has its own high-bandwidth memory (HBM), firmware, and thermal profile, and is the hottest, most expensive, most-monitored component.
• Host CPU(s): run the OS, the in-band agents, and feed the accelerators. 1–2 sockets per node.
• NICs / DPUs: the data-plane network interface. AI nodes often have multiple high-speed NICs (one per accelerator for RDMA) plus a separate management NIC. A DPU (SmartNIC) is a NIC with its own CPU/OS that can offload networking, storage, and security — itself a managed device with firmware.
• Scale-up fabric: NVSwitch trays (or vendor equivalent) that wire accelerators into one coherent fabric. In an NVL72 these are dedicated trays with their own management.
• Top-of-Rack (ToR) data switch: the scale-out Ethernet/InfiniBand switch connecting this rack's nodes to the rest of the data center.
• Management switch: a separate, smaller switch carrying only out-of-band management traffic (BMCs, PDUs, CDU controller) — see Chapter 3/6.
• Power shelf / PSUs: AC→DC conversion. In OCP racks, a power shelf feeds a vertical DC busbar that runs the height of the rack; nodes tap the busbar with blind-mate connectors. In legacy racks, each server has its own PSUs fed by PDUs (power distribution units — essentially smart power strips).
• PDU(s): distribute and (if "smart"/metered/switched) measure and control power per outlet or per phase. A managed device you talk to over SNMP/Redfish (Phase 03/04).
• CDU (Coolant Distribution Unit): for liquid cooling — pumps coolant through the rack's cold plates, exchanges heat with facility water, and reports flow/temperature/pressure. A managed device (Chapter 5).
• BMC per node (and often a rack manager / RMC): the always-on management microcontrollers (Chapter 3).

A useful sanity image: an NVL72 rack ≈ 18 compute trays (2 Blackwell GPUs + 1 Grace CPU each → 72 GPUs) + 9 NVSwitch trays + power shelves + busbar + manifold/CDU plumbing + ToR + management switch. ~1.36 metric tons, ~120 kW. That is "rack-scale."

Chapter 3: The Management Plane — BMC, OOB vs In-Band

This is the single most important concept for your job, so go slow.

The BMC (Baseboard Management Controller) is a small, independent computer soldered onto the server's motherboard — typically an ARM SoC (e.g., ASPeed AST2600) running its own Linux (often OpenBMC) with its own memory, its own network port, and its own power rail (standby power). It is the computer inside the computer. It runs even when the host is powered off (as long as the rack has standby power).

What the BMC does:

• Power control: power on/off/cycle the host, regardless of the host OS state.
• Sensors: read temperatures, voltages, fan speeds, power draw — independent of the host.
• Console: Serial-over-LAN (SoL) and KVM (remote keyboard/video/mouse) so you can watch the host boot.
• Virtual media: mount an ISO over the network so you can reinstall the OS remotely.
• Logs: the System Event Log (SEL) and crash/RAS captures (Phase 08).
• Firmware: it can update host firmware (BIOS/UEFI) and its own.

Out-of-band (OOB) vs in-band — the distinction your whole control plane rests on:

• In-band = management that goes through the host OS and its main NIC: SSH to the host, an agent reading /proc, the data NIC. Works only when the host is up and healthy.
• Out-of-band = management that goes through the BMC over the management network, independent of the host OS. Works when the host is off, hung, kernel-panicked, or mid-reinstall.

Why OOB exists / why you cannot live without it: the moments you most need to manage a machine are exactly the moments it's not healthy — it won't boot, it kernel-panicked, it needs an OS reinstall. In-band management is useless then. OOB (Redfish/IPMI to the BMC, Chapter 6 / Phase 03) is how you power-cycle a wedged node, watch why it won't POST, and reinstall it — from a continent away. The JD's "remote diagnostics," "firmware updates," and "support of deployed systems" all live on the OOB plane.

Mental model: every node is really two computers on two networks — the host (in-band, data network) and the BMC (out-of-band, management network). Rack management software spends most of its time talking to the second one.

Misconception: "the BMC is part of the OS." No — it's separate silicon with separate firmware and power. You can redfish GET a node's sensors while its host OS is completely dead. That independence is the entire point.

Chapter 4: Power — From the Feed to the Chip

Power is where rack-scale AI breaks everything that worked at enterprise scale. You must be able to do this math (Lab 02).

The path: utility → data hall → 3-phase AC to the rack → PDU/power-shelf → PSUs (AC→DC) → busbar/cables → node → VRMs → chip.

3-phase power, minimally: data centers distribute 3-phase AC because it carries more power per conductor. A common rack feed is, e.g., 3-phase 415V 30A. Usable power ≈ √3 × V_line-to-line × I × power-factor ≈ 1.732 × 415 × 30 × ~1 ≈ 21.6 kW per feed (regional voltages vary; the point is the formula). Two feeds for redundancy → ~43 kW available, but see the 80% rule.

The 80% rule (NEC continuous-load derating): a breaker rated for continuous load may only be loaded to 80% of its rating. A 30A circuit → budget to 24A. This is not optional; exceed it and the breaker trips (an outage) or you create a fire risk. Every power-budget calculation a rack manager does derates by 0.8. Lab 02 enforces this.

Redundancy schemes (memorize):

• N: just enough capacity, no spare. One failure = outage.
• N+1: one spare unit; survives a single PSU/feed failure. The common default.
• 2N: fully duplicated (two independent feeds, each able to carry the whole load). Survives a whole side failing. Used for the most critical gear.

The trap: in N+1, the load must fit on N units after one fails. If you fill the rack assuming N+1 capacity, a single failure overloads the survivors and cascades. Lab 02's redundancy check is exactly this.

Power density, then and now:

• Enterprise rack (2010s): 5–10 kW.
• Dense compute: 15–20 kW.
• AI air-cooled: 30–40 kW (near the limit of what air can remove).
• AI liquid-cooled (NVL72-class): 60–130+ kW per rack, heading toward 250 kW–1 MW in next-gen designs.

That ~10× jump is the reason this role exists: at 120 kW, power and cooling are first-class software-managed budgets, not facility afterthoughts.

Nameplate vs rated vs measured:

• Nameplate (TDP/max): the worst-case the component could draw. Sizing breakers naively to nameplate strands huge capacity.
• Rated/derated: what you provision for after derating.
• Measured (telemetry): actual draw, read from PDU/BMC (Phase 04/07). Almost always well below nameplate.

Power capping: the BMC/accelerator can be told to cap power (clock/voltage throttle) to fit a budget — trading performance for thermal/power fit. Rack management software uses capping to (a) safely oversubscribe based on measured behavior and (b) ride out a cooling/power degradation without a hard outage. Lab 02's extension implements capping-to-fit.

Chapter 5: Cooling — Why AI Racks Went Liquid

Every watt of power becomes a watt of heat. A 120 kW rack produces 120 kW of heat that must leave the building, or the chips throttle and then die.

The conversion to know cold: 1 kW = 3,412 BTU/hr. A 120 kW rack dumps ~409,000 BTU/hr. For intuition: that's roughly the heat of ~34 home ovens running flat-out, in one cabinet.

Why air stops working: air has low heat capacity; removing 30–40 kW from a rack already requires gale-force airflow and chilled aisles. Beyond ~40 kW, air physically can't keep modern accelerators below their throttle temperature. So the industry moved to liquid.

Direct Liquid Cooling (DLC): coolant runs through cold plates bolted directly onto the hot chips (accelerators, sometimes CPUs/VRMs). Liquid carries ~3,500× the heat per volume of air. Variants: cold-plate (most common), rear-door heat exchangers (a radiator on the rack's back door), and full immersion (the whole board in dielectric fluid — rarer).

The CDU (Coolant Distribution Unit): the heart of a liquid-cooled rack. It:

• pumps the secondary (rack-side) coolant loop through the cold plates,
• exchanges heat with the facility's primary (chilled-water) loop via a heat exchanger (keeping the clean rack loop isolated from facility water),
• reports flow rate (L/min), supply/return temperatures, ΔT, pressure, pump speed, and leak detection.

That last bullet is yours: the CDU is a managed device (Redfish/Modbus/SNMP) your software monitors. A failing pump or a rising ΔT is an early warning of a thermal event; leak detection is a safety-critical alert that may trigger an emergency drain/shutdown.

Thermal budget math (Lab 02): heat removed = flow × fluid heat capacity × ΔT (supply vs return temperature). If component heat load exceeds the CDU's rated removal at safe ΔT, the rack overheats. Your calculator checks heat load against cooling capacity exactly as it checks power against breakers.

PUE (Power Usage Effectiveness): total facility power ÷ IT power. 1.0 is perfect (all power does compute). Air-cooled enterprise: ~1.5–2.0. Liquid-cooled AI halls push toward ~1.1. PUE is a facility KPI, but rack-level cooling efficiency feeds it — and "tokens per joule" / "performance per watt" is increasingly how AI infra is judged.

Misconception: "cooling is the facility team's problem." At 120 kW with liquid loops, cooling telemetry (CDU flow/ΔT/leak) is part of the rack's health that your software monitors and acts on. A leak or pump failure is a software-visible, software-actionable event.

Chapter 6: Networks — Three of Them, Not One

A common beginner mistake is imagining "the network." An AI rack has (at least) three logically separate networks, and conflating them is a security and reliability bug.

1. Data / compute network (in-band, scale-out): high-speed Ethernet or InfiniBand (e.g., 400/800 GbE, NDR IB) connecting nodes to the ToR and out to the cluster. Carries the actual AI traffic (gradients, activations, RDMA). Highest bandwidth, tenant-visible.
2. Scale-up fabric (intra-rack): NVLink/NVSwitch-class links wiring accelerators into one coherent memory fabric. Not "Ethernet"; managed as part of the rack's hardware.
3. Management network (out-of-band): a separate, modest (often 1 GbE) network connecting every BMC, PDU, CDU, and managed switch to the management/control plane. This is where Redfish/IPMI/SNMP live. It is physically and administratively isolated from tenant data.

Why management is separate (this is an interview favorite):

• Reliability: you must reach the BMC to fix a node even when its data network/host is down. If management rode the data network, a data-network outage would also blind your control plane — exactly when you need it.
• Security: BMCs are extraordinarily powerful (full power and firmware control) and historically under-hardened. Exposing them on the tenant network would be catastrophic. They live on an isolated management VLAN reachable only by the control plane (Phase 09).
• Blast-radius: separating planes means a problem in one doesn't take out the other.

The JD's "VLAN separation" (Phase 09) is, in large part, about keeping these planes — and different tenants within the data plane — isolated. Hold this thought; Phase 09 builds it for real.

Chapter 7: Failure Domains & Blast Radius

A failure domain is a set of components that fail together because they share something. The blast radius of a fault is everything its failure domain contains. Designing and operating racks is largely the art of knowing and bounding blast radius — and your inventory model (Lab 01) must encode it.

Failure domains in a rack:

• Power domain: everything fed by one PDU/feed/breaker. If a 24 kW feed dies and it powered 6 nodes without redundancy, those 6 nodes drop together. (N+1/2N exists to shrink this.)
• Cooling domain: everything cooled by one CDU/loop/manifold. A pump failure or leak can take the whole rack's liquid-cooled components thermal at once.
• Network domain: everything behind one ToR or one management switch. A ToR failure isolates the whole rack's data plane; a management-switch failure blinds your OOB control of the whole rack.
• The rack itself: shared roof, shared facility water valve, shared row PDU — correlated failures up the hierarchy (rack → row → hall → site).

Why this matters operationally: workload placement and maintenance must respect domains. You don't put all replicas of a service on nodes sharing one PDU. You drain a whole power domain before maintenance on its feed. When something fails, the first diagnostic question is "what's the failure domain?" — a single node, or a shared resource taking many nodes with it? Lab 01's find_blast_radius(component) query is the software form of this question, and it's the seed of correlation logic your telemetry (Phase 07) and RCA (Phase 10) will need.

Redundancy vs isolation (don't conflate): redundancy (N+1) lets a domain survive a failure; isolation (separate domains, VLANs, MIG-style partitioning) contains a failure or a tenant. You design for both.

Chapter 8: Reference Designs — OCP, MGX, NVL72, and Where Qualcomm Fits

You don't need to memorize every SKU, but a leader/engineer in this space speaks the reference-design vocabulary.

• OCP (Open Compute Project): the open-hardware community (started by Meta) that standardized hyperscale rack design: Open Rack (a wider 21-inch equipment space, 48 "OU"), the busbar/power-shelf model, and open management specs. Much of "modern rack" thinking is OCP-derived. Redfish, the OpenBMC firmware, and OCP's hardware management specs are the standards your software targets.
• NVIDIA HGX / DGX / MGX: HGX is the baseboard of 8 GPUs + NVSwitch; DGX is NVIDIA's complete server; MGX is a modular reference architecture OEMs/ODMs build to. GB200 NVL72 is the rack-scale design (72 Blackwell GPUs as one NVLink domain, liquid-cooled, ~120 kW) — the canonical "AI rack" today.
• Hyperscaler in-house: Google TPU pods, AWS Trainium/Inferentia racks, Meta's OCP-based designs, Microsoft Maia — each a rack-scale system with its own management software (the kind you'd build).
• Where Qualcomm fits (this JD): Qualcomm's data-center AI accelerators (the Cloud AI 100 line and its successors) deploy in rack-scale systems. This role is the software that provisions, monitors, and maintains those racks — interfacing with the BMCs, PDUs, CDUs, and PCIe switches of accelerator products, integrating with orchestration and telemetry, and co-developed with ODM partners (the JD names Pegatron). The accelerator is the payload; your software runs the rack around it.

The strategic insight: hardware vendors increasingly win or lose on the software that operates the hardware at scale. A great accelerator with a flaky rack-management stack is unsellable to a hyperscaler. That's why this role is senior/staff and why the JD stresses production debugging and operational excellence.

Chapter 9: The Rack Lifecycle and the Ecosystem (OEM/ODM/ISV)

Rack management software acts across a rack's entire life. Know the stages and the players.

The ecosystem:

• OEM (Original Equipment Manufacturer): sells the branded system (e.g., NVIDIA, Dell, HPE — or, for accelerators, Qualcomm).
• ODM (Original Design Manufacturer): designs/builds the hardware, often white-label, for OEMs and hyperscalers. Pegatron (named in the JD), Quanta, Wiwynn, Foxconn are ODMs. You co-define interfaces, the software EBOM, and protocol behavior with them (Phase 12).
• ISV (Independent Software Vendor): third-party software (orchestrators, DCIM, monitoring) you integrate with.

The lifecycle (and where your software acts):

1. Design / NPI (New Product Introduction): define hardware + the management interfaces (Redfish schemas, sensor lists, the EBOM). Staff engineers contribute here (Phase 12).
2. Manufacture & integration (L6–L11): the ODM builds boards (L6), nodes (L10), and integrates the full rack (L11), running factory tests. Your software may provide the test/validation tooling.
3. Ship & install: rack arrives, is bolted down, powered, plumbed (liquid), and cabled.
4. Commission / burn-in: discovery (find every BMC/PDU/CDU on the management network), inventory capture, firmware baseline, stress/burn-in tests, validation against the EBOM. This is rack-management software's first big job.
5. Provision: install OS/firmware, configure networking, join the orchestrator (Phase 05/06).
6. Operate: monitor (Phase 07), enforce tenancy/QoS (Phase 09), serve workloads.
7. Maintain: firmware updates (Phase 08), diagnostics/RCA (Phase 08/10), component RMA (replace a failed accelerator/PSU/pump and re-validate).
8. Decommission: secure wipe, asset retirement, update inventory.

Why a Staff engineer cares about the whole arc: bugs and design gaps surface at the seams between stages (e.g., a firmware baseline mismatch found only at burn-in). Owning "implementation through deployment and debugging in real-world conditions" (the JD's words) means your code and your mental model span this entire lifecycle.

Chapter 10: What Rack-Management Software Must Model

Synthesize the above into the thing you build first (Lab 01): the model. Get this right and everything composes; get it wrong and every feature fights the schema.

Entities (the nouns):

• Site → Hall → Row → Rack → Tray/Node → Component (accelerator, CPU, NIC/DPU, NVMe, PSU, fan, BMC) — a containment hierarchy.
• Shared rack-level objects: PDU, Busbar, PowerShelf, CDU, ToRSwitch, MgmtSwitch, NVSwitchTray.

Identity (how you name a thing, durably):

• Asset tag (your inventory ID), serial number (manufacturer), FRU (Field Replaceable Unit) data, MAC/UUID, and location (rack + U position). Critical distinction: identity ≠ location — a node keeps its serial when it moves U positions or racks; your model must track both and reconcile them (a swapped-but-not-updated location is a classic data-integrity bug).

Relationships (the edges — this is what makes it a graph, not a table):

• contains (rack contains node), powers (PDU powers node), cools (CDU cools node), connects (ToR connects node), manages (BMC manages node). These edges are what find_blast_radius traverses.

State:

• Power state (on/off/standby), health (ok/warning/critical), lifecycle state (discovered/provisioning/ready/maintenance/decommissioned), firmware versions per component.

Telemetry (time-series, attached to components): temperature, power, fan/pump speed, errors — fed by Phases 04/07.

Invariants (validation rules your model enforces — Lab 01):

• Every node has at least one power feed and (if liquid) a cooling path.
• Power draw ≤ derated budget (links to Lab 02).
• No two components claim the same U position.
• Every managed device has a reachable management address.

Declared vs discovered: your model holds the declared intent (the EBOM / what should be there) and the discovered reality (what the management network actually found). The diff between them is gold: missing components, firmware drift, mislabeled locations. This declared-vs-discovered reconciliation is the conceptual seed of the Kubernetes operator pattern in Phase 06 and the RCA work in Phase 10.

Lab Walkthrough Guidance

Order: Lab 01 → Lab 02. Lab 01 builds the model (the nouns and edges); Lab 02 reasons over it (the budgets). Don't invert — the calculator consumes the model.

Lab 01 (Rack inventory engine):

1. Read solution.py top-down before running; the dataclasses are Chapter 10. Predict what the validation report will flag.
2. Run it; check the rack it builds against your mental Chapter-2 picture.
3. Extensions in order: (a) add a CDU + cools edges (Chapter 5); (b) implement find_blast_radius(component) by traversing powers/cools/connects edges (Chapter 7); (c) JSON export shaped like Redfish's Chassis/Systems tree (foreshadows Phase 03).

Lab 02 (Power/thermal budget):

1. python solution.py — read the budget report for the populated rack.
2. Add nodes until you trip the 80% breaker rule; predict the failing feed before the tool tells you.
3. Extensions: (a) fail one PSU in an N+1 shelf and confirm the survivors stay within budget; fail two and watch it go critical; (b) apply per-node power capping to fit a rack that's 10% over budget — the performance/fit tradeoff (Chapter 4).

The capstone question for the phase: "A partner proposes 18 trays at 6.5 kW each in a rack with two 30A 415V feeds, N+1, liquid-cooled with a 130 kW CDU. Does it fit — power and thermal — and what's the blast radius of one feed failing?" If you can answer that with numbers, Phase 01 did its job.

Success Criteria

You're done with this phase when — without notes:

• You can draw an AI rack and name every box and which of the three networks each sits on (Ch. 2, 6)
• You can explain OOB vs in-band and give two reasons the management network is separate (Ch. 3, 6)
• You can size a 3-phase feed, apply the 80% rule, and check N+1 redundancy in your head (Ch. 4)
• You can convert kW→BTU/hr and explain why >40 kW forced liquid cooling and what a CDU does (Ch. 5)
• You can define a failure domain and compute a simple blast radius (Ch. 7)
• You can name the rack lifecycle stages and what rack-management software does in each (Ch. 9)
• Your inventory engine builds a valid rack, flags invariant violations, and answers a blast-radius query (Lab 01)
• Your budget calculator enforces derating + redundancy and supports capping-to-fit (Lab 02)

Interview Q&A

Q1: Walk me through the components of an AI rack and how it's powered and cooled. A: Top-level, the rack holds compute trays (each with host CPUs, 4–8 AI accelerators with HBM, local NVMe, and multiple NICs/DPUs), scale-up fabric trays (NVSwitch-class) wiring the accelerators into one coherent fabric, a top-of-rack data switch for scale-out, and a separate management switch. Power comes in as 3-phase AC to a power shelf or PDUs; in OCP designs a power shelf feeds a vertical DC busbar that nodes blind-mate onto, with N+1 or 2N PSU redundancy. Because these racks draw 60–130+ kW — about 10× a legacy rack — they're liquid-cooled: a CDU pumps coolant through cold plates on the chips and exchanges heat with facility water, reporting flow, ΔT, pressure, and leak status. Every node also has a BMC on the management network. The key framing: it's not "N servers," it's a system with rack-level shared resources (power, cooling, fabric) that I manage as first-class objects.

Q2: What's the difference between out-of-band and in-band management, and why does it matter for your software? A: In-band management goes through the host OS and its data NIC — SSH, an agent reading /proc — and only works when the host is healthy. Out-of-band goes through the BMC, a separate always-on microcontroller with its own NIC on a separate management network, and works even when the host is off, hung, or kernel-panicked. It matters because the moments you most need to manage a node — won't boot, panicked, needs reinstall — are exactly when in-band is dead. So rack-management software does power control, firmware updates, console access, and diagnostics over OOB (Redfish/IPMI to the BMC). My control plane assumes the host may be down and reaches everything through the management network.

Q3: A rack has two 3-phase 415V 30A feeds, configured N+1. How much IT load can you safely place, and why not 100%? A: Per feed, usable ≈ √3 × 415 × 30 ≈ 21.6 kW, but the 80% continuous-load rule (NEC) caps a breaker at 80% of rating, so budget ≈ 17.3 kW per feed. With N+1, the surviving feed must carry the whole load after one fails, so the safe IT load is bounded by a single derated feed ≈ 17.3 kW — not the sum. I never budget to the combined or the nameplate capacity: exceed 80% and the breaker trips (an outage); budget to the sum and a single feed failure overloads the survivor and cascades. Measured draw is usually well below nameplate, so I can oversubscribe carefully using telemetry and power capping, but the guaranteed safe number is the derated single-feed figure.

Q4: Why are AI racks liquid-cooled when enterprise racks weren't, and what does that add to your job? A: Power density. Enterprise racks ran 5–10 kW, removable by air. AI racks run 60–130+ kW, and air physically can't remove that much heat past ~40 kW — 1 kW is 3,412 BTU/hr, so a 120 kW rack dumps ~409,000 BTU/hr. Liquid carries thousands of times more heat per volume, so coolant runs through cold plates on the chips, with a CDU pumping it and exchanging heat with facility water. For my software it adds a whole managed subsystem: I monitor CDU flow, supply/return ΔT, pressure, pump health, and leak detection; a rising ΔT or failing pump is an early thermal-event warning, and a leak is a safety-critical alert that may trigger emergency shutdown. Cooling becomes a software-visible, software-actionable part of rack health, not just a facility concern.

Q5: What's a failure domain? Give an example and how your software should treat it. A: A failure domain is a set of components that fail together because they share a resource; the blast radius is everything in it. Example: six nodes fed by one non-redundant PDU/breaker — if that feed dies, all six drop simultaneously. My software treats domains in three ways: (1) the inventory model encodes the powers/cools/connects edges so I can compute blast radius; (2) placement respects domains — don't put all replicas of a service behind one PDU or one ToR; (3) during incidents, the first question is "single node or shared resource?" — correlating alerts by domain turns a flood of node alarms into one root cause. Redundancy (N+1) lets a domain survive a failure; isolation (separate domains/VLANs) contains it — I design for both.

Q6: Design the data model for a rack inventory system. What are the entities, relationships, and the hard parts? A: Entities are a containment hierarchy — Site→Hall→Row→Rack→Node→Component — plus rack-level shared objects (PDU, busbar/power shelf, CDU, ToR, management switch, NVSwitch trays). Relationships are a graph, not just containment: powers, cools, connects, manages — those edges drive blast-radius queries. Each thing needs durable identity (asset tag, serial, FRU, MAC/UUID) kept separate from location (rack + U), because identity persists when a unit moves and a stale location is a classic bug. State covers power, health, lifecycle stage, and per-component firmware versions; telemetry is time-series attached to components. The hard parts: (1) identity-vs-location reconciliation; (2) declared-vs-discovered diffing — the model holds intent (the EBOM) and reality (what discovery found), and the diff catches missing parts and firmware drift; (3) validation invariants — every node has power and cooling, no two parts share a U, every managed device is reachable. That declared-vs-discovered reconcile is conceptually the same loop a Kubernetes operator runs.

Q7 (Staff): How would you approach commissioning a brand-new rack from an ODM partner? A: Commissioning is discovery → inventory → baseline → validate → hand-off. First, on the isolated management network, discover every managed endpoint (BMCs, PDUs, CDU, switches) — typically via DHCP + a Redfish/IPMI sweep. Capture full inventory (serials, FRU, firmware versions, accelerator topology) and reconcile it against the declared EBOM agreed with the ODM (e.g., Pegatron) — any missing/extra/mismatched part is a finding raised back to them. Set the firmware baseline (Phase 08) so the whole rack is at known-good versions. Run burn-in/stress and validate power and thermal behavior against budget (Phase 01 calculator) and the spec. Capture a golden inventory + telemetry baseline so future drift is detectable. Only then mark the rack "ready" and hand it to provisioning. The Staff value-add is making this repeatable and automated — a commissioning pipeline with a clear pass/fail and a documented escalation path to the partner — because at fleet scale you commission racks continuously, not once.

References

• Open Compute Project — Open Rack & hardware management specs — https://www.opencompute.org/
• DMTF Redfish — Schema & Specification (the management data model you'll meet in Phase 03) — https://www.dmtf.org/standards/redfish
• NVIDIA GB200 NVL72 — architecture overview — https://www.nvidia.com/en-us/data-center/gb200-nvl72/
• ASHRAE TC 9.9 — Thermal Guidelines for Data Processing Environments (cooling/ΔT references)
• Uptime Institute — Tier topology & redundancy (N, N+1, 2N) concepts — https://uptimeinstitute.com/
• The Open19 / OCP power & busbar design notes — for the power-shelf/busbar model
• Barroso, Hölzle, Ranganathan, The Datacenter as a Computer, 3rd ed. — the canonical systems view of warehouse-scale machines (free online)
• Qualcomm Cloud AI — product context for the accelerators this role's racks carry — https://www.qualcomm.com/products/technology/processors/cloud-artificial-intelligence
• Cross-track: Head of Software Engineering — GPU, Phase 01 WARMUP — the silicon inside the accelerators you manage

🛸 Hitchhiker's Guide — Phase 01: Rack-Scale AI Systems

Read this if: you can write software but a data-center rack is still a black box labeled "servers go here." This is the practitioner's compressed tour — the numbers, mental models, and war stories the WARMUP derives from first principles. Skim it first, then read the WARMUP slowly.

0. The 30-second mental model

An AI rack is one machine the size of a wardrobe that draws as much power as a small apartment building (60–130+ kW), is plumbed with liquid coolant, and contains two networks worth of computers: the hosts (on the data network) and a shadow fleet of BMCs/PDUs/CDU controllers (on a separate management network). Your software lives on that second network and treats the rack as a managed object with a power budget, a cooling budget, a failure-domain graph, and a lifecycle. If you remember one sentence: you manage the rack out-of-band, as a system, not the servers in-band, one at a time.

1. The org chart of an AI rack

RACK (≈42U / 48 OU, ~1.2–1.4 t, 60–130+ kW, liquid-cooled)
├── Compute trays ×N            ← host CPU(s) + 4–8 accelerators (HBM) + NVMe + NICs/DPUs
│     └── BMC per node          ← always-on ARM SoC (OpenBMC); the OOB brain
├── NVSwitch / scale-up trays   ← wire accelerators into one NVLink-class fabric
├── ToR data switch             ← scale-out: 400/800 GbE or InfiniBand to the cluster
├── Management switch           ← OOB only: BMCs, PDUs, CDU  (separate, isolated)
├── Power shelf + busbar / PDUs ← 3-phase AC → DC; N+1 / 2N; metered + switched
└── CDU + manifolds + cold plates ← pumps coolant, exchanges heat w/ facility water

Two ratios to tattoo on your arm:

• Power density jump: enterprise 5–10 kW → AI liquid 60–130+ kW ≈ ~10×. That jump is the whole reason this role exists.
• 1 kW = 3,412 BTU/hr. A 120 kW rack = ~409,000 BTU/hr of heat to remove. Air can't; hence liquid.

2. OOB vs in-band in 4 bullets

• Every node is two computers: the host (in-band, data NIC) and the BMC (out-of-band, mgmt NIC, standby power).
• The BMC works when the host is off, hung, or panicked — which is exactly when you need it.
• Redfish/IPMI/SNMP (Phase 03) ride the management network; SSH/agents ride the data network.
• Rule: assume the host is down and reach everything through OOB. Your control plane's reliability depends on it.

3. The power math you'll actually use

War story shape: "the rack tripped a breaker at 2 a.m." almost always traces to budgeting against nameplate or the sum of redundant feeds instead of the derated single feed. The fix is a budgeting tool that bakes in 0.8 and N+1 (Lab 02) — and blocks a placement that violates it.

4. Cooling, quantified

• Heat removed by liquid = flow (L/min) × fluid heat capacity × ΔT (return − supply). Watch ΔT rising at constant load → fouling/flow problem → pre-failure signal.
• CDU = pump + heat exchanger + sensors. It isolates the clean rack loop from facility water and reports flow/ΔT/pressure/pump-speed/leak.
• Leak detection is safety-critical: it can trigger an automated drain/shutdown. Treat it like a smoke alarm in code: highest priority, fail-safe.
• PUE = facility power ÷ IT power. Liquid AI halls aim ~1.1; the metric the business cares about is increasingly performance/watt and tokens/joule.

5. The three networks (don't conflate them)

Putting BMCs on the tenant network is the classic catastrophic mistake: a BMC owns power and firmware: full physical control. Phase 09 isolates these for real with VLANs.

6. Failure domains: the first question in every incident

When alerts flood in, ask "what's the shared resource?" before "what's wrong with node X?":

• 6 nodes down at once, same PDU → power domain, not 6 separate faults.
• Whole rack thermal-throttling → cooling domain (CDU/pump/loop).
• Rack unreachable in-band but BMCs answer → ToR/data-network domain, hosts are fine.
• Rack unreachable OOB → management-switch domain; you may be blind, escalate to hands-on.

Encoding the powers/cools/connects edges (Lab 01) turns a 30-alarm storm into one root cause. This is the bridge to Phase 07 (alert correlation) and Phase 10 (RCA).

7. The lifecycle, and where you write code

design/NPI → ODM build (Pegatron) → ship/install → COMMISSION/burn-in → PROVISION → OPERATE → MAINTAIN/RMA → decommission
   (P12)         (EBOM, P12)                         (P01/03/08 here!)    (P05/06)   (P07/09)  (P08/10)

The under-appreciated one is commissioning: discover every endpoint on the mgmt network, capture inventory, reconcile against the EBOM, set the firmware baseline, burn-in, validate power/thermal, then mark "ready." At fleet scale you commission racks continuously, so it must be an automated pipeline — a great Staff-level thing to own.

8. Vocabulary that makes you sound like you've been there

• FRU (Field Replaceable Unit) — the granularity of "swap this part."
• U / OU — rack unit (1.75") / OpenRack unit (taller, wider 21" space).
• Busbar — the vertical DC rail nodes blind-mate onto in OCP racks.
• DLC / cold plate / RDHx / immersion — direct liquid cooling variants.
• DPU/SmartNIC — a NIC with its own CPU/OS; itself a managed, firmwared device.
• DCIM — Data Center Infrastructure Management; the facility-level system your rack software feeds and integrates with.
• L6/L10/L11 — manufacturing integration levels (board / node / rack).
• NPI — New Product Introduction; the pre-production phase where interfaces and the EBOM get defined with the ODM.
• Blast radius / failure domain — say these and people know you've held a pager.

9. The mistakes that mark a beginner (avoid in interviews)

1. "Just SSH to the node" — for a node that won't boot, there's no SSH; it's a Redfish/SoL problem.
2. Budgeting power to nameplate or to the sum of redundant feeds.
3. Treating cooling as "the facility's problem" when CDU telemetry is part of rack health your code acts on.
4. One flat inventory table with no relationship edges → you can't compute blast radius or reconcile.
5. Conflating identity (serial) with location (rack+U) → stale location bugs after every RMA.
6. Putting management traffic on the data network → reliability and security disaster.

10. How this phase pays off later

• The inventory model (Lab 01) is literally reused/expanded in the Phase 13 capstone.
• The declared-vs-discovered idea becomes the Phase 06 operator reconcile loop.
• The failure-domain graph powers Phase 07 alert correlation and Phase 10 RCA.
• The OOB-first mindset is the premise of Phases 03 (protocols), 04 (hardware), and 08 (firmware/diag).
• The power/thermal budgets reappear as live telemetry thresholds in Phase 07.

You're ready for the WARMUP. Read it slowly — the numbers in it are the ones interviewers probe.

Lab 01 — Rack Data Model & Inventory Engine

Phase: 01 — Rack-Scale Foundations | Difficulty: ⭐⭐☆☆☆ | Time: 3–4 hours Language: Python (stdlib only) | Hardware: none

Concept primer: ../WARMUP.md Ch. 7 (failure domains) & Ch. 10 (what to model), ../HITCHHIKERS-GUIDE.md §1, §6.

Run

python3 solution.py     # builds a rack, validates it, prints blast radius, self-tests

0. The mission

Build the data model that every other phase plugs into. A rack manager is, at its core, a system that keeps an accurate model of physical reality and acts on it. If the model is wrong or thin, every feature above it leaks. You will build a typed object graph of a realistic AI rack, validate it against invariants, and answer the question that begins every incident: "if this fails, what goes with it?"

Two questions you must be able to answer at the end:

1. Why is inventory a graph (powers/cools/connects edges), not a flat table?
2. Why must identity (serial/asset tag) be separate from location (rack + U)?

1. What the code models

2. What the run shows

=== Rack rack-ai-01 (8 nodes) ===
  total power draw : 14.3 kW
  total heat load  : 14.3 kW
  ...
  validation       : PASS (all invariants hold)

--- Blast radius examples ---
  if PDU-A fails -> 4 node(s) affected   ← N+1 balance: PDU-A feeds the even nodes
  if CDU-1 fails -> 8 node(s) affected   ← single CDU = whole-rack cooling domain
  if TOR-1 fails -> 8 node(s) affected   ← single ToR = whole-rack data domain

Read those three blast-radius lines carefully — they are the failure-domain lesson (WARMUP Ch. 7). A PDU failure is half the rack (because power is balanced N+1 across two feeds); the CDU and ToR are single points of failure for the whole rack. That asymmetry is exactly what drives redundancy decisions and workload placement.

3. Extension exercises (do them — this is the lab)

1. Model the cooling loop properly: today cools is a single edge from the CDU to each node. Add a Manifold/coolant loop object between CDU and nodes so a single manifold failure has a smaller blast radius than the whole CDU. Re-run blast radius and explain the new numbers.
2. N+1-aware blast radius: find_blast_radius ignores redundancy. Add a find_blast_radius_redundant() that, for power, only counts a node as "down" if it loses all its power feeds. Then give each node a second powers edge to the other PDU and show that a single PDU failure now affects 0 nodes — the point of N+1.
3. Declared vs discovered: add a discover() that returns a slightly different rack (a missing accelerator, a firmware drift) and a reconcile(declared, discovered) that reports the diff. This is the conceptual seed of the Phase 06 operator loop and Phase 10 RCA.
4. Identity vs location bug: write a test that moves a node to a new U position but keeps its serial, and assert that queries by serial still resolve while location-based queries see the move. Then introduce the classic bug (update location but reuse a stale asset_tag) and show your validator catches the duplicate.

4. Common pitfalls

1. Flattening the graph — if you store relationships as columns on a node row, you can't traverse them; blast radius and reconciliation become impossible. Keep edges.
2. Putting u_position in Identity — then a part "changes identity" when it moves. Location is state of the slot, identity belongs to the part.
3. Forgetting rack-level objects — PDUs, CDU, switches belong to no single node; model them at the rack level or you can't compute shared-resource blast radius.
4. Validating only the happy path — the value of validate() is catching the broken configs (no power feed, U collision, dangling edge). Test those explicitly.

5. What this lab proves about you

You can design the schema a rack-management product is built on — the part that looks "boring" in an interview but is where real systems live or die. When asked "design the data model for a rack inventory system" (a near-certain system-design prompt), you don't hand-wave a table; you draw a graph with identity/location separation, invariants, and a blast-radius traversal, and you can say which objects are single points of failure and why. The Phase 13 capstone reuses this model directly.

Lab 02 — Power & Thermal Budget Calculator

Phase: 01 — Rack-Scale Foundations | Difficulty: ⭐⭐⭐☆☆ | Time: 3–4 hours Language: Python (stdlib only) | Hardware: none

Concept primer: ../WARMUP.md Ch. 4 (power) & Ch. 5 (cooling), ../HITCHHIKERS-GUIDE.md §3–§4.

Run

python3 solution.py     # evaluates three rack scenarios + self-tests

0. The mission

Build the tool that prevents 2 a.m. outages: a calculator that decides whether a rack configuration is safe on power and cooling, with redundancy modeled correctly. This is the math a rack manager applies before every deployment and every "can we add a node?" request — and getting it wrong (budgeting to nameplate, or to the sum of redundant feeds) is one of the most common, most expensive infrastructure mistakes.

Two questions you must be able to answer at the end:

1. Why is the safe load on two N+1 feeds equal to one derated feed, not two?
2. Why do you multiply every breaker by 0.8 before you do anything else?

1. The three rules encoded

2. What the run shows

Three scenarios, same hardware, different answers — the whole lesson in one screen:

• Scenario A (fits): 8 nodes at 1.8 kW = 15 kW on N+1 (usable ≈ 16.9 kW). Passes, but note the low power headroom (+1.9 kW) vs huge cooling headroom — power is the binding constraint here, as it usually is in dense AI racks.
• Scenario B (over budget): denser nodes push it to 19.8 kW > 16.9 kW usable. The tool fails it and proposes a per-node cap (2.04 kW) that would make it fit — trading a little performance for a safe deployment.
• Scenario C (no redundancy): the same 19.8 kW load "fits" if you drop to N (no spare) — usable jumps to 33.8 kW. This is the dangerous shortcut a tired engineer reaches for; the tool makes the redundancy tradeoff explicit instead of hidden.

The contrast between B and C is the point: redundancy is a choice with a power cost, and your tooling should surface it, not silently assume it away.

3. Extension exercises (do them — this is the lab)

1. Simulate a feed failure: add PowerBudget.with_failed_feed(name) and re-check the load under the surviving feeds. Confirm that a config sized for N+1 still passes after one feed dies, and that an N-sized config goes critical. This is the redundancy promise, verified.
2. Real cooling at risk: drop the CDU flow (a failing pump) from 120 to 40 L/min and re-evaluate. Watch a previously-fine rack become cooling-bound. Then raise max_return_temp_c and explain why running hotter buys capacity but shrinks the safety margin to throttle.
3. Per-component power, not flat per-node: replace node_power_kw with a list of components (accelerators at 350 W, CPU at 350 W, NIC at 25 W) and compute draw from the bill of materials — then wire this lab to Lab 01's model so the calculator consumes the inventory graph directly.
4. Live headroom monitor: instead of a static node_power_kw, accept measured draw (you'll get this from the Phase 04 PDU/CDU simulator) and emit a Prometheus-style rack_power_headroom_kw gauge (foreshadows Phase 07). Alert when headroom < 10%.

4. Common pitfalls

1. Budgeting to the sum of N+1 feeds — the #1 real-world outage cause. Usable is one derated feed, because the survivor must carry everything.
2. Skipping the 0.8 derate — a breaker sized to its nameplate trips under sustained load. Derate first, always.
3. Confusing rated capacity with ΔT-limited capacity — a CDU rated 130 kW only removes that much if flow and ΔT allow; at low flow or low ΔT it removes far less. removable_kw_at_dt() takes the min of the two for a reason.
4. Assuming nameplate = draw — measured power is usually 40–70% of nameplate; budgeting to nameplate strands capacity, budgeting to measured needs telemetry and capping as a safety net.

5. What this lab proves about you

You can do — in your head and in code — the power/cooling math that gates every rack deployment, and you understand redundancy as an explicit, costed tradeoff rather than a checkbox. In an interview, "two 30A 415V feeds, N+1, how much load?" is a fast filter; you answer with √3, the 0.8 rule, and "one derated feed ≈ 17 kW, because the survivor carries it all" — and then you mention capping and measured-vs-nameplate as the levers that let you push utilization safely. That's the difference between a coder and someone who can be trusted with a 120 kW rack.

Phase 02 — Production Systems Programming (Python, C++, Shell, Node.js)

Difficulty: ⭐⭐⭐☆☆ | Estimated Time: 2 weeks Roles supported: Rack Management SWE (Senior/Staff), Infra/Systems SWE Hardware needed: none — a C++17 compiler and Python 3 on any Linux/macOS box

Why This Phase Exists

The JD's hard requirement is blunt: "Proficiency in Python, C++, and shell scripting" and "Write high‑quality, secure, and maintainable code following established coding standards and best practices." Node.js is "a plus." This is the phase where you stop writing scripts and start writing infrastructure software — code that runs unattended, close to hardware, in production, for years, on someone else's pager.

The difference is not syntax. It is robustness under failure: a rack daemon must survive a BMC that stops responding mid-poll, a sensor that returns garbage, a SIGTERM during a critical section, a disk that fills, and a malicious or malformed input — and it must do so observably, with logs and exit codes an on-call engineer can reason about at 3 a.m. The same code must be testable without the hardware (you can't ship a BMC to CI) and maintainable by a distributed team you've never met.

This phase establishes the engineering bar that every other phase's labs are held to. You build two pieces of real infrastructure software — a C++ sensor daemon and a Python rack CLI — that demonstrate the production patterns: RAII and resource safety, typed interfaces, structured logging, retries with backoff, graceful shutdown, safe subprocess handling, and the testability that lets you mock hardware.

Concepts

The language matrix — why each tool, where

• C++ — close-to-hardware, long-running, performance- and resource-sensitive code: daemons, agents, anything that talks to a device library or parses binary protocols. RAII, smart pointers, the rule of zero/five, const-correctness, error handling (exceptions vs std::expected/error codes), modern C++17/20, CMake, sanitizers (ASan/UBSan/TSan), threading.
• Python — orchestration, clients, tooling, CLIs, simulators, parsers, glue: most of the rack-management surface. Type hints + mypy, packaging (pyproject.toml), virtualenvs, argparse/click, structured logging, retries/backoff, subprocess safety, asyncio vs threads, pytest, ruff/black.
• Shell — provisioning glue, CI steps, bootstrap, and "do this on 500 nodes" one-liners. Safe scripting (set -euo pipefail), quoting, idempotency, trap/cleanup, when to stop using shell and switch to Python.
• Node.js (the "plus") — tooling and lightweight services (a telemetry/status web service, a dashboard backend). When JS/TS is the right call and when it isn't.

Production engineering (language-agnostic)

• Error handling philosophy: fail fast vs degrade gracefully; the error taxonomy (transient vs permanent vs programmer error)
• Retries, exponential backoff, jitter, timeouts, circuit breakers (deeper in Phase 10)
• Resource lifetime: RAII in C++, context managers in Python, defer-style cleanup; never leak fds/handles/locks
• Concurrency: threads vs async vs processes; the GIL; data races; lock discipline
• Daemon mechanics: signals, graceful shutdown/drain, PID/lifecycle, systemd units, journald, readiness/liveness
• Structured logging (key/value, correlation IDs), log levels, what to log and what not to (no secrets)
• Configuration: env vs file vs flags; precedence; validation at startup
• Security: input validation, shell-injection avoidance, least privilege, secrets handling, integer/buffer safety in C++, dependency/supply-chain hygiene
• Testability: dependency injection, mocking the hardware boundary, deterministic time, the seam that lets you test without a BMC
• Coding standards & style: clang-format/clang-tidy, ruff/black, code review norms (Phase 12)

Labs

Lab 01 — C++ Sensor-Polling Daemon

Lab 02 — Production Python Rack CLI

Deliverables Checklist

• C++ daemon builds, runs, shuts down cleanly on SIGTERM, and passes tests against a fake sensor
• C++ daemon is clean under -fsanitize=address,undefined
• You can explain RAII and show where a leak would occur without it
• Python CLI runs, logs structured JSON, retries transient failures, and has correct exit codes
• Python CLI never uses shell=True; you can explain the injection risk
• You can state the config precedence order and why
• Both labs have a clear test seam at the hardware boundary (you can test without hardware)
• You can write a safe Bash script with set -euo pipefail and proper quoting

Interview Relevance

• "Write a daemon that polls a sensor every second and exposes the latest value. How do you handle a sensor that hangs?"
• "What is RAII and why does it matter here? Show me a resource leak and how RAII prevents it."
• "Why is subprocess.run(cmd, shell=True) dangerous? Rewrite it safely."
• "How do you make this code testable when the real dependency is a BMC you don't have in CI?"
• "Your daemon gets a SIGTERM in the middle of a poll. Walk me through what happens."
• "Exceptions vs error codes vs std::expected — when do you reach for each?"
• "How do you handle retries without hammering a struggling BMC?" (backoff + jitter + circuit breaker)

Warmup Guide — Production Systems Programming

Zero-to-expert primer for Phase 02. You can already write code; this guide makes it production infrastructure code — robust under failure, secure, testable without hardware, and maintainable by a team you've never met. Every later lab is held to this bar.

Table of Contents

• Chapter 1: The Language Matrix — Why Three (Plus One)
• Chapter 2: Error Handling as a Design Decision
• Chapter 3: RAII and Resource Lifetime (the C++ core idea)
• Chapter 4: Modern C++ for Infra Code
• Chapter 5: Production Python
• Chapter 6: Concurrency — Threads, Async, Processes
• Chapter 7: Daemons, Signals, and systemd
• Chapter 8: Security in Infra Code
• Chapter 9: Testability — The Hardware Boundary Seam
• Chapter 10: Safe Shell and When to Stop Using It
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: The Language Matrix — Why Three (Plus One)

The JD lists Python, C++, and shell as required and Node.js as a plus. This is not a wish list; it reflects the layers of a rack-management stack and where each tool earns its place.

• C++ lives at the bottom, close to hardware: device libraries, binary protocol parsers, performance-critical daemons, anything linking a vendor C SDK. You choose C++ when you need deterministic resource control, low overhead, or to call into a hardware library. The cost is that you own memory and lifetime; the discipline (RAII, Chapter 3) is what makes that safe.
• Python is the broad middle: clients (Redfish/IPMI/SNMP, Phase 03), exporters (Phase 07), operators (Phase 06), CLIs, simulators, parsers (Phase 08), and orchestration glue. You choose Python for velocity, libraries, and readability — and accept that it's slower and needs typing discipline (mypy) to stay maintainable at scale.
• Shell is the glue and bootstrap: PXE/kickstart hooks (Phase 05), CI steps (Phase 11), and "run this on every node." Powerful for orchestrating other programs, dangerous for logic. The skill is knowing the line where shell stops and Python begins (Chapter 10).
• Node.js is optional tooling/services: a status web UI, a small telemetry API, a dashboard backend. TypeScript brings types; the event loop suits I/O-bound services. You reach for it when the team/ecosystem favors JS or you're serving a web surface.

The Staff-level framing: choosing the language is an engineering decision with consequences for the team, not a preference. "We wrote the hot path in C++ behind a stable C ABI and the control logic in typed Python, with shell only for bootstrap" is the kind of layered answer that signals seniority.

Misconception: "use one language for everything." A rack stack that's all-Python can't meet a hard latency/resource budget at the device layer; an all-C++ stack moves at a fraction of the velocity for orchestration code. Polyglot-by-layer is the norm, and the seams between languages (a stable ABI, a JSON contract) are where you spend design care.

Chapter 2: Error Handling as a Design Decision

Production infra code is mostly error handling. The naive happy path is 20% of the work; the other 80% is what happens when the BMC times out, the sensor returns 0xFFFF, the disk is full, or the config is malformed.

The error taxonomy (decide the response per class):

• Transient (network blip, BMC busy, lock contention): retry with backoff (Chapter 6, Phase 10). Do not crash.
• Permanent (auth failure, 404, malformed firmware image): fail fast and loud; retrying just wastes time and hammers the target.
• Programmer error (precondition violated, null where impossible): assert/abort in dev; in prod, fail the operation, never silently continue with corrupt state.
• Degraded (one sensor of many failed): continue with reduced function, mark the bad part stale, surface a warning. A rack manager that dies because one of 40 sensors is flaky is useless.

Fail-fast vs degrade-gracefully is contextual: a startup misconfiguration should fail fast (don't run misconfigured); a runtime partial failure should usually degrade (keep managing the healthy 95%). The art is knowing which is which.

Mechanism by language:

• C++: exceptions for truly exceptional/unwindable errors; error-returning types (std::optional, std::expected in C++23, or a Result-like type) for expected failures like "sensor read failed." Don't use exceptions for control flow; don't ignore return codes.
• Python: exceptions are idiomatic; catch specific exceptions, never bare except:; wrap at boundaries; use try/finally or context managers for cleanup.

The cardinal sin: swallowing errors (except: pass, ignoring a return code). Every swallowed error is a future 3 a.m. mystery. Log it with context or propagate it; never drop it.

Chapter 3: RAII and Resource Lifetime (the C++ core idea)

RAII = Resource Acquisition Is Initialization. The single most important idea in C++ infra code, and a frequent interview question. The principle: tie a resource's lifetime to an object's scope. Acquire in the constructor, release in the destructor. When the object goes out of scope — for any reason, including an exception — the destructor runs and the resource is freed. No finally, no manual cleanup, no leak path.

Resources this governs: heap memory, file descriptors, sockets, mutex locks, device handles, a BMC session, a firmware-update transaction.

// Without RAII: a leak waits at every early return / throw.
int fd = open(path, O_RDONLY);
if (parse(fd) < 0) return -1;     // <-- fd leaked
close(fd);

// With RAII: the guard's destructor closes fd on ANY exit path.
class FdGuard {
    int fd_;
public:
    explicit FdGuard(int fd) : fd_(fd) {}
    ~FdGuard() { if (fd_ >= 0) ::close(fd_); }
    int get() const { return fd_; }
    FdGuard(const FdGuard&) = delete;            // non-copyable: one owner
    FdGuard& operator=(const FdGuard&) = delete;
};
FdGuard f(open(path, O_RDONLY));
if (parse(f.get()) < 0) return -1;   // fd still closed — destructor runs

Smart pointers are RAII for memory: std::unique_ptr (single owner, zero overhead — your default), std::shared_ptr (reference-counted shared ownership — use sparingly; cycles leak), std::weak_ptr (breaks cycles). The rule: never new/delete raw in modern C++. Use std::make_unique/make_shared.

std::lock_guard/std::scoped_lock are RAII for mutexes — the lock releases when the guard leaves scope, even on exception. This is why your daemon's shared state (Lab 01) is protected with a guard, not manual lock/unlock (which leaks the lock on an early return → deadlock).

The rule of zero/three/five: if your class owns no raw resources (it holds unique_ptrs, strings, vectors — all RAII themselves), write no destructor/copy/move (rule of zero) — the compiler-generated ones are correct. If you do manage a raw resource directly, you must handle destructor + copy + move + their assignments (rule of five), usually by deleting copy and defining move. Aim for the rule of zero: let RAII members do the work.

Chapter 4: Modern C++ for Infra Code

Beyond RAII, the modern-C++ toolkit that keeps systems code safe and readable:

• const-correctness: mark everything you don't mutate const. It documents intent, enables optimization, and the compiler catches accidental mutation. A const-correct sensor reader can't be accidentally made to mutate the device.
• References over pointers where null is not a valid state; pointers (or optional) only where absence is meaningful.
• std::optional<T> for "maybe a value" (a sensor that may not have a reading) and an error type (expected/a Result) for "value or why it failed."
• std::string_view/std::span for non-owning views — pass without copying, but never outlive the owner.
• Value semantics & move: prefer values + move over manual pointer juggling; std::move transfers ownership cheaply.
• enum class for type-safe states (power state, health) — no implicit int conversions.
• Build & quality gates: CMake (or a clean Makefile), -Wall -Wextra -Werror, clang-tidy (catches bugs), clang-format (style — never argue about it in review), and the sanitizers:
  • ASan (AddressSanitizer): use-after-free, buffer overflow, leaks.
  • UBSan: undefined behavior (signed overflow, bad shifts, null deref).
  • TSan: data races. Run your daemon's tests under TSan to prove your locking is correct. Sanitizers in CI catch the bugs that otherwise become irreproducible production crashes — exactly the JD's "debugging complex issues."

Misconception: "modern C++ is just C with classes." Modern C++ (17/20) with RAII, smart pointers, optional/expected, and sanitizers is a memory-safe-by-discipline language. The bugs that haunt infra code (UAF, leaks, races) are preventable with these tools, and a Staff engineer enforces them as standards (Phase 12).

Chapter 5: Production Python

Python's velocity is a trap without discipline; here's the discipline that keeps a 50k-line rack toolchain maintainable.

• Type hints + mypy: annotate function signatures and dataclasses; run mypy --strict in CI. Types are documentation that can't go stale and catch a whole class of bugs (passing a str where an int belongs) before runtime. For infra code consumed by a distributed team, this is non-negotiable.
• Packaging: a pyproject.toml, a real package (not a pile of scripts), pinned dependencies, a virtualenv. pip install -e . for dev. Reproducible installs are the difference between "works on my machine" and a deployable tool.
• Structured logging: not print(). Use logging with a structured (JSON) formatter and key/value context (node id, request id). Logs are queried by machines (Phase 07); structure makes them searchable and correlatable.
• Configuration precedence: a clear order — flags > environment > config file > built-in default — validated at startup. Surprises in config precedence cause outages; make it explicit and tested (Lab 02).
• Retries/backoff: a decorator that retries transient errors with exponential backoff + jitter and a cap; never an unbounded retry loop that DDoSes a struggling BMC.
• subprocess safety: pass a list of args, never shell=True with an interpolated string (Chapter 8). Set timeouts. Check return codes.
• Exit-code discipline: 0 = success, non-zero = specific failure classes. Tools are composed in scripts and CI; your exit code is an API.
• Quality gates: ruff (lint), black (format), pytest (test), mypy (types) — all in CI.

The GIL, briefly: CPython's Global Interpreter Lock means threads don't give you CPU parallelism (only one runs Python bytecode at a time), but they do help with I/O-bound work (waiting on 100 BMCs). For CPU-bound work use processes; for high-concurrency I/O use asyncio. Knowing this prevents the classic "why didn't my threads speed up my CPU-bound loop" mistake (Chapter 6).

Chapter 6: Concurrency — Threads, Async, Processes

A rack manager is inherently concurrent: poll 100 nodes, each independently, while serving status queries and handling signals. Choose the model deliberately.

• Threads: shared memory, cheap context, but you own synchronization (locks, races). In C++ great for true parallelism; in Python (GIL) only useful for I/O-bound concurrency. Use a thread pool with bounded size — never spawn one thread per node for 10,000 nodes.
• Async (asyncio / coroutines): single-threaded cooperative concurrency, excellent for many concurrent I/O waits (the BMC poll fan-out) with low overhead and no locks for most state. The cost: "async all the way down" and no help for CPU-bound work.
• Processes: isolation and true parallelism (bypass the GIL), at the cost of memory and IPC. Use for CPU-bound work or fault isolation.

The lock discipline (C++ Lab 01): shared state (latest readings) is guarded by a mutex; every access takes a lock_guard. Keep critical sections tiny (copy out under lock, process outside). Never hold a lock across I/O or a callback. Run under TSan to prove there are no races — data races are undefined behavior and the source of the worst, least-reproducible bugs.

Bounded concurrency: whatever the model, cap it (a semaphore, a pool size). Polling 10,000 BMCs at once will exhaust fds and overwhelm the management network. Bounded fan-out (e.g., 50 at a time) is both kinder to the hardware and more predictable.

Chapter 7: Daemons, Signals, and systemd

A daemon is a long-running process that must behave well for years. The mechanics:

• Signals: SIGTERM (graceful stop — the one systemctl stop sends), SIGINT (Ctrl-C), SIGHUP (reload config), SIGKILL (uncatchable — last resort). Install handlers that set an atomic "stop" flag; the main loop checks it and drains (finish the in-flight poll, flush logs, release resources via RAII) before exiting. Never do heavy work in a signal handler (async-signal-safety) — just flip a flag.
• Graceful shutdown: on SIGTERM, stop accepting new work, finish or cancel in-flight work cleanly, persist any needed state, exit 0. A daemon that corrupts a firmware-update transaction on shutdown is a disaster; design shutdown as carefully as startup.
• systemd integration: a unit file with Restart=on-failure, resource limits (MemoryMax, TasksMax), Type=notify (the daemon calls sd_notify(READY=1) when actually ready — true readiness, not "the process started"), and WatchdogSec (the daemon pings; if it hangs, systemd restarts it — a free liveness probe).
• journald: log to stdout/stderr with structure; systemd captures it into the journal, queryable with journalctl -u sensord. No log-file rotation code to write.
• Readiness vs liveness: ready = can serve requests (mirrors Kubernetes readiness, Phase 06); alive = not hung. Expose both; the watchdog handles liveness, sd_notify handles readiness.

Why this matters: the JD's "operational support," "observability," and "deployed systems" all assume your software runs as a well-behaved daemon. A process that ignores SIGTERM, can't be restarted safely, or hangs without the watchdog noticing is an operational liability.

Chapter 8: Security in Infra Code

The JD says "secure" three times. Rack-management software runs with high privilege on the management network — a bug here is a fleet-wide risk. The essentials:

• Shell-injection (the classic): never build a shell command by string-interpolating untrusted input. subprocess.run(f"ipmitool -H {host} ...", shell=True) lets host = "x; rm -rf /" run anything. Fix: pass an arg list and shell=False (the default) — ["ipmitool", "-H", host, ...] — so host can never be parsed as shell syntax. Same idea in C++: never system() a constructed string; use posix_spawn/exec with an argv array.
• Input validation: validate everything from the network/config/user at the boundary — lengths, ranges, types, allow-lists (not deny-lists). A Redfish response, a firmware image header, a config value: all untrusted until validated.
• Memory safety (C++): bounds-check, avoid strcpy/sprintf (use snprintf/std::string), beware integer overflow in size calculations (a classic heap-overflow precursor), and run ASan/UBSan in CI. Most CVEs in C/C++ infra code are buffer/integer bugs.
• Least privilege: run the daemon as a dedicated non-root user; grant only the capabilities it needs (CAP_NET_ADMIN, not full root); scope its filesystem access (systemd ProtectSystem, ReadOnlyPaths).
• Secrets: never in code, logs, or argv (argv is world-readable via /proc). Read from a file with tight perms, an env var injected by the orchestrator, or a secrets manager (Phase 09). Redact secrets in logs.
• Supply chain: pin dependencies, verify checksums/signatures, minimize the dependency tree (every dep is attack surface), scan with pip-audit/cargo audit/SCA in CI.

The mindset: assume every input is hostile and every privilege you hold can be abused. Phase 09 goes deep on the system-level security (secure boot, tenancy); this chapter is the code-level security the JD demands in every commit.

Chapter 9: Testability — The Hardware Boundary Seam

You cannot put a BMC, PDU, or CDU in CI. So the central testability technique for rack software is the seam at the hardware boundary: define an interface for the hardware interaction and program against it, so tests inject a fake.

struct SensorSource {                 // the seam
    virtual std::optional<double> read_temp(int id) = 0;
    virtual ~SensorSource() = default;
};
class RealBmcSource : public SensorSource { /* talks to the BMC */ };
class FakeSensorSource : public SensorSource { /* returns scripted values, failures */ };

The daemon takes a SensorSource& (dependency injection). Production wires the real one; tests wire the fake to simulate a hang, garbage data, intermittent failure — deterministically. This is exactly how Lab 01 tests retry and staleness without a BMC, and it scales to every phase: a fake Redfish service (Phase 03/11), a fake firmware target (Phase 08), a fake CDU (Phase 04).

Determinism: also inject time (a clock interface) so you can test "mark stale after 5s" without sleeping 5s. Real-clock tests are flaky; injected-clock tests are instant and reliable.

The test pyramid (deeper in Phase 11): many fast unit tests against fakes, fewer integration tests against emulators (Redfish mock, ipmi_sim), a thin top of hardware-in-the-loop tests on real gear. The seam is what makes the wide base possible.

Misconception: "infra code can't be unit-tested because it needs hardware." Wrong — it can't be unit-tested if you didn't design a seam. The seam is a design choice you make up front, and it's a hallmark of senior infra code.

Chapter 10: Safe Shell and When to Stop Using It

Shell is unavoidable (bootstrap, CI, provisioning hooks) and dangerous. The safety basics:

• set -euo pipefail at the top of every script: -e exit on error, -u error on unset variable, -o pipefail a pipeline fails if any stage fails. This turns silent failures into loud ones.
• Quote everything: "$var", "${array[@]}". Unquoted variables word-split and glob — the source of countless "it worked until a path had a space" bugs and injection vectors.
• trap cleanup EXIT for RAII-like cleanup (remove temp files, release locks) on any exit.
• Idempotency: a provisioning script may run twice; design it to converge (check-then-act, mkdir -p, "create if not exists") rather than assume a clean slate.
• shellcheck in CI — it catches the quoting and portability bugs you'll miss.

When to stop: the moment you need data structures, real error handling, arithmetic, JSON, or testing — switch to Python. A rule of thumb: >50 lines, any non-trivial control flow, or any logic you'd want a unit test for → it's a Python program, not a shell script. Staff engineers are known for deleting sprawling shell and replacing it with tested Python; that's a maturity signal, not a rewrite-for-its-own-sake.

Lab Walkthrough Guidance

Order: Lab 01 (C++ daemon) → Lab 02 (Python CLI). The daemon teaches resource safety and the hardware seam at the hardest layer; the CLI applies the same robustness ideas where you'll write most code.

Lab 01 (C++ sensor daemon):

1. make && ./sensord --once — one poll, then exits. Read the output.
2. Find the three load-bearing pieces in sensord.cpp: the SensorSource interface (the seam, Ch. 9), the RAII guard / lock_guard (Ch. 3), and the signal handler that flips the stop flag (Ch. 7).
3. make test — the fake source injects a hang and garbage; tests assert retry/backoff and staleness marking.
4. make asan && ./sensord_asan --once — confirm clean under AddressSanitizer.
5. Extensions: a Result/expected error type; a Unix-socket status endpoint; a systemd unit with Type=notify.

Lab 02 (Python rack CLI):

1. python3 -m rackctl status --rack rack-ai-01 — read the JSON-structured logs.
2. Trace a command from dispatch → retry decorator → subprocess wrapper → exit code.
3. python3 -m pytest — see the retry, config-precedence, and subprocess-safety tests.
4. Try to make it run an injected shell command through a node name; confirm it can't (arg list, no shell=True).
5. Extensions: --output json, an asyncio bounded fan-out over 100 mock nodes, a universal --dry-run.

The phase capstone question: "Take the C++ daemon and the Python CLI and explain, for each, the single change that most improves its production-readiness." (Good answers: the daemon — Type=notify + watchdog so the orchestrator knows true readiness/liveness; the CLI — bounded-concurrency async fan-out + structured-log correlation IDs so it scales and is debuggable.)

Success Criteria

You're done with this phase when — without notes:

• You can explain and demonstrate RAII, and identify the leak/deadlock it prevents (Ch. 3)
• You can state the rule of zero/five and when each applies (Ch. 3)
• You can classify an error as transient/permanent/programmer/degraded and pick the response (Ch. 2)
• You can pick threads vs async vs processes for a given workload and justify it, including the GIL (Ch. 6)
• You can write a graceful-shutdown daemon and a systemd unit with Type=notify + watchdog (Ch. 7)
• You can spot and fix a shell-injection and explain least privilege/secrets handling (Ch. 8)
• You can design a hardware-boundary seam and test failure paths without hardware (Ch. 9)
• Both labs build, pass tests, and (C++) run clean under ASan/UBSan
• You can write a safe Bash script and say when to switch to Python (Ch. 10)

Interview Q&A

Q1: What is RAII, and why does it matter for a rack daemon? A: RAII ties a resource's lifetime to an object's scope — acquire in the constructor, release in the destructor — so the resource is freed on every exit path, including exceptions, with no manual cleanup. For a daemon it's the difference between leaking file descriptors / BMC sessions / mutex locks on error paths and being correct by construction. Concretely: a lock_guard releases the mutex even if the critical section throws (no deadlock); a unique_ptr/fd-guard closes the handle on early return. The rule is "no raw new/delete, no manual lock/unlock" — let destructors do the work. It also gives the rule of zero: a class holding only RAII members needs no destructor or copy/move at all.

Q2: subprocess.run(f"ipmitool -H {host} power status", shell=True) — what's wrong and how do you fix it? A: It's a shell-injection vulnerability. With shell=True the string is parsed by /bin/sh, so host = "h; rm -rf /" runs arbitrary commands with the daemon's privileges — which on the management network are high. The fix is to pass an argument list with shell=False (the default): subprocess.run(["ipmitool", "-H", host, "power", "status"], timeout=10, check=True). Now host is a single argv element that can never be reinterpreted as shell syntax. I'd also validate host against an allow-list/format, set a timeout, and check the return code. Same principle in C++: never system() a built string; use exec/posix_spawn with an argv array.

Q3: How do you unit-test code whose real dependency is a BMC you don't have in CI? A: Put a seam at the hardware boundary — define an interface (e.g., SensorSource with read_temp) and program against it via dependency injection. Production injects the real BMC client; tests inject a fake that returns scripted values, simulates a hang, garbage data, or intermittent failure — deterministically. I also inject the clock so I can test "mark stale after 5s" instantly without sleeping. That gives a wide base of fast, reliable unit tests; above it, a few integration tests run against an emulator (a Redfish mock or ipmi_sim), and a thin top runs on real hardware. The key insight is that testability is a design decision — the seam — made up front, not something you bolt on.

Q4: Your daemon receives SIGTERM mid-poll. Walk me through correct behavior. A: The SIGTERM handler does the minimum async-signal-safe work: set an atomic stop flag (and maybe write to a self-pipe/eventfd to wake the loop). It does not do logging or cleanup inside the handler. The main loop notices the flag, stops accepting new work, lets the in-flight poll finish (or cancels it cleanly if it's safe), flushes logs, and returns — at which point RAII guards release fds, sessions, and locks, and the process exits 0. If there's a transactional operation in progress (a firmware update), shutdown must either complete it or leave it in a safe, resumable state — never a corrupt half-state. With systemd I'd also use Type=notify so readiness is real and a WatchdogSec so a hang during shutdown gets escalated to SIGKILL.

Q5: Threads, async, or processes — how do you choose, and where does the GIL bite? A: Match the model to the workload. I/O-bound concurrency (poll 100 BMCs that mostly wait) → async/coroutines (low overhead, no locks) or a bounded thread pool. CPU-bound work → processes for true parallelism. Fault isolation → processes. The GIL means CPython threads don't run bytecode in parallel, so threads won't speed up CPU-bound Python — a common mistake; they do help I/O-bound work because the GIL is released during I/O. In C++ threads give real parallelism but I own synchronization: tiny critical sections under lock_guard, never hold a lock across I/O, and prove it with TSan. Whatever the model, I bound concurrency — a semaphore or pool size — so I don't exhaust fds or flood the management network polling thousands of nodes at once.

Q6: What does "secure code" mean concretely for this role? A: At the code level: no shell injection (arg lists, never shell=True/system()), validate all external input at the boundary with allow-lists, memory safety in C++ (no strcpy/sprintf, watch integer overflow in size math, ASan/UBSan in CI), least privilege (dedicated non-root user, scoped capabilities, systemd sandboxing), secrets never in code/logs/argv (argv leaks via /proc), and supply-chain hygiene (pinned, checksum-verified, minimal, scanned dependencies). It matters more here than in most software because rack-management code runs with high privilege on the management network, so a single buffer overflow or injected command is a fleet-wide compromise. Security is a per-commit discipline enforced by standards and CI gates, not an afterthought.

Q7 (Staff): Your team's rack tooling is a 4,000-line pile of Bash that's become unmaintainable. How do you approach it? A: First, stabilize and understand, not rewrite blindly: add set -euo pipefail and shellcheck to stop the bleeding, and capture what it actually does (its behavior is the spec). Then carve along seams — identify the parts doing real logic (data structures, retries, parsing, anything you'd want tested) and migrate those to a typed, tested Python package incrementally, leaving shell only for thin bootstrap/glue. I'd add tests as I go so each migrated piece is pinned, ship behind a flag, and run old and new in parallel to validate. The Staff value is doing this incrementally and safely with the team — not a big-bang rewrite that stalls feature work — and codifying the new standard (lint/format/type/test gates) so it doesn't regress. The deliverable is a maintainable tool and a raised engineering bar.

References

• Scott Meyers, Effective Modern C++ — the canonical modern-C++ practices (RAII, smart pointers, move, const)
• ISO C++ Core Guidelines — https://isocpp.github.io/CppCoreGuidelines/ (esp. the Resource and Concurrency sections)
• CMake & sanitizers — https://clang.llvm.org/docs/AddressSanitizer.html , UBSan, ThreadSanitizer docs
• Brett Slatkin, Effective Python — production Python idioms (typing, concurrency, packaging)
• Python subprocess security — https://docs.python.org/3/library/subprocess.html#security-considerations
• systemd.service & sd_notify — https://www.freedesktop.org/software/systemd/man/systemd.service.html
• Google Shell Style Guide & shellcheck — https://google.github.io/styleguide/shellguide.html , https://www.shellcheck.net/
• OWASP — Command Injection & Secrets Management cheat sheets — https://cheatsheetseries.owasp.org/
• Cross-track: Security Engineer — AppSec/SDLC phase for deeper secure-coding practice

🛸 Hitchhiker's Guide — Phase 02: Production Systems Programming

Read this if: you write working code but aren't sure what separates a script from infrastructure software. This is the compressed field guide — the patterns, the footguns, and the one-liners that show up in code review and on the pager.

0. The 30-second mental model

Production infra code is mostly the failure paths. The happy path is the easy 20%. What makes you senior is the other 80%: what happens when the BMC hangs, the sensor lies, the disk fills, the config is wrong, SIGTERM arrives mid-write, and the input is hostile. The four habits that get you there: RAII/cleanup-on-every-path, a seam at the hardware boundary so you can test without hardware, structured logs + exit codes so on-call can reason about it, and never trust input. If you remember one sentence: write the code you'd want to debug at 3 a.m. on someone else's machine.

1. The language decision, in one table

Staff move: choose by layer and design the seams between them (a stable C ABI, a JSON contract). "All one language" is a junior answer.

2. RAII in four lines

• Resource lifetime = object scope. Destructor frees it on every exit (return, throw, break).
• unique_ptr is your default owner; shared_ptr only for genuine shared ownership (watch cycles → weak_ptr).
• lock_guard/scoped_lock for mutexes — never manual lock/unlock (early return leaks the lock → deadlock).
• Rule of zero: hold RAII members, write no destructor/copy/move. If you manage a raw resource, rule of five (usually: delete copy, define move).

3. The error taxonomy → response map

Cardinal sin: except: pass / ignoring a return code. Every swallowed error is a future mystery.

4. The hardware-boundary seam (the testability superpower)

[ daemon logic ]  ──uses──▶  interface (SensorSource / RedfishClient / FirmwareTarget)
                                 ├── RealBmcSource     (prod)
                                 └── FakeSource        (tests: hang, garbage, flaky)

Inject the dependency and the clock. Now you test retry, staleness, and timeouts in milliseconds with no hardware. This single pattern is reused in Phases 03, 04, 08, 11.

5. subprocess / shell injection — the 10-second fix

# BAD — shell parses the string; node="x; rm -rf /" runs
subprocess.run(f"ipmitool -H {node} power status", shell=True)
# GOOD — argv list, shell=False (default); node can't be reinterpreted
subprocess.run(["ipmitool", "-H", node, "power", "status"], timeout=10, check=True)

Same rule in C++: never system("..."); use exec/posix_spawn with an argv array.

6. The daemon checklist

• SIGTERM/SIGINT → set atomic stop flag (nothing heavy in the handler) → main loop drains.
• Graceful shutdown: stop new work, finish/cancel in-flight cleanly, never leave a transaction half-done.
• systemd: Restart=on-failure, Type=notify (real readiness via sd_notify), WatchdogSec (free liveness), MemoryMax/TasksMax, sandboxing (ProtectSystem, non-root User=).
• Log to stdout/stderr (journald captures it) — don't write your own log rotation.

7. Concurrency cheat sheet

• I/O-bound (poll 100 BMCs) → asyncio or bounded thread pool. Bound it (semaphore/pool) — don't open 10,000 sockets.
• CPU-bound → processes (the GIL blocks thread parallelism in CPython).
• C++ threads → real parallelism, tiny critical sections under lock_guard, prove with TSan.
• Never hold a lock across I/O or a callback.

8. The CI quality gates (per language)

If it's not in CI, it doesn't hold. Standards you can't enforce are wishes (Phase 12).

9. Security one-liners that matter here

• Validate at the boundary with allow-lists, not deny-lists.
• C++: no strcpy/sprintf; watch integer overflow in size = a * b before malloc.
• Secrets: never in code, logs, or argv (argv is world-readable via /proc/<pid>/cmdline).
• Least privilege: dedicated user, scoped capabilities, read-only FS where possible.
• Pin + checksum + minimize + scan dependencies — every dep is attack surface.

10. The "you've done this before" tells

• "Where's the seam?" (testability) · "What happens on SIGTERM?" (lifecycle) · "Is that arg list or shell-interpolated?" (security) · "Is the critical section under a guard?" (RAII/locks) · "Transient or permanent?" (retry policy) · "Did we bound the concurrency?" (fan-out safety).

11. How this phase pays off later

• The seam + fake pattern powers every later lab's tests (esp. Phases 03, 08, 11).
• The daemon mechanics are how the Phase 07 exporter and Phase 13 capstone agent run.
• The retry/backoff and error taxonomy become the Phase 10 reliability patterns.
• The security discipline is the per-commit half of Phase 09's system security.

Lab 01 — C++ Sensor-Polling Daemon

Phase: 02 — Production Systems Programming | Difficulty: ⭐⭐⭐☆☆ | Time: 4–6 hours Language: C++17 | Hardware: none (fake sensor source) | Needs: a C++17 compiler + make

Concept primer: ../WARMUP.md Ch. 3 (RAII), Ch. 7 (daemons/signals), Ch. 9 (the seam), ../HITCHHIKERS-GUIDE.md §2, §4, §6.

Run

make            # builds ./sensord  (-Wall -Wextra -Werror-clean)
./sensord --once   # one poll cycle against the fake source, then exits
./sensord --run    # daemon loop; Ctrl-C (SIGINT) or SIGTERM to stop cleanly
make test          # ./sensord --self-test — hermetic unit tests, no hardware
make asan          # build + run tests under AddressSanitizer + UBSan

0. The mission

Write a daemon the way infrastructure software is actually written: it polls a sensor source on an interval, keeps the latest readings, survives a sensor that fails or lies, shuts down cleanly on SIGTERM, and is fully unit-tested without any real hardware. This is the C++ bar every close-to-hardware lab in the curriculum is held to.

Two questions you must answer at the end:

1. Where is the seam that lets you test retry/staleness logic with no BMC?
2. What exactly happens to in-flight work and held resources when SIGTERM arrives?

1. The load-bearing pieces (read these in sensord.cpp)

2. What the tests prove (make test)

Five behaviors, all without hardware, all deterministic:

1. Transient recovery — a sensor that fails twice then succeeds is retried exactly to success (asserts the call count).
2. Permanent failure — no configured value → all retries exhausted → no reading (not a crash).
3. Graceful degradation — a sensor that had a value then fails keeps its last value but is marked stale (a rack manager must not lose all state because one read failed).
4. Staleness by clock — advancing the injected clock past the window flips stale with no real sleep.
5. RAII move — moving a SessionGuard doesn't double-release.

This is the whole point of Chapter 9: the seam + injected clock turn "needs a BMC and a stopwatch" into millisecond, repeatable unit tests.

3. Extension exercises (do them — this is the lab)

1. A Result/expected error type: replace std::optional<double> with a type that carries why a read failed (timeout vs auth vs garbage), and route the response by error class (retry timeouts, fail-fast on auth). This is the error taxonomy in code.
2. Expose readings over a Unix socket: add a tiny request/response endpoint so other processes can query the latest readings — and write an integration test that connects to it. (Mind the lock: don't hold mu_ across the socket I/O.)
3. systemd integration: write a sensord.service unit with Type=notify, Restart=on-failure, WatchdogSec=, a non-root User=, and ProtectSystem=strict. Call sd_notify(0, "READY=1") after the first successful poll and ping the watchdog each loop. Now the orchestrator knows true readiness and a hang gets restarted.
4. Prove the locking with TSan: build with -fsanitize=thread, spawn a reader thread calling get() while the poll loop runs, and confirm zero races. Then deliberately remove a lock_guard and watch TSan catch the data race — the bug you'd otherwise chase in production for a week.

4. Common pitfalls

1. Work in the signal handler — calling std::cout/malloc/cleanup from on_signal is undefined behavior (not async-signal-safe). Only the atomic flag is legal.
2. Holding the lock across I/O — if get() did socket I/O under mu_, the poll loop would block on a slow client. Copy out under the lock, do I/O outside.
3. Real sleeps in tests — a test that actually waits 5s for staleness is slow and flaky. Inject the clock.
4. Raw new/delete — use values, unique_ptr, and RAII guards. The compiler- generated special members are correct when you hold only RAII members (rule of zero).
5. Unbounded retry — retrying forever on a permanent failure hammers the BMC. Cap retries and back off.

5. What this lab proves about you

You can write C++ that belongs in a production fleet: resource-safe by construction, testable without the hardware, well-behaved as a service, and clean under sanitizers. In an interview, "write a daemon that polls a sensor and handles a hang" is a senior screen; you answer with the seam, the atomic-flag shutdown, the lock discipline, and "here's how I'd test every failure path in milliseconds" — and you can show the ASan/TSan runs that prove it. That credibility is exactly what the JD's "high-quality, secure, maintainable code" and "debugging complex issues" lines are testing for.

Lab 02 — Production Python Rack CLI (rackctl)

Phase: 02 — Production Systems Programming | Difficulty: ⭐⭐⭐☆☆ | Time: 4–6 hours Language: Python 3 (stdlib only) | Hardware: none (fake BMC client) | Needs: pytest, mypy (optional)

Concept primer: ../WARMUP.md Ch. 5 (production Python), Ch. 8 (security), Ch. 9 (the seam), ../HITCHHIKERS-GUIDE.md §5, §9.

Run

python3 -m rackctl status --rack rack-ai-01            # node power states (text)
python3 -m rackctl status --rack rack-ai-01 --output json
python3 -m rackctl power cycle --node node-01          # JSON logs go to stderr
python3 -m rackctl power off  --node node-00 --dry-run # mutating command, no effect
python3 -m pytest -q                                   # 14 tests, no hardware
python3 -m mypy rackctl                                # strict types clean

0. The mission

Build the tool a rack engineer lives in — rackctl — to the production bar: typed, packaged, structured-logged, retry-aware, subprocess-safe, with explicit config precedence and disciplined exit codes. It's deliberately stdlib-only so you see the patterns, not a framework's magic.

Two questions you must answer at the end:

1. Why does the CLI take a BmcClient it doesn't construct itself?
2. What's the config precedence order, and why must "flag not provided" differ from "flag provided as a falsy value"?

1. The package, file by file

2. The two ideas that make it production-grade

The seam (BmcClient) — main(argv, client=None) lets tests (and the capstone) inject a fake fleet. FakeBmcClient can be told to make a node flaky (make_flaky) so the retry path is exercised deterministically. Production would inject a real Redfish/IPMI client (Phase 03). The CLI logic never changes.

Config precedence with SUPPRESS — argparse has a famous gotcha: with parents=, a subparser's default clobbers a value the main parser already set. The fix here is default=argparse.SUPPRESS on the shared flags, so an attribute exists only if the user actually provided it — which is exactly the "flags override env override file" semantics load_config needs (None means "not provided", so it doesn't override). This is the kind of subtle correctness an interviewer probes with "what if I pass --retries after the subcommand?"

3. What the run shows

status prints a table (or JSON with --output json). Every operation emits a structured JSON log line to stderr (stdout stays clean for piping) — try python3 -m rackctl power cycle --node node-01 and read the {"ts":...,"level":...} log. --dry-run on a mutating command logs "would execute" and changes nothing. Unknown node → exit code 3 (permanent); a flaky node recovers via retries → exit 0.

4. Extension exercises (do them — this is the lab)

1. Async bounded fan-out: add status --all that polls 100 mock nodes concurrently with asyncio and a Semaphore(50) cap (WARMUP Ch. 6). Measure it against a serial loop. The bound matters: 10,000 unbounded connections would exhaust fds and flood the management network.
2. Correlation IDs: generate a request id per invocation and attach it to every log line via a logging filter, so a single rackctl run is greppable end-to-end (foreshadows Phase 07 tracing).
3. Universal --dry-run + confirmation: make every mutating command support --dry-run and require --yes for destructive actions (power off in production).
4. A real subprocess path: add an exec subcommand that runs ipmitool/redfishtool via runner.run([...]), and write a test proving a malicious --node "x; reboot" cannot inject (it's one argv element).
5. Wire the CI gates: add ruff, black --check, mypy --strict, and pytest to a make check target — the Phase 11 quality gates in miniature.

5. Common pitfalls

1. shell=True — the whole reason runner.run refuses strings. Interpolating a node name into a shell string is remote code execution waiting to happen.
2. Bare except: — catch TransientError/PermanentError specifically; a bare catch hides programmer errors and turns bugs into silent wrong behavior.
3. print() for logs — unstructured, unsearchable, and mixes with stdout data. Use the JSON logger to stderr.
4. Retrying permanent failures — an auth error or unknown node won't fix itself; retrying just delays the error and hammers the BMC. The retry_on allow-list prevents it.
5. Secrets in argv/logs — argv is world-readable via /proc; the logger redacts secret-named fields, but never pass a password as a flag in the first place.

6. What this lab proves about you

You can write the Python that is a rack-management product's operator surface: typed, testable, safe, and operable. In an interview, "write a CLI to power-cycle a node and handle a flaky BMC" becomes a showcase — you reach for the seam (testable without hardware), the retry-by-error-class (don't retry auth failures), the arg-list subprocess (no injection), structured logs (debuggable), and exit codes (composable). The same patterns scale to the Phase 03 protocol clients, the Phase 07 exporter, and the Phase 13 capstone — rackctl is the seed of all of them.

Phase 03 — Management Protocols: Redfish, IPMI, SNMP

Difficulty: ⭐⭐⭐☆☆ | Estimated Time: 2 weeks Roles supported: Rack Management SWE (Senior/Staff), Firmware/Platform SWE, DCIM Engineer Hardware needed: none — labs run a Redfish mock and protocol simulators in-process

Why This Phase Exists

This is the phase the JD is most explicit about. It names the protocols directly: "Interface with rack hardware (BMC, PDU, CDU, PCIe switches) using Redfish, SNMP, and IPMI protocols" and requires "Experience with Redfish APIs and DCIM tools." These three protocols are the language your software speaks to every managed device on the out-of-band network (Phase 01, Ch. 3/6). Without them, you cannot read a temperature, power-cycle a node, or learn that a PSU failed.

You must know all three, and — critically — when each applies and why the industry is migrating from IPMI/SNMP to Redfish. Redfish is the modern, structured, secure standard (RESTful, JSON, schema-defined, HTTPS); IPMI is the legacy binary out-of-band standard (deprecated by Intel in 2019 but everywhere in deployed fleets); SNMP is the venerable polling/trap protocol still ubiquitous on PDUs, switches, and older gear. A senior rack engineer writes a Redfish-first abstraction that also speaks IPMI and SNMP for the hardware that needs them.

By the end you will have written a real Redfish client that navigates the data model, performs actions, and subscribes to events against a live (mock) Redfish service, plus an IPMI sensor/SEL reader and an SNMP poller/trap receiver — the exact toolkit a rack manager is built from.

Concepts

Redfish (the modern standard — the one to know cold)

• The DMTF Redfish model: RESTful, JSON, HTTPS, schema-defined, HATEOAS (hypermedia/links)
• The resource tree: ServiceRoot (/redfish/v1) → Systems, Chassis, Managers, UpdateService, EventService, SessionService, TaskService
• ComputerSystem (the host), Chassis (the physical box + Power/Thermal sub-resources), Manager (the BMC itself)
• Actions (@Redfish.Actions): ComputerSystem.Reset, power control, etc. (POST)
• Collections, @odata annotations, Members, navigation links, $expand
• Authentication: HTTP Basic vs Session (X-Auth-Token) vs the SessionService
• Events: subscriptions (push, EventDestination) vs SSE (Server-Sent Events) vs polling
• Tasks: long-running operations (TaskService, Task monitors) — firmware updates return a Task
• Telemetry model: MetricReport, MetricDefinition (the structured-telemetry future)
• Redfish for PDUs/CDUs: the DMTF Power/Thermal equipment schemas; OutletGroup, Outlet, CoolingLoop

IPMI (the legacy out-of-band standard — still everywhere)

• The architecture: BMC, IPMB, ipmitool, IPMI-over-LAN (RMCP+), the LAN session
• Sensors and the SDR (Sensor Data Repository); sensor types, thresholds, readings
• The SEL (System Event Log); decoding event records
• Chassis control (power on/off/cycle), boot device selection, SoL (Serial-over-LAN)
• Why IPMI is deprecated (security history, cleartext, complexity) and why it persists

SNMP (polling + traps — PDUs, switches, environmental gear)

• The model: manager/agent, MIB, OID tree, GET/GETNEXT/GETBULK/WALK, traps/informs
• MIBs and OIDs: the dotted-number namespace, enterprise OIDs, standard MIBs (ENTITY, ENTITY-SENSOR, host-resources)
• SNMP versions: v1/v2c (community strings, cleartext) vs v3 (auth/priv — the secure one)
• Traps vs polling; the trap receiver; why you need both
• Mapping SNMP/IPMI/Redfish into one unified model (foreshadows the Phase 07 exporter)

Cross-cutting (the senior framing)

• Redfish-first with IPMI/SNMP fallback; the driver-abstraction pattern (a Transport/Driver seam)
• Protocol security: HTTPS/cert validation, SNMPv3, never IPMI/SNMPv2 on an untrusted net
• Idempotency, retries, and rate-limiting against fragile BMCs (Phase 02 patterns applied)
• Vendor variance: OEM extensions, schema gaps, the "every BMC is subtly different" reality

Labs

Lab 01 — Redfish Client Against a Live Mock

Lab 02 — SNMP Poller/Trap Receiver + IPMI Sensors

Deliverables Checklist

• Redfish client navigates ServiceRoot → Systems → Chassis → Power/Thermal by following links
• Client performs a ComputerSystem.Reset action and observes the state change
• Client creates an EventService subscription (and you can explain push vs SSE vs poll)
• You can explain why a firmware update returns a Task and how you'd monitor it
• You can decode an IPMI sensor threshold reading and a SEL event
• You can perform an SNMP WALK conceptually and explain GETNEXT lexicographic ordering
• You can state SNMPv2c vs v3 and IPMI security caveats and the Redfish-first rationale
• You normalized all three protocols into one unified reading model

Interview Relevance

• "What is Redfish and why is the industry moving to it from IPMI?"
• "Walk me through how you'd discover a server's power state and CPU temperature with Redfish."
• "How does Redfish authentication work? Basic vs Session tokens?"
• "A firmware update is a long operation — how does Redfish model that?" (Tasks)
• "How do you get an event when a PSU fails — poll or push?" (EventService/SSE vs polling)
• "What's an OID? Walk me through an SNMP GETNEXT/WALK."
• "Your fleet has new Redfish servers and old IPMI/SNMP gear. How do you build one tool?" (driver abstraction)
• "Why should you never run SNMPv2c or IPMI on the tenant network?"

Warmup Guide — Management Protocols: Redfish, IPMI, SNMP

Zero-to-expert primer for Phase 03. You need no prior protocol knowledge: this guide starts at "what is a BMC API" and ends with you navigating a Redfish tree, decoding an IPMI sensor, and walking an SNMP MIB — and knowing exactly when to use which. These protocols are how all of your software touches hardware (Phase 01, Ch. 3).

Table of Contents

• Chapter 1: The Three Protocols and Why There Are Three
• Chapter 2: Redfish — The Data Model
• Chapter 3: Redfish — Navigation, Actions, Auth
• Chapter 4: Redfish — Events and Tasks
• Chapter 5: IPMI — The Legacy You Still Must Speak
• Chapter 6: SNMP — Polling, MIBs, and Traps
• Chapter 7: Security Across the Three
• Chapter 8: The Driver Abstraction — Redfish-First With Fallback
• Chapter 9: Vendor Variance and the Real World
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: The Three Protocols and Why There Are Three

A BMC (and a PDU, a CDU, a switch) is a tiny networked computer that exposes its hardware over a management API. Three such APIs dominate, born in three eras:

• IPMI (Intelligent Platform Management Interface, 1998): a binary out-of-band standard. You send byte-encoded commands to the BMC over a UDP session (IPMI-over-LAN / RMCP+) and get sensor readings, power control, and event logs back. Powerful, ubiquitous in deployed fleets — and deprecated (Intel stopped developing it in 2019) because it was complex and had a bad security history.
• SNMP (Simple Network Management Protocol, 1988): a polling protocol from the networking world. A manager polls an agent for values identified by numeric OIDs, or the agent sends unsolicited traps. Still everywhere on PDUs, switches, UPSs, and environmental sensors.
• Redfish (DMTF, 2015): the modern replacement — a RESTful, JSON, HTTPS, schema-defined API. Human-readable, hypermedia-driven, secure by default, and the standard all new hardware (including AI accelerator platforms) ships. This is the one to know cold.

Why three, still? Because a real fleet is heterogeneous: new accelerator nodes speak Redfish, the rack PDU speaks SNMP, an older switch speaks SNMP, and some BMCs still need IPMI for a specific operation Redfish doesn't yet expose. The senior move is Redfish- first: build your abstraction around Redfish's model and treat IPMI/SNMP as fallback drivers for the gear that needs them (Chapter 8).

Mental model: Redfish is "the hardware as a REST API with a typed schema." IPMI is "the hardware as a binary command set." SNMP is "the hardware as a tree of numbered variables you poll." Same goal (read/control hardware OOB), three idioms.

Chapter 2: Redfish — The Data Model

Redfish models a system as a tree of JSON resources at predictable URLs, all reachable from one entry point. Internalize this tree; it's the spine of the whole protocol.

/redfish/v1                         ← ServiceRoot (the entry point, unauthenticated GET)
├── /Systems        → collection of ComputerSystem   (the HOSTs: power, boot, processors, memory)
│   └── /Systems/1  → one ComputerSystem
│       ├── Actions: #ComputerSystem.Reset           (power on/off/cycle)
│       ├── Processors, Memory, EthernetInterfaces, Storage
│       └── Links → Chassis, ManagedBy(Manager)
├── /Chassis        → collection of Chassis           (the physical boxes)
│   └── /Chassis/1
│       ├── /Power    → PowerSupplies, voltages, PowerControl (draw, cap)
│       └── /Thermal  → Temperatures, Fans
├── /Managers       → collection of Manager            (the BMC ITSELF)
│   └── /Managers/1  → BMC firmware, network, SerialConsole, VirtualMedia
├── /UpdateService  → firmware update (SimpleUpdate, multipart push) → returns a Task
├── /EventService   → event subscriptions (push) / SSE
├── /TaskService    → long-running operations (Tasks)
├── /SessionService → login/logout, sessions
└── /TelemetryService → MetricReports, MetricDefinitions (structured telemetry)

Three resource types carry most of the weight, and beginners conflate them:

• ComputerSystem = the host (the OS-running machine): power state, boot order, CPUs, memory, NICs. "Reset this server" acts here.
• Chassis = the physical enclosure: its Power and Thermal sub-resources hold PSUs, voltages, temperatures, fans. "What's the inlet temperature" reads here.
• Manager = the BMC itself: its own firmware, network config, virtual media, serial console. "Update the BMC firmware" / "mount an ISO" acts here.

A single physical node thus appears as (at least) a ComputerSystem + a Chassis + a Manager, linked together via Links. Knowing which resource owns which capability is the single most useful Redfish skill.

@odata annotations (Redfish builds on OData): @odata.id is a resource's canonical URL (you follow these — never hardcode paths), @odata.type names its schema, and collections carry Members (a list of {@odata.id} links) and Members@odata.count.

Chapter 3: Redfish — Navigation, Actions, Auth

Navigation is hypermedia (HATEOAS): you GET /redfish/v1, read the @odata.id of the Systems collection, GET that, read a member's @odata.id, GET that, follow its Links/Actions. You never hardcode /redfish/v1/Systems/1/... — you follow links, because vendors lay out IDs differently (Chapter 9). Lab 01 does exactly this.

Reading is plain GET returning JSON. $expand lets you pull a collection's members inline in one request (fewer round-trips, bigger payloads — a tradeoff at fleet scale).

Actions are the "do something" verbs, declared in a resource's Actions object and invoked with POST:

POST /redfish/v1/Systems/1/Actions/ComputerSystem.Reset
{ "ResetType": "ForceRestart" }     # GracefulRestart | ForceOff | On | PowerCycle ...

The available ResetType values are advertised in ResetType@Redfish.AllowableValues — you read what's allowed, you don't guess.

Writing config is PATCH (partial update) on a resource (e.g., set the boot device).

Authentication — three levels:

• Unauthenticated: only ServiceRoot and a few metadata resources.
• HTTP Basic: username/password on every request — simple, fine for scripts, but re-auths each call.
• Session-based (the production choice): POST /SessionService/Sessions with credentials → you get an X-Auth-Token header → send it on subsequent requests → DELETE the session to log out. Cheaper than Basic per-call and revocable. Always over HTTPS with certificate validation (Chapter 7).

Chapter 4: Redfish — Events and Tasks

Two features separate a toy client from a production one.

Events — how you learn a PSU failed without polling every BMC constantly:

• Subscription (push): POST /EventService/Subscriptions with a Destination URL (your listener) and the event types you want. The BMC then POSTs events to you. Scales well but requires your endpoint to be reachable by every BMC.
• SSE (Server-Sent Events): connect to the ServerSentEventUri and the BMC streams events over a long-lived HTTP connection. No inbound endpoint needed; one connection per BMC.
• Polling: just GET the relevant resources on an interval. Simplest, but high load and high latency at fleet scale. Use it as a fallback / for resources without events.

The tradeoff (an interview favorite): push/SSE give low latency and low steady-state load but add connection/endpoint management; polling is trivial but doesn't scale to tens of thousands of BMCs at low latency. Production telemetry (Phase 07) usually blends scheduled polling for metrics with event subscriptions for state changes/faults.

Tasks — how Redfish models long-running operations (a firmware update can take minutes): the action returns HTTP 202 Accepted with a Task resource URL (a "task monitor"). You GET that Task to watch TaskState (New→Running→Completed/Exception) and PercentComplete, and read Messages for results. Anything slow returns a Task — firmware updates (Phase 08), some resets, BIOS config jobs. A client that blocks waiting for a synchronous response to a firmware push is broken; the correct pattern is "submit → poll the Task → handle completion/failure" (Lab 01 extension).

Chapter 5: IPMI — The Legacy You Still Must Speak

IPMI is deprecated but pervasive; you will read sensors and SELs from IPMI BMCs for years.

Architecture: the BMC hosts the IPMI stack; you talk to it locally (KCS interface) or remotely over IPMI-over-LAN using RMCP+ (an authenticated, optionally encrypted UDP session — IPMI v2.0). The ubiquitous client is ipmitool:

ipmitool -I lanplus -H <bmc-ip> -U <user> -P <pass> chassis power status
ipmitool -I lanplus -H <bmc-ip> -U <user> -P <pass> sdr list      # sensors
ipmitool -I lanplus -H <bmc-ip> -U <user> -P <pass> sel list      # event log

(lanplus = RMCP+/IPMI 2.0; plain lan is insecure IPMI 1.5 — avoid.)

Sensors and the SDR: the Sensor Data Repository describes every sensor (type, units, and thresholds). A reading comes with its thresholds:

• lnr/lc/lnc = lower non-recoverable / critical / non-critical
• unc/uc/unr = upper non-critical / critical / non-recoverable

So a CPU temp reading isn't just "78°C" — it's "78°C, upper-non-critical 85, upper-critical 95," which lets you classify health (OK/Warning/Critical) the way the hardware intends. Lab 02 models this.

The SEL (System Event Log): a BMC-maintained log of hardware events (a PSU lost input, a DIMM corrected-error threshold, a thermal trip). Each record has a timestamp, sensor, and event type. Parsing the SEL is core to diagnostics (Phase 08).

Other IPMI: chassis control (power on/off/cycle/reset), boot-device override (boot once from PXE — Phase 05), and SoL (Serial-over-LAN) to watch a host boot remotely.

Why deprecated: cleartext options, a history of vulnerabilities (cipher-zero auth bypass, password-hash exposure), and protocol complexity. Never expose IPMI to an untrusted network; keep it on the isolated management VLAN (Phase 09) and prefer Redfish where available.

Chapter 6: SNMP — Polling, MIBs, and Traps

SNMP is how you talk to PDUs, switches, UPSs, and environmental gear that don't (yet) speak Redfish.

The model: a manager (your software) polls an agent (the device). Every value the agent exposes is named by an OID (Object Identifier) — a dotted sequence of numbers locating a node in a global tree, e.g. 1.3.6.1.2.1.1.3.0 is sysUpTime. A MIB (Management Information Base) is the human-readable schema mapping names (sysUpTime) to OIDs and types. Enterprise-specific values live under 1.3.6.1.4.1.<enterprise-number> (e.g., a vendor's PDU outlet metering).

The operations:

• GET — fetch one OID's value.
• GETNEXT — fetch the next OID in lexicographic order. This is how you discover the tree without knowing it in advance.
• WALK — repeated GETNEXT to enumerate a subtree (e.g., all outlets' power draw). The ordering is lexicographic by OID, which is why snmpwalk returns values in a predictable, sorted sequence. Lab 02 implements this ordering.
• GETBULK (v2c+) — fetch many OIDs in one request (efficient WALK).
• SET — write a value (e.g., switch an outlet off — if the PDU allows).

Traps and informs: unsolicited messages the agent sends to your trap receiver (UDP 162) when something happens (a PSU failed). Traps are fire-and-forget; informs are acknowledged. You run a trap receiver to get low-latency notifications and poll for the full picture — neither alone is sufficient (traps can be lost; polling is slow).

Versions (security-critical):

• v1 / v2c: authentication is a cleartext community string ("public"/"private"). No encryption. Never on an untrusted network.
• v3: real security — USM (User-based Security Model) with authentication (HMAC) and privacy (encryption). The only version to use where security matters.

Chapter 7: Security Across the Three

These protocols control power and firmware; a compromise here is catastrophic (Phase 09). The rules:

• Redfish: HTTPS only, validate certificates (don't disable TLS verification — a shockingly common bug), session tokens over Basic, least-privilege Redfish roles (ReadOnly for telemetry, Operator/Administrator only where needed), and never log the token.
• IPMI: lanplus (RMCP+) only, strong passwords, disable cipher-suite 0 (the auth- bypass), and treat it as last-resort. Management VLAN only.
• SNMP: v3 with auth+priv where it matters; if stuck on v2c (read-only PDUs), restrict by ACL to the management network and never use it for writes/secrets.
• All three: live only on the isolated management network (Phase 01, Ch. 6; Phase 09 builds the isolation). The single biggest real-world failure is a BMC/PDU reachable from a tenant or the internet — Shodan is full of exposed IPMI BMCs.

Chapter 8: The Driver Abstraction — Redfish-First With Fallback

A heterogeneous fleet demands one interface, many backends — the same seam idea as Phase 02 Ch. 9, applied to protocols:

class DeviceDriver(Protocol):
    def power_state(self) -> str: ...
    def set_power(self, action: str) -> None: ...
    def read_sensors(self) -> list[Reading]: ...

class RedfishDriver(DeviceDriver): ...   # preferred
class IpmiDriver(DeviceDriver): ...      # fallback for IPMI-only BMCs
class SnmpDriver(DeviceDriver): ...      # PDUs, switches

Your rack-management code programs against DeviceDriver; a factory picks the driver per device (capability detection: try Redfish ServiceRoot, fall back to IPMI/SNMP). This:

• isolates protocol quirks behind a stable interface,
• lets you migrate a device class from IPMI to Redfish without touching callers,
• makes the whole stack testable with a FakeDriver (Phase 02 seam),
• is exactly the structure of the Phase 13 capstone and the Phase 07 exporter.

The unified Reading/state model (Lab 02) is the other half: normalize Redfish Thermal.Temperatures, IPMI SDR readings, and SNMP sensor OIDs into one shape so the rest of your system doesn't care where a temperature came from.

Chapter 9: Vendor Variance and the Real World

The dirty secret: "Redfish-compliant" hardware varies. A senior engineer plans for it.

• Layout differs: System IDs are 1, System.Embedded.1, or a UUID depending on vendor — which is why you follow links, never hardcode paths.
• Schema gaps & OEM extensions: vendors add Oem blocks for features not in the standard schema, and sometimes omit standard fields. Code defensively (.get() with defaults; validate presence).
• Bugs & quirks: some BMCs return 200 with an error body, rate-limit aggressively, drop sessions, or implement an action subtly wrong. Apply Phase 02 robustness: retries with backoff, timeouts, idempotency, and a conformance test per vendor (Phase 11).
• Firmware drift: the same model behaves differently across BMC firmware versions — another reason for the firmware-baseline discipline (Phase 08).

This is where "experience with Redfish APIs" in the JD really means experience with real, imperfect Redfish implementations — and the abstraction (Chapter 8) plus defensive coding (Phase 02) is how you tame it.

Lab Walkthrough Guidance

Order: Lab 01 (Redfish) → Lab 02 (IPMI/SNMP). Learn the modern model first; it gives you the vocabulary (resources, sensors, events) that the legacy protocols map onto.

Lab 01 (Redfish client + mock):

1. python3 solution.py — it starts an in-process Redfish mock and walks it. Read the printed tree.
2. Trace the client: GET /redfish/v1 → follow Systems → a System → its Chassis link → Power/Thermal. Notice it follows @odata.id, never hardcodes.
3. Watch the ComputerSystem.Reset action POST change the System's PowerState.
4. Extensions: poll a Task to completion; add Session-token auth; point it at the real DMTF Redfish-Mockup-Server or OpenBMC sushy-emulator (the README shows how).

Lab 02 (IPMI/SNMP):

1. python3 solution.py — read the SNMP OID-tree WALK (note the lexicographic order) and the IPMI SDR/SEL decode.
2. See all three protocols normalized into one Reading with a health classification from thresholds.
3. Extensions: parse real ipmitool sdr/sel text; run against snmpsim; emit the unified readings in Prometheus exposition format (the Phase 07 bridge).

Phase capstone question: "A node speaks Redfish, its PDU speaks SNMPv2c, and an older peer BMC only speaks IPMI. Sketch the one interface and three drivers, and say how you'd get a low-latency alert when any PSU fails." (Answer: DeviceDriver with Redfish/IPMI/SNMP backends + a unified Reading; PSU alerts via Redfish EventService subscription/SSE and SNMP traps, with scheduled polling as the backstop.)

Success Criteria

You're done with this phase when — without notes:

• You can draw the Redfish tree and say which resource owns power vs temperature vs BMC firmware (Ch. 2)
• You can navigate by following @odata.id links and explain why you never hardcode paths (Ch. 3, 9)
• You can invoke an Action (POST) and read its AllowableValues (Ch. 3)
• You can explain Session vs Basic auth and HTTPS/cert validation (Ch. 3, 7)
• You can explain why firmware updates return a Task and how to monitor one (Ch. 4)
• You can choose push/SSE vs polling for events and justify it at fleet scale (Ch. 4)
• You can decode an IPMI sensor with thresholds and a SEL event (Ch. 5)
• You can perform/explain an SNMP WALK and the lexicographic GETNEXT ordering, and v2c vs v3 (Ch. 6)
• You can sketch the Redfish-first driver abstraction with IPMI/SNMP fallback (Ch. 8)

Interview Q&A

Q1: What is Redfish and why is the industry moving to it from IPMI? A: Redfish is the DMTF's modern out-of-band management standard: a RESTful, JSON, HTTPS, schema-defined API that models a system as a tree of resources (ServiceRoot → Systems, Chassis, Managers, …). The industry is moving to it because IPMI is a complex binary protocol with a poor security history (Intel deprecated it in 2019): Redfish is human-readable, hypermedia-driven (you follow links, so it's vendor-portable), secure by default (HTTPS, session tokens, role-based auth), and extensible via schemas and OEM blocks. It also models modern needs natively — Tasks for long operations, an EventService for push notifications, a TelemetryService for structured metrics, and Power/Thermal schemas that extend to PDUs and CDUs. New hardware ships Redfish; IPMI/SNMP remain as fallbacks for deployed and non-server gear.

Q2: Walk me through reading a server's power state and CPU temperature with Redfish. A: GET /redfish/v1 (ServiceRoot, no auth needed) and read the @odata.id of the Systems collection. GET that, take a member's @odata.id, GET the ComputerSystem — its PowerState field is on/off. For temperature, follow the System's Links to its Chassis, GET the Chassis, follow its Thermal sub-resource, and read Temperatures[] (each has a ReadingCelsius and threshold fields). The key discipline is that I follow @odata.id links rather than hardcoding /Systems/1/..., because vendors lay out IDs differently — that's what makes Redfish portable. I'd do it over HTTPS with a session token and certificate validation.

Q3: A firmware update takes minutes. How does Redfish handle that, and how do you write the client? A: It's an asynchronous operation. You POST to the UpdateService (SimpleUpdate or a multipart push) and the service responds 202 Accepted with a Task resource URL — a task monitor. The client then polls that Task, reading TaskState (New→Running→Completed or Exception) and PercentComplete, and inspects Messages for the result. So the correct pattern is submit → poll the Task with backoff → handle completion or failure (and surface progress) — never block expecting a synchronous response. This same Task pattern covers BIOS config jobs and some resets. I'd also make the submit idempotent and handle the BMC dropping the session mid-update (Phase 08 goes deeper on staging and rollback).

Q4: How do you get notified when a PSU fails — poll or push? A: Both, layered. For low-latency fault notification I subscribe via the Redfish EventService — either a push subscription (the BMC POSTs events to my listener) or SSE (I hold a long-lived connection per BMC and the BMC streams events). Push scales to many BMCs but needs my endpoint reachable; SSE needs no inbound endpoint but is one connection per BMC. For non-Redfish gear (PDUs), I run an SNMP trap receiver on UDP 162. But events can be missed (lost traps, dropped connections), so I also poll the relevant resources on a schedule as a backstop and to get the full state. At fleet scale (tens of thousands of BMCs) pure low-latency polling is too expensive, so the design is event-driven for state changes plus scheduled polling for metrics — exactly the Phase 07 telemetry pattern.

Q5: What's an OID, and what happens in an SNMP WALK? A: An OID (Object Identifier) is a dotted sequence of numbers naming a node in SNMP's global tree — e.g., 1.3.6.1.2.1.1.3.0 is sysUpTime. A MIB is the schema mapping human names to OIDs and types; vendor-specific values live under 1.3.6.1.4.1.<enterprise>. A WALK enumerates a subtree by repeatedly issuing GETNEXT, which returns the next OID in lexicographic order; you keep going until you leave the subtree. So snmpwalk over a PDU's outlet-power subtree returns each outlet's draw in sorted OID order. v2c uses GETBULK to fetch many at once. The security caveat: v1/v2c authenticate with a cleartext community string and no encryption, so they're management- network-only; v3 (USM) adds real auth and privacy and is what you use where security matters.

Q6: Your fleet has new Redfish nodes, SNMP PDUs, and some IPMI-only BMCs. How do you build one tool? A: A driver abstraction: define one DeviceDriver interface (power_state, set_power, read_sensors, …) and implement RedfishDriver, IpmiDriver, and SnmpDriver behind it, with a factory that detects capability per device (try Redfish ServiceRoot; fall back to IPMI/SNMP). All rack-management logic programs against the interface, so protocol quirks are isolated, a device can migrate from IPMI to Redfish without touching callers, and I can inject a FakeDriver for hermetic tests. I'd pair it with a unified Reading/state model that normalizes Redfish Thermal entries, IPMI SDR readings, and SNMP sensor OIDs into one shape, plus defensive coding for vendor variance (follow links, .get() with defaults, retries/timeouts, per-vendor conformance tests). That's the structure of a real rack-management control plane and of this track's capstone.

Q7 (Staff): "Redfish-compliant" hardware from two ODMs behaves differently. How do you keep your platform sane? A: Treat the standard as necessary but not sufficient. First, never hardcode paths — follow @odata.id links so layout differences (System.Embedded.1 vs a UUID) don't break me. Second, code defensively: .get() with defaults, validate presence, tolerate OEM blocks and missing optional fields, and read AllowableValues instead of assuming. Third, build a conformance harness (Phase 11): a suite that exercises each vendor/firmware combo against my expected behavior, run in CI against their emulators, so a quirk is caught before deployment, not in production. Fourth, work the relationship: feed concrete defects and schema-gap requests back to the ODM (e.g., Pegatron) as part of the protocol documentation and EBOM process (Phase 12) — driving them toward conformance is part of the Staff job. Finally, pin and track BMC firmware versions (Phase 08) because the same model drifts across firmware. The abstraction layer plus the conformance suite plus the partner feedback loop is how a platform stays maintainable across a heterogeneous, imperfect fleet.

References

• DMTF Redfish — Specification, Schema Index, and the Redfish School — https://www.dmtf.org/standards/redfish
• DMTF Redfish-Mockup-Server (a real Redfish service to test against) — https://github.com/DMTF/Redfish-Mockup-Server
• OpenBMC sushy-emulator (a Redfish emulator) — https://docs.openstack.org/sushy/latest/
• DMTF Redfish Power/Thermal & Outlet equipment schemas (PDU/CDU) — https://redfish.dmtf.org/schemas/
• IPMI v2.0 Specification (Intel/Dell/HP/NEC) and ipmitool — https://github.com/ipmitool/ipmitool
• IETF SNMP — RFC 1157 (v1), RFC 3416 (v2c ops), RFC 3414/3826 (v3 USM/priv)
• Net-SNMP tools (snmpwalk, snmptrapd) & snmpsim simulator — http://www.net-snmp.org/ , https://github.com/etingof/snmpsim
• "Penetration Testing a BMC" / IPMI security background (Rapid7, Dan Farmer's IPMI papers) — for Chapter 7
• Cross-track: Security Engineer — hardware/firmware device security

🛸 Hitchhiker's Guide — Phase 03: Management Protocols

Read this if: you need to talk to a BMC/PDU/CDU and "Redfish, IPMI, SNMP" is an alphabet soup. This is the compressed field guide — the tree, the commands, the gotchas.

0. The 30-second mental model

Three ways to talk to hardware out-of-band, three eras:

• Redfish = the hardware as a REST/JSON tree (modern, secure, the one to master).
• IPMI = the hardware as a binary command set (legacy, deprecated, still everywhere).
• SNMP = the hardware as a tree of numbered variables you poll (PDUs/switches).

Senior move: Redfish-first with IPMI/SNMP as fallback drivers behind one interface. If you remember one rule: follow @odata.id links, never hardcode Redfish paths.

1. The Redfish tree (tattoo this)

/redfish/v1                     ServiceRoot (GET, no auth)
├─ /Systems/{id}                the HOST    → PowerState, boot, CPUs; Actions: Reset
├─ /Chassis/{id}                the BOX     → /Power (PSUs, draw, cap), /Thermal (temps, fans)
├─ /Managers/{id}               the BMC     → its firmware, network, VirtualMedia, SerialConsole
├─ /UpdateService               firmware    → returns a TASK
├─ /EventService                push/SSE events
├─ /TaskService                 long ops
└─ /SessionService              login → X-Auth-Token

Which resource owns what (the #1 confusion): System = host power/boot, Chassis = temps/PSUs, Manager = the BMC itself. One node = a System + a Chassis + a Manager.

2. Redfish verbs

Read …@Redfish.AllowableValues for what a field/action accepts — don't guess.

3. IPMI in five commands (ipmitool -I lanplus -H ip -U u -P p ...)

chassis power status|on|off|cycle|reset
sdr list            # sensors + readings + thresholds
sensor list         # detailed sensors
sel list            # the System Event Log (hardware events)
chassis bootdev pxe # boot-once override (Phase 05)
sol activate        # Serial-over-LAN console

Sensor reading = value + thresholds: lnr lc lnc | reading | unc uc unr. Health is "which threshold did it cross." Always lanplus (RMCP+); never plain lan/cipher-0.

4. SNMP in one breath

• Value = OID (1.3.6.1.2.1.1.3.0 = sysUpTime). MIB = the name↔OID schema. Vendor stuff under 1.3.6.1.4.1.<enterprise>.
• GET one · GETNEXT the next-in-lexicographic-order · WALK a subtree · GETBULK many · SET write · trap = device → you (UDP 162).
• snmpwalk -v3 -l authPriv -u user -a SHA -A authpw -x AES -X privpw host OID
• v1/v2c = cleartext community string (mgmt-net only); v3 = auth+priv (use this).

5. Events: poll vs push vs SSE

Real telemetry (Phase 07): poll for metrics + subscribe for state changes/faults.

6. Security non-negotiables

• Redfish: HTTPS + validate the cert (disabling TLS verify is the classic bug), session tokens, least-privilege roles, never log the token.
• IPMI: lanplus only, disable cipher-0, strong creds, last resort.
• SNMP: v3 where it matters; v2c read-only + ACL otherwise.
• All three live ONLY on the isolated management VLAN (Phase 09). Exposed BMCs are a Shodan staple and a fleet-wide compromise.

7. The driver abstraction (the senior pattern)

DeviceDriver(power_state, set_power, read_sensors)
   ├─ RedfishDriver   (preferred)
   ├─ IpmiDriver      (fallback)
   └─ SnmpDriver      (PDUs/switches)
+ unified Reading model so callers don't care where a temp came from

This is the Phase 13 capstone and Phase 07 exporter in one diagram.

8. Vendor variance survival kit

• Follow links (IDs differ: 1 vs System.Embedded.1 vs UUID).
• .get() with defaults; tolerate Oem blocks and missing optional fields.
• Retries/backoff/timeouts (BMCs are flaky); idempotent actions.
• A per-vendor conformance suite in CI (Phase 11); pin BMC firmware (Phase 08).

9. The "done this before" tells

"Which resource owns that — System, Chassis, or Manager?" · "Are you following the link or hardcoding the path?" · "Is that a Task? Poll it." · "v2c or v3?" · "lanplus, right?" · "Is TLS verification on?" · "Event or poll — and what's the backstop?"

10. How this phase pays off later

• The driver + unified Reading is the foundation of the Phase 07 exporter and Phase 13 capstone.
• Tasks are how Phase 08 firmware updates are monitored.
• EventService/traps feed Phase 07 alerting and Phase 10 RCA.
• Protocol security is enforced by the Phase 09 management-network isolation.

Lab 01 — Redfish Client Against a Live Mock

Phase: 03 — Management Protocols | Difficulty: ⭐⭐⭐☆☆ | Time: 4–6 hours Language: Python 3 (stdlib only) | Hardware: none (in-process Redfish mock)

Concept primer: ../WARMUP.md Ch. 2–4, ../HITCHHIKERS-GUIDE.md §1–§2, §5.

Run

python3 solution.py     # starts a Redfish mock, walks it, performs Reset, subscribes

Run it for REAL against the DMTF Redfish-Mockup-Server (the extension — verified)

redfish_real.py reuses the exact same RedfishClient against an independent, real Redfish service — proving the link-following design (the client doesn't change):

git clone https://github.com/DMTF/Redfish-Mockup-Server
pip install -r Redfish-Mockup-Server/requirements.txt
python3 Redfish-Mockup-Server/redfishMockupServer.py \
    -D Redfish-Mockup-Server/public-rackmount1 -S -p 8000 &     # -S = short-form mockup
python3 redfish_real.py http://localhost:8000

Verified output against DMTF's public-rackmount1 mockup:

System: id=437XR1138R2 model=3500 power=On  (id is a vendor string, not hardcoded)
  temp  CPU1 Temp                41C
  temp  CPU2 Temp                NoneC          # a real sensor with no reading -> code defensively
  power System Input Power       344 W
  reset allowable: ['On','ForceOff','GracefulShutdown','GracefulRestart','ForceRestart','Nmi',...]

The System id is 437XR1138R2 — not node-00 or 1. That's exactly the vendor variance the "follow @odata.id links, never hardcode" rule (WARMUP Ch. 3, 9) exists for; the client handles it unchanged, and .get() tolerates the None-reading sensor. (You can also point it at OpenBMC's sushy-emulator or a real BMC — same code.)

0. The mission

Write the Redfish client a rack manager is built on — and write it the right way: by following @odata.id links from the ServiceRoot, not by hardcoding paths. To make it runnable with zero setup, the lab spins up a small in-process Redfish HTTP mock; the client code is the real thing and works unchanged against the DMTF mockup server, OpenBMC's sushy-emulator, or a real BMC.

Two questions you must answer at the end:

1. Why follow links instead of constructing /redfish/v1/Systems/1/...?
2. Which resource holds power state, which holds temperature, and which holds BMC firmware?

1. What the client does (read RedfishClient)

Notice the System owns PowerState and the Reset action; the Chassis owns Power (PSUs, draw, cap) and Thermal (temps, fans); the Manager owns BMC firmware. Putting the right capability on the right resource is the core Redfish skill (WARMUP Ch. 2).

2. What the run shows

System node-00 (AI Compute Node 00) power=On health=OK
  power: 1480 W consumed, cap 2000 W, PSUs ['OK', 'OK']
  temp: Inlet        27C (crit 45)
  temp: Accelerator0 71C (crit 90)
  reset(GracefulShutdown) -> HTTP 204     ← action accepted (No Content)
  power now: Off                          ← state actually changed
  subscription created -> HTTP 201 at .../Subscriptions/sub-1

The self-tests additionally prove that an illegal ResetType is rejected with 400 (the server checks AllowableValues), and that the subscription appears in the collection.

3. Extension exercises (do them — this is the lab)

1. Run it against a real Redfish service. Install the DMTF mockup server (pip install redfish-mockup-server or clone DMTF/Redfish-Mockup-Server) or OpenBMC's sushy-emulator, start it, and point RedfishClient(base) at it. Your client should work unchanged — that's the payoff of following links.
2. Tasks: add an UpdateService SimpleUpdate that returns 202 + a Task, and make the client poll the Task (TaskState, PercentComplete) to completion. This is the exact pattern Phase 08 firmware updates use.
3. Session auth: add a SessionService to the mock; have the client POST credentials, capture the X-Auth-Token, send it on every request, and DELETE the session at the end. Then add HTTPS with certificate validation (and a test that rejects a bad cert).
4. $expand: implement $expand=. on the Systems collection so one GET returns all members inline, and discuss the round-trips-vs-payload tradeoff at fleet scale.
5. Fold into the driver abstraction: make RedfishClient implement the DeviceDriver interface from WARMUP Ch. 8 so it's swappable with the IPMI/SNMP drivers from Lab 02.

4. Common pitfalls

1. Hardcoding paths — /redfish/v1/Systems/1 works on one vendor and 404s on the next (IDs differ: 1, System.Embedded.1, a UUID). Follow @odata.id.
2. Assuming fields exist — real BMCs omit optional fields and add Oem blocks. Use .get() with defaults; validate presence.
3. Blocking on async operations — a firmware update returns a Task, not a result. Poll it; don't wait for a synchronous body.
4. Disabling TLS verification — the single most common Redfish security bug. Validate the certificate; pin or trust the management CA.
5. Re-authenticating per call — use a session token, not Basic auth on every request, at fleet scale.

5. What this lab proves about you

You can write the Redfish client at the heart of any rack-management product — correctly (link-following, action validation, events) and portably (works against any conformant service). In an interview, "discover a server's power and temperature over Redfish" becomes a confident walk of ServiceRoot → System → Chassis → Power/Thermal, with the right notes on Sessions, HTTPS, Tasks, and events. This client becomes a DeviceDriver backend in the Phase 13 capstone and the data source for the Phase 07 exporter.

Lab 02 — SNMP Poller/Trap Receiver + IPMI Sensors

Phase: 03 — Management Protocols | Difficulty: ⭐⭐⭐☆☆ | Time: 4–6 hours Language: Python 3 (stdlib only) | Hardware: none (simulated agents)

Concept primer: ../WARMUP.md Ch. 5 (IPMI) & Ch. 6 (SNMP), ../HITCHHIKERS-GUIDE.md §3–§4.

Run

python3 solution.py     # SNMP WALK + trap decode + IPMI SDR/SEL + unified view

0. The mission

Learn the two legacy protocols by implementing their semantics: a lexicographic SNMP WALK, an IPMI SDR sensor with real thresholds and a health verdict, a SEL decode, and — the senior part — normalizing all of it into one Reading model so the rest of your system doesn't care whether a temperature came from Redfish, IPMI, or SNMP.

Two questions you must answer at the end:

1. Why does an SNMP WALK return .2 before .10 (and what breaks if you sort OIDs as strings)?
2. How does an IPMI sensor reading become "OK / Warning / Critical"?

1. The real commands this models

2. The two ideas that matter

Lexicographic OID ordering — OIDs sort numerically per component, so ...2.1.2 comes before ...2.1.10. Sorting OIDs as plain strings is a classic bug (you'd get .1, .10, .2). _oid_key splits on . and compares integer lists; WALK then collects a subtree in the order a real agent returns it. This is why snmpwalk output is predictably ordered.

IPMI threshold health — a sensor reading is meaningless without its thresholds. Each IpmiSensor carries lnr/lc/lnc (lower non-recoverable/critical/non-critical) and unc/uc/unr (upper). health() walks them from most to least severe and returns OK/Warning/Critical — exactly how the hardware intends a reading to be interpreted. The "Accel0 Temp 92°C → Warning" line is the upper-non-critical band in action.

3. What the run shows

A WALK in correct numeric order, a decoded PSU-failure trap, IPMI sensors classified by threshold (note the hot accelerator flagged Warning), the SEL hardware-event log decoded to human strings, and finally 9 readings from two protocols collapsed into one model with a health histogram — the seed of the Phase 07 telemetry pipeline.

4. Extension exercises (do them — this is the lab)

1. Point SNMP at snmpsim. pip install snmpsim, run snmpsimd with a recorded community, and replace the PDU_AGENT dict with real urllib/pysnmp GETNEXT calls. Your WALK logic stays the same.
2. Parse real ipmitool output. Capture ipmitool sdr list / sel list text and write a parser that produces IpmiSensor/SEL records — real fleets often shell out to ipmitool (safely, via an arg list — Phase 02 Ch. 8) rather than speak the binary protocol directly.
3. A real trap receiver. Bind a UDP socket on 162 (or a high port) and decode incoming SNMP trap PDUs; integrate with decode_trap. Discuss trap loss and why you still poll.
4. SNMPv3. Add the USM concepts (auth + priv) and explain why v2c community strings are management-network-only.
5. Emit Prometheus. Render unified_view() in Prometheus exposition format (pdu_outlet_power_watts{outlet="1"} 410) — the literal bridge to the Phase 07 exporter.

5. Common pitfalls

1. String-sorting OIDs — .10 sorts before .2 as strings; WALK breaks. Sort by integer components.
2. Reading without thresholds — "78°C" alone can't be classified; carry the SDR thresholds with the value.
3. Trap-only or poll-only — traps can be lost (UDP, fire-and-forget); polling is slow. Production does both.
4. v2c on an untrusted net — the community string is cleartext; it's mgmt-VLAN-only, and v3 where security matters.
5. Per-protocol data shapes leaking upward — normalize to one Reading at the driver boundary so callers (alerting, dashboards) stay protocol-agnostic.

6. What this lab proves about you

You understand the protocols a heterogeneous fleet actually speaks, down to the WALK ordering and the threshold model — and you can fuse them into one coherent telemetry view. In an interview, "how do you monitor a PDU and an old IPMI BMC alongside Redfish nodes?" becomes a clear answer: per-protocol drivers, threshold-aware health, traps + polling, and one unified model — which is exactly the pipeline you build in Phase 07 and the capstone.

Phase 04 — Hardware Interfacing: BMC, PDU, CDU, PCIe Switches

Difficulty: ⭐⭐⭐⭐☆ | Estimated Time: 2 weeks Roles supported: Rack Management SWE (Senior/Staff), Firmware/Platform SWE, HW-bringup SWE Hardware needed: none — simulators + your own machine's lspci

Why This Phase Exists

Phase 03 gave you the protocols; this phase gives you the devices those protocols talk to. The JD names them explicitly: "Interface with rack hardware (BMC, PDU, CDU, PCIe switches)." Each is a distinct managed component with its own model, failure modes, and control surface, and a rack manager must understand all four:

• BMC — the management brain of every node (power, sensors, console, virtual media, firmware).
• PDU — power distribution and metering; the source of per-outlet/per-phase power truth.
• CDU — the coolant distribution unit of a liquid-cooled rack; flow/ΔT/pressure/leak (Phase 01 Ch. 5).
• PCIe switches — the fabric wiring accelerators, NICs, and CPUs together; topology, lanes, hot-plug, errors.

"Understanding of server hardware, networking, and storage systems" (a JD requirement) is exactly this layer. You'll learn what each device is, how to read and control it, what breaks, and how to model it — then build the tools: a BMC interaction toolkit, a PDU/CDU telemetry simulator, and a PCIe topology mapper that turns your own machine's lspci into a fabric graph.

Concepts

BMC / OpenBMC (deeper than Phase 01)

• The BMC SoC (e.g., ASPeed AST2600), its RTOS/Linux, standby power, the host↔BMC interfaces (KCS, LPC/eSPI, PCIe VDM, NC-SI for shared NIC)
• OpenBMC architecture: D-Bus, phosphor-* services, the sensor/Entity model, obmc mapping to Redfish/IPMI
• Capabilities: power/reset control, sensor aggregation, SoL, KVM/iKVM, virtual media (mount ISO), SEL, firmware (host BIOS + BMC self-update)
• The BMC as a security boundary (Phase 09): it can do anything; protect it accordingly

PDU (Power Distribution Unit)

• Types: basic, metered, switched, metered-and-switched; per-outlet vs per-phase metering
• Busbar vs PDU power topologies (OCP); 3-phase balance; the metering data model
• Reading draw, switching outlets, alarms; protocols (SNMP common, Redfish PDU schema emerging)
• Why per-outlet metering is the ground truth for the Phase 01 budget calculator

CDU (Coolant Distribution Unit) & liquid cooling telemetry

• The secondary (rack) loop vs primary (facility) loop; heat exchanger; pumps; manifolds; cold plates
• Telemetry: supply/return temperature, ΔT, flow (L/min), pressure, pump speed, leak detection
• Failure modes: pump failure, fouling (ΔT rises at constant load), blockage, leak → emergency action
• Protocols: Modbus/SNMP/Redfish; the safety-critical nature of leak handling

PCIe switches & fabric

• PCIe basics: lanes, links, generations (Gen4/5/6 bandwidth), the tree topology (root complex → switches → endpoints)
• PCIe switches (e.g., Broadcom/Microchip): fanout, lane allocation, hot-plug, peer-to-peer (P2P) for GPU-direct
• Reading topology: BDF (bus:device.function), lspci, sysfs (/sys/bus/pci/devices), capabilities, link speed/width
• Errors: AER (Advanced Error Reporting), correctable vs uncorrectable, link down/degraded (a degraded x16→x4 link is a silent perf killer) — feeds Phase 08 RAS
• The scale-up fabric (NVSwitch-class) vs PCIe — where each lives

Cross-cutting

• The device-driver abstraction extended to all four device classes (Phase 03 Ch. 8)
• Modeling them in the Phase 01 inventory graph; relating telemetry to components
• Safety: power and cooling actions are physical and irreversible-ish — confirmation, interlocks, dry-run

Labs

Lab 01 — BMC Power Control, Sensors & Virtual Media

Lab 02 — PDU/CDU Telemetry Simulator + PCIe Topology Mapper

Deliverables Checklist

• You can list OpenBMC capabilities and explain the host↔BMC interfaces
• BMC toolkit reads sensors and controls power with a dry-run/confirmation guard
• You can describe the virtual-media remote-reinstall flow end to end
• PDU simulator aggregates per-outlet draw into a rack total (feeds Phase 01)
• CDU simulator alarms on leak and on ΔT rising at constant load
• PCIe mapper builds a topology graph from lspci and flags a degraded link
• You can explain why a degraded x16→x4 link is a silent performance killer
• You can relate each device to the Phase 01 inventory model and Phase 08 RAS

Interview Relevance

• "What is a BMC made of, and how does the host talk to it?"
• "How would you reinstall an OS on a server you can't physically touch?" (virtual media)
• "A rack's cooling is degrading — what telemetry tells you before it fails?" (ΔT, flow, pump)
• "How do you handle a coolant leak signal in software?" (safety-critical, emergency action)
• "Walk me through reading PCIe topology on Linux. What's a BDF?"
• "An accelerator is slow but 'healthy' — how could PCIe be the cause?" (degraded link width/speed)
• "How do you make a power-off operation safe to expose in a tool?"

Warmup Guide — Hardware Interfacing: BMC, PDU, CDU, PCIe Switches

Zero-to-expert primer for Phase 04. Phase 03 taught the protocols; this teaches the devices — what each one is, how to read and control it, how it fails, and how to model it. By the end you can reason about a node's BMC, its power and cooling, and its PCIe fabric the way a hardware-fluent software engineer must.

Table of Contents

• Chapter 1: The BMC, Under the Hood
• Chapter 2: OpenBMC — How the Management Stack Is Built
• Chapter 3: BMC Operations — Power, Console, Virtual Media
• Chapter 4: PDUs and Power Distribution
• Chapter 5: CDUs and Liquid-Cooling Telemetry
• Chapter 6: PCIe — The Fabric Inside the Node
• Chapter 7: Reading PCIe Topology on Linux
• Chapter 8: Modeling and Safety
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: The BMC, Under the Hood

The BMC (Phase 01 Ch. 3) is a real computer; now look inside. A typical server BMC is an ASPeed AST2500/2600 — an ARM SoC with its own RAM, flash, Ethernet MAC, and a pile of hardware interfaces to the host. It runs on standby power (alive whenever the rack has any power, even with the host off) and boots its own firmware (often OpenBMC, a Linux distribution) independently of the host.

How the BMC senses and controls the host — the interfaces matter because they're where bugs and security boundaries live:

• KCS / BT (LPC or eSPI): the legacy in-band channel the host OS uses to talk IPMI to the BMC locally.
• eSPI / LPC: the bus carrying low-pin-count peripherals, TPM, and host↔BMC messaging.
• PECI: the Intel interface the BMC uses to read CPU temperature/telemetry directly.
• I²C / PMBus: how the BMC reads voltage regulators, PSUs, temperature sensors, and the FRU EEPROMs (the parts inventory — Phase 01 Ch. 10).
• NC-SI: lets the BMC share the host's NIC for management traffic (vs a dedicated mgmt port) — a shared-NIC design has security implications (Phase 09).
• PCIe VDM / MCTP: modern in-band management messaging (e.g., to NICs, accelerators).

The key idea: the BMC aggregates every low-level sensor and control into one management endpoint, then exposes them via Redfish/IPMI (Phase 03). When you GET a temperature over Redfish, the BMC read it over PECI/I²C and translated it. Understanding this chain is what lets you debug "the sensor reads garbage" (Phase 10) — is it the sensor, the I²C bus, the BMC firmware, or the Redfish mapping?

Chapter 2: OpenBMC — How the Management Stack Is Built

OpenBMC (a Linux Foundation project, used by Meta, Google, IBM, Microsoft, and many ODMs including for accelerator platforms) is worth knowing because it's the firmware your software talks to, and increasingly the one you might contribute to alongside an ODM (Phase 12).

Architecture, briefly:

• It's embedded Linux + a set of services communicating over D-Bus (the IPC bus).
• The phosphor-* services each own a domain: phosphor-hwmon (sensors from kernel hwmon), phosphor-state-manager (host/chassis power state), phosphor-host-ipmid (the IPMI daemon), entity-manager (discovers what hardware is present from config + EEPROMs), phosphor-logging (events/SEL).
• bmcweb is the HTTP server that exposes Redfish (and a web UI), translating D-Bus state into the Redfish resource tree you navigated in Phase 03. So "the Redfish Thermal resource" is bmcweb rendering phosphor-hwmon's D-Bus sensor objects.

Why this matters: when a Redfish field is missing or wrong on an OpenBMC system, the chain is sensor → hwmon → D-Bus → bmcweb → Redfish JSON. Knowing the chain turns "vendor bug" into a specific, reportable defect (and sometimes a patch you send upstream). It's also why "every BMC is subtly different" (Phase 03 Ch. 9): different entity-manager configs and firmware versions render different trees.

Chapter 3: BMC Operations — Power, Console, Virtual Media

The operations your toolkit (Lab 01) performs, and their semantics:

Power control — the host has a small state machine the BMC drives:

• On / ForceOff (yank power — abrupt) / GracefulShutdown (ask the OS to shut down — needs a cooperating OS) / ForceRestart / GracefulRestart / PowerCycle.
• The subtlety: GracefulShutdown can fail silently if the host OS is hung (nothing to ask). Robust tooling tries graceful, waits, then escalates to force — exactly the retry/escalation pattern from Phase 02.

Console — see what the host is doing without a monitor:

• SoL (Serial-over-LAN): the host's serial console redirected over the BMC's network — text, scriptable, the workhorse for watching boot/POST and kernel panics (Phase 08/10).
• KVM / iKVM: full remote keyboard-video-mouse (graphical), for BIOS screens.

Virtual media — the killer OOB feature: the BMC presents a network ISO/image as if it were a physical USB/CD drive to the host. Combined with boot-device override, this is how you reinstall an OS on a server you can't touch: mount the installer ISO over Redfish, set next-boot to the virtual CD, power-cycle, and the host boots your installer. This is the backbone of bare-metal provisioning's "rescue path" (Phase 05) and a primary reason OOB exists. Lab 01 implements the mount→set-boot→cycle→eject lifecycle.

Firmware — the BMC can update the host's BIOS/UEFI/NIC firmware and self-update its own firmware (Phase 08), all over the management network.

Chapter 4: PDUs and Power Distribution

The PDU is where your power truth comes from. Types (in increasing capability):

• Basic: dumb power strip. No telemetry. (Rare in modern racks.)
• Metered: reports power draw (per-phase and/or per-outlet). Read-only truth for budgeting.
• Switched: can turn individual outlets on/off remotely (e.g., hard power-cycle a wedged device whose BMC is also wedged).
• Metered + switched: both — the common AI-rack choice.

Per-outlet vs per-phase metering: per-outlet tells you which device draws what (great for the Phase 01 budget and for attributing power to tenants); per-phase tells you balance across the 3 phases (an unbalanced load wastes capacity and can trip a phase). A good rack manager reads both.

Busbar designs (OCP): in OCP racks the power shelf feeds a DC busbar and nodes blind-mate on; metering may live in the power shelf rather than discrete PDUs. Same data (draw, per-feed), different physical form.

Protocols: SNMP is the historical standard for PDUs (Phase 03 Ch. 6 — you walk an outlet subtree); the DMTF Redfish PDU/Outlet schema is the modern path. Switching an outlet is a physical, disruptive action — guard it (Chapter 8).

Why it feeds Phase 01: the budget calculator's "measured draw" is exactly per-outlet PDU telemetry. Nameplate is a guess; the PDU is ground truth. Lab 02 simulates this and you can feed it back into the Phase 01 calculator.

Chapter 5: CDUs and Liquid-Cooling Telemetry

The CDU (Phase 01 Ch. 5) is a managed device you monitor and, in emergencies, act on.

The two loops: the secondary (rack-side) loop is clean coolant the CDU pumps through cold plates; the primary (facility) loop is chilled water. The CDU's heat exchanger moves heat between them while keeping the clean loop isolated from facility water (which may be dirty/corrosive). You manage the secondary loop's behavior.

Telemetry you read (and what each tells you):

• Supply / return temperature and ΔT (return − supply): the workhorse signal. Heat removed ∝ flow × ΔT. ΔT rising at constant IT load is the classic early warning of reduced flow or fouling — a pre-failure signal you alert on (Lab 02).
• Flow (L/min): drops indicate a failing/throttled pump or a blockage.
• Pressure: out-of-range indicates a leak, blockage, or pump issue.
• Pump speed / status: redundant pumps (N+1) — losing one is tolerable, losing the spare is not.
• Leak detection: rope/spot sensors. Safety-critical — a leak near live electronics can destroy hardware and is a hazard. Software response is decisive: alarm at top priority, and (per policy/interlock) trigger an emergency action (reduce load via power capping, or controlled shutdown/valve action). Treat it like a fire alarm in code.

Protocols: CDUs commonly speak Modbus (industrial) and/or SNMP, with Redfish CoolingLoop/CoolingUnit schemas emerging. Your software normalizes whichever into the unified model (Phase 03 Ch. 8).

Failure modes to recognize (interview-ready): pump failure (flow→0, temps spike), fouling/scaling (ΔT creeps up over weeks), air in the loop (erratic flow/pressure), blockage (high pressure, low flow), leak (pressure drop + leak sensor). Each has a distinct telemetry signature — that's why you collect all of it.

Chapter 6: PCIe — The Fabric Inside the Node

PCIe (Peripheral Component Interconnect Express) wires the CPU, accelerators, NICs, and NVMe together inside a node. You must understand it because (a) accelerators and NICs hang off it, (b) its topology determines performance (P2P, NUMA), and (c) its errors are a major RAS source (Phase 08).

The basics:

• A lane is a serial differential pair (TX+RX). A link bundles lanes: x1, x4, x8, x16 (the common GPU/accelerator width). More lanes = more bandwidth.
• Generations double per-lane bandwidth: Gen4 ≈ 2 GB/s/lane, Gen5 ≈ 4 GB/s/lane, Gen6 ≈ 8 GB/s/lane. So a Gen5 x16 link ≈ 64 GB/s each way. (Recall Phase 01: PCIe is the narrow tier vs HBM/NVLink.)
• Topology is a tree: a Root Complex (in the CPU) → PCIe switches (fan-out) → endpoints (accelerators, NICs, NVMe). Switches let one root port fan out to many devices and enable peer-to-peer (P2P) transfers (e.g., GPUDirect RDMA: NIC→GPU without bouncing through CPU memory).
• BDF addressing: every function is Bus:Device.Function (e.g., 0000:3b:00.0), the coordinate you use to identify a device in lspci/sysfs.

PCIe switches (Broadcom/Microchip et al.) are managed devices in AI systems: they allocate lanes, support hot-plug (swap an NVMe/accelerator without reboot), and report errors. In some designs the switch fabric is reconfigurable (lanes reassigned between hosts).

Errors and AER (the RAS hook for Phase 08): PCIe Advanced Error Reporting classifies errors as correctable (recovered by the link layer — but a rising rate predicts failure) vs uncorrectable (fatal/non-fatal — data integrity at risk). A degraded link — negotiated down from x16 to x4, or Gen5 to Gen1 — is the silent killer: the device works and reports "healthy," but bandwidth collapsed, so an accelerator mysteriously runs at a fraction of speed. Detecting link width/speed below capability is a high-value diagnostic (Lab 02).

Chapter 7: Reading PCIe Topology on Linux

The concrete skills (Lab 02 makes them software):

• lspci: lists PCIe devices. lspci -tv shows the tree; lspci -vvv shows capabilities including LnkCap (what the link can do) vs LnkSta (what it negotiated). The diagnostic gold is LnkSta width/speed < LnkCap → degraded link. lspci -mm -nn gives machine-parseable output with vendor/device IDs.
• sysfs: /sys/bus/pci/devices/<BDF>/ exposes current_link_speed, current_link_width, max_link_speed, max_link_width, aer_dev_correctable, aer_dev_fatal, and numa_node. This is what your monitoring code reads (no shelling out).
• NUMA: numa_node tells you which CPU socket a device is "near." Placing a workload's GPU and NIC on the same NUMA node avoids cross-socket PCIe hops — a real perf lever and a scheduling input (Phase 06).

A rack manager that maps topology can answer: "are all 8 accelerators at full x16 Gen5?", "which NIC is P2P-adjacent to which GPU?", "is this node's slow training a degraded-link problem?" — questions that otherwise take an engineer hours on-site.

Chapter 8: Modeling and Safety

Modeling: each device class slots into the Phase 01 inventory graph — BMCs manage nodes, PDUs power them, CDUs cool them, PCIe switches connect components — and their telemetry attaches to components with thresholds (Phase 03 unified Reading). The driver abstraction (Phase 03 Ch. 8) extends to all four: a DeviceDriver per class, one model above.

Safety — this phase is the first where your software causes physical, hard-to-undo effects: powering off a node mid-job, switching a PDU outlet (kills whatever's on it), acting on a leak. The disciplines:

• Dry-run by default for mutating operations; require explicit confirmation (--yes) for destructive ones (Phase 02 Lab 02 pattern). Lab 01 enforces this.
• Interlocks: don't allow a power-on if cooling is failed; don't switch an outlet feeding a node that's mid-firmware-update.
• Blast-radius awareness (Phase 01 Ch. 7): "switch off outlet 3" might drop a whole shelf — show the operator the blast radius before acting.
• Idempotency & confirmation of effect: after a power action, read back the state to confirm it happened (don't trust the request succeeded).
• Audit: every physical action is logged with who/what/when (Phase 07/12).

The leak case is the archetype: software that can act on the physical world must be more careful, not less, than software that only moves bytes.

Lab Walkthrough Guidance

Order: Lab 01 (BMC) → Lab 02 (PDU/CDU/PCIe). The BMC is the control surface; the others are the power/cooling/fabric truth around it.

Lab 01 (BMC toolkit):

1. python3 solution.py — runs against the bundled BMC mock; read the power state machine.
2. Find the dry-run/confirm guard on every mutating call — this is the safety discipline.
3. Trace the virtual-media flow: mount ISO → set next-boot → power-cycle → eject.
4. Extensions: SoL capture; power-on→POST-watch; point at OpenBMC sushy-emulator.

Lab 02 (PDU/CDU sim + PCIe mapper):

1. python3 solution.py — read the PDU draw aggregation and the CDU telemetry with leak/ΔT alarms.
2. See the PCIe tree built from sample lspci output, with a degraded x16→x4 link flagged.
3. Run it on your own machine: pipe real lspci -mm -nn in and watch it build your topology.
4. Extensions: read /sys/bus/pci/devices link state and AER counts; feed PDU draw into the Phase 01 calculator as live data; emit Prometheus metrics.

Phase capstone question: "An accelerator node passes all health checks but trains 4× slow. Walk me from symptom to root cause using this phase's tools." (Answer: check PCIe LnkSta vs LnkCap for a degraded link / check NUMA adjacency of GPU+NIC / check CDU ΔT for thermal throttling / check PDU draw for power capping — a degraded x16→x4 link or thermal throttle is the usual culprit, and your topology mapper + telemetry find it in minutes.)

Success Criteria

You're done with this phase when — without notes:

• You can name the BMC's host interfaces (PECI/I²C/eSPI/NC-SI) and the sensor→Redfish chain (Ch. 1–2)
• You can describe OpenBMC's D-Bus/phosphor/bmcweb architecture at a high level (Ch. 2)
• You can explain the virtual-media remote-reinstall flow (Ch. 3)
• You can distinguish PDU types and per-outlet vs per-phase metering (Ch. 4)
• You can list CDU telemetry and the signature of each cooling failure mode, incl. leak handling (Ch. 5)
• You can explain PCIe lanes/gens/topology/BDF and why a degraded link is a silent killer (Ch. 6–7)
• You can read LnkCap vs LnkSta and sysfs link state to find a degraded link (Ch. 7)
• You can state the safety disciplines for physical actions (dry-run, interlocks, read-back) (Ch. 8)

Interview Q&A

Q1: What is a BMC physically, and how does it read a CPU temperature you then see in Redfish? A: The BMC is an independent ARM SoC (often an ASPeed AST2600) on the motherboard with its own RAM, flash, NIC, and standby power, running its own firmware (frequently OpenBMC). It reads the CPU temperature over PECI (the Intel telemetry interface) — or board sensors over I²C/PMBus — and exposes it through its stack: on OpenBMC, the kernel hwmon driver → phosphor-hwmon → a D-Bus sensor object → bmcweb renders it into the Redfish Thermal resource. So "GET the temperature over Redfish" is the end of a chain that starts at PECI. Knowing that chain is how I debug a wrong/missing sensor: is it the sensor, the I²C bus, the hwmon driver, the D-Bus object, or the Redfish mapping?

Q2: How do you reinstall an OS on a server you can't physically touch? A: Out-of-band via the BMC's virtual media. Over Redfish I mount the installer ISO from a network share as a virtual CD/USB on the BMC, set the next boot device to that virtual media (a one-time boot override), and power-cycle the host through the BMC. The host boots the installer; I watch progress over Serial-over-LAN, and when done I eject the media and let it boot from disk. This works even if the host has no OS or a broken one — that's the whole point of OOB. It's also the rescue path behind PXE provisioning in Phase 05.

Q3: A liquid-cooled rack's cooling is degrading. What telemetry warns you before an outage, and how should software respond to a leak? A: I watch the CDU: ΔT (return − supply) rising at constant IT load is the classic early warning of reduced flow or fouling — heat removed is ∝ flow × ΔT, so if load is flat and ΔT climbs, flow is dropping. I also watch flow (a failing pump), pressure (blockage/leak), and pump status (N+1 — losing the spare is critical). For a leak, software treats it as safety-critical, like a fire alarm: highest-priority alarm immediately, and per policy/ interlock trigger an emergency response — reduce heat load via power capping and/or a controlled shutdown or valve action — because a leak near live electronics is a hazard and can destroy hardware. The principle is that software which can act on the physical world must be more careful, with interlocks and confirmed effect, not less.

Q4: An accelerator is "healthy" but runs at a fraction of its speed. How could PCIe be the cause, and how do you find it? A: The PCIe link likely negotiated down — a degraded link, e.g., x16→x4 or Gen5→Gen1 — which collapses bandwidth while the device still enumerates and reports healthy, so it's silent. I find it by comparing LnkSta (negotiated width/speed) against LnkCap (what the link can do): in lspci -vvv or sysfs (current_link_width/current_link_speed vs max_link_width/max_link_speed). If they don't match, that's the smoking gun (often a reseat/cable/connector or a marginal lane). I'd also check NUMA adjacency — if the GPU and its NIC are on different sockets, cross-socket PCIe hops hurt — and PCIe AER correctable-error counts, since a rising rate precedes hard failures. My topology mapper automates exactly this check across all devices.

Q5: What's the difference between a metered and a switched PDU, and why do you care about per-outlet data? A: A metered PDU reports power draw (per-phase and/or per-outlet); a switched PDU can turn outlets on/off remotely; metered+switched does both, which is the AI-rack norm. I care about per-outlet metering because it's the ground truth for the power budget (Phase 01) — nameplate is a guess, the outlet is reality — and it lets me attribute power to specific devices/tenants and spot anomalies. Switching matters operationally: if a device and its BMC are both wedged, a switched outlet is my last-resort hard power-cycle. But outlet switching is physical and disruptive, so I guard it with dry-run, confirmation, blast-radius display, and read-back.

Q6 (Staff): You're integrating a new accelerator platform with an ODM. What hardware-interface issues do you anticipate and how do you de-risk them? A: I expect variance and gaps: incomplete/incorrect Redfish sensor mappings (the sensor→hwmon→D-Bus→bmcweb chain has bugs), PDU/CDU protocol differences (Modbus vs SNMP vs Redfish), PCIe topology surprises (lane allocation, degraded links under load, hot-plug quirks), and firmware-version-dependent behavior. To de-risk: agree a precise interface spec and software EBOM up front (Phase 12) — exact sensor list, Redfish schema coverage, OEM extensions, error semantics; build a conformance harness (Phase 11) that runs against their emulator/early hardware in CI so deviations are caught before deployment; instrument PCIe link state and AER from day one so degraded-link/thermal issues are visible during bring-up; and establish a tight defect-feedback loop with the ODM (Pegatron and peers), including upstreaming OpenBMC fixes where applicable. The abstraction layer (one model, per- class drivers) means their quirks stay contained while we drive them toward conformance.

References

• OpenBMC project & architecture (D-Bus, phosphor, bmcweb) — https://github.com/openbmc/openbmc , https://github.com/openbmc/docs
• ASPeed AST2600 BMC SoC overview — vendor datasheet / OpenBMC hardware docs
• DMTF Redfish PDU/Outlet & CoolingUnit/CoolingLoop schemas — https://redfish.dmtf.org/schemas/
• PCI-SIG — PCI Express Base Specification (lanes, links, AER); link speeds by generation
• Linux PCIe: lspci(8), sysfs PCI docs (/sys/bus/pci/devices), PCIe AER kernel docs — https://docs.kernel.org/PCI/
• Open Compute Project — Open Rack & power/busbar design — https://www.opencompute.org/
• Modbus (for CDUs) — https://modbus.org/specs.php
• Cross-track: Head of SWE — GPU, Phase 04 (Linux drivers/containers) — the driver side of these devices

🛸 Hitchhiker's Guide — Phase 04: Hardware Interfacing

Read this if: you can call Redfish/IPMI (Phase 03) but the devices behind them — BMC, PDU, CDU, PCIe switch — are still abstractions. This is the compressed device tour.

0. The 30-second mental model

Four device classes you manage: the BMC (control brain), the PDU (power truth), the CDU (cooling truth + a safety system), and the PCIe fabric (how everything connects and where silent perf bugs hide). Each is a managed device with telemetry and a control surface; each slots into the Phase 01 graph and the Phase 03 driver abstraction. One rule that earns respect: physical actions need guards (dry-run, interlocks, read-back).

1. BMC internals cheat sheet

Host CPU  --PECI-->   BMC (ARM SoC, standby power, OpenBMC)
DIMMs/VRM --I²C/PMBus--> BMC  --renders--> Redfish/IPMI
FRU EEPROM --I²C-->   BMC (inventory)
Host OS   --KCS/eSPI--> BMC (in-band IPMI)
Shared NIC --NC-SI-->  BMC (mgmt traffic)

OpenBMC chain for a sensor: hwmon → phosphor-hwmon → D-Bus → bmcweb → Redfish JSON. Debug a bad sensor by walking that chain.

2. BMC power state machine (don't confuse these)

On · ForceOff (yank) · GracefulShutdown (asks OS — fails silently if host hung) · ForceRestart · GracefulRestart · PowerCycle. Robust tooling: try graceful → wait → escalate to force → read back state to confirm.

3. The killer BMC feature: virtual media

Mount ISO over Redfish → set next-boot = virtual CD → power-cycle → host boots installer → watch over SoL → eject. This is how you reinstall a server you can't touch. Backbone of Phase 05 provisioning's rescue path.

4. PDU in one line

Metered = reads draw (per-outlet truth for the Phase 01 budget); Switched = toggles outlets (last-resort hard power-cycle); read per-phase too (balance). Switching = physical = guard it.

5. CDU: the telemetry and what it means

Treat leak like a fire alarm in code: top priority, fail-safe, interlock the power-on.

6. PCIe in five facts

• lane = 1 serial pair; link = x1/x4/x8/x16; Gen5 ≈ 4 GB/s/lane → Gen5 x16 ≈ 64 GB/s.
• tree: Root Complex → switches → endpoints (accelerators/NICs/NVMe). Switches enable P2P (GPUDirect).
• BDF = Bus:Device.Function (0000:3b:00.0) — the device coordinate.
• degraded link = LnkSta < LnkCap (x16→x4 or Gen5→Gen1): device "healthy", bandwidth collapsed → the silent perf killer. Check it always.
• AER: correctable (recovered, but rising rate = pre-failure) vs uncorrectable (fatal).

7. Reading PCIe on Linux (the commands)

lspci -tv                       # the tree
lspci -vvv | grep -A2 LnkSta    # negotiated vs capable width/speed
cat /sys/bus/pci/devices/<BDF>/current_link_width   # vs max_link_width
cat /sys/bus/pci/devices/<BDF>/numa_node            # GPU/NIC adjacency
cat /sys/bus/pci/devices/<BDF>/aer_dev_correctable  # error counts

8. Safety disciplines for physical actions

dry-run by default · --yes for destructive · interlocks (no power-on if cooling failed) · show blast radius before acting · read back to confirm effect · audit who/what/when.

9. The "done this before" tells

"Is that link at full width?" · "What's the ΔT trend?" · "Did you read back the power state?" · "Which NUMA node is the NIC on?" · "Is the leak interlock wired?" · "Per-outlet or per-phase?"

10. How this phase pays off later

PCIe AER + link state → Phase 08 RAS/diagnostics. PDU/CDU telemetry → Phase 07 dashboards and Phase 01 live budgets. Safety/guards → Phase 10 reliability. Device drivers → Phase 13 capstone.

Lab 01 — BMC Power Control, Sensors & Virtual Media

Phase: 04 — Hardware Interfacing | Difficulty: ⭐⭐⭐☆☆ | Time: 3–5 hours Language: Python 3 (stdlib only) | Hardware: none (BMC mock)

Concept primer: ../WARMUP.md Ch. 1–3, Ch. 8 (safety), ../HITCHHIKERS-GUIDE.md §1–§3, §8.

Run

python3 solution.py     # sensors, safe power-off (incl. hung-host escalation), vmedia reinstall

0. The mission

Build the toolkit behind "manage a server you can't physically touch": read sensors, power it safely, and reinstall its OS over the network with virtual media — with a dry-run/confirmation guard on every mutating operation. The BMC mock makes it runnable; the toolkit logic is the real pattern (swap in a Phase 03 Redfish/IPMI driver).

1. The three things to study

1. The safe power-off: GracefulShutdown → if the host is hung it silently no-ops → detect that (read back) → escalate to ForceOff → confirm. This graceful→escalate→verify pattern is the difference between a toy and a tool (WARMUP Ch. 3).
2. The virtual-media lifecycle: mount ISO → set next-boot to virtual CD → power-cycle → (installer runs, watched over SoL) → eject → restore boot-to-disk. This is remote provisioning's rescue path (Phase 05).
3. The guard: _guard() records intent (audit), honors dry_run (no mutation), and refuses destructive ops without confirm. Physical actions demand this (WARMUP Ch. 8).

2. Extensions

1. Point it at OpenBMC's sushy-emulator (real Redfish): implement set_power/virtual-media over Redfish actions; the toolkit stays the same.
2. Add SoL capture: stream the host serial console during the reinstall and assert you see the bootloader.
3. Add a power-on → POST-watch flow: power on, then poll for the host reaching a ready state (mirrors Phase 02 readiness).
4. Add an interlock: refuse power on if a (Phase 04 Lab 02) CDU leak/critical signal is active.

3. What this lab proves

You can implement OOB server management correctly and safely — the operations the JD's "remote diagnostics," "operational support of deployed systems," and "firmware updates" all rest on. "Reinstall a server you can't touch" becomes a precise answer: virtual media + boot override + power-cycle + SoL, guarded and verified.

Lab 02 — PDU/CDU Telemetry Simulator + PCIe Topology Mapper

Phase: 04 — Hardware Interfacing | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–6 hours Language: Python 3 (stdlib only) | Hardware: none (your own lspci optional)

Concept primer: ../WARMUP.md Ch. 4 (PDU), Ch. 5 (CDU), Ch. 6–7 (PCIe), ../HITCHHIKERS-GUIDE.md §4–§7.

Run

python3 solution.py                          # PDU + CDU telemetry + PCIe topology demo
lspci -mm -nn | python3 solution.py --lspci  # parse YOUR machine's PCIe devices (Linux)

0. The mission

Three tools that turn the power/cooling/fabric layer into software: a PDU meter (feeds the Phase 01 budget), a CDU monitor that alarms on the pre-failure signals (ΔT rising, low flow, lost pump) and on a safety-critical leak, and a PCIe topology mapper that finds the silent accelerator-perf killers — degraded links and AER errors.

1. The three lessons

• PDU: per-outlet draw → rack total → compare to the 80%-derated capacity. This is the measured truth nameplate can't give you (WARMUP Ch. 4; ties to Phase 01 Lab 02).
• CDU: ΔT = return − supply; ΔT rising at constant load → flow/fouling warning (the early signal), low flow → critical, leak → dominate-everything safety alarm (WARMUP Ch. 5). Read the three samples (healthy/degrading/leaking) and the alarms each raises.
• PCIe: a device is degraded when LnkSta < LnkCap (negotiated width/speed below capability). The demo fabric has Accelerator 5 at x4 instead of x16 — 16 GB/s where it should be 64 — and Accelerator 2 with high AER correctable errors. Both "look healthy" but aren't (WARMUP Ch. 6–7).

2. The output to internalize

The PCIe tree prints with a <-- DEGRADED flag showing the bandwidth you're actually getting vs capable, and a <-- AER flag for error-prone links. This is the tool that turns "the accelerator is mysteriously slow" into a 30-second diagnosis.

3. Extensions

1. Read your real machine's link state: parse /sys/bus/pci/devices/<BDF>/ current_link_width/max_link_width (and _speed) to detect degraded links on actual hardware, and aer_dev_correctable/aer_dev_fatal for errors.
2. NUMA adjacency: read numa_node and report GPU/NIC pairs that are cross-socket (a scheduling input for Phase 06).
3. Feed Phase 01: pipe PDU draw into the Phase 01 budget calculator as live "measured" data and turn it into a headroom monitor.
4. Emit Prometheus: render PDU/CDU/PCIe metrics in exposition format (cdu_delta_t_celsius, pcie_link_width{bdf=...}) — the Phase 07 bridge.
5. Build the full lspci -t parser: infer parent/child from bus numbers to construct the real tree from your machine.

4. Common pitfalls

1. Trusting "healthy" — a degraded x16→x4 link and a thermally-throttled accelerator both report healthy. Check link state and ΔT explicitly.
2. Treating leak like any alarm — it's safety-critical; it must dominate and (per policy) trigger emergency action.
3. Nameplate budgeting — use measured PDU draw, not nameplate (Phase 01).
4. Ignoring correctable AER — correctable errors are "recovered," but a rising rate predicts a hard failure (Phase 08 RAS).

5. What this lab proves

You can turn the physical power/cooling/fabric layer into actionable software signals and catch the failures that are invisible to naive health checks. "An accelerator passes health checks but is 4× slow — find it" becomes a confident walk: check link width/speed, ΔT, and power capping — the exact tools you just built. These feed Phase 07 dashboards, Phase 08 RAS, and the Phase 13 capstone.

Phase 05 — Provisioning & Configuration Management

Difficulty: ⭐⭐⭐⭐☆ | Estimated Time: 2 weeks Roles supported: Rack Management SWE (Senior/Staff), Infra/Platform SWE, SRE Hardware needed: none for the core labs (simulators + real config files); a VM/QEMU for the optional PXE extension

Why This Phase Exists

A rack arrives from the ODM as bare metal. Getting it from "powered on" to "running tenant workloads" is provisioning, and the JD asks for it directly: "Implement, test, and maintain software solutions for rack provisioning" and "Integrate rack management software with orchestration platforms (e.g., Kubernetes, Terraform, Foreman)" plus "Familiarity with Foreman, Ansible, Chef, and Puppet for provisioning and configuration."

This phase covers the two halves of turning hardware into a usable node:

1. Bare-metal provisioning — the PXE/iPXE boot chain that installs an OS onto a diskless machine over the network (DHCP → TFTP/HTTP → bootloader → kernel → automated installer), orchestrated by tools like Foreman.
2. Configuration management — bringing a freshly-installed node to a declared desired state (packages, drivers, network, services) idempotently, with Ansible/Chef/ Puppet, plus Terraform for infrastructure-as-code.

The unifying idea is declared vs discovered state and convergence (Phase 01 Ch. 10): you declare what a rack should be; the system makes reality match, repeatably. You'll build a provisioning state machine and a config-convergence engine, and write the real iPXE/ kickstart/Ansible/Terraform artifacts.

Concepts

Bare-metal provisioning

• The boot chain: PXE vs iPXE; DHCP options (next-server/66, filename/67, UEFI vs BIOS, arch detection); TFTP vs HTTP boot; the bootloader (pxelinux/iPXE/GRUB); kernel + initrd
• Automated install: kickstart (RHEL), preseed/autoinstall (Debian/Ubuntu), cloud-init (first-boot config); unattended OS install
• UEFI HTTP boot and the interaction with Secure Boot (Phase 09) — signed shim/bootloader
• The discovery/inventory step (a "discovery image" that reports hardware before install)
• Image-based (golden image / immutable) vs config-based provisioning; tradeoffs

Foreman (the provisioning lifecycle manager)

• Host groups, provisioning templates, the discovery plugin, smart proxies (DHCP/TFTP/DNS), Katello for content
• Where Foreman sits: it orchestrates DHCP/TFTP/installer and tracks host lifecycle

Configuration management

• Declarative desired-state & idempotency (the central concept — apply twice = same result)
• Ansible: agentless, push, SSH, YAML playbooks, modules, inventory, roles; idempotent modules
• Chef (agent, pull, Ruby DSL, resources/recipes/cookbooks) and Puppet (agent, pull, declarative manifests, catalog compilation) — the model and when each fits
• Convergence, drift detection, and remediation
• Secrets in CM (Ansible Vault, etc.); ordering vs declarative dependencies

Terraform / Infrastructure-as-Code

• Providers, resources, state, plan/apply, drift, modules; the declarative graph
• Bare-metal/Foreman/Redfish providers; what IaC means for physical racks
• Idempotency and the plan-before-apply discipline

The rack onboarding flow (the synthesis)

• discover → inventory/EBOM reconcile → firmware baseline (Phase 08) → OS install → configure → validate → join orchestrator (Phase 06) → ready
• Idempotent, resumable, observable; the state machine and its failure handling

Labs

Lab 01 — PXE/iPXE Bare-Metal Provisioning Flow

Lab 02 — Ansible + Terraform Rack Onboarding

Deliverables Checklist

• You can draw the PXE/iPXE boot chain end to end (DHCP → TFTP/HTTP → bootloader → kernel → installer)
• You can explain DHCP options 66/67 and BIOS vs UEFI PXE
• Provisioning state machine is resumable: a mid-flow failure resumes from the last good state
• You can explain idempotency and show a convergence that's a no-op on the second run
• Convergence engine detects and remediates drift
• You can compare Ansible vs Chef vs Puppet (push/pull, agent/agentless, language) and pick one
• You can explain Terraform state, plan/apply, and drift
• You can connect declared-vs-discovered here to the Phase 06 reconcile loop

Interview Relevance

• "Walk me through what happens when a diskless server PXE-boots."
• "How do you install an OS on 500 servers without touching them?"
• "What is idempotency and why does it matter in configuration management?"
• "Ansible vs Puppet vs Chef — when would you choose each?"
• "What does Terraform's state file do, and what is drift?"
• "Your provisioning fails halfway on node 237 of 500. What happens?" (resumable/idempotent)
• "How do you onboard a brand-new rack from bare metal to running workloads?"

Warmup Guide — Provisioning & Configuration Management

Zero-to-expert primer for Phase 05. You need no prior ops background: this guide starts at "how does a diskless machine get an OS" and ends with you designing a resumable rack onboarding pipeline using PXE/Foreman + Ansible/Terraform, grounded in one idea: declare the desired state, then converge reality to it.

Table of Contents

• Chapter 1: The Provisioning Problem
• Chapter 2: The PXE/iPXE Boot Chain
• Chapter 3: Automated OS Install
• Chapter 4: Foreman and Lifecycle Management
• Chapter 5: Configuration Management & Idempotency
• Chapter 6: Ansible vs Chef vs Puppet
• Chapter 7: Terraform and Infrastructure-as-Code
• Chapter 8: The Rack Onboarding Pipeline
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: The Provisioning Problem

A new rack's nodes have no operating system — bare metal with empty (or wiped) disks. You cannot SSH to a machine with no OS. You cannot insert a USB stick into 500 servers in a data center hall in Riyadh from your desk. So provisioning must be remote, networked, and automated, and it builds on the OOB tools you already have (Phase 04: virtual media is one path; network boot is the scalable one).

The job: take N bare-metal nodes and, with zero physical touches, end with each running a known OS, configured to spec, validated, and joined to the orchestrator — repeatably (you do this for every rack, forever) and resumably (failures are normal at scale; node 237 of 500 will fail mid-install, and the pipeline must recover, not restart everything).

Two broad strategies (know the tradeoff):

• Image-based / immutable: lay down a complete golden disk image, identical everywhere. Fast, deterministic, easy rollback (reimage); but images are large and any change means a rebuild. Favored for fleets at scale and immutable-infra shops.
• Config-based: install a base OS, then converge it to desired state with a CM tool. Flexible, smaller artifacts, drift-correctable; but more moving parts and slower per node.

Real fleets blend them: a thin golden image + cloud-init/CM for the last mile.

Chapter 2: The PXE/iPXE Boot Chain

PXE (Preboot eXecution Environment) is firmware in the NIC that can boot a machine over the network. The chain, step by step (memorize it — it's the #1 interview question here):

1. Power on → NIC PXE firmware runs (you select "network boot" in BIOS/UEFI, or it's the default boot order — and the BMC can force it once, Phase 04).
2. DHCP: the NIC broadcasts a DHCP request. The DHCP server replies with an IP and two special options: option 66 / next-server (the TFTP/boot server address) and option 67 / filename (the boot file to fetch). UEFI vs BIOS clients send a client- arch tag so the server hands the right bootloader (this arch-detection is a common gotcha).
3. TFTP (or HTTP): the client downloads the bootloader named in option 67 — classic pxelinux.0 (BIOS) or a UEFI binary, or (better) iPXE.
4. iPXE (the modern upgrade): a richer bootloader that can boot over HTTP/HTTPS (faster, reliable, firewall-friendly vs TFTP), run scripts, and make decisions (chainloading). You often PXE-boot a tiny loader that then chainloads iPXE, which fetches an iPXE script over HTTP telling it what kernel/installer to load.
5. Kernel + initrd: the bootloader downloads the installer kernel and initial ramdisk and boots them.
6. Installer runs with an automated answer file (Chapter 3).

TFTP vs HTTP: TFTP is simple but slow and UDP-fragile; iPXE-over-HTTP is the modern default for speed and reliability at scale. UEFI HTTP Boot removes PXE/TFTP entirely on newer firmware — DHCP hands a URL, the firmware fetches over HTTP directly.

Secure Boot interaction (Phase 09): with Secure Boot on, the firmware will only run a signed bootloader (the Microsoft-signed shim → signed GRUB → signed kernel chain). Your PXE/iPXE artifacts must be signed, or provisioning fails at boot — a real gotcha when enabling Secure Boot on a previously-working PXE setup.

Chapter 3: Automated OS Install

The installer must run unattended — no human clicking "Next." Each OS family has an answer-file format:

• Kickstart (RHEL/Rocky/Alma): a ks.cfg specifying disk partitioning, packages, users, network, and %post scripts. The kernel cmdline points at it (inst.ks=http://.../ks.cfg).
• preseed (Debian) / autoinstall (Ubuntu Subiquity, YAML): the same idea, different syntax.
• cloud-init: not an installer but a first-boot configurator — when a node boots an OS image, cloud-init applies user-data (users, SSH keys, packages, run-commands, write files). Ubiquitous for cloud and increasingly bare metal (with the right datasource).

The pattern: the boot chain delivers kernel+initrd+answer-file; the installer partitions, installs the base OS, drops an SSH key / joins CM, reboots into the installed system. From there, configuration management (Chapter 5) takes over.

Idempotency starts here: a good answer file + post-script should be safe to re-run, and the provisioning orchestrator must track which node reached which step so a failure resumes rather than reinstalls the whole rack (Lab 01).

Chapter 4: Foreman and Lifecycle Management

Doing the above by hand-editing DHCP/TFTP configs doesn't scale. Foreman is the open- source tool that orchestrates it and tracks host lifecycle (the JD names it). What it gives you:

• Smart proxies: Foreman drives DHCP, TFTP, DNS, and the installer via proxy services near the hardware (so the control plane stays central, the services stay local — a recurring rack-software pattern).
• Host groups & provisioning templates: declare "this class of node gets this OS, this partitioning, this kickstart" once; apply to many.
• The discovery plugin: unprovisioned nodes PXE-boot a discovery image that inventories the hardware and registers it with Foreman before you install — the software form of Phase 01 commissioning (discover → reconcile against EBOM → provision).
• Katello (add-on): content/repo and lifecycle-environment management (dev/test/prod content views) — relevant to controlled OS/package rollouts.

Foreman is one option; hyperscalers often build bespoke equivalents (MaaS, Tinkerbell, or in-house). The concepts — discovery, templated provisioning, smart proxies, lifecycle states — transfer regardless of tool, which is what an interviewer probes.

Chapter 5: Configuration Management & Idempotency

Once the OS is installed, the node still needs packages, drivers (accelerator/NIC firmware hooks, Phase 08), kernel params, network config, users, services, and security hardening. Configuration management (CM) brings it to a declared desired state.

Idempotency is the central idea: you declare the end state ("package X present, service Y running, file Z with these contents"), not the steps, and applying the configuration any number of times converges to that state and is a no-op once converged. This is what makes CM safe to run continuously and what lets it correct drift (someone changed a config by hand → the next run puts it back). Lab 02 proves it: apply → no-op on re-apply → drift → re-converge.

Contrast with imperative scripts: a shell script that runs useradd bob fails the second time (bob exists); the idempotent CM resource "user bob present" succeeds every time. That difference is why CM exists.

Convergence + drift: the system continuously (or on-demand) compares declared vs actual (declared-vs-discovered again — Phase 01 Ch. 10) and remediates the difference. This is exactly the reconcile loop a Kubernetes operator runs (Phase 06) — CM is the same control pattern applied to OS config.

Chapter 6: Ansible vs Chef vs Puppet

The JD lists all three; know the model and the tradeoffs (a guaranteed question).

How to choose (the senior answer): Ansible when you want agentless simplicity, ad-hoc runs, and orchestration of multi-step flows (often the rack-provisioning default); Puppet when you want a strong declarative model enforced continuously by agents at large scale; Chef when the team wants programmatic flexibility. Many shops standardize on Ansible for its agentless model and pair it with image-based provisioning. The key insight to convey: they're all idempotent declarative convergence engines — the architecture (push vs pull, agent vs agentless) is the real differentiator, not the capability.

Chapter 7: Terraform and Infrastructure-as-Code

Terraform declares infrastructure as code: you write the desired set of resources (in HCL), and Terraform makes reality match. Core concepts:

• Providers: plugins for a platform (AWS, a bare-metal/Foreman provider, a Redfish provider that configures BMCs, a Kubernetes provider). Providers are the bridge to the real world.
• State: Terraform records what it created in a state file — the map between your declared resources and real-world IDs. State is the source of truth for what Terraform manages; it must be stored safely (remote backend, locked) and never hand-edited.
• plan then apply: plan shows the diff between declared and actual (create/change/ destroy) before changing anything — the discipline that prevents surprise destruction. apply executes it.
• Drift: if reality changes outside Terraform, the next plan shows the drift; you re-apply to reconcile (declared-vs-discovered, again).

For physical racks, Terraform's role is configuring the infrastructure layer — BMC settings via the Redfish provider, network config, Foreman host definitions, then the Kubernetes/RKE2 cluster (Phase 06). It's the same declarative-convergence idea as CM, at the infra level.

CM vs IaC, briefly: IaC (Terraform) provisions/declares infrastructure (machines, networks, BMC config); CM (Ansible/Puppet/Chef) configures inside the OS. They compose: Terraform stands up the node and its infra; Ansible configures the OS; the orchestrator (Phase 06) schedules workloads.

Chapter 8: The Rack Onboarding Pipeline

Synthesize everything into the pipeline you build (Lab 01) — the software the JD's "rack provisioning" really means:

[discover] → [inventory/EBOM reconcile] → [firmware baseline] → [OS install] →
[configure] → [validate] → [join orchestrator] → [READY]
   P05/P04        P01/P12                  P08          P05         P05/P02     P06

Design properties that separate a real pipeline from a script:

• State machine, persisted: each node has a state; transitions are recorded so the pipeline is resumable after a crash or a node failure (node 237 resumes at its last good state, not from zero). Lab 01 implements this.
• Idempotent steps: re-running a step is safe; partial progress doesn't corrupt.
• Validation gates: a node doesn't advance to "ready" until it passes checks (firmware at baseline, sensors nominal, network reachable, accelerators enumerated at full PCIe width — Phase 04). A bad node is quarantined for RCA (Phase 10), not shipped to tenants.
• Observable: every transition emits structured logs/metrics (Phase 07) so you can watch 500 nodes onboard and spot the stuck ones.
• Reconciled against intent: the discovered hardware is checked against the EBOM (Phase 01/12); discrepancies are findings, not silent acceptance.

This pipeline is the seed of the Phase 13 capstone, and "design a bare-metal provisioning service" is a core system-design prompt (see system-design/05).

Lab Walkthrough Guidance

Order: Lab 01 (provisioning state machine) → Lab 02 (config convergence). Provisioning gets the OS on; CM brings it to spec — the natural sequence.

Lab 01 (PXE/iPXE flow):

1. python3 solution.py — watch a node move discover→…→ready, then a run where a step fails and the pipeline resumes from the last good state.
2. Read the real iPXE script, the dnsmasq/DHCP options, and the kickstart file — the artifacts the state machine would drive.
3. Extension: do a real PXE install in QEMU with dnsmasq + iPXE (instructions in the README).

Lab 02 (Ansible + Terraform):

1. python3 solution.py — converge a node; re-run (a no-op — idempotency); introduce drift; re-converge (remediation).
2. Read the real Ansible playbook and Terraform example.
3. Extension: run the playbook against a container/VM with real ansible.

Phase capstone question: "Design the software to onboard a 500-node fleet from bare metal, where ~3% of nodes will fail somewhere mid-flow. What are the must-have properties?" (Answer: a persisted per-node state machine that's resumable + idempotent steps + validation gates that quarantine bad nodes + EBOM reconciliation + full observability — not a big sequential script.)

Success Criteria

You're done with this phase when — without notes:

• You can recite the PXE/iPXE boot chain and explain DHCP options 66/67 and arch detection (Ch. 2)
• You can explain TFTP vs HTTP boot and Secure Boot's effect on PXE (Ch. 2)
• You can describe automated install (kickstart/cloud-init) and where CM takes over (Ch. 3)
• You can explain what Foreman/discovery/smart-proxies give you (Ch. 4)
• You can define idempotency and demonstrate a no-op re-apply and drift remediation (Ch. 5)
• You can compare Ansible/Puppet/Chef by architecture and choose one with reasons (Ch. 6)
• You can explain Terraform state, plan/apply, and drift (Ch. 7)
• You can design a resumable, validated, observable rack onboarding pipeline (Ch. 8)

Interview Q&A

Q1: Walk me through what happens when a diskless server PXE-boots. A: On power-on the NIC's PXE firmware runs and broadcasts a DHCP request. The DHCP server replies with an IP plus option 66 (next-server, the boot server) and option 67 (filename, the boot file), using the client's arch tag to pick BIOS vs UEFI. The client fetches that bootloader over TFTP (or HTTP) — typically iPXE — which then fetches an iPXE script over HTTP that tells it which installer kernel and initrd to load. It boots the kernel with a cmdline pointing at an automated answer file (kickstart/autoinstall), the installer partitions and lays down the base OS unattended, drops an SSH key or joins config management, and reboots into the installed system, where CM converges it to desired state. With Secure Boot on, the bootloader chain (shim→GRUB→kernel) must be signed or it won't boot.

Q2: How do you install an OS on 500 servers without touching them, and what happens when some fail? A: Network provisioning driven by a state machine. Each node PXE/iPXE-boots and runs an automated installer (kickstart/cloud-init) orchestrated by something like Foreman (DHCP/TFTP/ templates) or an in-house equivalent. The critical design point is failure: at 500 nodes, some will fail mid-flow, so I don't run a big sequential script — I run a persisted per-node state machine with idempotent steps, so a node that fails at "configure" resumes at "configure," not at "reinstall the rack." Bad nodes hit a validation gate (firmware baseline, sensors, PCIe link width, network) and get quarantined for RCA rather than shipped to tenants, while the rest proceed. Everything emits structured telemetry so I can watch the fleet onboard and triage the stuck ones.

Q3: What is idempotency, and why is it the core of configuration management? A: Idempotency means applying the same configuration any number of times converges to the same end state and is a no-op once you're there. CM is built on it because you declare the desired state ("package present, service running, file with these contents"), not imperative steps, so the tool can safely run continuously, correct drift (someone hand-edits a config → the next run restores it), and be re-run after a partial failure without corruption. The classic contrast: a script doing useradd bob fails the second time, but the idempotent resource "user bob present" succeeds every time. It's the same control pattern as a Kubernetes reconcile loop — declared vs discovered, continuously converged.

Q4: Ansible vs Puppet vs Chef — when would you choose each? A: They're all idempotent declarative convergence engines; the real difference is architecture. Ansible is agentless and push-based over SSH with YAML playbooks — fastest to adopt, great for orchestration and ad-hoc runs, often the rack-provisioning default. Puppet is agent-based and pull-based with a strong declarative DSL and a master compiling catalogs — excellent for continuously enforced state at large scale. Chef is agent/pull with a Ruby DSL — flexible and programmer-friendly. I'd pick Ansible when I want no agents and simple orchestration (and pair it with image-based provisioning), Puppet when I want agents continuously enforcing declared state across a big fleet, and Chef when the team values programmatic flexibility. The capability is similar; push-vs-pull and agent-vs-agentless drive the decision.

Q5: What does Terraform's state file do, and what is drift? A: The state file is Terraform's record of what it created — the mapping between your declared HCL resources and real-world resource IDs. It's how Terraform knows what it manages and computes the diff on the next run. Drift is when reality changes outside Terraform (someone edits a BMC setting by hand); the next plan detects the difference between declared and actual, and apply reconciles it. The disciplines around state are: store it in a locked remote backend (never local for teams), never hand-edit it, and always plan before apply so you see create/change/destroy before it happens. It's declared-vs-discovered convergence at the infrastructure layer.

Q6 (Staff): Design the provisioning system for continuously onboarding new racks at fleet scale. A: I'd build a control plane around a persisted state machine per node with the pipeline discover → EBOM reconcile → firmware baseline → OS install → configure → validate → join orchestrator → ready. Properties: every step idempotent and resumable (crash-safe); smart proxies near the hardware for DHCP/TFTP/HTTP/installer while the control logic stays central (so it scales across sites — relevant to a globally distributed team); a discovery image that inventories hardware and reconciles against the EBOM agreed with the ODM, raising discrepancies as findings; firmware brought to a known baseline (Phase 08) before install; image-based install (a thin golden image + cloud-init) for determinism and easy reimage-as- rollback; CM (Ansible) for the last-mile config and continuous drift correction; validation gates that quarantine nodes failing health/PCIe/firmware checks; and end-to-end observability (Phase 07) so I can watch hundreds of nodes onboard and SLO the time-to-ready. I'd expose it as a declarative API (desired rack state) so it composes with Terraform/IaC and the orchestrator. The whole thing is a convergence loop — declared rack vs discovered rack — which is the same pattern as the Phase 06 operator and is what makes it testable and maintainable.

References

• iPXE — open network bootloader, scripting, HTTP boot — https://ipxe.org/
• PXE specification & DHCP options (66/67, client-arch) — RFC 2131/4578; SYSLINUX/pxelinux docs
• Kickstart (RHEL), Ubuntu autoinstall (Subiquity), cloud-init — https://cloudinit.readthedocs.io/
• Foreman & the Discovery plugin; Katello — https://theforeman.org/ , https://theforeman.org/plugins/foreman_discovery/
• Ansible docs (modules, roles, idempotency, Vault) — https://docs.ansible.com/
• Puppet & Chef overviews (declarative resources, pull model) — https://www.puppet.com/docs , https://docs.chef.io/
• Terraform — providers, state, plan/apply; the Redfish provider — https://developer.hashicorp.com/terraform
• Tinkerbell / Canonical MAAS — alternative bare-metal provisioning stacks — https://tinkerbell.org/ , https://maas.io/
• Cross-track: Phase 06 (Kubernetes/RKE2) — where provisioning hands off

🛸 Hitchhiker's Guide — Phase 05: Provisioning & Config Management

Read this if: "PXE, Foreman, Ansible, Terraform" is a list of words. This is the compressed map — the boot chain, the one big idea, and the tool tradeoffs.

0. The 30-second mental model

Bare metal → running node has two halves: provisioning (network-boot an OS onto a diskless box: DHCP → bootloader → installer) and configuration (converge the OS to a declared state, idempotently). The one idea under all of it — PXE, Ansible, Terraform, and the Phase 06 operator alike — is declare desired state, converge reality to it. At fleet scale the must-have is a resumable per-node state machine, not a big script.

1. The PXE/iPXE boot chain (the #1 question)

power on → NIC PXE firmware
   → DHCP (gives IP + option 66 next-server + option 67 filename; arch tag = BIOS/UEFI)
   → TFTP/HTTP fetch bootloader (pxelinux / iPXE / UEFI binary)
   → iPXE script over HTTP → fetch installer kernel + initrd
   → installer runs UNATTENDED (kickstart / autoinstall / cloud-init)
   → reboot into installed OS → config management takes over

Gotchas: TFTP is slow/UDP-fragile (prefer iPXE-over-HTTP); UEFI vs BIOS needs the right bootloader (arch detection); Secure Boot requires signed shim→GRUB→kernel or it won't boot.

2. Automated install answer files

3. Foreman in one line

Orchestrates DHCP/TFTP/DNS/installer via smart proxies (local services, central control), with host groups + templates and a discovery image that inventories unprovisioned nodes — the software form of Phase 01 commissioning.

4. Idempotency (the core idea)

Declare end state, not steps → apply N times = same result, no-op once converged, corrects drift. useradd bob (breaks on re-run) vs resource "user bob present" (always fine). This is the same loop as a k8s operator and Terraform.

5. Ansible vs Puppet vs Chef

They're all idempotent convergence engines — architecture is the differentiator.

6. Terraform in four words

Providers · State (the map; lock it, never hand-edit) · plan (see the diff, incl. destroy) · apply (reconcile). Drift = reality diverged → next plan shows it. IaC declares infrastructure; CM configures inside the OS; they compose.

7. The onboarding pipeline (what to build)

discover → EBOM reconcile → firmware baseline → OS install → configure → validate → join orch → READY

Must-haves: persisted resumable state machine · idempotent steps · validation gates (quarantine bad nodes) · EBOM reconciliation · observability. Not a sequential script.

8. The "done this before" tells

"Is the step idempotent?" · "Does it resume from the last state?" · "BIOS or UEFI PXE?" · "Is the boot chain signed for Secure Boot?" · "What does the validation gate check?" · "Where's the Terraform state stored?" · "Did you plan before apply?"

9. How this phase pays off later

The state-machine + declared-vs-discovered pattern is the Phase 06 operator and the Phase 13 capstone. Validation gates use Phase 04 (PCIe/sensors) and Phase 08 (firmware). Observability is Phase 07. "Design a provisioning service" is system-design/05.

Lab 01 — PXE/iPXE Bare-Metal Provisioning Flow

Phase: 05 — Provisioning & Config Mgmt | Difficulty: ⭐⭐⭐⭐☆ | Time: 5–7 hours Language: Python 3 (state machine) + real iPXE/dnsmasq/kickstart | Hardware: none (QEMU for the optional real install)

Concept primer: ../WARMUP.md Ch. 2–3, Ch. 8, ../HITCHHIKERS-GUIDE.md §1–§3, §7.

Run

python3 solution.py        # the provisioning state machine: clean run, resume, quarantine

Files

0. The mission

Build the software the JD's "rack provisioning" means — not a script, a resumable state machine that takes bare metal → ready, with a validation gate that quarantines bad nodes — and pair it with the real boot artifacts (boot.ipxe, dnsmasq.conf, ks.cfg) it drives. The boot chain and the orchestrator are the two halves of provisioning.

1. What the state machine demonstrates

• Resumability: Node B hits an injected failure at OS_INSTALLED and resumes from FIRMWARE_BASELINED — it never re-runs the earlier steps (the self-test asserts INVENTORIED happened exactly once). At 500 nodes, this is the difference between a 3% failure rate being a non-event and being a re-provision-the-whole-rack disaster.
• Idempotency: re-provisioning an already-READY node is a no-op.
• Validation gate: Node C's discovered inventory (3 accelerators) mismatches the EBOM (4) → it's QUARANTINED with a reason, not shipped to tenants. This is Phase 01 commissioning enforced in code.

2. The boot artifacts (read them)

dnsmasq.conf shows the two-stage PXE trick (PXE → iPXE binary → HTTP iPXE script) and arch detection; boot.ipxe shows HTTP boot with a serial console so the BMC's SoL can watch (Phase 04); ks.cfg shows unattended install ending with an SSH-key drop and a callback to the orchestrator ("os-installed") — the seam between the boot chain and the state machine.

3. Extensions (do them)

1. A real PXE install in QEMU. Install dnsmasq, grab ipxe.efi/undionly.kpxe, serve the artifacts over a tiny HTTP server, and boot a diskless QEMU VM (-boot n, -netdev). Watch it DHCP → chainload iPXE → install Ubuntu unattended.
2. UEFI HTTP Boot: skip TFTP entirely — DHCP hands a URL, the firmware fetches over HTTP.
3. EBOM reconciliation: wire the INVENTORIED step to the Phase 01 inventory model and produce a real declared-vs-discovered diff.
4. Persist state to disk/DB: make the state machine truly crash-safe (process dies → restart → resumes), the way a production provisioner stores per-node state.
5. Secure Boot: make the boot chain use signed shim→GRUB→kernel and discuss what breaks.

4. What this lab proves

You can design the provisioning control plane, not just paste a kickstart. "Onboard 500 nodes where 3% fail mid-flow" becomes a confident answer — resumable state machine, idempotent steps, validation gates, EBOM reconciliation — and you can show the real iPXE/DHCP/kickstart that the control plane drives. This is the seed of system-design/05 and the Phase 13 capstone.

Lab 02 — Ansible + Terraform Rack Onboarding

Phase: 05 — Provisioning & Config Mgmt | Difficulty: ⭐⭐⭐☆☆ | Time: 4–6 hours Language: Python 3 (engine) + real Ansible/Terraform | Hardware: none (a VM/container for real Ansible)

Concept primer: ../WARMUP.md Ch. 5–7, ../HITCHHIKERS-GUIDE.md §4–§6.

Run

python3 solution.py     # converge -> no-op -> drift -> remediate (idempotency proven)

Files

0. The mission

Internalize the one idea under every CM/IaC tool — declare desired state, converge, be a no-op once converged, correct drift — by building the engine and reading the real artifacts (playbook.yml, main.tf) that do it against real systems.

1. What the engine proves

A "resource" is (check, apply): check if already in the desired state; apply only if not. The run shows the three behaviors that define configuration management:

1. First converge changes everything (fresh node).
2. Second converge is a no-op — idempotency.
3. Drift (sshd stopped, config hand-edited) is detected and remediated back to desired.

This is literally the same control loop as Terraform (plan/apply/drift) and a Kubernetes operator (Phase 06) — declared vs discovered, converged.

2. The real artifacts

• playbook.yml: every Ansible task uses a declarative state: so re-running reports "ok" (no change). Note --check for a dry-run (plan), handlers for restart-on-change, and that secrets never appear in the file.
• main.tf: Terraform manages the infrastructure layer (BMC boot order via Redfish) while Ansible configures inside the OS — they compose. Note plan before apply, sensitive variables, ssl_insecure = false (validate the cert — Phase 03 Ch. 7), and that state is the locked source of truth.

3. Extensions (do them)

1. Run the playbook for real against a container/VM: ansible-playbook -i inventory.ini playbook.yml, then run it again and confirm the second run reports zero changes (idempotency on real hardware). Hand-edit /etc/rack/node.conf and re-run to watch drift correction.
2. Ansible Vault: encrypt a secret and reference it, so credentials never sit in plaintext.
3. A Terraform plan discipline: change a value, run terraform plan, and read the create/change/destroy diff before applying.
4. Scheduled drift detection: run the engine on a timer and alert (Phase 07) when drift is found — the basis of continuous compliance.
5. Compare pull-based: sketch how Puppet/Chef agents would do the same convergence on a pull schedule, and the tradeoff vs Ansible's push.

4. What this lab proves

You understand configuration management at the level that lets you reason about any tool, not just recite one. "What is idempotency / Ansible vs Puppet / Terraform state and drift" become precise answers backed by an engine you built — and you can connect it to the Phase 06 operator and the Phase 13 capstone as the same convergence pattern.

Phase 06 — Orchestration: Kubernetes & RKE2

Difficulty: ⭐⭐⭐⭐☆ | Estimated Time: 2 weeks Roles supported: Rack Management SWE (Senior/Staff), Platform/Infra SWE, SRE Hardware needed: none for the core labs (simulators + real manifests); optional kind/k3d for the extensions

Why This Phase Exists

The JD: "Integrate rack management software with orchestration platforms (e.g., Kubernetes, Terraform, Foreman)" · "Implement... software solutions for... workload deployment" · "Knowledge of AI workload orchestration tools (e.g., K8s, RKE2)." Once a rack's nodes are provisioned (Phase 05), the orchestrator is what schedules tenant AI workloads onto them and manages their lifecycle. Rack-management software both runs on and integrates with Kubernetes — and the operator pattern (a controller running a reconcile loop) is the modern way to encode rack-management logic into the cluster itself.

The deep idea you take from this phase is the reconcile loop: a controller continuously drives actual state toward declared (CRD) state — the same declared-vs-discovered convergence you met in inventory (Phase 01), provisioning (Phase 05), and config management (Phase 05). You'll build a RackNode operator that reconciles, and walk the device-plugin + scheduling path that gets accelerators to pods.

Concepts

Kubernetes core (enough to be dangerous, then deep where it matters)

• The control plane: API server, etcd, scheduler, controller-manager; the declarative API
• Nodes, kubelet, the container runtime (containerd/CRI), pods, the pod lifecycle
• Workload objects: Deployment/ReplicaSet, DaemonSet (the rack-agent pattern), StatefulSet, Job
• Scheduling: requests/limits, node selectors, taints/tolerations, affinity/anti-affinity, topology spread
• Services, the networking model (CNI), namespaces, RBAC, resource quotas (multi-tenancy)

The operator / controller pattern (the heart of the phase)

• CRDs (CustomResourceDefinitions) — extend the API with your own objects (a RackNode)
• The reconcile loop: observe → diff (desired vs actual) → act → requeue; level-triggered, idempotent
• Controllers, informers/watches, the work queue, status vs spec, finalizers, owner references
• Why operators encode operational knowledge as software (the "automated SRE" idea)

Accelerators in Kubernetes

• The device plugin API — how a node advertises accelerators as schedulable resources
• Node Feature Discovery; extended resources; the GPU/accelerator operator pattern
• Topology-aware scheduling (NUMA/PCIe adjacency — Phase 04); MIG-style partitioning concepts

RKE2 and production distros

• What RKE2 is (Rancher's secure, hardened, CIS-compliant k8s distro) and why hardened distros exist
• RKE2 vs upstream/k3s/OpenShift; air-gapped install; embedded etcd; the security posture
• Cluster lifecycle: node join/cordon/drain/upgrade; the relationship to provisioning (Phase 05)

Multi-tenancy & integration

• Namespaces + RBAC + quotas + network policy as tenant boundaries (pairs with Phase 09)
• How rack-management software integrates: as a DaemonSet agent, an operator, and an API consumer

Labs

Lab 01 — A RackNode Kubernetes Operator (reconcile loop)

Lab 02 — Device Plugin & Accelerator Scheduling

Deliverables Checklist

• You can describe the k8s control plane (API server, etcd, scheduler, controllers) and the declarative model
• Operator reconcile loop converges desired→actual, is idempotent, and requeues on failure
• You can explain level- vs edge-triggered and why operators are level-triggered
• You can explain spec vs status and what a finalizer does
• Scheduler sim enforces capacity, taints/tolerations, and topology preference
• You can explain the device-plugin path from hardware to a schedulable resource
• You can say what RKE2 is and why a hardened distro exists; node drain/upgrade lifecycle
• You can connect the reconcile loop to Phase 05 CM and Phase 01 declared-vs-discovered

Interview Relevance

• "What happens when you kubectl apply a Deployment?" (control plane → scheduler → kubelet)
• "Explain the operator pattern and the reconcile loop. Why level-triggered?"
• "What's the difference between spec and status? What is a finalizer for?"
• "How does a GPU/accelerator get scheduled to a pod?" (device plugin → extended resource)
• "Why might a pod be stuck Pending?" (capacity, taints, affinity, no matching node)
• "What is RKE2 and why use it over upstream k8s?" (hardened, CIS, air-gap)
• "How would you safely upgrade k8s on a 100-node rack?" (cordon/drain/upgrade, canary)

Warmup Guide — Orchestration: Kubernetes & RKE2

Zero-to-expert primer for Phase 06. You need no Kubernetes background: this guide starts at "what does an orchestrator do" and ends with you building an operator reconcile loop and reasoning about accelerator scheduling. The through-line is the reconcile loop — declared-vs-discovered convergence, the same idea from Phases 01 and 05.

Table of Contents

• Chapter 1: Why Orchestration, and Where Rack Software Fits
• Chapter 2: The Kubernetes Control Plane
• Chapter 3: Workloads, Nodes, and the DaemonSet Agent
• Chapter 4: The Operator Pattern and the Reconcile Loop
• Chapter 5: Scheduling — How a Pod Lands on a Node
• Chapter 6: Accelerators — Device Plugins & Topology
• Chapter 7: RKE2 and Production Distros
• Chapter 8: Multi-Tenancy & Cluster Lifecycle
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Why Orchestration, and Where Rack Software Fits

Once nodes are provisioned (Phase 05), something must decide what runs where, restart failed work, scale it, and manage its lifecycle across the fleet. That's orchestration, and Kubernetes (k8s) is the de-facto standard, including for AI workloads (often via hardened distros like RKE2 and add-ons for accelerators).

Rack-management software relates to k8s in three ways — keep them distinct:

1. It runs on k8s: your agents/exporters deploy as pods/DaemonSets (one per node).
2. It integrates with k8s: it consumes the API to cordon/drain a node for maintenance, label nodes with hardware facts, or react to events.
3. It extends k8s: you write operators (controllers + CRDs) that teach the cluster rack-specific operations — "a RackNode should be at firmware baseline X and healthy" — so rack lifecycle becomes declarative, self-healing software.

The mental anchor: Kubernetes is a declarative, level-triggered control system. You declare desired state (objects); controllers continuously reconcile reality to it. If that sounds like config management (Phase 05) and inventory reconciliation (Phase 01) — it is the same idea, which is why this curriculum keeps returning to it.

Chapter 2: The Kubernetes Control Plane

The control plane is a set of components around a database:

• etcd: the consistent key-value store holding all cluster state (the source of truth).
• API server (kube-apiserver): the only thing that talks to etcd; a REST API where everything is an object (validated, versioned, RBAC-controlled). All reads/writes go through it. kubectl apply is an API write.
• Scheduler (kube-scheduler): watches for unscheduled pods and assigns each to a node (Chapter 5). It only decides; it doesn't run anything.
• Controller manager: runs the built-in controllers (Deployment, ReplicaSet, Node, …), each a reconcile loop driving actual toward desired.
• On each node: kubelet (runs/monitors the node's pods via the container runtime over CRI) and kube-proxy (service networking).

The flow for kubectl apply -f deploy.yaml: API server validates and writes the Deployment to etcd → the Deployment controller creates a ReplicaSet → the ReplicaSet controller creates Pods (desired count) → the scheduler binds each Pod to a node → that node's kubelet pulls the image and starts the container → controllers keep watching and correcting. No component gives orders imperatively; each watches state and reconciles. That's the whole design.

Chapter 3: Workloads, Nodes, and the DaemonSet Agent

The objects you'll use:

• Pod: the smallest deployable unit (one or more co-located containers). Usually you don't create pods directly; a controller does.
• Deployment → ReplicaSet → Pods: stateless apps; declare replicas: N, get N pods, self-healed and rolling-updatable.
• DaemonSet: one pod per node (optionally a node subset). This is the rack-agent pattern — your node-level rack-management agent, the metrics exporter (Phase 07), the device plugin, and the firmware agent all deploy as DaemonSets, so every node runs exactly one copy automatically as nodes join/leave.
• StatefulSet: stable identity/storage for stateful workloads (databases).
• Job/CronJob: run-to-completion / scheduled tasks (a one-shot firmware update job).

Each pod declares requests (guaranteed resources, used for scheduling) and limits (caps). Getting requests right is how the scheduler bin-packs without overcommitting — and it's where accelerators plug in (Chapter 6).

Chapter 4: The Operator Pattern and the Reconcile Loop

This is the chapter that matters most for this role, because it's how you encode rack operations as cluster-native software.

CRD (CustomResourceDefinition): extends the API with your own object kind. Define a RackNode CRD and now kubectl get racknodes works and the API server stores/validates them like built-ins. The CRD has a spec (desired state, written by users/automation) and a status (observed state, written by your controller).

The reconcile loop (the operator's core):

loop forever:
  obj = observe(current state of the RackNode + the real world)
  diff = desired(spec) - actual(status/world)
  if diff: act to reduce it (idempotently)
  update status
  requeue (after a delay, or on the next event)

Properties that make it robust (and are interview gold):

• Level-triggered, not edge-triggered: it reacts to current state, not to the event that changed it. So a missed event isn't fatal — the next reconcile sees reality and fixes it. Edge-triggered (act on the change notification) breaks if you miss an event; level- triggered is self-healing. This is why Kubernetes is reliable.
• Idempotent: reconcile may run many times; each run converges and is a no-op once matched (Phase 05's idea, exactly).
• Requeue + backoff: a failed action requeues for retry rather than crashing.
• Finalizers: a marker on an object that blocks deletion until your controller does cleanup (e.g., decommission the physical node, remove it from inventory) — then removes the finalizer to allow deletion. Without finalizers, deleting the object would orphan real-world resources.
• Owner references: child objects owned by a parent are garbage-collected with it.

A RackNode operator might reconcile: "spec says firmware baseline 1.2 and role=compute; actual says firmware 1.1 → trigger a firmware job (Phase 08), update status, requeue until matched; on delete, run the decommission finalizer." That's rack lifecycle as self-healing software — the "automated SRE" promise of operators. Lab 01 builds this loop.

Chapter 5: Scheduling — How a Pod Lands on a Node

The scheduler runs two phases per unscheduled pod:

1. Filter (predicates): eliminate nodes that can't run it — insufficient requests-vs-allocatable, missing required labels (nodeSelector/affinity), or taints the pod doesn't tolerate.
2. Score (priorities): rank the survivors — spread vs bin-pack, affinity preferences, topology spread, image locality — and pick the best.

Key levers a rack operator uses:

• requests/limits: requests drive filtering/bin-packing; a pod requesting more than any node's allocatable stays Pending forever (a classic "why is my pod Pending" answer).
• Taints & tolerations: taint a node (maintenance:NoSchedule, or dedicated=tenantA) and only pods tolerating it land there. This is how you drain a node for maintenance (cordon = unschedulable; taint+drain evicts), and how you dedicate accelerator nodes.
• Affinity/anti-affinity & topology spread: co-locate or spread pods (e.g., spread replicas across failure domains — Phase 01 — so one PDU/ToR loss doesn't take all replicas).
• Topology-aware placement: for accelerators, place a pod where its GPU and NIC share a NUMA node / PCIe root (Phase 04) for performance.

"Why is this pod Pending?" → walk the filter phase: capacity, taints, affinity, no matching node. Lab 02 makes this concrete.

Chapter 6: Accelerators — Device Plugins & Topology

Kubernetes doesn't natively know about GPUs/AI accelerators; the device plugin API teaches it:

• A device-plugin DaemonSet runs on each node, discovers the accelerators, and advertises them to the kubelet as an extended resource (e.g., qualcomm.com/ai-accelerator: 4).
• Pods request them like any resource: resources.limits["qualcomm.com/ai-accelerator"]: 1.
• The scheduler now treats accelerators as schedulable capacity (filter/score), and the kubelet, via the plugin, allocates specific devices to the container (and sets up cgroups/device access).

Beyond basic counting:

• Node Feature Discovery (NFD) labels nodes with hardware facts (PCIe IDs, capabilities) so scheduling can target them.
• Topology-aware allocation places a pod's accelerators and NIC on the same NUMA node/PCIe root (Phase 04) — a real performance lever the Topology Manager handles.
• Partitioning (MIG-style fractional accelerators) is exposed as multiple smaller resources — the multi-tenancy/utilization tradeoff.

The "accelerator operator" pattern bundles the device plugin, NFD, driver management, and monitoring into one operator — the kind of thing this role builds or integrates.

Chapter 7: RKE2 and Production Distros

Upstream Kubernetes is a kit; production clusters run a distribution. RKE2 (Rancher Kubernetes Engine 2, aka RKE Government) is a hardened, secure-by-default distro — relevant because the JD names it and because secure/sovereign deployments (think a government data center in the Kingdom) demand exactly its properties:

• CIS-hardened out of the box; FIPS-capable; designed to pass security benchmarks.
• Simple, self-contained install (a single binary, embedded etcd) — good for air-gapped deployments (no internet during install), a recurring requirement in sovereign/regulated environments.
• containerd runtime, bundled CNI, no Docker dependency.

Context for the family: k3s (lightweight, edge), RKE2 (hardened, datacenter/gov), OpenShift (enterprise, opinionated), upstream/kubeadm (DIY). They're all Kubernetes; the differences are security posture, install model, and bundled components. Knowing why RKE2 (hardening + air-gap + CIS) signals you understand the deployment context, not just the API.

Chapter 8: Multi-Tenancy & Cluster Lifecycle

Multi-tenancy in k8s (pairs with Phase 09's network/VLAN isolation):

• Namespaces partition objects; RBAC controls who can do what; ResourceQuotas cap a tenant's consumption; NetworkPolicies restrict pod-to-pod traffic; PodSecurity standards constrain what pods can do.
• Soft multi-tenancy (namespaces + quotas + RBAC) suits trusted tenants; hard isolation (separate clusters/nodes, or strong sandboxing) suits untrusted ones. For accelerators, not co-placing competing tenants on one physical device (Phase 04/09) is part of this.

Cluster lifecycle (where rack-management software acts on k8s):

• Node join: a freshly provisioned node (Phase 05) joins the cluster.
• cordon: mark a node unschedulable (no new pods) without evicting.
• drain: evict pods (respecting PodDisruptionBudgets) to free a node for maintenance — what you do before a firmware update (Phase 08) or a hardware swap.
• upgrade: roll the k8s/OS/driver version node-by-node — cordon → drain → upgrade → uncordon, canaried (Phase 11), image-pinned for rollback.

These verbs are how your rack operator safely takes a node out of service and back. The maintenance flow (cordon → drain → act → validate → uncordon) is a core operational primitive and a frequent system-design detail.

Lab Walkthrough Guidance

Order: Lab 01 (operator) → Lab 02 (scheduling). The reconcile loop is the conceptual core; scheduling is how workloads (and your agents) land on the hardware.

Lab 01 (RackNode operator):

1. python3 solution.py — a controller reconciles RackNodes: provisions to desired, corrects drift, and deletes with a finalizer-driven cleanup.
2. Read the reconcile function; note it's level-triggered (acts on current state) and idempotent (re-reconcile is a no-op).
3. Inject drift and a transient action failure; watch it requeue and converge.
4. Extensions: reimplement with kopf against a real kind cluster + a CRD YAML.

Lab 02 (device plugin + scheduling):

1. python3 solution.py — a mini scheduler places accelerator-requesting pods across nodes honoring capacity, taints, and topology.
2. Over-subscribe accelerators and watch pods go Pending; add a maintenance taint and watch placement avoid that node.
3. Read the real device-plugin DaemonSet + pod manifests.
4. Extensions: run NVIDIA's device plugin on a real GPU; add MIG-style fractional resources; add gang scheduling.

Phase capstone question: "Design how rack-management runs and integrates with Kubernetes for an accelerator fleet." (Answer: node agent + exporter as DaemonSets; a RackNode operator encoding lifecycle/firmware/health as a reconcile loop; device plugin + NFD + topology-aware scheduling for accelerators; cordon/drain/upgrade for maintenance; namespaces+RBAC+quota+ NetworkPolicy for tenancy; RKE2 for a hardened, air-gappable base.)

Success Criteria

You're done with this phase when — without notes:

• You can trace kubectl apply through the control plane to a running container (Ch. 2)
• You can explain the DaemonSet rack-agent pattern (Ch. 3)
• You can write/explain a reconcile loop and why it's level-triggered and idempotent (Ch. 4)
• You can explain spec vs status, finalizers, and owner references (Ch. 4)
• You can diagnose a Pending pod via the filter phase (capacity/taint/affinity) (Ch. 5)
• You can explain the device-plugin path from hardware to schedulable resource + topology (Ch. 6)
• You can say what RKE2 is and why hardened/air-gapped distros exist (Ch. 7)
• You can describe the cordon→drain→upgrade maintenance flow (Ch. 8)

Interview Q&A

Q1: What happens when you kubectl apply a Deployment? A: The request hits the API server, which authenticates, validates, and writes the Deployment object to etcd. The Deployment controller (a reconcile loop) sees it and creates a ReplicaSet; the ReplicaSet controller creates the desired number of Pods. The scheduler watches for unscheduled Pods and, for each, filters nodes (capacity, taints, affinity) and scores the survivors, then binds the Pod to a node. That node's kubelet sees the binding, pulls the image via the container runtime (CRI/containerd), and starts the container, reporting status back. From then on the controllers keep watching and reconciling — if a pod dies, the ReplicaSet controller makes a new one. Nothing is imperative; every component watches state and converges it, which is the whole Kubernetes design.

Q2: Explain the operator pattern and the reconcile loop, and why it's level-triggered. A: An operator extends Kubernetes with a CRD (a custom object like RackNode) plus a controller that runs a reconcile loop: observe current state, diff desired (spec) against actual (status/world), act idempotently to close the gap, update status, and requeue. It's level-triggered — it reacts to current state, not to the event that changed it — so if a notification is missed or duplicated, the next reconcile still sees reality and corrects it. Edge-triggered logic (act on the change event) breaks when you miss an event; level-triggered is self-healing, which is why it's reliable. The loop must be idempotent (many runs converge, no-op once matched), requeue with backoff on failure, and use finalizers to do cleanup before an object is deleted. It's the same declared-vs-discovered convergence as configuration management — operators just apply it to cluster-native objects.

Q3: What's the difference between spec and status, and what is a finalizer for? A: Spec is the desired state, written by users or automation; status is the observed state, written by the controller. The controller's job is to make status (and the real world) match spec. A finalizer is a key in an object's metadata that blocks its deletion: when you delete the object, Kubernetes sets a deletion timestamp but keeps the object until all finalizers are removed. The controller sees the deletion timestamp, performs cleanup (e.g., decommission the physical node, remove it from inventory, release licenses), then removes its finalizer, allowing the object to be deleted. Without finalizers, deleting a RackNode object would orphan the real-world resources it represented.

Q4: How does an accelerator get scheduled to a pod, and why might a pod be Pending? A: A device-plugin DaemonSet on each node discovers the accelerators and advertises them to the kubelet as an extended resource (e.g., qualcomm.com/ai-accelerator: 4). A pod requests some via resources.limits, the scheduler treats them as schedulable capacity (filter + score), and the kubelet (via the plugin) allocates specific devices and wires up access. A pod goes Pending when the scheduler can't find a node that passes the filter phase: not enough free accelerators/CPU/memory anywhere, a required nodeSelector/affinity matches no node, or every candidate node has a taint the pod doesn't tolerate (e.g., maintenance, or dedicated to another tenant). For accelerators specifically, requesting more than any single node has, or topology constraints that can't be satisfied, leaves it Pending. The fix is reading the scheduler events and the node's allocatable vs requested.

Q5: What is RKE2 and why choose it over upstream Kubernetes? A: RKE2 is Rancher's hardened Kubernetes distribution — CIS-benchmark-hardened and FIPS-capable by default, with a simple single-binary, embedded-etcd install and bundled containerd/CNI. You choose it when security posture and deployment constraints matter: it passes security benchmarks out of the box and installs cleanly air-gapped (no internet), which is exactly what sovereign, regulated, or government data centers require — relevant for a data center in the Kingdom. Upstream Kubernetes is a kit you'd have to harden and assemble yourself; RKE2 gives you a secure, supportable base. It's still Kubernetes — same API and operators — so my rack operators and DaemonSets run on it unchanged.

Q6 (Staff): How do you safely upgrade Kubernetes (or a driver/firmware) across a 100-node accelerator rack with running tenant workloads? A: Treat it as a controlled, canaried, reversible rollout (Phase 11 thinking). First, ensure PodDisruptionBudgets so draining respects availability. Then per node, in waves: cordon (stop new scheduling) → drain (evict pods, honoring PDBs, so tenant work reschedules elsewhere) → perform the upgrade (k8s/driver/firmware — Phase 08) → validate (node healthy, accelerators enumerated at full PCIe width per Phase 04, smoke test) → uncordon. Start with a 1-node canary, bake, then widen the wave size, with automatic halt-and-rollback if health/ SLOs regress (image-pinned so rollback is fast). Respect failure domains (Phase 01) — don't drain a whole PDU/ToR's worth at once. Drive the whole thing from an operator/job so it's declarative, observable (Phase 07), and resumable. The Staff point is that the process — gated, canaried, domain-aware, reversible — matters more than the mechanics, because the cost of a bad fleet-wide upgrade is enormous.

References

• Kubernetes docs — Concepts (control plane, controllers, scheduling) — https://kubernetes.io/docs/concepts/
• Kubernetes — Operator pattern & the controller/reconcile model — https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
• Kubebuilder Book (CRDs, controllers, finalizers, owner refs) — https://book.kubebuilder.io/
• kopf (Kubernetes operators in Python) — https://kopf.readthedocs.io/
• Kubernetes Device Plugins & Topology Manager — https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/
• NVIDIA GPU Operator / device plugin (the accelerator-operator pattern) — https://github.com/NVIDIA/k8s-device-plugin
• RKE2 documentation (hardening, air-gap, CIS) — https://docs.rke2.io/
• Cross-track: Head of SWE — GPU, Phase 06 (scheduling/virtualization)

🛸 Hitchhiker's Guide — Phase 06: Orchestration

Read this if: you can kubectl get pods but the control plane and operators are fuzzy. Compressed map of the parts, the reconcile loop, and how accelerators get scheduled.

0. The 30-second mental model

Kubernetes is a declarative, level-triggered control system: you write desired state (objects); controllers continuously reconcile reality to it. Rack software relates three ways — it runs on k8s (DaemonSet agents), integrates with it (cordon/drain/label), and extends it (operators encoding rack lifecycle). The one idea to carry: the reconcile loop is the same declared-vs-discovered convergence as Phases 01 and 05.

1. Control plane in one diagram

kubectl ─▶ API server ─▶ etcd (truth)
                │
   ┌────────────┼─────────────┐
 scheduler   controllers     (watch + reconcile)
   │              │
   ▼              ▼
  node: kubelet ─▶ containerd ─▶ pods

No imperative orders — every component watches state and converges it.

2. Workload objects (pick the right one)

3. The reconcile loop (memorize)

observe → diff(desired spec, actual status/world) → act idempotently → update status → requeue

• level-triggered (acts on current state, not the event) → self-healing, survives missed events.
• idempotent (no-op once matched) · requeue+backoff on failure · finalizers for delete-time cleanup · owner refs for GC.
• spec = desired (users write) · status = observed (controller writes).

4. Why a pod is Pending (the filter phase)

capacity (requests > allocatable anywhere) · a taint it doesn't tolerate · nodeSelector/ affinity matches no node · topology can't be satisfied. Read scheduler events first.

5. Accelerators → pods

device-plugin DaemonSet discovers HW → advertises vendor.com/accel: N extended resource → pod resources.limits requests it → scheduler treats as capacity → kubelet allocates the device. Add NFD (labels), Topology Manager (GPU+NIC same NUMA — Phase 04), MIG (fractional).

6. Maintenance verbs (how rack software acts on k8s)

cordon (no new pods) → drain (evict, honor PDBs) → upgrade/firmware (Phase 08) → validate → uncordon. Canary + domain-aware (Phase 01) + image-pinned rollback (Phase 11).

7. RKE2 in one line

Hardened (CIS/FIPS), single-binary, embedded-etcd, air-gappable Kubernetes — the distro for secure/sovereign data centers. Same API; your operators run unchanged. (k3s=edge, OpenShift=enterprise, kubeadm=DIY.)

8. Multi-tenancy boundaries

namespaces + RBAC + ResourceQuota + NetworkPolicy + PodSecurity (soft); separate clusters/ nodes or strong sandboxing (hard). Pairs with Phase 09 VLAN/QoS isolation.

9. The "done this before" tells

"Is it level- or edge-triggered?" · "Is the reconcile idempotent?" · "Did you add a finalizer for cleanup?" · "Why is it Pending — capacity, taint, or affinity?" · "DaemonSet for the agent, right?" · "Did you drain before the upgrade?"

10. How this phase pays off later

The operator/reconcile loop is the Phase 13 capstone's control core. cordon/drain is the safe path for Phase 08 firmware updates. Tenancy pairs with Phase 09. DaemonSet is how the Phase 07 exporter ships.

Lab 01 — A RackNode Kubernetes Operator (reconcile loop)

Phase: 06 — Orchestration | Difficulty: ⭐⭐⭐⭐☆ | Time: 5–7 hours Language: Python 3 (sim) + a real CRD | Hardware: none (kind for the extension)

Concept primer: ../WARMUP.md Ch. 4, ../HITCHHIKERS-GUIDE.md §3.

Run

python3 solution.py     # provision -> idempotent re-reconcile -> drift -> delete (finalizer)

Files

• solution.py — the reconcile loop (the operator's core), as a runnable simulator.
• racknode-crd.yaml — the real CRD + an example RackNode (status subresource enabled).
• operator.py — the real kopf operator (same logic, real Kubernetes handlers).
• run-kind.sh + requirements.txt — stand up a real cluster and run the operator.

Run it for REAL on a live cluster (the extension — verified)

Needs Docker + kind + kubectl. The simulator's logic, as an actual operator:

./run-kind.sh up                                   # kind cluster + CRD
pip install -r requirements.txt
kopf run operator.py --verbose                     # the operator (leave running)
# in another shell:
kubectl apply -f racknode-crd.yaml                 # creates node-00
kubectl get racknodes -w                           # PHASE -> Ready, FIRMWARE -> desired
kubectl patch racknode node-00 --type merge \
  -p '{"spec":{"desiredFirmware":"2.2.0"}}'        # drift -> re-converges
kubectl delete racknode node-00                    # kopf finalizer cleanup runs
./run-kind.sh down

This was run end-to-end on a kind v1.31 cluster — observed output:

NAME      PHASE   FIRMWARE   DESIRED
node-00   Ready   2.1.0      2.1.0      # reconciled to baseline (with a level-triggered re-run)
node-00   Ready   2.2.0      2.2.0      # after a desiredFirmware patch: drift corrected
# on delete: "decommissioned node-00 (finalizer cleanup done)"

Note: kopf logs each TemporaryError("converging firmware") as failed temporarily — that's the requeue mechanism (level-triggered re-run), not a real failure. kopf manages its own finalizer because operator.py has an @kopf.on.delete handler.

0. The mission

Build the loop every operator runs and that the JD's "integrate with Kubernetes" and rack lifecycle automation depend on: observe → diff(spec, status/world) → act idempotently → update status → requeue. Make it level-triggered (self-healing) and prove it converges, no-ops when matched, corrects drift, requeues on failure, and cleans up via a finalizer.

1. The five behaviors to internalize

1. Convergence: a RackNode with desiredFirmware: 2.1.0 drives the world to 2.1.0 and reaches Ready.
2. Idempotency / level-triggered: re-reconciling a converged node adds nothing and asks for no requeue — it reacts to current state, not events.
3. Drift correction: an out-of-band firmware downgrade is detected and remediated on the next reconcile (this is why level-triggered beats edge-triggered).
4. Requeue on transient failure: a flaky firmware action raises, the controller requeues, and it eventually converges — never crashes, never half-applies.
5. Finalizer cleanup: deleting the object runs the decommission cleanup before the object disappears, so the real node isn't orphaned.

2. Extensions (do them)

1. Make it real with kopf + kind: kind create cluster, kubectl apply -f racknode-crd.yaml, then write a kopf handler whose @kopf.on.create/update/delete functions mirror reconcile(). Watch kubectl get racknodes show the status converge.
2. Status conditions with timestamps (Kubernetes-style condition objects: type/status/ reason/lastTransitionTime).
3. Owner references + GC: have the RackNode own a child Job (the firmware update) so deleting the RackNode garbage-collects it.
4. Backoff: replace the naive retry with exponential backoff + a max-requeue policy (Phase 02).

3. What this lab proves

You understand the operator pattern at the level that lets you build one, not just use one — the modern way rack-management logic lives in the cluster. "Explain the reconcile loop and why it's level-triggered" becomes a demonstration. This loop is the control core of the Phase 13 capstone, and it's the same convergence pattern as Phase 05 CM and Phase 01 inventory.

Lab 02 — Device Plugin & Accelerator Scheduling

Phase: 06 — Orchestration | Difficulty: ⭐⭐⭐☆☆ | Time: 4–5 hours Language: Python 3 (sim) + real manifests | Hardware: none (kind/GPU for extensions)

Concept primer: ../WARMUP.md Ch. 5–6, ../HITCHHIKERS-GUIDE.md §4–§5.

Run

python3 solution.py     # place pods across nodes; capacity, taints, topology, Pending

Files

• solution.py — a mini scheduler (filter → score → bind) for accelerator pods.
• manifests.yaml — the real device-plugin DaemonSet + a pod requesting an accelerator.

0. The mission

Understand how an accelerator goes from hardware to a scheduled pod, and why a pod ends up Pending — by building the scheduler's two-phase placement (filter then score) with capacity, taints/tolerations, and topology (NUMA/PCIe from Phase 04).

1. What the sim shows

• Capacity: too-big (16 accelerators) fits nowhere → Pending with the per-node reasons (like kubectl describe pod).
• Taints/tolerations: a maintenance-tainted node only accepts pods that tolerate it (maint-tool can; infer-b can't) — the basis of cordon/drain.
• Topology: want_topology pods are scored to land where accelerators share the NIC's NUMA node (Phase 04 performance).
• Binding consumes capacity: placement mutates node usage, so later pods see less room.

2. The real manifests

manifests.yaml shows the device-plugin DaemonSet (one per node, privileged, tolerates everything so it runs even on tainted nodes), a tenant pod requesting qualcomm.com/ai-accelerator: 1 with anti-affinity to spread replicas across PDUs (Phase 01 failure domains), and the taint/cordon/drain commands for maintenance.

3. Extensions (do them)

1. Run a real device plugin: on a kind cluster with a fake-device plugin (or a real GPU node + NVIDIA's plugin), kubectl describe node to see the advertised resource, then schedule a pod that requests it.
2. MIG-style fractional resources: advertise accel-slice units and schedule sub-device workloads — the utilization/isolation tradeoff (Phase 09).
3. Gang scheduling: require all N accelerators of a multi-GPU job to be placed together or none (avoid partial deadlock) — a real AI-training need.
4. Topology Manager: read /sys NUMA from Phase 04 Lab 02 and feed real adjacency into the scorer.

4. What this lab proves

You can reason about accelerator scheduling end to end — the device-plugin path, why a pod is Pending, and topology-aware placement — which is exactly how rack-management software gets tenant AI workloads onto the right hardware. Combined with Lab 01's operator, you have the two halves of "integrate rack management with Kubernetes."

Phase 07 — Telemetry & Observability

Difficulty: ⭐⭐⭐☆☆ | Estimated Time: 2 weeks Roles supported: Rack Management SWE (Senior/Staff), SRE, Platform/Infra SWE Hardware needed: none — the exporter + Prometheus/Grafana run locally (Docker optional)

Why This Phase Exists

The JD is explicit and repeated: "Develop dashboards and unified telemetry pipelines using Prometheus and Grafana" and "Implement observability features such as logging, metrics, and tracing to support troubleshooting and operational reliability." You cannot operate — let alone debug in production (Phase 10) — what you cannot see. Observability is how a rack fleet stops being a black box.

The hard, role-specific problem is unification: a rack's signals come from Redfish nodes, IPMI BMCs, SNMP PDUs/switches, and CDUs (Phases 03–04), each with its own model. Your job is to normalize them into one telemetry pipeline — metrics (Prometheus), dashboards (Grafana), plus logs and traces — so an operator sees one coherent view and alerts fire on the right signals (the ΔT/PCIe/power thresholds from Phases 01/04).

You'll build a Redfish→Prometheus exporter that serves real exposition-format metrics, with PromQL alert rules and a Grafana dashboard, and an OpenTelemetry tracing + structured logging demo that makes a multi-step rack workflow debuggable.

Concepts

The three pillars (and the fourth)

• Metrics: numeric time-series (temperature, power, error counts); cheap, aggregatable, the basis of dashboards and alerts
• Logs: discrete structured events (Phase 02); the detail when a metric says "something's wrong"
• Traces: the path of a request/workflow across components (where time went, what failed)
• (Events/alerts): state changes and the notifications they trigger

Prometheus (the metrics standard)

• The pull model: Prometheus scrapes /metrics HTTP endpoints on an interval (vs push)
• Exposition format: metric_name{label="value"} number — the text format an exporter serves
• Metric types: counter, gauge, histogram, summary; when to use each
• Exporters: the adapter pattern — node_exporter, redfish_exporter, snmp_exporter, ipmi_exporter; you write one for rack devices
• PromQL: rate(), aggregation (sum/avg by label), threshold queries; recording rules; alerting rules → Alertmanager
• Cardinality (the #1 Prometheus footgun): labels explode series count; design labels carefully
• Service discovery (k8s/file-based); scrape configs; the TSDB

Grafana (dashboards)

• Data sources, panels, variables/templating; dashboards-as-code (JSON); per-rack/per-fleet views
• Designing a useful dashboard: the USE/RED methods; what to put on a rack overview

Logs & tracing

• Structured logging (Phase 02) → aggregation (Loki/ELK); correlation IDs
• OpenTelemetry: traces, spans, context propagation, trace_id/span_id; the OTel collector
• Tying logs↔traces↔metrics with shared IDs (exemplars); the debugging payoff

Operational practice (the senior layer)

• SLI/SLO/error budgets; alert on symptoms (SLOs) not causes; alert fatigue
• The unified pipeline architecture: collectors near hardware → central TSDB/log store → dashboards/alerts
• What to alert on for a rack: thermal (ΔT, throttle), power (over-budget, PSU), PCIe (AER, degraded link), node down, firmware drift

Labs

Lab 01 — Redfish → Prometheus Exporter + Grafana

Lab 02 — OpenTelemetry Tracing + Structured Logging

Deliverables Checklist

• Exporter serves valid Prometheus exposition format on /metrics
• You can choose counter vs gauge vs histogram correctly and explain why
• You can explain the cardinality trap and design labels to avoid it
• Alert rules fire on the right rack symptoms (thermal/power/PCIe/node-down)
• You can read/build a Grafana dashboard and explain USE/RED
• Tracing demo shows a span tree with timings and a marked failure
• Logs carry the trace_id so logs↔traces correlate
• You can articulate SLI/SLO and "alert on symptoms, not causes"

Interview Relevance

• "Design a telemetry pipeline for a rack fleet with Redfish, IPMI, and SNMP devices."
• "Prometheus pull vs push — why pull? When would you push?"
• "Counter vs gauge vs histogram — give an example of each for a rack."
• "What is metric cardinality and how do you keep it under control?"
• "What would you alert on for an AI rack, and how do you avoid alert fatigue?"
• "Metrics say latency spiked. How do traces and logs help you find why?"
• "What's an SLO and an error budget, and why alert on symptoms not causes?"

Warmup Guide — Telemetry & Observability

Zero-to-expert primer for Phase 07. You need no monitoring background: this guide starts at "what is a metric" and ends with you designing a unified rack telemetry pipeline, writing PromQL alerts, and correlating metrics/logs/traces. This is how you see the fleet — the prerequisite for operating and debugging it (Phase 10).

Table of Contents

• Chapter 1: Observability vs Monitoring; the Three Pillars
• Chapter 2: Metrics and the Prometheus Model
• Chapter 3: Metric Types and the Exposition Format
• Chapter 4: Exporters — The Unification Problem
• Chapter 5: PromQL, Recording Rules, and Alerting
• Chapter 6: Cardinality — The Footgun
• Chapter 7: Grafana and Dashboard Design
• Chapter 8: Logs, Traces, and Correlation
• Chapter 9: SLIs/SLOs and Alerting Philosophy
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Observability vs Monitoring; the Three Pillars

Monitoring = watching known signals for known problems (is CPU > 90%?). Observability = being able to ask new questions about your system's behavior from its outputs, including ones you didn't anticipate ("why is this specific rack's tail latency high since the firmware update?"). You build monitoring; you design for observability.

The classic three pillars:

• Metrics — numeric time-series sampled over time (temperature, power, request rate). Cheap to store, easy to aggregate and alert on; the backbone of dashboards. Weakness: they tell you that something is wrong, rarely why at fine grain.
• Logs — timestamped, ideally structured (Phase 02) events. The detail behind a metric anomaly. Weakness: volume and cost; hard to aggregate.
• Traces — the end-to-end path of one request/workflow across components, broken into spans with timings. The answer to "where did the time go / where did it fail." Weakness: needs instrumentation and (usually) sampling.

The power is in correlation: a metric alert → jump to the traces in that window → jump to the logs for the failing span, all tied by shared IDs. For a rack fleet, metrics catch the thermal/power/PCIe problems, logs explain the specific failure, traces show which step of a provisioning/firmware workflow stalled.

Chapter 2: Metrics and the Prometheus Model

Prometheus is the de-facto open metrics system (the JD names it). Its defining choice is the pull model: Prometheus periodically scrapes an HTTP /metrics endpoint on each target and stores the samples in its time-series database (TSDB).

Why pull (a guaranteed interview question):

• Targets are dumb; the monitoring system controls scrape frequency and knows the target list (via service discovery) — so a missing target is itself a signal ("up == 0").
• No push infrastructure for thousands of targets to overwhelm; backpressure is natural.
• Easy local testing: curl /metrics shows exactly what Prometheus sees.

When you do push: short-lived batch jobs that may finish before any scrape (use the Pushgateway), and across network boundaries you can't scrape (remote-write to a central Prometheus/Mimir/Thanos). For a rack, the agents/exporters expose /metrics and Prometheus scrapes them; ephemeral provisioning jobs might push to a Pushgateway.

A time-series is identified by its name + a set of labels: node_temp_celsius{rack="r1", node="n0",sensor="accel0"}. Labels are the dimensions you slice and aggregate by — and the source of cardinality danger (Chapter 6).

Chapter 3: Metric Types and the Exposition Format

The exposition format is plain text an exporter serves:

# HELP node_temp_celsius Temperature in Celsius
# TYPE node_temp_celsius gauge
node_temp_celsius{rack="r1",node="n0",sensor="accel0"} 71
node_temp_celsius{rack="r1",node="n0",sensor="inlet"} 27
# TYPE pcie_correctable_errors_total counter
pcie_correctable_errors_total{node="n0",bdf="0000:13:00.0"} 250

The four metric types — choosing correctly is a senior signal:

• Counter: monotonically increasing (only goes up, resets on restart). Errors, bytes, requests. You almost always query rate(counter[5m]), never the raw value. PCIe AER errors, packets, firmware-update attempts.
• Gauge: a value that goes up and down. Temperature, power draw, fan RPM, free memory, CDU ΔT. Most rack hardware telemetry is gauges.
• Histogram: samples bucketed into ranges + a sum + a count; lets you compute quantiles (p99) server-side via histogram_quantile. Scrape latency, provisioning step duration.
• Summary: client-side quantiles (less flexible for aggregation than histograms; prefer histograms when you'll aggregate across instances).

The rule of thumb: did it only ever increase? → counter; can it decrease? → gauge; do you need a distribution/quantile? → histogram.

Chapter 4: Exporters — The Unification Problem

Prometheus scrapes HTTP; your hardware speaks Redfish/IPMI/SNMP/Modbus (Phases 03–04). An exporter bridges the gap: it pulls from the device's native protocol and re-exposes the data as Prometheus metrics on /metrics. The ecosystem has node_exporter (host metrics), snmp_exporter, ipmi_exporter, redfish_exporter — and for a rack with heterogeneous gear, you build a unified exporter that uses the Phase 03 driver abstraction (Redfish-first, IPMI/SNMP fallback) and normalizes everything into one consistent metric/label scheme.

This is the JD's "unified telemetry pipelines" in concrete form, and it's Lab 01. The design choices that matter:

• Consistent naming/labels across protocols (a temperature is node_temp_celsius regardless of whether it came from Redfish Thermal or IPMI SDR) — so dashboards and alerts don't care about the source.
• Scrape-time vs background-poll: BMCs are slow/fragile (Phase 03 Ch. 9), so the exporter usually polls devices on its own schedule (with caching, retries, timeouts — Phase 02) and serves the cached values on scrape, rather than hammering a BMC synchronously on every Prometheus scrape.
• An up/health metric per device so you can alert on "exporter can't reach this BMC."

Chapter 5: PromQL, Recording Rules, and Alerting

PromQL queries the TSDB. The patterns you'll use constantly:

• Aggregate across a label: avg by (rack) (node_temp_celsius), sum by (rack) (node_power_watts).
• Rate of a counter: rate(pcie_correctable_errors_total[5m]) — errors per second.
• Threshold for an alert: node_temp_celsius > 85, sum by (rack)(node_power_watts) > 17000 (the Phase 01 budget!).
• up == 0 — a target Prometheus couldn't scrape (device/exporter down).

Recording rules precompute expensive/常用 queries into new series (e.g., per-rack power) so dashboards and alerts are fast.

Alerting rules are PromQL conditions with a for: duration (must hold N minutes to fire, avoiding flapping), labels (severity), and annotations (description/runbook link → Phase 12). They fire to Alertmanager, which dedupes, groups, silences, and routes (page/Slack/email). Example rack alerts:

- alert: AcceleratorHot
  expr: node_temp_celsius{sensor=~"accel.*"} > 85
  for: 5m
  labels: { severity: warning }
  annotations: { summary: "Accelerator hot on {{$labels.node}}", runbook: "..." }
- alert: RackPowerOverBudget
  expr: sum by (rack)(node_power_watts) > 17000
  for: 2m
  labels: { severity: critical }
- alert: NodeDown
  expr: up{job="rack-exporter"} == 0
  for: 1m
  labels: { severity: critical }

Chapter 6: Cardinality — The Footgun

The single most important Prometheus operational lesson. Cardinality = the number of distinct time-series, which is the product of every label's distinct values. Each unique label-set is a separate series consuming memory and disk.

The trap: putting a high-cardinality value in a label — a timestamp, a request ID, a serial number on a high-churn metric, a full error message — multiplies series explosively and can OOM Prometheus. http_requests_total{user_id="..."} with a million users = a million series per metric.

The discipline:

• Labels are for bounded, low-cardinality dimensions you aggregate/filter by: rack, node, sensor, severity, job. Not unbounded identifiers.
• Put high-cardinality detail in logs/traces (Chapter 8), not metric labels.
• Watch series count; alert on it; review new metrics for label explosions in code review.

For a rack fleet (thousands of nodes × dozens of sensors), even legitimate labels add up — which is why naming and label design is a real engineering task, not an afterthought.

Chapter 7: Grafana and Dashboard Design

Grafana visualizes metrics (and logs/traces) from data sources like Prometheus. Dashboards are JSON (dashboards-as-code — version them, review them, Phase 11). Design matters more than prettiness:

• Variables/templating: a $rack/$node dropdown so one dashboard serves the whole fleet.
• Two proven methods: USE (Utilization, Saturation, Errors) for resources (per accelerator/PDU/CDU); RED (Rate, Errors, Duration) for services (the control-plane API). For a rack overview: power vs budget, temps with thresholds, PCIe error rates, node up/down, CDU ΔT/flow, firmware-version spread.
• The hierarchy: a fleet view (which racks are unhealthy) → a rack view (which nodes) → a node view (which sensor) — so an operator drills from symptom to cause.
• Avoid the "wall of graphs nobody reads"; every panel should answer a question or back an alert.

Chapter 8: Logs, Traces, and Correlation

Logs (Phase 02 structured JSON) get aggregated (Loki, Elasticsearch) so you can query across the fleet. The key for correlation: include a trace_id and stable labels (node, rack, request_id) in every line.

Traces (OpenTelemetry — the vendor-neutral standard) capture a workflow as a tree of spans, each with a name, start/end (duration), attributes, and a parent — sharing a trace_id and each having a span_id. Context propagation carries the trace_id/ span_id across function calls and across services (via headers), so a provisioning workflow that spans the orchestrator, a BMC call, and a firmware job is one trace. Traces answer "where did the time go / which step failed," which metrics and logs alone don't.

Correlation is the payoff: a metric alert → the traces in that window (via exemplars linking a metric sample to a trace) → the failing span → its logs (same trace_id). This is the debugging loop (Phase 10) that turns "the fleet is slow" into "step 3 of provisioning on node-237 times out calling the BMC, here's the log." The OTel collector receives, processes (sampling), and exports telemetry to backends (Jaeger/Tempo for traces, Prometheus for metrics) — one pipeline, vendor-neutral. Lab 02 builds the tracing + correlation.

Chapter 9: SLIs/SLOs and Alerting Philosophy

The operational maturity layer:

• SLI (Service Level Indicator): a measured quantity reflecting user/operator experience (e.g., "fraction of provisioning runs that succeed," "p99 control-plane API latency," "fraction of nodes healthy").
• SLO (Objective): a target for an SLI over a window ("99% of provisions succeed," "99.9% node availability/month").
• Error budget: 1 − SLO. The allowed failure. Spend it on velocity (ship changes) until it's low, then slow down — a shared language between dev and ops.

Alert on symptoms, not causes. Page on the thing that matters (SLO breach: provisioning failing, a rack offline, accelerators throttling) — not on every cause (a single retry, a transient blip). Cause-based alerts produce alert fatigue: too many pages, people stop reading them, the real one gets missed. Every page should be actionable and link a runbook (Phase 12). For a rack: page on node-down, rack-over-budget, sustained accelerator overheat, CDU leak (Phase 04), and SLO breaches; don't page on a single correctable PCIe error (dashboard it, alert only on a rate).

Lab Walkthrough Guidance

Order: Lab 01 (exporter/metrics) → Lab 02 (tracing/logs). Metrics are the backbone; tracing is the deep-dive when metrics flag a problem.

Lab 01 (Redfish→Prometheus exporter):

1. python3 solution.py, then curl localhost:9101/metrics — read the exposition format.
2. Note the metric types (gauge for temp/power, counter for AER errors) and the bounded labels (rack/node/sensor) — and that there are no high-cardinality labels (Ch. 6).
3. Read alerts.yml (the PromQL rack alerts) and dashboard.json (Grafana).
4. Extensions: run real Prometheus scraping it + import the dashboard into Grafana; wire it to the Phase 04 PDU/CDU/PCIe sim.

Lab 02 (OTel tracing + logs):

1. python3 solution.py — a traced provisioning-style workflow with an injected failure.
2. Read the span tree (timings, parent/child) and the structured logs carrying the trace_id.
3. See how the failed span + its correlated logs pinpoint the problem.
4. Extensions: export OTLP to Jaeger/Tempo; add a metric exemplar linking a slow metric to its trace.

Phase capstone question: "Design the telemetry pipeline for a 10,000-node accelerator fleet with Redfish/IPMI/SNMP devices. What do you alert on, and how do you keep it scalable and debuggable?" (Answer in system-design/02: unified exporters near hardware → Prometheus (federated/remote-write to Mimir/Thanos for scale) → Grafana fleet→rack→node dashboards → Alertmanager on SLO symptoms with runbooks; logs to Loki and traces via OTel, correlated by IDs; ruthless cardinality discipline.)

Success Criteria

You're done with this phase when — without notes:

• You can explain the three pillars and what each is good/bad at (Ch. 1)
• You can explain Prometheus pull vs push and when to push (Ch. 2)
• You can pick counter/gauge/histogram for a given rack signal and read exposition format (Ch. 3)
• You can describe the exporter pattern and how you'd unify Redfish/IPMI/SNMP (Ch. 4)
• You can write PromQL for an aggregation, a rate, and an alert threshold (Ch. 5)
• You can explain cardinality and avoid the label-explosion trap (Ch. 6)
• You can design a fleet→rack→node Grafana dashboard with USE/RED (Ch. 7)
• You can explain traces/spans/context propagation and log↔trace correlation (Ch. 8)
• You can define SLI/SLO/error budget and "alert on symptoms not causes" (Ch. 9)

Interview Q&A

Q1: Design a telemetry pipeline for a rack fleet with Redfish, IPMI, and SNMP devices. A: I'd build unified exporters near the hardware that use the Phase 03 driver abstraction (Redfish-first, IPMI/SNMP fallback), poll each device on their own schedule with caching/ retries (BMCs are fragile), and re-expose everything as Prometheus metrics with a consistent naming/label scheme so a temperature is node_temp_celsius regardless of source. Prometheus scrapes the exporters (pull); at fleet scale I federate or remote-write to a scalable backend (Mimir/Thanos). Grafana provides fleet→rack→node dashboards (USE for hardware, RED for the control plane). Alertmanager fires on SLO symptoms (node down, rack over power budget, sustained overheat, CDU leak) with runbook links, deduped and routed. Logs go to Loki and traces via OpenTelemetry, all correlated by node/rack/trace_id. The two disciplines that keep it healthy: strict cardinality control (bounded labels only) and alerting on symptoms not causes.

Q2: Why does Prometheus pull instead of push, and when would you push? A: Pull means the monitoring system owns the target list (via service discovery) and the scrape schedule, so a target simply being unreachable is itself a signal (up == 0), there's no push infrastructure to overwhelm, backpressure is natural, and I can curl /metrics to see exactly what Prometheus sees. I push in two cases: short-lived batch jobs that may finish before any scrape (via the Pushgateway), and crossing network boundaries I can't scrape (remote-write to a central store). For a rack, exporters expose /metrics and Prometheus pulls; an ephemeral provisioning job might push its result to a Pushgateway.

Q3: Counter vs gauge vs histogram — give a rack example of each. A: A counter only increases (resets on restart) and you query its rate — e.g., pcie_correctable_errors_total, queried as rate(...[5m]) to see error velocity. A gauge goes up and down — e.g., node_temp_celsius or cdu_delta_t_celsius or node_power_watts; most hardware telemetry is gauges. A histogram buckets samples so you can compute quantiles server-side — e.g., provisioning_step_duration_seconds so I can alert on p99 step latency across the fleet. The rule: only-ever-up → counter, up-and-down → gauge, need-a-distribution → histogram.

Q4: What is cardinality and how do you keep it from blowing up? A: Cardinality is the number of distinct time-series, which is the product of each label's distinct values — every unique label-set is a separate series costing memory/disk. It blows up when you put an unbounded value in a label (a request ID, timestamp, full error string, or a serial on a high-churn metric), which can OOM Prometheus. The discipline: labels are only for bounded, low-cardinality dimensions you aggregate/filter by (rack, node, sensor, severity); high-cardinality detail goes in logs/traces, not metric labels; and I watch the series count, alert on it, and review new metrics in code review for label explosions. At thousands of nodes × dozens of sensors, even legitimate labels add up, so label design is real engineering.

Q5: What would you alert on for an AI rack, and how do you avoid alert fatigue? A: Alert on symptoms that matter, with a for: duration to avoid flapping and a runbook on each: node/exporter down (up==0), rack over power budget (sum by(rack)(power) > derated), sustained accelerator overheat or thermal throttling, CDU leak or ΔT/flow anomaly (Phase 04), and SLO breaches (provisioning success rate, control-plane latency). I avoid fatigue by not paging on every cause — a single correctable PCIe error or one retry gets dashboarded, and I alert only on a rate or sustained condition; non-urgent issues go to a ticket, not a page. Every page must be actionable; if it isn't, it becomes a dashboard or a tuned threshold. Symptom-based, SLO-driven alerting keeps the signal-to-noise high so the real page gets read.

Q6: Metrics show provisioning latency spiked. Walk me from alert to root cause. A: The alert (an SLO symptom: p99 provisioning duration breached) is the entry point. I pivot to traces in that window — each provisioning run is a trace of spans (discover, firmware, OS install, configure, validate). The span tree shows which step got slow or errored; say the firmware span's duration ballooned. I follow the span's trace_id into the logs (structured, correlated) to see the specific error — e.g., the BMC returning 503s. I cross-check the metrics for that node/BMC (up, scrape latency, BMC error rate) to confirm it's the device, not my code. So: metric alert → trace to localize the step → logs to get the specific failure → metrics to confirm scope. That metrics→traces→logs correlation, tied by IDs, is exactly why you invest in all three pillars (and it's the Phase 10 RCA loop).

References

• Prometheus docs — data model, exposition format, metric types, PromQL — https://prometheus.io/docs/
• Prometheus — instrumentation & naming best practices; histograms vs summaries — https://prometheus.io/docs/practices/
• Brian Brazil, Prometheus: Up & Running — the canonical book
• Grafana docs — dashboards, variables, provisioning (as-code) — https://grafana.com/docs/
• OpenTelemetry — traces/metrics/logs, context propagation, the collector — https://opentelemetry.io/docs/
• Google SRE Book — SLIs/SLOs/error budgets; "alerting on symptoms" — https://sre.google/books/
• The USE method (Brendan Gregg) & the RED method (Tom Wilkie) — dashboard frameworks
• redfish_exporter / snmp_exporter / ipmi_exporter — reference exporter implementations
• Cross-track: Head of SWE — GPU, Phase 10 (observability/production)

🛸 Hitchhiker's Guide — Phase 07: Telemetry & Observability

Read this if: "Prometheus, Grafana, tracing" is a stack you've seen but not built. Compressed map of metrics, the cardinality trap, and the metrics→traces→logs loop.

0. The 30-second mental model

Three pillars: metrics (cheap time-series — that something's wrong), logs (the detail — what), traces (the path — where the time went / what failed). Tie them by shared IDs and you can debug a fleet. The role-specific hard part is unification: one exporter normalizing Redfish/IPMI/SNMP/CDU into consistent metrics. Two laws keep you sane: bound your label cardinality and alert on symptoms, not causes.

1. Prometheus in one breath

Pull model: Prometheus scrapes /metrics on a schedule → TSDB. A series = name{labels}. Pull (not push) because the system owns the target list (so up==0 is a signal), no push infra to overwhelm, and curl /metrics shows what it sees. Push only for batch jobs (Pushgateway) or across boundaries (remote-write).

2. Metric types (pick right)

Exposition format: name{label="v"} 123. Only-up→counter; up/down→gauge; need p99→histogram.

3. The exporter pattern (the unification job)

Hardware speaks Redfish/IPMI/SNMP; Prometheus speaks HTTP. An exporter bridges it — use the Phase 03 driver abstraction, poll devices on your own schedule (cache, retry, timeout — BMCs are fragile), and serve consistent metrics so a temp is node_temp_celsius regardless of source. Add an up/health metric per device.

4. PromQL you'll actually write

avg by (rack) (node_temp_celsius)               # aggregate
sum by (rack) (node_power_watts) > 17000         # over the Phase 01 budget
rate(pcie_correctable_errors_total[5m]) > 1      # error velocity
up{job="rack-exporter"} == 0                      # device/exporter down

Recording rules = precompute; alerting rules = PromQL + for: + severity + runbook → Alertmanager.

5. The cardinality law

series = product of label values. Never put unbounded values (IDs, timestamps, messages) in labels → OOM. Labels = bounded dimensions (rack/node/sensor/severity). Detail → logs/traces.

6. Grafana dashboards

JSON (version it). $rack/$node variables. USE (Utilization/Saturation/Errors) for hardware, RED (Rate/Errors/Duration) for services. Hierarchy: fleet → rack → node. Every panel answers a question or backs an alert.

7. Traces + correlation (the debug loop)

Trace = tree of spans (name, duration, parent) sharing a trace_id; context propagation carries it across calls/services. Logs carry the trace_id. The loop: metric alert → traces in window (exemplars) → failing span → its logs. OTel = the vendor-neutral standard; the collector exports to Jaeger/Tempo/Prometheus.

8. SLO discipline

SLI (measured) → SLO (target) → error budget (1−SLO). Alert on symptoms (SLO breach), not causes — cause alerts → fatigue → missed real page. Every page is actionable + has a runbook (Phase 12).

9. The "done this before" tells

"Counter or gauge?" · "What's the cardinality of that label?" · "Are we alerting on a symptom or a cause?" · "Does the log carry the trace_id?" · "Pull or push for that?" · "What's the SLO?"

10. How this phase pays off later

The exporter is reused in the Phase 13 capstone; alerts/runbooks feed Phase 10 RCA and Phase 12 ops; the fleet pipeline is system-design/02; thresholds come from Phases 01/04.

Lab 01 — Unified Rack Telemetry → Prometheus Exporter

Phase: 07 — Telemetry & Observability | Difficulty: ⭐⭐⭐☆☆ | Time: 4–6 hours Language: Python 3 (stdlib) + real Prometheus/Grafana config | Hardware: none

Concept primer: ../WARMUP.md Ch. 3–7, ../HITCHHIKERS-GUIDE.md §1–§6.

Run

python3 solution.py     # starts the exporter, prints /metrics, self-tests
# in another shell while it runs:  curl localhost:<port>/metrics

Files

• solution.py — the exporter (serves real Prometheus exposition format on /metrics).
• alerts.yml — Prometheus alerting rules (the rack symptoms you page on).
• dashboard.json — a Grafana dashboard-as-code (fleet→rack hardware view).

0. The mission

Build the "unified telemetry pipeline" the JD asks for: one exporter that turns normalized Redfish/IPMI/SNMP rack telemetry (Phase 03) into Prometheus metrics, with the alert rules and Grafana dashboard that make it operable. The point is unification — one consistent metric/label scheme regardless of which protocol the data came from.

1. What to study in the output

• Metric types: temp/power are gauges, AER errors a counter (queried as a rate), and a per-node up gauge — the right type for each signal (WARMUP Ch. 3).
• Bounded labels only: rack, node, sensor, job — the self-test asserts no other labels exist, so cardinality can't explode (WARMUP Ch. 6). No serials, IDs, or timestamps in labels.
• No stale data for down devices: node-02 is unreachable → up=0 and we don't emit a fake temperature for it.
• The hot accelerator (92°C) is exactly what alerts.yml's AcceleratorHot (>85, for 5m) fires on; node-01's power pushes toward the RackPowerOverBudget rule (the Phase 01 budget).

2. The real config

alerts.yml shows symptom-based alerting (for: to avoid flapping, severity, runbook links, and rate for PCIe errors not raw count). dashboard.json shows a $rack variable and USE- style panels (nodes up/down, power vs budget threshold, accel temps with 85/95 thresholds, PCIe error rate).

3. Extensions (do them)

1. Run the real stack: docker run prom/prometheus with a scrape config pointing at this exporter, load alerts.yml, run docker run grafana/grafana, add Prometheus as a data source, and import dashboard.json. Watch the alert fire on the 92°C accelerator.
2. Wire it to Phase 04: pull live PDU/CDU/PCIe values from the Phase 04 Lab 02 simulator so the metrics reflect ΔT, leak state, and degraded links.
3. Add a histogram: scrape_duration_seconds and provisioning_step_duration_seconds; query p99 with histogram_quantile.
4. Background polling + cache: poll the (fake) BMCs on a timer with retries/timeouts (Phase 02) and serve cached values on scrape — the production pattern for fragile BMCs.
5. Cardinality experiment: add a high-cardinality label (a timestamp) and observe how the series count would explode; then remove it and explain why.

4. What this lab proves

You can build the metrics backbone of a rack-management platform — correctly typed, cardinality- safe, unified across protocols, alertable, and dashboardable. "Design a telemetry pipeline for a rack fleet" becomes a demonstration, and this exporter is reused in the Phase 13 capstone and expanded in system-design/02 (the fleet-scale pipeline).

Lab 02 — OpenTelemetry Tracing + Structured Logging

Phase: 07 — Telemetry & Observability | Difficulty: ⭐⭐⭐☆☆ | Time: 3–5 hours Language: Python 3 (stdlib) | Hardware: none

Concept primer: ../WARMUP.md Ch. 8, ../HITCHHIKERS-GUIDE.md §7.

Run

python3 solution.py     # a traced workflow (success + failure) with correlated logs

0. The mission

Make a multi-step rack workflow debuggable: emit a tree of spans with timings and a propagated trace context, and structured logs that carry the active trace_id — so "where did it fail and how long did each step take?" is answerable in seconds, and any log line links to its span. This is the deep-dive half of observability (metrics flag that; traces show where).

1. What to study

• Context propagation: a contextvars current-span carries the trace across nested with span(...) blocks — no manual passing — exactly like OpenTelemetry's implicit context.
• The span tree: provision is the root; discover/firmware/os_install/configure/validate are children sharing one trace_id. The printed tree shows per-step duration_ms.
• Failure pinpointing: node-B's os_install span is marked ✗ ERROR, and the error propagates to the root span — you instantly see which step failed and how long the others took.
• Log↔trace correlation: every log(...) line auto-includes the active trace_id/span_id, so from a log you jump to the trace and vice-versa.

2. Extensions (do them)

1. Use real OpenTelemetry: pip install opentelemetry-sdk opentelemetry-exporter-otlp, replace the toy tracer, and export OTLP to Jaeger or Tempo (docker run jaegertracing/ all-in-one); view the trace tree in the UI.
2. Cross-service propagation: split the workflow across a real HTTP call and propagate the traceparent header so the trace spans both processes.
3. Metric exemplars: link a slow provisioning_step_duration_seconds sample (Lab 01) to its trace, so a dashboard spike jumps straight to the trace (the metrics→traces leap).
4. Sampling: add head/tail sampling and discuss the cost/visibility tradeoff at fleet scale.

3. What this lab proves

You can instrument a workflow so production failures are diagnosable — the JD's "logging, metrics, and tracing... to support troubleshooting." "Latency spiked — find why" becomes the metrics→traces→logs loop (Phase 10 RCA), demonstrated end to end. Combined with Lab 01's metrics/alerts, you have the full observability surface a rack platform needs.

Phase 08 — Firmware Lifecycle, Crash Dumps & RAS Diagnostics

Difficulty: ⭐⭐⭐⭐☆ | Estimated Time: 2 weeks Roles supported: Rack Management SWE (Senior/Staff), Firmware/Platform SWE, RAS/Reliability SWE Hardware needed: none — firmware orchestrator + RAS log parser run as simulators

Why This Phase Exists

The JD names this directly: "Support firmware updates, crash dump parsing, and remote diagnostics across rack components." This is some of the highest-stakes software a rack engineer writes: a botched firmware update can brick a $300k node from a continent away, and the ability to read a crash dump or RAS log is the difference between "RMA the whole node" and "it's a single DIMM / a degraded PCIe lane." It's also where the JD's repeated theme — debugging complex issues in deployed production systems — lives.

Two disciplines:

1. Firmware lifecycle — updating BIOS/UEFI, BMC, NIC, accelerator, PSU, and PCIe-switch firmware safely across a fleet: staging, signature verification, A/B (redundant) banks, rollback, and the Redfish Task model (Phase 03).
2. RAS (Reliability, Availability, Serviceability) — reading the hardware's own error telemetry: PCIe AER, CPU MCE/MCA, ACPI/APEI, the BMC SEL, accelerator crash/Xid-style logs — and turning a flood of cryptic records into an actionable root cause.

You'll build a firmware-update orchestrator with rollback and a RAS/crash-dump parser that classifies errors into a diagnostics verdict.

Concepts

Firmware lifecycle

• The firmware zoo: BIOS/UEFI, BMC, NIC/DPU, accelerator, PSU, PCIe switch, CPLD — each independently versioned (the firmware EBOM baseline, Phase 12)
• Update mechanisms: Redfish UpdateService (SimpleUpdate, multipart push) → a Task (Phase 03 Ch. 4); vendor tools; in-band vs out-of-band updates
• A/B (redundant) banks: update the inactive bank, switch on success, fall back on failure
• Staging, verification, rollback: download → verify signature/checksum → stage → apply → verify → activate (often a reboot) → confirm → rollback on failure
• Firmware signing & secure update: signed images, anti-rollback (don't downgrade past a security fix), the chain of trust (pairs with Phase 09 secure boot)
• Fleet rollout: ordering/dependencies (BMC before BIOS?), canary, drain-first (Phase 06), maintenance windows, the cost of a fleet-wide bad firmware
• Failure modes: bricking, partial update, power loss mid-flash, version skew

RAS — reading the hardware's pain

• PCIe AER (Advanced Error Reporting): correctable vs uncorrectable (fatal/non-fatal); decoding the error bits; a rising correctable rate as a pre-failure signal (Phase 04)
• CPU MCE/MCA (Machine Check Exception / Architecture): decoding a machine-check bank; correctable ECC vs uncorrectable; which component (memory/cache/interconnect)
• ACPI/APEI (Platform Error Interfaces): how firmware reports hardware errors to the OS (HEST/BERT/ERST); the BMC's role
• The SEL (Phase 03) and BMC crash dumps; accelerator crash logs (Xid-style codes)
• Memory errors: correctable ECC (logged, page-offline candidate) vs uncorrectable (fatal); DIMM sparing; predictive failure
• Remote diagnostics: SoL, crash capture, core/kernel dumps (kdump), collecting the evidence OOB when the host is dead

The synthesis (the senior layer)

• Turning many raw error records into one classified diagnosis (which FRU, how severe, what action) — the parser you build
• Correlation with telemetry (Phase 07) and failure domains (Phase 01) for fleet RCA (Phase 10)
• The decision tree: log-only / page / cordon+drain / RMA the FRU

Labs

Lab 01 — Firmware Update Orchestrator (staged + rollback)

Lab 02 — Crash-Dump & RAS Log Parser

Deliverables Checklist

• You can list the firmware components on a node and explain the EBOM baseline
• Orchestrator verifies signatures, uses A/B banks, validates after activation, and rolls back
• You can explain anti-rollback and the tension with needing to recover from a bad update
• You can explain why firmware updates return a Redfish Task and how you'd run a fleet rollout
• You can decode a PCIe AER correctable vs uncorrectable error and say what each means
• You can explain MCE/MCA and correctable-ECC vs uncorrectable memory errors
• Parser produces a FRU-level diagnosis with severity and a recommended action
• You can explain when to log-only vs page vs cordon vs RMA

Interview Relevance

• "How do you safely update firmware on 1,000 nodes without bricking them?"
• "What are A/B firmware banks and why do they exist?"
• "Walk me through verifying and rolling back a firmware update."
• "What is PCIe AER? Correctable vs uncorrectable — what do you do about each?"
• "A node logs a flood of correctable ECC errors. Is that a problem?" (rate → predictive)
• "How do you diagnose a node that crashed in production, remotely?" (SEL, kdump, RAS, SoL)
• "How do you decide whether to RMA a component?"

Warmup Guide — Firmware Lifecycle, Crash Dumps & RAS Diagnostics

Zero-to-expert primer for Phase 08. You need no firmware/RAS background: this guide starts at "what is firmware and why is updating it dangerous" and ends with you designing a safe fleet firmware rollout and reading a PCIe AER / MCE crash dump to a FRU-level diagnosis. This is the highest-stakes, most "debug the deployed system" part of the role.

Table of Contents

• Chapter 1: The Firmware Zoo and Why Updates Are Dangerous
• Chapter 2: Update Mechanisms and the Redfish Task Model
• Chapter 3: A/B Banks, Staging, Verification, Rollback
• Chapter 4: Firmware Signing & Anti-Rollback
• Chapter 5: Fleet Rollout Strategy
• Chapter 6: RAS — How Hardware Reports Its Pain
• Chapter 7: Decoding PCIe AER and CPU MCE
• Chapter 8: Remote Diagnostics and the Decision Tree
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: The Firmware Zoo and Why Updates Are Dangerous

Firmware is the software baked into a hardware component that makes it work. A single AI node has many independently-firmwared components, each a thing you may need to update:

• BIOS/UEFI (host platform firmware), BMC (the management controller itself, Phase 04),
• NIC/DPU firmware, accelerator firmware, PSU firmware, PCIe switch firmware, CPLD, drive firmware.

The set of correct versions across all of these is the firmware baseline — part of the software EBOM you agree with the ODM (Phase 12). Keeping a fleet at a known baseline is a core rack-management job (and the Phase 01 commissioning step).

Why updates are dangerous — this is the framing that makes you careful:

• A failed firmware flash can brick the component (and sometimes the node) — and you're doing it to a machine you can't physically touch, possibly on another continent.
• The riskiest target is the BMC itself: if you brick the BMC, you lose your OOB lifeline to the node (Phase 04) — the very tool you'd use to recover it.
• Power loss or a crash mid-flash can corrupt the firmware image.
• A bad image deployed fleet-wide is a catastrophe at the scale of "thousands of nodes offline." Firmware rollouts are where a single bug becomes a major incident.

So the entire discipline of this phase is making an inherently dangerous operation safe: redundancy (A/B banks), verification (signatures, before and after), reversibility (rollback), and caution (staging, canary, drain-first). Respect for this danger is itself a senior signal.

Chapter 2: Update Mechanisms and the Redfish Task Model

How an update is delivered:

• Redfish UpdateService (the modern standard, Phase 03): SimpleUpdate (point the service at an image URL) or a multipart HTTP push (upload the image directly). Because flashing is slow, the action returns HTTP 202 + a Task (Phase 03 Ch. 4) — you poll the Task's TaskState/PercentComplete to completion. A client that blocks on a synchronous response is wrong.
• Vendor tools / in-band agents: some components update from inside the host OS (an agent flashes the NIC/accelerator). In-band is faster but requires a healthy host; OOB (via the BMC) works even on a sick node and is preferred for platform firmware.
• Activation: many updates only take effect after a reboot or an explicit activation step — so the flow is apply → activate (reboot) → verify the new version is running.

The Task model also gives you progress and result messages, which your orchestrator surfaces to operators (Phase 07) and uses to decide success/failure.

Chapter 3: A/B Banks, Staging, Verification, Rollback

The core safety mechanism, and the heart of Lab 01.

A/B (redundant) banks: the component has two firmware slots. One is active (running); the other is inactive. You flash the inactive bank, then switch the active pointer to it on success. If the new firmware fails to boot or fails validation, you switch back to the known-good bank. This means a bad flash never destroys your working firmware — the single most important firmware-safety idea. (Not every component has A/B; where it doesn't, the risk and the need for caution are higher.)

The safe update state machine (what your orchestrator implements):

1. Download the image (to a staging area).
2. Verify the image: checksum + signature (Chapter 4) + version/compatibility + anti-rollback. Reject before touching the device if anything fails.
3. Stage: flash the inactive bank. (Active firmware still running — no disruption yet.)
4. Activate: switch the active pointer / reboot into the new bank.
5. Verify post-activation: confirm the device booted, reports the new version, and is healthy (sensors nominal, links up — Phase 04). Verify after, not just before — a flash can succeed but the firmware misbehave.
6. Commit or rollback: if healthy, commit; if not, switch back to the previous bank (and reboot), restoring the known-good state.

Idempotency & resumability (Phase 05 again): the orchestrator tracks state so a crash mid-update resumes safely, and re-running a completed update is a no-op.

Chapter 4: Firmware Signing & Anti-Rollback

Firmware is a prime attack target (malicious firmware is persistent and below the OS — Phase 09). So:

• Signed images: the vendor signs the firmware; the component (or your orchestrator) verifies the signature against a trusted public key before flashing. An unsigned/tampered image is rejected. (Lab 01 models this with HMAC/signature verification.)
• The chain of trust: signed firmware ties into secure boot (Phase 09) — each stage verifies the next. A compromised firmware update would break the chain.
• Anti-rollback: you must not be able to downgrade firmware past a version that fixed a security vulnerability, or an attacker would "update" you to a known-exploitable old version. Components track a minimum version (a monotonic counter); the orchestrator enforces it.
• The tension (a good interview point): anti-rollback protects security but can trap you on a bad new version. The resolution is the A/B bank (roll back to the other good bank, not a downgrade to an old vulnerable one) plus a vendor recovery path for genuine emergencies.

Chapter 5: Fleet Rollout Strategy

Updating one node safely is Lab 01; updating ten thousand without a fleet-wide incident is the Staff problem (and system-design/03).

• Ordering/dependencies: some firmware must update in a specific order (e.g., BMC before BIOS if the new BIOS needs new BMC support; or a NIC firmware that requires a driver version). Encode dependencies; don't flash blindly.
• Drain first (Phase 06): cordon + drain the node (move tenant workloads off) before an update that requires a reboot — never reboot a node running tenant work.
• Canary + progressive rollout (Phase 11): update 1 node, bake/validate, then 1%, then 10%, then the rest — with automatic halt-and-rollback if health/SLOs (Phase 07) regress. A bad image caught at 1% is an annoyance; at 100% it's an outage.
• Failure-domain awareness (Phase 01): don't update a whole power/cooling/ToR domain at once (correlated downtime).
• Maintenance windows & comms: tenant-visible disruption is scheduled and communicated.
• Observability: track version distribution across the fleet (a Phase 07 metric: firmware_version{component,version}), so drift and rollout progress are visible.

The mental model: a firmware rollout is a deployment with the blast radius of physical hardware — apply software-deployment discipline (canary, rollback, observability) with extra caution because you can't just kubectl rollout undo a bricked BMC.

Chapter 6: RAS — How Hardware Reports Its Pain

RAS = Reliability, Availability, Serviceability: the hardware/firmware machinery that detects, logs, and sometimes corrects errors, and the software that reads it. This is how you "debug the deployed system" when something physical goes wrong. The sources:

• PCIe AER (Advanced Error Reporting): errors on the PCIe fabric (Phase 04) — correctable (recovered by the link layer; a rising rate predicts failure) vs uncorrectable (fatal or non-fatal; data integrity at risk). Reported via the OS (dmesg/sysfs) and/or the BMC.
• CPU MCE/MCA (Machine Check Exception / Architecture): the CPU's error reporting — memory ECC errors, cache errors, interconnect errors — organized into banks you decode to find which subsystem erred. Correctable (logged) vs uncorrectable (often a crash).
• ACPI/APEI (Platform Error Interfaces — HEST/BERT/ERST): how platform firmware reports hardware errors to the OS, including a BERT (Boot Error Record Table) that surfaces the cause of the last crash after a reboot — gold for diagnosing a node that mysteriously reset.
• The SEL (Phase 03): the BMC's event log — PSU input lost, thermal trips, DIMM thresholds.
• Accelerator crash logs: vendor error codes (NVIDIA's Xid is the canonical example) for GPU/accelerator faults — ECC errors, falls-off-the-bus, thermal, app errors.
• Memory errors specifically: correctable ECC (single-bit, auto-fixed — but a DIMM throwing many is a candidate for page-offlining or replacement) vs uncorrectable (a multi-bit error — data loss, usually fatal).
• kdump: the Linux mechanism that captures a kernel crash dump (vmcore) into reserved memory on panic — your evidence for a host kernel crash.

Chapter 7: Decoding PCIe AER and CPU MCE

The concrete skill (Lab 02). The raw records are cryptic; decoding them to meaning is the value.

PCIe AER: an error is reported with a class and a set of status bits. The decode you need:

• Correctable (e.g., Receiver Error, Bad TLP, Bad DLLP, Replay Timer Timeout): the link recovered. One is noise; a high rate on one device signals a marginal lane/cable/connector — a pre-failure indicator (ties to Phase 04 degraded links).
• Uncorrectable non-fatal (e.g., Completion Timeout, Completer Abort): the transaction failed but the link survives; the device/driver may recover or error up.
• Uncorrectable fatal (e.g., Malformed TLP, link errors): the link is unreliable; often a device reset or node impact. The BDF (Phase 04) tells you which device.

CPU MCE: a machine check is decoded from the bank number + status/address registers:

• The bank identifies the subsystem (memory controller, cache, interconnect).
• Correctable (CE): logged, system continues (e.g., a corrected DRAM ECC error → which DIMM, via the address). A rising CE rate on one DIMM → predictive replacement / page-offline.
• Uncorrectable (UC): data integrity lost; often a panic/crash. The bank+address point to the failing FRU.

The parser's job (Lab 02): take a mixed stream of AER/MCE/SEL records, decode each to a human meaning, map it to a FRU (which DIMM, which PCIe device, which PSU), assign severity, detect predictive patterns (rising correctable rates), and emit a recommended action. That turns "500 lines of dmesg horror" into "DIMM A2 has a rising correctable-ECC rate — schedule replacement; PCIe device 0000:13:00.0 had one correctable error — monitor."

Chapter 8: Remote Diagnostics and the Decision Tree

When a node misbehaves in production and you can't walk up to it, your evidence is all OOB/ remote:

• SoL (Phase 04): watch the console for a panic/POST error in real time.
• The SEL + BERT: read what the BMC logged and what caused the last crash after a reboot.
• kdump vmcore: analyze the kernel crash dump (with crash/drgn) for a host panic.
• RAS logs (AER/MCE): the parser's input.
• Telemetry (Phase 07): correlate with temperature/power/error-rate trends around the event.
• Failure domain (Phase 01): is this one node or a shared-resource event?

The decision tree your software (and you) applies to a diagnosis:

• Log-only: a single correctable error / known-benign event → record, don't alert.
• Monitor/predictive: a rising correctable rate → flag the FRU for proactive replacement at the next window (avoid an outage).
• Cordon + drain (Phase 06): a degrading-but-up node → take it out of service for tenants.
• RMA the FRU: an uncorrectable/persistent fault mapped to a specific part → replace it (a DIMM, an accelerator, a PSU, a cable) and re-validate (Phase 01 commissioning), then return to service. Replace the FRU, not the whole node — that's the serviceability payoff of good diagnosis.

This decision tree is the bridge to Phase 10 (systematic RCA) and Phase 12 (runbooks). The whole point of RAS software is to make the right one of these choices automatically or with minimal human effort, at fleet scale.

Lab Walkthrough Guidance

Order: Lab 01 (firmware) → Lab 02 (RAS). Firmware is the "act safely" half; RAS is the "diagnose" half — together they're the JD's firmware/crash-dump/diagnostics line.

Lab 01 (firmware orchestrator):

1. python3 solution.py — watch the happy path, a bad-signature reject, a failed-activation rollback (A/B bank restored), and an anti-rollback (downgrade) block.
2. Read the state machine: verify-before → stage-to-inactive → activate → verify-after → commit/ rollback.
3. Extensions: drive a real Redfish UpdateService + poll the Task; add canary + drain-first.

Lab 02 (RAS parser):

1. python3 solution.py — parse a mixed AER/MCE/SEL stream into a classified report.
2. See the predictive flag on a rising correctable-ECC rate and the action per record.
3. Extensions: parse real dmesg/mcelog/ras-mc-ctl; correlate with Phase 07 telemetry.

Phase capstone question: "A node in production reboots randomly under load. Walk me from the symptom to a FRU-level root cause and the fix — remotely." (Answer: read the SEL + BERT for the last-crash cause; pull AER/MCE via the parser; if MCE points at a DIMM with rising correctable ECC under thermal load → predictive DIMM fault; cordon+drain, schedule DIMM replacement, re-validate, return to service — all OOB, no site visit.)

Success Criteria

You're done with this phase when — without notes:

• You can list a node's firmware components and explain the baseline/EBOM (Ch. 1)
• You can explain why firmware updates are dangerous and why the BMC is the riskiest target (Ch. 1)
• You can describe the Redfish UpdateService/Task update flow (Ch. 2)
• You can explain A/B banks and the verify→stage→activate→verify→rollback state machine (Ch. 3)
• You can explain signing and anti-rollback and the tension it creates (Ch. 4)
• You can design a safe fleet rollout (canary, drain-first, domain-aware, rollback) (Ch. 5)
• You can decode PCIe AER (correctable vs uncorrectable) and CPU MCE to a FRU (Ch. 6–7)
• You can apply the log/monitor/cordon/RMA decision tree to a diagnosis (Ch. 8)

Interview Q&A

Q1: How do you safely update firmware across a fleet without bricking nodes? A: Per node, a verify-first, reversible state machine on A/B banks: download → verify (checksum + signature + version/compat + anti-rollback) before touching the device → flash the inactive bank (active firmware keeps running) → activate (switch pointer/reboot) → verify after (new version reported, sensors/links healthy — Phase 04) → commit, or roll back to the previous bank on any failure. Because flashing is slow it's driven through the Redfish UpdateService as a Task I poll. Across the fleet: drain the node first (Phase 06) for reboots, respect firmware dependencies (e.g., BMC before BIOS), and roll out progressively — 1 node → 1% → 10% → rest — with automatic halt-and-rollback if health/SLOs regress, never updating a whole failure domain at once. The mindset is that a firmware rollout is a deployment with the blast radius of physical hardware, so I apply canary/rollback/observability discipline with extra caution because I can't rollout undo a bricked BMC.

Q2: What are A/B firmware banks and why do they matter? A: A component with A/B banks has two firmware slots — one active (running), one inactive. I flash the inactive bank and only switch the active pointer to it after the new firmware validates; if it fails to boot or fails health checks, I switch back to the known-good bank. The point is that a bad flash never destroys my working firmware, so recovery is a pointer switch + reboot rather than a brick + truck roll. It's the single most important firmware-safety mechanism, especially for the BMC, because bricking the BMC would cost me the OOB lifeline I'd use to recover the node.

Q3: Signed firmware and anti-rollback — what's the tension and how do you resolve it? A: Firmware is signed by the vendor and verified before flashing, so tampered/unsigned images are rejected — and it ties into the secure-boot chain of trust (Phase 09). Anti-rollback prevents downgrading past a version that fixed a security hole, so an attacker can't "update" you to a known-exploitable old version; components enforce a monotonic minimum version. The tension is that anti-rollback could trap you on a bad new release. The resolution is the A/B bank — you roll back to the other good bank (the prior version you were already running), not a downgrade to an old vulnerable image — plus a vendor-authenticated recovery path for genuine emergencies. So you get both security and recoverability.

Q4: What is PCIe AER, and what do you do about correctable vs uncorrectable errors? A: AER (Advanced Error Reporting) is PCIe's mechanism for reporting fabric errors per device (by BDF). Correctable errors (Bad TLP, Replay Timeout, Receiver Error) were recovered by the link layer — one is noise, but a high rate on one device signals a marginal lane/cable/connector, a pre-failure indicator I'd flag for proactive service (and it correlates with a degraded link from Phase 04). Uncorrectable errors are worse: non-fatal (e.g., completion timeout) means the transaction failed but the link survives and the driver may recover; fatal (malformed TLP, link errors) means the link is unreliable and usually forces a reset or impacts the node. The BDF tells me which device, so I map it to a FRU and decide monitor vs cordon vs RMA.

Q5: A node logs a flood of correctable ECC memory errors. Is that a problem? A: A single correctable ECC error is benign — the ECC fixed it. But a flood, or a rising rate on one DIMM, is a strong predictive-failure signal: that DIMM is degrading and will likely throw an uncorrectable (fatal) error soon. So I don't ignore it and I don't panic-RMA on one event — I decode the MCE to identify the specific DIMM (via the bank + address), flag it for proactive replacement at the next maintenance window, and consider page-offlining the affected pages to reduce exposure. That's the serviceability win: catch it predictively, replace one DIMM in a window, and avoid an unplanned crash. It's exactly the "rate, not raw count" lesson from Phase 07 alerting applied to hardware.

Q6 (Staff): A fleet of nodes started crashing after a firmware rollout. What do you do, and how do you prevent it next time? A: First, stop the bleeding: halt the rollout immediately and roll back the affected nodes to the previous A/B bank (the reason A/B + progressive rollout exist). Quantify blast radius via the fleet firmware-version metric (Phase 07) and correlate the crashes (RAS parser: common MCE/AER/ SEL signature? same component/firmware version?) to confirm causation, not coincidence. Drain and recover affected nodes, validate them back to baseline, and communicate to stakeholders. Then the prevention/RCA (Phase 10/12): the rollout should have been caught at the 1% canary with automated health/SLO gates — so I'd tighten the canary criteria and bake time, add the specific failure signature to the pre-promotion checks, expand the conformance/HIL test matrix (Phase 11) to cover the condition that slipped through (likely a hardware/firmware combo we didn't test), and feed the defect back to the vendor/ODM (Phase 12). The Staff lesson is that the process (canary, gates, rollback, version observability) is what bounds the damage of an inevitable bad firmware — and a blameless postmortem that hardens the process is the deliverable, not just the fix.

References

• DMTF Redfish — UpdateService, SimpleUpdate, multipart push, Tasks — https://www.dmtf.org/standards/redfish
• UEFI/PI specs & firmware update (capsule update), A/B / fault-tolerant update concepts — https://uefi.org/specifications
• NIST SP 800-147 / 800-193 — Platform Firmware Resiliency (signing, recovery, anti-rollback) — https://csrc.nist.gov/
• PCIe AER — PCI Express Base Spec; Linux PCIe AER docs — https://docs.kernel.org/PCI/pcieaer-howto.html
• Linux EDAC / mcelog / rasdaemon (ras-mc-ctl) — memory/MCE error reporting — https://github.com/mchehab/rasdaemon
• ACPI APEI (HEST/BERT/ERST) — ACPI spec; Linux APEI docs
• Linux kdump/crash/drgn — kernel crash dump capture and analysis
• NVIDIA Xid errors (accelerator crash codes, as a canonical example) — https://docs.nvidia.com/deploy/xid-errors/
• Cross-track: Security Engineer — hardware/firmware device security

🛸 Hitchhiker's Guide — Phase 08: Firmware, Crash Dumps & RAS

Read this if: "firmware updates and crash-dump parsing" sounds scary. It should — this is where you can brick a node or save a fleet. Compressed map of safe updates and reading hardware pain.

0. The 30-second mental model

Two jobs: update firmware without bricking (A/B banks + verify-before-and-after + rollback, canaried across the fleet) and read RAS (PCIe AER, CPU MCE, SEL, accelerator codes) to a FRU-level diagnosis so you replace one DIMM, not a whole node. The mindset: a firmware rollout is a deployment with the blast radius of physical hardware — you can't rollout undo a bricked BMC.

1. The firmware zoo

BIOS/UEFI · BMC · NIC/DPU · accelerator · PSU · PCIe switch · CPLD · drives. Each is independently versioned; the correct set = the baseline / EBOM (Phase 12). The BMC is the riskiest target — brick it and you lose your OOB lifeline.

2. The safe update state machine

download → VERIFY (checksum+signature+version+anti-rollback)   [reject here, before touching HW]
→ stage to INACTIVE A/B bank   [active firmware still running]
→ activate (switch pointer / reboot)
→ VERIFY AFTER (new version + healthy: sensors/links)
→ commit  OR  ROLLBACK to previous bank

Verify before and after. A/B means a bad flash never destroys the good firmware. Idempotent + resumable (Phase 05).

3. Redfish update = a Task

POST UpdateService (SimpleUpdate / multipart) → 202 + Task → poll TaskState/PercentComplete. Activation usually needs a reboot. Don't block on a synchronous response.

4. Signing & anti-rollback

Signed images (verify vs trusted key) tie into secure boot (Phase 09). Anti-rollback blocks downgrading past a security fix. Tension (trapped on bad new version) → resolved by A/B (roll back to the other good bank, not an old vulnerable one) + vendor recovery path.

5. Fleet rollout

drain-first (Phase 06) for reboots · dependency order (BMC before BIOS) · canary 1→1%→10%→ rest with auto halt-and-rollback on SLO/health regress (Phase 07) · don't update a whole failure domain at once (Phase 01) · track firmware_version{component,version} as a metric.

6. RAS sources (reading hardware pain)

7. Decode to a FRU + an action

Parser job: decode each record → map to a FRU (which DIMM/device/PSU) → severity → predictive flag (rising correctable rate) → action. Decision tree:

single correctable / benign      → log-only
rising correctable rate          → monitor → proactive FRU replacement (avoid outage)
degrading-but-up                 → cordon + drain (Phase 06)
uncorrectable / persistent fault → RMA the FRU + re-validate (Phase 01) → return to service

8. Remote diagnosis toolkit (host is dead — all OOB)

SoL console · SEL + BERT (last-crash cause after reboot) · kdump vmcore (crash/drgn) · AER/MCE via the parser · Phase 07 telemetry correlation · Phase 01 failure-domain check.

9. The "done this before" tells

"A/B banks, right?" · "Did you verify after activation, not just before?" · "Is it canaried and drained first?" · "Correctable rate or a one-off?" · "Which FRU does that MCE bank point to?" · "Replace the DIMM, not the node."

10. How this phase pays off later

Safe updates → the Phase 06 cordon/drain flow + Phase 11 canary/rollback. RAS verdicts → Phase 10 RCA and Phase 12 runbooks/RMA. Firmware baseline → Phase 01 commissioning + Phase 12 EBOM. The fleet update system is system-design/03.

Lab 01 — Firmware Update Orchestrator (staged + rollback)

Phase: 08 — Firmware & RAS | Difficulty: ⭐⭐⭐⭐☆ | Time: 5–7 hours Language: Python 3 (stdlib) | Hardware: none

Concept primer: ../WARMUP.md Ch. 2–5, ../HITCHHIKERS-GUIDE.md §2–§5.

Run

python3 solution.py     # happy path, bad-signature reject, failed-activation rollback, anti-rollback

0. The mission

Make the most dangerous operation in the role — flashing firmware on a remote node — safe: verify before touching hardware, stage to the inactive A/B bank, activate, verify after, and roll back to the previous bank on failure. Plus signature verification and anti-rollback.

1. The four behaviors (read the state machine)

1. Happy path v10→v11: stage to inactive bank → activate → verify → commit (advance the anti-rollback floor).
2. Tampered image: HMAC signature check fails → rejected before any flash (the device is never touched).
3. Bad new firmware: passes signature but fails post-activation health → automatic rollback to the previous bank (v10 restored). This is why A/B banks exist and why you verify after activation, not just before.
4. Downgrade: a v9 image below the v10 floor → anti-rollback reject (no downgrading past a security fix).

2. Why each piece matters

• Verify before AND after: a flash can succeed yet the firmware misbehave; only post- activation health (sensors/links — Phase 04) confirms real success.
• A/B banks: a bad flash never destroys the working firmware → recovery is a pointer switch, not a truck roll. Critical for the BMC (don't lose your OOB lifeline).
• Anti-rollback + A/B together: you roll back to the other good bank, never to an old vulnerable version — resolving the security-vs-recoverability tension.

3. Extensions

1. Drive a real Redfish UpdateService (SimpleUpdate) against the Phase 03 mock + poll the Task to completion.
2. Add fleet canary + drain-first (Phases 06/11): 1 node → 1% → rest with auto halt-and- rollback on SLO regress.
3. Model power-loss mid-flash: crash between stage and activate, then prove resume leaves a bootable bank.
4. Real signing: replace HMAC with RSA/ECDSA signature verification.

4. What this lab proves

You can build firmware-update software that won't brick a fleet — verify, A/B, rollback, anti- rollback — and explain every safety decision. "How do you update firmware on 1,000 nodes safely" becomes a demonstration. This orchestrator + the Phase 06 drain flow + Phase 11 canary = the fleet firmware system (system-design/03).

Lab 02 — Crash-Dump & RAS Log Parser

Phase: 08 — Firmware & RAS | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–6 hours Language: Python 3 (stdlib) | Hardware: none

Concept primer: ../WARMUP.md Ch. 6–8, ../HITCHHIKERS-GUIDE.md §6–§8.

Run

python3 solution.py     # parse mixed AER/MCE/SEL/Xid logs -> FRU-level diagnoses + actions

0. The mission

Turn the hardware's cryptic error telemetry into a decision: parse PCIe AER, CPU MCE, BMC SEL, and accelerator (Xid-style) records, decode each to a meaning, map it to a FRU, assign severity, detect predictive patterns (rising correctable rate), and recommend an action — log-only / monitor / cordon / RMA the FRU. This is "crash dump parsing / remote diagnostics" from the JD.

1. The decoders to study

• AER: correctable = noise individually (log-only); uncorrectable fatal (MalformedTLP/ LinkError) → RMA, non-fatal → cordon. BDF identifies the device.
• MCE: the bank maps to a subsystem; bank 7 (memory controller) + address → a specific DIMM. Correctable ECC = log; uncorrectable = RMA.
• SEL/Xid: PSU input-lost → monitor; a fatal Xid (GPU off the bus) → RMA.

2. The key insight: rate, not count

predictive_escalation is the senior move: one correctable error is noise, but 60 corrected ECC errors on DIMM B1 is a degrading part → escalate to a scheduled replacement (RMA) before it throws an uncorrectable (a crash). Same lesson as Phase 07 alerting (alert on rates), applied to hardware — and it's how you turn unplanned outages into planned maintenance.

3. The output

A per-record diagnosis list (critical/warning surfaced), rate-based predictive escalations, and a summary verdict (how many FRUs to RMA / monitor / log). That's the difference between "RMA the whole node" and "replace DIMM B1 and the accelerator at the next window."

4. Extensions

1. Parse real dmesg (AER lines), mcelog/ras-mc-ctl --error-count, and ipmitool sel output.
2. Correlate with Phase 07 telemetry (did the DIMM errors rise with temperature? → thermal root cause).
3. Feed verdicts into the Phase 10 RCA workflow and auto-open an RMA ticket (Phase 12).
4. Add BERT (last-crash cause after reboot) parsing for diagnosing random reboots.

5. What this lab proves

You can do remote hardware diagnosis — the JD's most-repeated theme — turning a flood of RAS records into a precise FRU-level verdict and the right action. "A node crashes randomly in production — find the root cause remotely" becomes a methodical walk (SEL/BERT → AER/MCE decode → predictive flag → RMA the DIMM), no site visit required. Feeds Phase 10 and the capstone.

Phase 09 — Networking, Secure Boot & Multi-Tenancy

Difficulty: ⭐⭐⭐⭐☆ | Estimated Time: 2 weeks Roles supported: Rack Management SWE (Senior/Staff), Infra Security SWE, Platform SWE Hardware needed: none — simulators + real ip/tc commands (Linux/VM for the live extensions)

Why This Phase Exists

The JD: "Ensure secure boot, VLAN separation, and QoS enforcement for multi-tenant environments." When a rack hosts multiple tenants (or separates the management plane from tenant data — Phase 01 Ch. 6), isolation isn't a feature, it's a contract: tenant A must not see tenant B's traffic, the management network must be unreachable from tenants, no tenant can starve another of bandwidth, and the software/firmware that boots must be trusted. These are the security and fairness primitives of a shared rack.

Three pillars:

1. Network isolation — VLANs, network namespaces, the management-vs-data plane separation, and segmentation that keeps tenants and the control plane apart.
2. QoS — guaranteeing/limiting bandwidth so one tenant can't degrade another (and the management plane stays responsive).
3. Secure/measured boot & attestation — a chain of trust from firmware to OS so you know a node booted trusted software, with remote attestation (TPM) to prove it — the foundation for sovereign/regulated deployments (Phase 06 RKE2).

You'll build a tenant-isolation + QoS model (with the real ip/tc commands) and a secure/measured-boot + attestation simulator (TPM PCRs, signature chain).

Concepts

Network isolation

• The three planes again (Phase 01 Ch. 6): management (OOB) vs data (tenant) vs scale-up; why management is physically/logically isolated
• VLANs (802.1Q): tags, trunk vs access ports, the broadcast-domain boundary; VLAN-per-tenant
• Linux network namespaces, veth, bridges, VLAN sub-interfaces — how isolation is implemented on a host
• Segmentation & microsegmentation; firewalls/ACLs; the BMC/management VLAN as a hard boundary
• VXLAN/overlays (briefly) for cross-rack tenant networks; SR-IOV/VFs for accelerator NICs in tenant pods
• mTLS, network policy (Phase 06), secrets management, least privilege on the control plane

QoS (Quality of Service)

• Why QoS: the noisy-neighbor problem; protecting the management plane and paying tenants
• Linux tc (traffic control): qdiscs (HTB, fq_codel), classes, filters; rate limiting, shaping, prioritization
• DSCP/marking; bandwidth guarantees vs limits; per-tenant fairness
• QoS for storage/PCIe (briefly) and for the management network

Secure & measured boot, attestation

• UEFI Secure Boot: signed shim → bootloader → kernel; the key hierarchy (PK/KEK/db/dbx); what it does and doesn't protect
• Measured boot: each stage measures (hashes) the next into a TPM PCR before executing it; the difference from Secure Boot (measure vs enforce)
• TPM: PCRs (extend-only hash registers), the root of trust, sealing, quotes
• Remote attestation: a node proves its boot state (a signed PCR quote) to a verifier, which checks it against known-good values before trusting it (e.g., before handing it tenant work/keys)
• The chain of trust from hardware root → firmware → OS; anti-rollback (Phase 08) as part of it
• Sovereign/air-gapped/regulated deployment (Phase 06 RKE2): why attestation matters in the Kingdom's data centers

Multi-tenancy synthesis

• Soft vs hard isolation; defense in depth (VLAN + namespace + RBAC + NetworkPolicy + attestation)
• Mapping tenant boundaries across network, compute (Phase 06), and accelerators (Phase 04 MIG/partitioning)
• The threat model: a malicious tenant, a compromised node, an exposed BMC

Labs

Lab 01 — VLAN Separation & QoS Tenant Isolation

Lab 02 — Secure/Measured Boot + Attestation

Deliverables Checklist

• You can explain the three planes and why management is a hard isolation boundary
• You can explain VLANs (802.1Q) and how Linux namespaces/bridges implement isolation
• Policy engine enforces no-cross-tenant and no-tenant→management invariants
• You can explain the noisy-neighbor problem and how tc/HTB QoS solves it
• You can explain Secure Boot (enforce) vs measured boot (measure) and the key hierarchy
• You can explain what a TPM PCR is and how measured boot extends it
• Attestation verifier admits only known-good boot states; you can explain why that gates trust
• You can describe defense-in-depth tenant isolation across network/compute/accelerator