🛸 Hitchhiker's Guide — Phase 01: Rack-Scale AI Systems

Read this if: you can write software but a data-center rack is still a black box labeled "servers go here." This is the practitioner's compressed tour — the numbers, mental models, and war stories the WARMUP derives from first principles. Skim it first, then read the WARMUP slowly.

0. The 30-second mental model

An AI rack is one machine the size of a wardrobe that draws as much power as a small apartment building (60–130+ kW), is plumbed with liquid coolant, and contains two networks worth of computers: the hosts (on the data network) and a shadow fleet of BMCs/PDUs/CDU controllers (on a separate management network). Your software lives on that second network and treats the rack as a managed object with a power budget, a cooling budget, a failure-domain graph, and a lifecycle. If you remember one sentence: you manage the rack out-of-band, as a system, not the servers in-band, one at a time.

1. The org chart of an AI rack

RACK (≈42U / 48 OU, ~1.2–1.4 t, 60–130+ kW, liquid-cooled)
├── Compute trays ×N            ← host CPU(s) + 4–8 accelerators (HBM) + NVMe + NICs/DPUs
│     └── BMC per node          ← always-on ARM SoC (OpenBMC); the OOB brain
├── NVSwitch / scale-up trays   ← wire accelerators into one NVLink-class fabric
├── ToR data switch             ← scale-out: 400/800 GbE or InfiniBand to the cluster
├── Management switch           ← OOB only: BMCs, PDUs, CDU  (separate, isolated)
├── Power shelf + busbar / PDUs ← 3-phase AC → DC; N+1 / 2N; metered + switched
└── CDU + manifolds + cold plates ← pumps coolant, exchanges heat w/ facility water

Two ratios to tattoo on your arm:

• Power density jump: enterprise 5–10 kW → AI liquid 60–130+ kW ≈ ~10×. That jump is the whole reason this role exists.
• 1 kW = 3,412 BTU/hr. A 120 kW rack = ~409,000 BTU/hr of heat to remove. Air can't; hence liquid.

2. OOB vs in-band in 4 bullets

• Every node is two computers: the host (in-band, data NIC) and the BMC (out-of-band, mgmt NIC, standby power).
• The BMC works when the host is off, hung, or panicked — which is exactly when you need it.
• Redfish/IPMI/SNMP (Phase 03) ride the management network; SSH/agents ride the data network.
• Rule: assume the host is down and reach everything through OOB. Your control plane's reliability depends on it.

3. The power math you'll actually use

War story shape: "the rack tripped a breaker at 2 a.m." almost always traces to budgeting against nameplate or the sum of redundant feeds instead of the derated single feed. The fix is a budgeting tool that bakes in 0.8 and N+1 (Lab 02) — and blocks a placement that violates it.

4. Cooling, quantified

• Heat removed by liquid = flow (L/min) × fluid heat capacity × ΔT (return − supply). Watch ΔT rising at constant load → fouling/flow problem → pre-failure signal.
• CDU = pump + heat exchanger + sensors. It isolates the clean rack loop from facility water and reports flow/ΔT/pressure/pump-speed/leak.
• Leak detection is safety-critical: it can trigger an automated drain/shutdown. Treat it like a smoke alarm in code: highest priority, fail-safe.
• PUE = facility power ÷ IT power. Liquid AI halls aim ~1.1; the metric the business cares about is increasingly performance/watt and tokens/joule.

5. The three networks (don't conflate them)

Putting BMCs on the tenant network is the classic catastrophic mistake: a BMC owns power and firmware: full physical control. Phase 09 isolates these for real with VLANs.

6. Failure domains: the first question in every incident

When alerts flood in, ask "what's the shared resource?" before "what's wrong with node X?":

• 6 nodes down at once, same PDU → power domain, not 6 separate faults.
• Whole rack thermal-throttling → cooling domain (CDU/pump/loop).
• Rack unreachable in-band but BMCs answer → ToR/data-network domain, hosts are fine.
• Rack unreachable OOB → management-switch domain; you may be blind, escalate to hands-on.

Encoding the powers/cools/connects edges (Lab 01) turns a 30-alarm storm into one root cause. This is the bridge to Phase 07 (alert correlation) and Phase 10 (RCA).

7. The lifecycle, and where you write code

design/NPI → ODM build (Pegatron) → ship/install → COMMISSION/burn-in → PROVISION → OPERATE → MAINTAIN/RMA → decommission
   (P12)         (EBOM, P12)                         (P01/03/08 here!)    (P05/06)   (P07/09)  (P08/10)

The under-appreciated one is commissioning: discover every endpoint on the mgmt network, capture inventory, reconcile against the EBOM, set the firmware baseline, burn-in, validate power/thermal, then mark "ready." At fleet scale you commission racks continuously, so it must be an automated pipeline — a great Staff-level thing to own.

8. Vocabulary that makes you sound like you've been there

• FRU (Field Replaceable Unit) — the granularity of "swap this part."
• U / OU — rack unit (1.75") / OpenRack unit (taller, wider 21" space).
• Busbar — the vertical DC rail nodes blind-mate onto in OCP racks.
• DLC / cold plate / RDHx / immersion — direct liquid cooling variants.
• DPU/SmartNIC — a NIC with its own CPU/OS; itself a managed, firmwared device.
• DCIM — Data Center Infrastructure Management; the facility-level system your rack software feeds and integrates with.
• L6/L10/L11 — manufacturing integration levels (board / node / rack).
• NPI — New Product Introduction; the pre-production phase where interfaces and the EBOM get defined with the ODM.
• Blast radius / failure domain — say these and people know you've held a pager.

9. The mistakes that mark a beginner (avoid in interviews)

1. "Just SSH to the node" — for a node that won't boot, there's no SSH; it's a Redfish/SoL problem.
2. Budgeting power to nameplate or to the sum of redundant feeds.
3. Treating cooling as "the facility's problem" when CDU telemetry is part of rack health your code acts on.
4. One flat inventory table with no relationship edges → you can't compute blast radius or reconcile.
5. Conflating identity (serial) with location (rack+U) → stale location bugs after every RMA.
6. Putting management traffic on the data network → reliability and security disaster.

10. How this phase pays off later

• The inventory model (Lab 01) is literally reused/expanded in the Phase 13 capstone.
• The declared-vs-discovered idea becomes the Phase 06 operator reconcile loop.
• The failure-domain graph powers Phase 07 alert correlation and Phase 10 RCA.
• The OOB-first mindset is the premise of Phases 03 (protocols), 04 (hardware), and 08 (firmware/diag).
• The power/thermal budgets reappear as live telemetry thresholds in Phase 07.

You're ready for the WARMUP. Read it slowly — the numbers in it are the ones interviewers probe.