Interview Bank 01 — Foundations & Systems Programming
Phases 01–02. Rack anatomy, power/cooling, the management plane, and production Python/C++/shell. Full-length model answers live in the phase WARMUPs' Interview Q&A; this bank is the high-yield set plus quick-fire drills.
Q1. Walk me through the components of an AI rack and how it's powered and cooled.
Compute trays (host CPUs + 4–8 accelerators with HBM + NICs/DPUs + NVMe), scale-up fabric (NVSwitch-class) wiring accelerators into one coherent fabric, a top-of-rack data switch, a separate management switch, power (3-phase AC → power shelf/PSUs → busbar, N+1/2N), and — because these draw 60–130+ kW (~10× a legacy rack) — liquid cooling via a CDU pumping coolant through cold plates and exchanging heat with facility water. Every node also has a BMC on the management network. Framing: it isn't "N servers," it's a system with rack-level shared resources (power, cooling, fabric) managed as first-class objects. (WARMUP P01 Ch. 2; Q1.)
Q2. Out-of-band vs in-band management — and why does it matter to your software?
In-band goes through the host OS/data NIC (only works when the host is healthy); out-of-band goes through the BMC on a separate management network with standby power (works when the host is off, hung, or panicked). It matters because the moments you most need to manage a node are when it's unhealthy, so power control, firmware, console, and diagnostics ride OOB. My control plane assumes the host may be down and reaches everything through the management network. (P01 Ch. 3; Q2.)
Q3. Two 30A 415V feeds, N+1 — how much load can you safely place, and why not 100%?
Per feed ≈ √3·415·30 ≈ 21.6 kW, but the 80% continuous-load rule caps a breaker at 80% → ~17.3 kW. With N+1 the survivor must carry the whole load after one feed dies, so the safe load is bounded by a single derated feed (~17.3 kW), not the sum. Budget to the sum and one failure overloads the survivor and cascades; budget to nameplate and the breaker trips. Measured draw is usually well below nameplate, so I oversubscribe carefully with telemetry + power capping, but the guaranteed safe number is the derated single feed. (P01 Ch. 4; Q3.)
Q4. Why are AI racks liquid-cooled, and what does that add to your job?
Power density: air can't remove heat past ~40 kW, and AI racks dump 60–130+ kW (1 kW = 3,412 BTU/hr). Liquid carries thousands× more heat/volume, so coolant runs through cold plates with a CDU. For software it adds a managed subsystem: I monitor CDU flow, supply/return ΔT, pressure, pump health, and leak detection — a rising ΔT at constant load is a pre-failure signal and a leak is a safety-critical alarm that may trigger emergency action. (P01 Ch. 5; Q4.)
Q5. What's a failure domain, and how does your software use it?
A set of components that fail together because they share a resource (a PDU, a CDU, a ToR); the blast radius is everything in it. Software uses it three ways: the inventory model encodes powers/cools/ connects edges to compute blast radius; placement spreads replicas across domains; and during incidents the first question is "single node or shared resource?" — correlating alerts by domain turns an alarm storm into one root cause. (P01 Ch. 7; Q5.)
Q6. What is RAII and why does it matter for a rack daemon?
RAII ties a resource's lifetime to an object's scope — acquire in the constructor, release in the
destructor — so fds, BMC sessions, and mutex locks are freed on every exit path including
exceptions, with no manual cleanup. For a daemon it's the difference between leaking handles/
deadlocking on error paths and being correct by construction (a lock_guard releases even if the
critical section throws). Rule of zero: hold RAII members, write no destructor/copy/move. (P02 Ch. 3;
Q1.)
Q7. subprocess.run(f"ipmitool -H {host} ...", shell=True) — what's wrong, fix it.
Shell injection: host="x; rm -rf /" runs arbitrary commands with the daemon's (high) privileges. Fix:
pass an argv list with shell=False (default) — ["ipmitool","-H",host,...] — so host can never
be reparsed as shell syntax; add a timeout, validate host, check the return code. Same rule in C++:
never system() a built string; use exec/posix_spawn. (P02 Ch. 8; Q2.)
Q8. How do you make rack code testable when the real dependency is a BMC you don't have in CI?
Put a seam at the hardware boundary — an interface (SensorSource/DeviceDriver) injected via DI —
so production wires the real client and tests wire a fake that scripts failures/timeouts/garbage
deterministically; inject the clock too so timing tests are instant. That gives a wide base of fast
unit tests; above it, integration tests run against emulators (Redfish mock, ipmi_sim), and a thin top
runs on real HIL. Testability is a design decision (the seam), not a bolt-on. (P02 Ch. 9; Q3.)
Q9. Threads, async, or processes — and where does the GIL bite?
I/O-bound concurrency (poll 100 BMCs) → asyncio or a bounded thread pool; CPU-bound → processes (CPython's GIL blocks thread parallelism for bytecode); fault isolation → processes. The GIL is released during I/O, so threads help I/O-bound work but not CPU-bound. In C++ threads give real parallelism but I own synchronization (tiny critical sections under a guard, prove with TSan). Whatever the model, bound the concurrency so I don't exhaust fds or flood the management network. (P02 Ch. 6; Q5.)
Quick-fire drills (know cold)
- 1 kW = 3,412 BTU/hr; AI rack ≈ 60–130+ kW.
- 3-phase usable ≈ √3·V·I·PF; then ×0.8 (breaker derate).
- N+1 safe load = one derated feed, not the sum.
- HBM : NVLink : PCIe ≈ 50 : 13 : 1 (cross-track GPU P01).
- BMC = always-on ARM SoC (OpenBMC) on standby power; works when the host is dead.
- RAII / rule of zero;
lock_guardnot manual lock/unlock; ASan/UBSan/TSan in CI. - No
shell=True; validate input with allow-lists; secrets never in argv (/procleaks). - The seam = the reason you can unit-test without hardware.