🛸 Hitchhiker's Guide — Phase 02: Production Systems Programming

Read this if: you write working code but aren't sure what separates a script from infrastructure software. This is the compressed field guide — the patterns, the footguns, and the one-liners that show up in code review and on the pager.

0. The 30-second mental model

Production infra code is mostly the failure paths. The happy path is the easy 20%. What makes you senior is the other 80%: what happens when the BMC hangs, the sensor lies, the disk fills, the config is wrong, SIGTERM arrives mid-write, and the input is hostile. The four habits that get you there: RAII/cleanup-on-every-path, a seam at the hardware boundary so you can test without hardware, structured logs + exit codes so on-call can reason about it, and never trust input. If you remember one sentence: write the code you'd want to debug at 3 a.m. on someone else's machine.

1. The language decision, in one table

Staff move: choose by layer and design the seams between them (a stable C ABI, a JSON contract). "All one language" is a junior answer.

2. RAII in four lines

• Resource lifetime = object scope. Destructor frees it on every exit (return, throw, break).
• unique_ptr is your default owner; shared_ptr only for genuine shared ownership (watch cycles → weak_ptr).
• lock_guard/scoped_lock for mutexes — never manual lock/unlock (early return leaks the lock → deadlock).
• Rule of zero: hold RAII members, write no destructor/copy/move. If you manage a raw resource, rule of five (usually: delete copy, define move).

3. The error taxonomy → response map

Cardinal sin: except: pass / ignoring a return code. Every swallowed error is a future mystery.

4. The hardware-boundary seam (the testability superpower)

[ daemon logic ]  ──uses──▶  interface (SensorSource / RedfishClient / FirmwareTarget)
                                 ├── RealBmcSource     (prod)
                                 └── FakeSource        (tests: hang, garbage, flaky)

Inject the dependency and the clock. Now you test retry, staleness, and timeouts in milliseconds with no hardware. This single pattern is reused in Phases 03, 04, 08, 11.

5. subprocess / shell injection — the 10-second fix

# BAD — shell parses the string; node="x; rm -rf /" runs
subprocess.run(f"ipmitool -H {node} power status", shell=True)
# GOOD — argv list, shell=False (default); node can't be reinterpreted
subprocess.run(["ipmitool", "-H", node, "power", "status"], timeout=10, check=True)

Same rule in C++: never system("..."); use exec/posix_spawn with an argv array.

6. The daemon checklist

• SIGTERM/SIGINT → set atomic stop flag (nothing heavy in the handler) → main loop drains.
• Graceful shutdown: stop new work, finish/cancel in-flight cleanly, never leave a transaction half-done.
• systemd: Restart=on-failure, Type=notify (real readiness via sd_notify), WatchdogSec (free liveness), MemoryMax/TasksMax, sandboxing (ProtectSystem, non-root User=).
• Log to stdout/stderr (journald captures it) — don't write your own log rotation.

7. Concurrency cheat sheet

• I/O-bound (poll 100 BMCs) → asyncio or bounded thread pool. Bound it (semaphore/pool) — don't open 10,000 sockets.
• CPU-bound → processes (the GIL blocks thread parallelism in CPython).
• C++ threads → real parallelism, tiny critical sections under lock_guard, prove with TSan.
• Never hold a lock across I/O or a callback.

8. The CI quality gates (per language)

If it's not in CI, it doesn't hold. Standards you can't enforce are wishes (Phase 12).

9. Security one-liners that matter here

• Validate at the boundary with allow-lists, not deny-lists.
• C++: no strcpy/sprintf; watch integer overflow in size = a * b before malloc.
• Secrets: never in code, logs, or argv (argv is world-readable via /proc/<pid>/cmdline).
• Least privilege: dedicated user, scoped capabilities, read-only FS where possible.
• Pin + checksum + minimize + scan dependencies — every dep is attack surface.

10. The "you've done this before" tells

• "Where's the seam?" (testability) · "What happens on SIGTERM?" (lifecycle) · "Is that arg list or shell-interpolated?" (security) · "Is the critical section under a guard?" (RAII/locks) · "Transient or permanent?" (retry policy) · "Did we bound the concurrency?" (fan-out safety).

11. How this phase pays off later

• The seam + fake pattern powers every later lab's tests (esp. Phases 03, 08, 11).
• The daemon mechanics are how the Phase 07 exporter and Phase 13 capstone agent run.
• The retry/backoff and error taxonomy become the Phase 10 reliability patterns.
• The security discipline is the per-commit half of Phase 09's system security.