Warmup Guide — Production Systems Programming

Zero-to-expert primer for Phase 02. You can already write code; this guide makes it production infrastructure code — robust under failure, secure, testable without hardware, and maintainable by a team you've never met. Every later lab is held to this bar.

Table of Contents

Chapter 1: The Language Matrix — Why Three (Plus One)

The JD lists Python, C++, and shell as required and Node.js as a plus. This is not a wish list; it reflects the layers of a rack-management stack and where each tool earns its place.

  • C++ lives at the bottom, close to hardware: device libraries, binary protocol parsers, performance-critical daemons, anything linking a vendor C SDK. You choose C++ when you need deterministic resource control, low overhead, or to call into a hardware library. The cost is that you own memory and lifetime; the discipline (RAII, Chapter 3) is what makes that safe.
  • Python is the broad middle: clients (Redfish/IPMI/SNMP, Phase 03), exporters (Phase 07), operators (Phase 06), CLIs, simulators, parsers (Phase 08), and orchestration glue. You choose Python for velocity, libraries, and readability — and accept that it's slower and needs typing discipline (mypy) to stay maintainable at scale.
  • Shell is the glue and bootstrap: PXE/kickstart hooks (Phase 05), CI steps (Phase 11), and "run this on every node." Powerful for orchestrating other programs, dangerous for logic. The skill is knowing the line where shell stops and Python begins (Chapter 10).
  • Node.js is optional tooling/services: a status web UI, a small telemetry API, a dashboard backend. TypeScript brings types; the event loop suits I/O-bound services. You reach for it when the team/ecosystem favors JS or you're serving a web surface.

The Staff-level framing: choosing the language is an engineering decision with consequences for the team, not a preference. "We wrote the hot path in C++ behind a stable C ABI and the control logic in typed Python, with shell only for bootstrap" is the kind of layered answer that signals seniority.

Misconception: "use one language for everything." A rack stack that's all-Python can't meet a hard latency/resource budget at the device layer; an all-C++ stack moves at a fraction of the velocity for orchestration code. Polyglot-by-layer is the norm, and the seams between languages (a stable ABI, a JSON contract) are where you spend design care.

Chapter 2: Error Handling as a Design Decision

Production infra code is mostly error handling. The naive happy path is 20% of the work; the other 80% is what happens when the BMC times out, the sensor returns 0xFFFF, the disk is full, or the config is malformed.

The error taxonomy (decide the response per class):

  • Transient (network blip, BMC busy, lock contention): retry with backoff (Chapter 6, Phase 10). Do not crash.
  • Permanent (auth failure, 404, malformed firmware image): fail fast and loud; retrying just wastes time and hammers the target.
  • Programmer error (precondition violated, null where impossible): assert/abort in dev; in prod, fail the operation, never silently continue with corrupt state.
  • Degraded (one sensor of many failed): continue with reduced function, mark the bad part stale, surface a warning. A rack manager that dies because one of 40 sensors is flaky is useless.

Fail-fast vs degrade-gracefully is contextual: a startup misconfiguration should fail fast (don't run misconfigured); a runtime partial failure should usually degrade (keep managing the healthy 95%). The art is knowing which is which.

Mechanism by language:

  • C++: exceptions for truly exceptional/unwindable errors; error-returning types (std::optional, std::expected in C++23, or a Result-like type) for expected failures like "sensor read failed." Don't use exceptions for control flow; don't ignore return codes.
  • Python: exceptions are idiomatic; catch specific exceptions, never bare except:; wrap at boundaries; use try/finally or context managers for cleanup.

The cardinal sin: swallowing errors (except: pass, ignoring a return code). Every swallowed error is a future 3 a.m. mystery. Log it with context or propagate it; never drop it.

Chapter 3: RAII and Resource Lifetime (the C++ core idea)

RAII = Resource Acquisition Is Initialization. The single most important idea in C++ infra code, and a frequent interview question. The principle: tie a resource's lifetime to an object's scope. Acquire in the constructor, release in the destructor. When the object goes out of scope — for any reason, including an exception — the destructor runs and the resource is freed. No finally, no manual cleanup, no leak path.

Resources this governs: heap memory, file descriptors, sockets, mutex locks, device handles, a BMC session, a firmware-update transaction.

// Without RAII: a leak waits at every early return / throw.
int fd = open(path, O_RDONLY);
if (parse(fd) < 0) return -1;     // <-- fd leaked
close(fd);

// With RAII: the guard's destructor closes fd on ANY exit path.
class FdGuard {
    int fd_;
public:
    explicit FdGuard(int fd) : fd_(fd) {}
    ~FdGuard() { if (fd_ >= 0) ::close(fd_); }
    int get() const { return fd_; }
    FdGuard(const FdGuard&) = delete;            // non-copyable: one owner
    FdGuard& operator=(const FdGuard&) = delete;
};
FdGuard f(open(path, O_RDONLY));
if (parse(f.get()) < 0) return -1;   // fd still closed — destructor runs

Smart pointers are RAII for memory: std::unique_ptr (single owner, zero overhead — your default), std::shared_ptr (reference-counted shared ownership — use sparingly; cycles leak), std::weak_ptr (breaks cycles). The rule: never new/delete raw in modern C++. Use std::make_unique/make_shared.

std::lock_guard/std::scoped_lock are RAII for mutexes — the lock releases when the guard leaves scope, even on exception. This is why your daemon's shared state (Lab 01) is protected with a guard, not manual lock/unlock (which leaks the lock on an early return → deadlock).

The rule of zero/three/five: if your class owns no raw resources (it holds unique_ptrs, strings, vectors — all RAII themselves), write no destructor/copy/move (rule of zero) — the compiler-generated ones are correct. If you do manage a raw resource directly, you must handle destructor + copy + move + their assignments (rule of five), usually by deleting copy and defining move. Aim for the rule of zero: let RAII members do the work.

Chapter 4: Modern C++ for Infra Code

Beyond RAII, the modern-C++ toolkit that keeps systems code safe and readable:

  • const-correctness: mark everything you don't mutate const. It documents intent, enables optimization, and the compiler catches accidental mutation. A const-correct sensor reader can't be accidentally made to mutate the device.
  • References over pointers where null is not a valid state; pointers (or optional) only where absence is meaningful.
  • std::optional<T> for "maybe a value" (a sensor that may not have a reading) and an error type (expected/a Result) for "value or why it failed."
  • std::string_view/std::span for non-owning views — pass without copying, but never outlive the owner.
  • Value semantics & move: prefer values + move over manual pointer juggling; std::move transfers ownership cheaply.
  • enum class for type-safe states (power state, health) — no implicit int conversions.
  • Build & quality gates: CMake (or a clean Makefile), -Wall -Wextra -Werror, clang-tidy (catches bugs), clang-format (style — never argue about it in review), and the sanitizers:
    • ASan (AddressSanitizer): use-after-free, buffer overflow, leaks.
    • UBSan: undefined behavior (signed overflow, bad shifts, null deref).
    • TSan: data races. Run your daemon's tests under TSan to prove your locking is correct. Sanitizers in CI catch the bugs that otherwise become irreproducible production crashes — exactly the JD's "debugging complex issues."

Misconception: "modern C++ is just C with classes." Modern C++ (17/20) with RAII, smart pointers, optional/expected, and sanitizers is a memory-safe-by-discipline language. The bugs that haunt infra code (UAF, leaks, races) are preventable with these tools, and a Staff engineer enforces them as standards (Phase 12).

Chapter 5: Production Python

Python's velocity is a trap without discipline; here's the discipline that keeps a 50k-line rack toolchain maintainable.

  • Type hints + mypy: annotate function signatures and dataclasses; run mypy --strict in CI. Types are documentation that can't go stale and catch a whole class of bugs (passing a str where an int belongs) before runtime. For infra code consumed by a distributed team, this is non-negotiable.
  • Packaging: a pyproject.toml, a real package (not a pile of scripts), pinned dependencies, a virtualenv. pip install -e . for dev. Reproducible installs are the difference between "works on my machine" and a deployable tool.
  • Structured logging: not print(). Use logging with a structured (JSON) formatter and key/value context (node id, request id). Logs are queried by machines (Phase 07); structure makes them searchable and correlatable.
  • Configuration precedence: a clear order — flags > environment > config file > built-in default — validated at startup. Surprises in config precedence cause outages; make it explicit and tested (Lab 02).
  • Retries/backoff: a decorator that retries transient errors with exponential backoff + jitter and a cap; never an unbounded retry loop that DDoSes a struggling BMC.
  • subprocess safety: pass a list of args, never shell=True with an interpolated string (Chapter 8). Set timeouts. Check return codes.
  • Exit-code discipline: 0 = success, non-zero = specific failure classes. Tools are composed in scripts and CI; your exit code is an API.
  • Quality gates: ruff (lint), black (format), pytest (test), mypy (types) — all in CI.

The GIL, briefly: CPython's Global Interpreter Lock means threads don't give you CPU parallelism (only one runs Python bytecode at a time), but they do help with I/O-bound work (waiting on 100 BMCs). For CPU-bound work use processes; for high-concurrency I/O use asyncio. Knowing this prevents the classic "why didn't my threads speed up my CPU-bound loop" mistake (Chapter 6).

Chapter 6: Concurrency — Threads, Async, Processes

A rack manager is inherently concurrent: poll 100 nodes, each independently, while serving status queries and handling signals. Choose the model deliberately.

  • Threads: shared memory, cheap context, but you own synchronization (locks, races). In C++ great for true parallelism; in Python (GIL) only useful for I/O-bound concurrency. Use a thread pool with bounded size — never spawn one thread per node for 10,000 nodes.
  • Async (asyncio / coroutines): single-threaded cooperative concurrency, excellent for many concurrent I/O waits (the BMC poll fan-out) with low overhead and no locks for most state. The cost: "async all the way down" and no help for CPU-bound work.
  • Processes: isolation and true parallelism (bypass the GIL), at the cost of memory and IPC. Use for CPU-bound work or fault isolation.

The lock discipline (C++ Lab 01): shared state (latest readings) is guarded by a mutex; every access takes a lock_guard. Keep critical sections tiny (copy out under lock, process outside). Never hold a lock across I/O or a callback. Run under TSan to prove there are no races — data races are undefined behavior and the source of the worst, least-reproducible bugs.

Bounded concurrency: whatever the model, cap it (a semaphore, a pool size). Polling 10,000 BMCs at once will exhaust fds and overwhelm the management network. Bounded fan-out (e.g., 50 at a time) is both kinder to the hardware and more predictable.

Chapter 7: Daemons, Signals, and systemd

A daemon is a long-running process that must behave well for years. The mechanics:

  • Signals: SIGTERM (graceful stop — the one systemctl stop sends), SIGINT (Ctrl-C), SIGHUP (reload config), SIGKILL (uncatchable — last resort). Install handlers that set an atomic "stop" flag; the main loop checks it and drains (finish the in-flight poll, flush logs, release resources via RAII) before exiting. Never do heavy work in a signal handler (async-signal-safety) — just flip a flag.
  • Graceful shutdown: on SIGTERM, stop accepting new work, finish or cancel in-flight work cleanly, persist any needed state, exit 0. A daemon that corrupts a firmware-update transaction on shutdown is a disaster; design shutdown as carefully as startup.
  • systemd integration: a unit file with Restart=on-failure, resource limits (MemoryMax, TasksMax), Type=notify (the daemon calls sd_notify(READY=1) when actually ready — true readiness, not "the process started"), and WatchdogSec (the daemon pings; if it hangs, systemd restarts it — a free liveness probe).
  • journald: log to stdout/stderr with structure; systemd captures it into the journal, queryable with journalctl -u sensord. No log-file rotation code to write.
  • Readiness vs liveness: ready = can serve requests (mirrors Kubernetes readiness, Phase 06); alive = not hung. Expose both; the watchdog handles liveness, sd_notify handles readiness.

Why this matters: the JD's "operational support," "observability," and "deployed systems" all assume your software runs as a well-behaved daemon. A process that ignores SIGTERM, can't be restarted safely, or hangs without the watchdog noticing is an operational liability.

Chapter 8: Security in Infra Code

The JD says "secure" three times. Rack-management software runs with high privilege on the management network — a bug here is a fleet-wide risk. The essentials:

  • Shell-injection (the classic): never build a shell command by string-interpolating untrusted input. subprocess.run(f"ipmitool -H {host} ...", shell=True) lets host = "x; rm -rf /" run anything. Fix: pass an arg list and shell=False (the default) — ["ipmitool", "-H", host, ...] — so host can never be parsed as shell syntax. Same idea in C++: never system() a constructed string; use posix_spawn/exec with an argv array.
  • Input validation: validate everything from the network/config/user at the boundary — lengths, ranges, types, allow-lists (not deny-lists). A Redfish response, a firmware image header, a config value: all untrusted until validated.
  • Memory safety (C++): bounds-check, avoid strcpy/sprintf (use snprintf/std::string), beware integer overflow in size calculations (a classic heap-overflow precursor), and run ASan/UBSan in CI. Most CVEs in C/C++ infra code are buffer/integer bugs.
  • Least privilege: run the daemon as a dedicated non-root user; grant only the capabilities it needs (CAP_NET_ADMIN, not full root); scope its filesystem access (systemd ProtectSystem, ReadOnlyPaths).
  • Secrets: never in code, logs, or argv (argv is world-readable via /proc). Read from a file with tight perms, an env var injected by the orchestrator, or a secrets manager (Phase 09). Redact secrets in logs.
  • Supply chain: pin dependencies, verify checksums/signatures, minimize the dependency tree (every dep is attack surface), scan with pip-audit/cargo audit/SCA in CI.

The mindset: assume every input is hostile and every privilege you hold can be abused. Phase 09 goes deep on the system-level security (secure boot, tenancy); this chapter is the code-level security the JD demands in every commit.

Chapter 9: Testability — The Hardware Boundary Seam

You cannot put a BMC, PDU, or CDU in CI. So the central testability technique for rack software is the seam at the hardware boundary: define an interface for the hardware interaction and program against it, so tests inject a fake.

struct SensorSource {                 // the seam
    virtual std::optional<double> read_temp(int id) = 0;
    virtual ~SensorSource() = default;
};
class RealBmcSource : public SensorSource { /* talks to the BMC */ };
class FakeSensorSource : public SensorSource { /* returns scripted values, failures */ };

The daemon takes a SensorSource& (dependency injection). Production wires the real one; tests wire the fake to simulate a hang, garbage data, intermittent failure — deterministically. This is exactly how Lab 01 tests retry and staleness without a BMC, and it scales to every phase: a fake Redfish service (Phase 03/11), a fake firmware target (Phase 08), a fake CDU (Phase 04).

Determinism: also inject time (a clock interface) so you can test "mark stale after 5s" without sleeping 5s. Real-clock tests are flaky; injected-clock tests are instant and reliable.

The test pyramid (deeper in Phase 11): many fast unit tests against fakes, fewer integration tests against emulators (Redfish mock, ipmi_sim), a thin top of hardware-in-the-loop tests on real gear. The seam is what makes the wide base possible.

Misconception: "infra code can't be unit-tested because it needs hardware." Wrong — it can't be unit-tested if you didn't design a seam. The seam is a design choice you make up front, and it's a hallmark of senior infra code.

Chapter 10: Safe Shell and When to Stop Using It

Shell is unavoidable (bootstrap, CI, provisioning hooks) and dangerous. The safety basics:

  • set -euo pipefail at the top of every script: -e exit on error, -u error on unset variable, -o pipefail a pipeline fails if any stage fails. This turns silent failures into loud ones.
  • Quote everything: "$var", "${array[@]}". Unquoted variables word-split and glob — the source of countless "it worked until a path had a space" bugs and injection vectors.
  • trap cleanup EXIT for RAII-like cleanup (remove temp files, release locks) on any exit.
  • Idempotency: a provisioning script may run twice; design it to converge (check-then-act, mkdir -p, "create if not exists") rather than assume a clean slate.
  • shellcheck in CI — it catches the quoting and portability bugs you'll miss.

When to stop: the moment you need data structures, real error handling, arithmetic, JSON, or testing — switch to Python. A rule of thumb: >50 lines, any non-trivial control flow, or any logic you'd want a unit test for → it's a Python program, not a shell script. Staff engineers are known for deleting sprawling shell and replacing it with tested Python; that's a maturity signal, not a rewrite-for-its-own-sake.

Lab Walkthrough Guidance

Order: Lab 01 (C++ daemon) → Lab 02 (Python CLI). The daemon teaches resource safety and the hardware seam at the hardest layer; the CLI applies the same robustness ideas where you'll write most code.

Lab 01 (C++ sensor daemon):

  1. make && ./sensord --once — one poll, then exits. Read the output.
  2. Find the three load-bearing pieces in sensord.cpp: the SensorSource interface (the seam, Ch. 9), the RAII guard / lock_guard (Ch. 3), and the signal handler that flips the stop flag (Ch. 7).
  3. make test — the fake source injects a hang and garbage; tests assert retry/backoff and staleness marking.
  4. make asan && ./sensord_asan --once — confirm clean under AddressSanitizer.
  5. Extensions: a Result/expected error type; a Unix-socket status endpoint; a systemd unit with Type=notify.

Lab 02 (Python rack CLI):

  1. python3 -m rackctl status --rack rack-ai-01 — read the JSON-structured logs.
  2. Trace a command from dispatch → retry decorator → subprocess wrapper → exit code.
  3. python3 -m pytest — see the retry, config-precedence, and subprocess-safety tests.
  4. Try to make it run an injected shell command through a node name; confirm it can't (arg list, no shell=True).
  5. Extensions: --output json, an asyncio bounded fan-out over 100 mock nodes, a universal --dry-run.

The phase capstone question: "Take the C++ daemon and the Python CLI and explain, for each, the single change that most improves its production-readiness." (Good answers: the daemon — Type=notify + watchdog so the orchestrator knows true readiness/liveness; the CLI — bounded-concurrency async fan-out + structured-log correlation IDs so it scales and is debuggable.)

Success Criteria

You're done with this phase when — without notes:

  • You can explain and demonstrate RAII, and identify the leak/deadlock it prevents (Ch. 3)
  • You can state the rule of zero/five and when each applies (Ch. 3)
  • You can classify an error as transient/permanent/programmer/degraded and pick the response (Ch. 2)
  • You can pick threads vs async vs processes for a given workload and justify it, including the GIL (Ch. 6)
  • You can write a graceful-shutdown daemon and a systemd unit with Type=notify + watchdog (Ch. 7)
  • You can spot and fix a shell-injection and explain least privilege/secrets handling (Ch. 8)
  • You can design a hardware-boundary seam and test failure paths without hardware (Ch. 9)
  • Both labs build, pass tests, and (C++) run clean under ASan/UBSan
  • You can write a safe Bash script and say when to switch to Python (Ch. 10)

Interview Q&A

Q1: What is RAII, and why does it matter for a rack daemon? A: RAII ties a resource's lifetime to an object's scope — acquire in the constructor, release in the destructor — so the resource is freed on every exit path, including exceptions, with no manual cleanup. For a daemon it's the difference between leaking file descriptors / BMC sessions / mutex locks on error paths and being correct by construction. Concretely: a lock_guard releases the mutex even if the critical section throws (no deadlock); a unique_ptr/fd-guard closes the handle on early return. The rule is "no raw new/delete, no manual lock/unlock" — let destructors do the work. It also gives the rule of zero: a class holding only RAII members needs no destructor or copy/move at all.

Q2: subprocess.run(f"ipmitool -H {host} power status", shell=True) — what's wrong and how do you fix it? A: It's a shell-injection vulnerability. With shell=True the string is parsed by /bin/sh, so host = "h; rm -rf /" runs arbitrary commands with the daemon's privileges — which on the management network are high. The fix is to pass an argument list with shell=False (the default): subprocess.run(["ipmitool", "-H", host, "power", "status"], timeout=10, check=True). Now host is a single argv element that can never be reinterpreted as shell syntax. I'd also validate host against an allow-list/format, set a timeout, and check the return code. Same principle in C++: never system() a built string; use exec/posix_spawn with an argv array.

Q3: How do you unit-test code whose real dependency is a BMC you don't have in CI? A: Put a seam at the hardware boundary — define an interface (e.g., SensorSource with read_temp) and program against it via dependency injection. Production injects the real BMC client; tests inject a fake that returns scripted values, simulates a hang, garbage data, or intermittent failure — deterministically. I also inject the clock so I can test "mark stale after 5s" instantly without sleeping. That gives a wide base of fast, reliable unit tests; above it, a few integration tests run against an emulator (a Redfish mock or ipmi_sim), and a thin top runs on real hardware. The key insight is that testability is a design decision — the seam — made up front, not something you bolt on.

Q4: Your daemon receives SIGTERM mid-poll. Walk me through correct behavior. A: The SIGTERM handler does the minimum async-signal-safe work: set an atomic stop flag (and maybe write to a self-pipe/eventfd to wake the loop). It does not do logging or cleanup inside the handler. The main loop notices the flag, stops accepting new work, lets the in-flight poll finish (or cancels it cleanly if it's safe), flushes logs, and returns — at which point RAII guards release fds, sessions, and locks, and the process exits 0. If there's a transactional operation in progress (a firmware update), shutdown must either complete it or leave it in a safe, resumable state — never a corrupt half-state. With systemd I'd also use Type=notify so readiness is real and a WatchdogSec so a hang during shutdown gets escalated to SIGKILL.

Q5: Threads, async, or processes — how do you choose, and where does the GIL bite? A: Match the model to the workload. I/O-bound concurrency (poll 100 BMCs that mostly wait) → async/coroutines (low overhead, no locks) or a bounded thread pool. CPU-bound work → processes for true parallelism. Fault isolation → processes. The GIL means CPython threads don't run bytecode in parallel, so threads won't speed up CPU-bound Python — a common mistake; they do help I/O-bound work because the GIL is released during I/O. In C++ threads give real parallelism but I own synchronization: tiny critical sections under lock_guard, never hold a lock across I/O, and prove it with TSan. Whatever the model, I bound concurrency — a semaphore or pool size — so I don't exhaust fds or flood the management network polling thousands of nodes at once.

Q6: What does "secure code" mean concretely for this role? A: At the code level: no shell injection (arg lists, never shell=True/system()), validate all external input at the boundary with allow-lists, memory safety in C++ (no strcpy/sprintf, watch integer overflow in size math, ASan/UBSan in CI), least privilege (dedicated non-root user, scoped capabilities, systemd sandboxing), secrets never in code/logs/argv (argv leaks via /proc), and supply-chain hygiene (pinned, checksum-verified, minimal, scanned dependencies). It matters more here than in most software because rack-management code runs with high privilege on the management network, so a single buffer overflow or injected command is a fleet-wide compromise. Security is a per-commit discipline enforced by standards and CI gates, not an afterthought.

Q7 (Staff): Your team's rack tooling is a 4,000-line pile of Bash that's become unmaintainable. How do you approach it? A: First, stabilize and understand, not rewrite blindly: add set -euo pipefail and shellcheck to stop the bleeding, and capture what it actually does (its behavior is the spec). Then carve along seams — identify the parts doing real logic (data structures, retries, parsing, anything you'd want tested) and migrate those to a typed, tested Python package incrementally, leaving shell only for thin bootstrap/glue. I'd add tests as I go so each migrated piece is pinned, ship behind a flag, and run old and new in parallel to validate. The Staff value is doing this incrementally and safely with the team — not a big-bang rewrite that stalls feature work — and codifying the new standard (lint/format/type/test gates) so it doesn't regress. The deliverable is a maintainable tool and a raised engineering bar.

References

  • Scott Meyers, Effective Modern C++ — the canonical modern-C++ practices (RAII, smart pointers, move, const)
  • ISO C++ Core Guidelines — https://isocpp.github.io/CppCoreGuidelines/ (esp. the Resource and Concurrency sections)
  • CMake & sanitizers — https://clang.llvm.org/docs/AddressSanitizer.html , UBSan, ThreadSanitizer docs
  • Brett Slatkin, Effective Python — production Python idioms (typing, concurrency, packaging)
  • Python subprocess security — https://docs.python.org/3/library/subprocess.html#security-considerations
  • systemd.service & sd_notify — https://www.freedesktop.org/software/systemd/man/systemd.service.html
  • Google Shell Style Guide & shellcheck — https://google.github.io/styleguide/shellguide.html , https://www.shellcheck.net/
  • OWASP — Command Injection & Secrets Management cheat sheets — https://cheatsheetseries.owasp.org/
  • Cross-track: Security Engineer — AppSec/SDLC phase for deeper secure-coding practice