Phase 02 — Production Systems Programming (Python, C++, Shell, Node.js)

Difficulty: ⭐⭐⭐☆☆ | Estimated Time: 2 weeks Roles supported: Rack Management SWE (Senior/Staff), Infra/Systems SWE Hardware needed: none — a C++17 compiler and Python 3 on any Linux/macOS box

Why This Phase Exists

The JD's hard requirement is blunt: "Proficiency in Python, C++, and shell scripting" and "Write high‑quality, secure, and maintainable code following established coding standards and best practices." Node.js is "a plus." This is the phase where you stop writing scripts and start writing infrastructure software — code that runs unattended, close to hardware, in production, for years, on someone else's pager.

The difference is not syntax. It is robustness under failure: a rack daemon must survive a BMC that stops responding mid-poll, a sensor that returns garbage, a SIGTERM during a critical section, a disk that fills, and a malicious or malformed input — and it must do so observably, with logs and exit codes an on-call engineer can reason about at 3 a.m. The same code must be testable without the hardware (you can't ship a BMC to CI) and maintainable by a distributed team you've never met.

This phase establishes the engineering bar that every other phase's labs are held to. You build two pieces of real infrastructure software — a C++ sensor daemon and a Python rack CLI — that demonstrate the production patterns: RAII and resource safety, typed interfaces, structured logging, retries with backoff, graceful shutdown, safe subprocess handling, and the testability that lets you mock hardware.

Concepts

The language matrix — why each tool, where

C++ — close-to-hardware, long-running, performance- and resource-sensitive code: daemons, agents, anything that talks to a device library or parses binary protocols. RAII, smart pointers, the rule of zero/five, const-correctness, error handling (exceptions vs std::expected/error codes), modern C++17/20, CMake, sanitizers (ASan/UBSan/TSan), threading.
Python — orchestration, clients, tooling, CLIs, simulators, parsers, glue: most of the rack-management surface. Type hints + mypy, packaging (pyproject.toml), virtualenvs, argparse/click, structured logging, retries/backoff, subprocess safety, asyncio vs threads, pytest, ruff/black.
Shell — provisioning glue, CI steps, bootstrap, and "do this on 500 nodes" one-liners. Safe scripting (set -euo pipefail), quoting, idempotency, trap/cleanup, when to stop using shell and switch to Python.
Node.js (the "plus") — tooling and lightweight services (a telemetry/status web service, a dashboard backend). When JS/TS is the right call and when it isn't.

Production engineering (language-agnostic)

Error handling philosophy: fail fast vs degrade gracefully; the error taxonomy (transient vs permanent vs programmer error)
Retries, exponential backoff, jitter, timeouts, circuit breakers (deeper in Phase 10)
Resource lifetime: RAII in C++, context managers in Python, defer-style cleanup; never leak fds/handles/locks
Concurrency: threads vs async vs processes; the GIL; data races; lock discipline
Daemon mechanics: signals, graceful shutdown/drain, PID/lifecycle, systemd units, journald, readiness/liveness
Structured logging (key/value, correlation IDs), log levels, what to log and what not to (no secrets)
Configuration: env vs file vs flags; precedence; validation at startup
Security: input validation, shell-injection avoidance, least privilege, secrets handling, integer/buffer safety in C++, dependency/supply-chain hygiene
Testability: dependency injection, mocking the hardware boundary, deterministic time, the seam that lets you test without a BMC
Coding standards & style: clang-format/clang-tidy, ruff/black, code review norms (Phase 12)

Labs

Lab 01 — C++ Sensor-Polling Daemon

Field	Value
Goal	Build a production-grade C++17 daemon that polls a (mocked) sensor source on an interval, exposes current readings, handles failures and signals, and is unit-testable without real hardware.
Concepts	RAII, smart pointers, the rule of zero, dependency injection at the hardware boundary, retries/backoff, graceful shutdown on SIGTERM/SIGINT, thread-safe state, structured logging, CMake/Make builds, sanitizers.
Steps	1) `make && ./sensord --once`. 2) Read `sensord.cpp`: find the `SensorSource` interface (the test seam), the RAII guard, the signal handler, the poll loop. 3) `make test` runs the unit tests against a fake source. 4) Run under ASan. 5) Extensions below.
Stack	C++17, `make` (or CMake), no external deps; optional `-fsanitize=address,undefined`
Output	A daemon that polls, degrades on sensor failure (stale + retry), shuts down cleanly, and passes unit tests against a fake sensor.
How to Test	`make test` — fake `SensorSource` injects failures/garbage; asserts verify retry, staleness marking, and clean shutdown.
Talking Points	Why RAII beats try/finally; why the hardware is behind an interface; what happens on SIGTERM mid-poll; why you never `system()` a string.
Resume Bullet	"Built a C++17 sensor-polling daemon with RAII resource management, injectable hardware boundary for hermetic unit tests, retry/backoff, and graceful SIGTERM shutdown; clean under ASan/UBSan."
Extensions	Add a `std::expected`-style error type; expose readings over a Unix socket; add a `systemd` unit with `Type=notify` readiness; add a TSan run to prove the lock discipline.

Lab 02 — Production Python Rack CLI

Field	Value
Goal	Build a real CLI tool (`rackctl`) — typed, packaged, with structured logging, retries/backoff, safe subprocess use, and config precedence — the kind of tool every rack engineer lives in.
Concepts	`argparse` subcommands, type hints + `mypy`, `pyproject.toml` packaging, structured JSON logging, retry decorator with backoff+jitter, `subprocess` without `shell=True`, config precedence (flags > env > file > default), exit-code discipline, `pytest`.
Steps	1) `python3 -m rackctl status --rack rack-ai-01`. 2) Read `rackctl/` — the command dispatch, the retry decorator, the logging setup, the subprocess wrapper. 3) `python3 -m pytest`. 4) Break a command and read the structured error. 5) Extensions below.
Stack	Python 3 (stdlib only — deliberately, to show the patterns without magic)
Output	A packaged CLI with subcommands, JSON logs, retried operations, and tests.
How to Test	`python3 -m pytest` — tests cover retry/backoff, config precedence, subprocess argument safety, and exit codes.
Talking Points	Why `shell=True` is a vulnerability; flags vs env vs file precedence; why structured logs beat printf; how you make time deterministic for tests.
Resume Bullet	"Built a typed, packaged Python rack-operations CLI with structured logging, retry/backoff, safe subprocess execution, and config precedence; fully unit-tested with mocked hardware boundaries."
Extensions	Add `--output json` for machine consumption; add an `asyncio` fan-out that polls 100 mock nodes concurrently with bounded concurrency; add a `--dry-run` to every mutating command.

Deliverables Checklist

C++ daemon builds, runs, shuts down cleanly on SIGTERM, and passes tests against a fake sensor
C++ daemon is clean under -fsanitize=address,undefined
You can explain RAII and show where a leak would occur without it
Python CLI runs, logs structured JSON, retries transient failures, and has correct exit codes
Python CLI never uses shell=True; you can explain the injection risk
You can state the config precedence order and why
Both labs have a clear test seam at the hardware boundary (you can test without hardware)
You can write a safe Bash script with set -euo pipefail and proper quoting

Interview Relevance

"Write a daemon that polls a sensor every second and exposes the latest value. How do you handle a sensor that hangs?"
"What is RAII and why does it matter here? Show me a resource leak and how RAII prevents it."
"Why is subprocess.run(cmd, shell=True) dangerous? Rewrite it safely."
"How do you make this code testable when the real dependency is a BMC you don't have in CI?"
"Your daemon gets a SIGTERM in the middle of a poll. Walk me through what happens."
"Exceptions vs error codes vs std::expected — when do you reach for each?"
"How do you handle retries without hammering a struggling BMC?" (backoff + jitter + circuit breaker)

Software Engineer — Rack Management (Senior / Staff)