Lab 02 — Production Python Rack CLI (rackctl)

Phase: 02 — Production Systems Programming | Difficulty: ⭐⭐⭐☆☆ | Time: 4–6 hours Language: Python 3 (stdlib only) | Hardware: none (fake BMC client) | Needs: pytest, mypy (optional)

Concept primer: ../WARMUP.md Ch. 5 (production Python), Ch. 8 (security), Ch. 9 (the seam), ../HITCHHIKERS-GUIDE.md §5, §9.

Run

python3 -m rackctl status --rack rack-ai-01            # node power states (text)
python3 -m rackctl status --rack rack-ai-01 --output json
python3 -m rackctl power cycle --node node-01          # JSON logs go to stderr
python3 -m rackctl power off  --node node-00 --dry-run # mutating command, no effect
python3 -m pytest -q                                   # 14 tests, no hardware
python3 -m mypy rackctl                                # strict types clean

0. The mission

Build the tool a rack engineer lives in — rackctl — to the production bar: typed, packaged, structured-logged, retry-aware, subprocess-safe, with explicit config precedence and disciplined exit codes. It's deliberately stdlib-only so you see the patterns, not a framework's magic.

Two questions you must answer at the end:

1. Why does the CLI take a BmcClient it doesn't construct itself?
2. What's the config precedence order, and why must "flag not provided" differ from "flag provided as a falsy value"?

1. The package, file by file

2. The two ideas that make it production-grade

The seam (BmcClient) — main(argv, client=None) lets tests (and the capstone) inject a fake fleet. FakeBmcClient can be told to make a node flaky (make_flaky) so the retry path is exercised deterministically. Production would inject a real Redfish/IPMI client (Phase 03). The CLI logic never changes.

Config precedence with SUPPRESS — argparse has a famous gotcha: with parents=, a subparser's default clobbers a value the main parser already set. The fix here is default=argparse.SUPPRESS on the shared flags, so an attribute exists only if the user actually provided it — which is exactly the "flags override env override file" semantics load_config needs (None means "not provided", so it doesn't override). This is the kind of subtle correctness an interviewer probes with "what if I pass --retries after the subcommand?"

3. What the run shows

status prints a table (or JSON with --output json). Every operation emits a structured JSON log line to stderr (stdout stays clean for piping) — try python3 -m rackctl power cycle --node node-01 and read the {"ts":...,"level":...} log. --dry-run on a mutating command logs "would execute" and changes nothing. Unknown node → exit code 3 (permanent); a flaky node recovers via retries → exit 0.

4. Extension exercises (do them — this is the lab)

1. Async bounded fan-out: add status --all that polls 100 mock nodes concurrently with asyncio and a Semaphore(50) cap (WARMUP Ch. 6). Measure it against a serial loop. The bound matters: 10,000 unbounded connections would exhaust fds and flood the management network.
2. Correlation IDs: generate a request id per invocation and attach it to every log line via a logging filter, so a single rackctl run is greppable end-to-end (foreshadows Phase 07 tracing).
3. Universal --dry-run + confirmation: make every mutating command support --dry-run and require --yes for destructive actions (power off in production).
4. A real subprocess path: add an exec subcommand that runs ipmitool/redfishtool via runner.run([...]), and write a test proving a malicious --node "x; reboot" cannot inject (it's one argv element).
5. Wire the CI gates: add ruff, black --check, mypy --strict, and pytest to a make check target — the Phase 11 quality gates in miniature.

5. Common pitfalls

1. shell=True — the whole reason runner.run refuses strings. Interpolating a node name into a shell string is remote code execution waiting to happen.
2. Bare except: — catch TransientError/PermanentError specifically; a bare catch hides programmer errors and turns bugs into silent wrong behavior.
3. print() for logs — unstructured, unsearchable, and mixes with stdout data. Use the JSON logger to stderr.
4. Retrying permanent failures — an auth error or unknown node won't fix itself; retrying just delays the error and hammers the BMC. The retry_on allow-list prevents it.
5. Secrets in argv/logs — argv is world-readable via /proc; the logger redacts secret-named fields, but never pass a password as a flag in the first place.

6. What this lab proves about you

You can write the Python that is a rack-management product's operator surface: typed, testable, safe, and operable. In an interview, "write a CLI to power-cycle a node and handle a flaky BMC" becomes a showcase — you reach for the seam (testable without hardware), the retry-by-error-class (don't retry auth failures), the arg-list subprocess (no injection), structured logs (debuggable), and exit codes (composable). The same patterns scale to the Phase 03 protocol clients, the Phase 07 exporter, and the Phase 13 capstone — rackctl is the seed of all of them.