🛸 Hitchhiker's Guide — Phase 11: CI/CD, Testing & Release Engineering
Read this if: you can write tests but "test software that needs a BMC" and "ship to 10,000 nodes safely" are open questions. Compressed map of the hardware-aware pyramid and safe rollouts.
0. The 30-second mental model
You can't put a BMC/PDU/CDU in CI, so: seam → fakes (unit), emulators (integration), HIL (real, gated). Ship to a fleet as a gated, canaried, reversible, domain-aware rollout — the same discipline as firmware (Phase 08) and k8s upgrades (Phase 06). Master that one pattern.
1. The no-hardware-in-CI answer
| tier | against | speed | enables |
|---|---|---|---|
| unit (~70%) | fakes via the seam (P02/P03) | ms, deterministic | the wide base |
| integration (~20%) | emulators (Redfish mock, ipmi_sim, snmpsim) | seconds | real wire behavior |
| HIL/E2E (~10%) | real hardware pool, gated | slow/scarce | vendor quirks, firmware, timing |
| soak | long-running | hours | leaks, drift, rare races |
Don't invert the pyramid (mostly E2E = slow + flaky + can't run per-commit).
2. CI pipeline stages (fail fast)
static (fmt/lint/mypy/clang-tidy) → build (-Werror) → unit → sanitizers (ASan/UBSan/
TSan) → integration (emulators as services) → security scan (pip-audit/SAST/secrets) → artifact
(build+sign+version) → deploy → HIL/soak (gated). Flaky test = broken test (quarantine+fix).
3. Versioning
semver (major=breaking) · the agent ↔ control-plane ↔ firmware matrix runs many versions at once → backward/forward-compatible APIs (additive, tolerate unknown fields like Redfish) · pin digests, not floating tags.
4. Safe rollout (the unifying pattern)
canary (1 → 1%) → bake → GATE on health/SLO (P07) → 10% → 50% → 100%
auto-rollback (image-pinned) if a wave regresses ; HALT the rollout
waves are FAILURE-DOMAIN-aware (P01) ; drain-first (P06) for reboots
strategies: rolling (default) · blue/green (fast rollback, 2x resources) · canary (gated subset) · feature flags (deploy ≠ release). "Can you undo it in 60s?" is the test.
5. Code review priorities
correctness (incl. failure paths) > security (P02) > tests + a seam > operability (P07) > style (let the linter win). Review = knowledge-sharing + mentoring (P12).
6. Continuous improvement loop
incident → RCA (P10) → regression test → new CI gate. A pipeline that grows a gate after each incident is a team that learns. Also: kill flakiness, speed it up, cover failure paths.
7. The "done this before" tells
"Where's the seam — can this run without hardware?" · "Unit or does it need the emulator?" · "Is it canaried?" · "What gates the promotion / triggers rollback?" · "Are waves domain-aware?" · "Can we roll back in 60s?" · "Did the incident get a test?"
8. How this phase pays off later
Same rollout discipline as Phase 08 firmware + Phase 06 k8s upgrade. The test seam comes
from Phases 02/03; gates come from Phase 07; new tests come from Phase 10 RCAs. The
capstone (Phase 13) ships through this pipeline. system-design/03 is fleet rollout at scale.