Phase 11 — CI/CD, Testing & Release Engineering
Difficulty: ⭐⭐⭐⭐☆ | Estimated Time: 1–2 weeks Roles supported: Rack Management SWE (Senior/Staff), SRE, Platform/Infra SWE Hardware needed: none — test pyramid + rollout simulators run locally
Why This Phase Exists
The JD: "Participate in code reviews, testing efforts, and continuous improvement of build, test, and deployment pipelines." Infrastructure software that runs close to hardware has a special testing problem — you can't put a BMC, PDU, CDU, or accelerator in CI — so the test strategy that works for web apps doesn't transfer. This phase builds the answer: a test pyramid anchored on the hardware-boundary seam (Phase 02/03), emulator-based CI (a Redfish mock in the pipeline), hardware-in-the-loop (HIL) at the top, and release engineering that ships changes to a fleet safely (versioning, canary, progressive rollout, rollback) — the same discipline that makes firmware rollouts (Phase 08) and k8s upgrades (Phase 06) safe.
You'll build a test pyramid with emulated hardware in CI and a fleet canary-rollout simulator with automatic rollback.
Concepts
Testing close-to-hardware software
- The test pyramid: many fast unit tests (against the Phase 02 seam/fakes), fewer
integration tests (against emulators — Redfish mock,
ipmi_sim,snmpsim), a thin top of HIL (hardware-in-the-loop) tests on real gear, plus soak/stress tests - Why the seam (Phase 02 Ch. 9) is what makes the wide base possible; deterministic time/IO
- Emulator-based CI: running the Redfish mock / IPMI sim in the pipeline so integration tests need no hardware
- HIL labs: a small pool of real hardware behind CI for the tests only real devices can validate; conformance suites per vendor/firmware (Phase 03 Ch. 9)
- Test types: unit, integration, contract, end-to-end, property-based, fuzzing (for parsers — Phase 08), soak
CI/CD pipelines
- The pipeline stages: lint/format/type → build → unit → integration (emulators) → security scan → artifact → deploy → HIL/soak
- Quality gates (Phase 02):
ruff/black/mypy/clang-tidy/sanitizers (ASan/UBSan/TSan), coverage - Artifact management & versioning (semver), reproducible builds, signing (Phase 08/09)
- GitHub Actions / GitLab CI mechanics; matrix builds; caching; secrets in CI (no leaks)
Release engineering (ship to a fleet safely)
- Versioning & compatibility (the agent ↔ control-plane ↔ firmware version matrix)
- Canary + progressive rollout (1 → 1% → 10% → 100%) with health/SLO gates (Phase 07) and automatic rollback
- Blue/green vs rolling vs canary; feature flags; drain-first (Phase 06); failure-domain-aware waves (Phase 01)
- Rollback strategy: image-pinned, fast, tested; the "can you undo it?" question
- The relationship to firmware rollout (Phase 08) and k8s upgrades (Phase 06) — same discipline
Code review & continuous improvement (the JD's words)
- Effective code review (correctness, security, tests, the seam); review as knowledge-sharing (Phase 12)
- Continuously improving the pipeline: flake reduction, speed, coverage, new gates after incidents (Phase 10 RCA → a test)
Labs
Lab 01 — CI With Emulated Hardware + Test Pyramid
| Field | Value |
|---|---|
| Goal | Build a test pyramid for rack software where the wide base runs against fakes/emulators (no hardware), and wire it into a CI pipeline that runs a Redfish mock as a service — so integration tests pass in CI with zero hardware. |
| Concepts | Test pyramid, the hardware seam, emulator-based integration tests, deterministic tests, CI stages/gates, fuzzing a parser. |
| Steps | 1) python3 run_tests.py — runs unit (fakes) + integration (against an in-process Redfish mock). 2) Read the pyramid and the seam usage. 3) Read .github/workflows/ci.yml (the real pipeline). 4) Extensions (real Redfish mock service, HIL). |
| Stack | Python 3 stdlib (test runner + emulator) + a real GitHub Actions workflow |
| Output | A runnable test pyramid (unit + emulator-integration) + a CI workflow that needs no hardware. |
| How to Test | python3 run_tests.py — the suite runs and reports per-tier results; the integration tier exercises a live (mock) Redfish service. |
| Talking Points | Why you can't put a BMC in CI and how the seam fixes it; the pyramid shape; emulators vs HIL; what to fuzz (the RAS/Redfish parsers). |
| Resume Bullet | "Designed a test pyramid for close-to-hardware rack software (unit on fakes, integration against Redfish/IPMI emulators, HIL on real gear) and an emulator-based CI pipeline requiring no hardware." |
| Extensions | Run the real DMTF Redfish-Mockup-Server as a CI service container; add mypy/ruff/ASan gates; add property-based fuzzing of the Phase 08 RAS parser; add a HIL stage gated to a hardware pool. |
Lab 02 — Fleet Canary & Rollback
| Field | Value |
|---|---|
| Goal | Build a progressive-rollout controller that ships a new agent/firmware version across a fleet in waves (1 → 1% → 10% → 100%), gates each wave on health/SLOs (Phase 07), and automatically rolls back on regression — failure-domain-aware. |
| Concepts | Canary, progressive rollout, health/SLO gates, automatic rollback, blast-radius/failure-domain waves, version pinning. |
| Steps | 1) python3 solution.py — a healthy rollout completes; a bad version is caught at the canary and rolled back. 2) Read the wave/gate/rollback logic. 3) See a domain-aware wave plan. 4) Extensions. |
| Stack | Python 3 stdlib |
| Output | A rollout controller demonstrating canary gating + auto-rollback, with the fleet never fully exposed to a bad version. |
| How to Test | Asserts verify: a good version reaches 100%; a bad version is halted at the canary and rolled back; waves respect failure domains; rollback restores the prior version. |
| Talking Points | Why 1% before 100%; what gates a promotion; image-pinned rollback; why you don't update a whole failure domain at once; same pattern as firmware/k8s. |
| Resume Bullet | "Built a failure-domain-aware progressive-rollout controller with health/SLO-gated promotion and automatic rollback, bounding the blast radius of a bad release across a fleet." |
| Extensions | Add error-budget-based halting (Phase 07); add a bake-time per wave; integrate with the Phase 08 firmware orchestrator and Phase 06 drain. |
Deliverables Checklist
- You can draw the test pyramid and place rack tests (unit/emulator-integration/HIL/soak)
- You can explain why the hardware seam makes the wide base possible
- Test suite runs unit (fakes) + integration (against an emulator) with no hardware
- You can name the CI stages/gates and what each catches
- Rollout controller does canary → progressive → 100% with health gates
- A bad version is caught at the canary and auto-rolled-back; the fleet is never fully exposed
- Waves respect failure domains (Phase 01)
- You can connect this discipline to firmware rollout (Phase 08) and k8s upgrade (Phase 06)
Interview Relevance
- "How do you test software that talks to hardware you don't have in CI?" (seam + emulators + HIL)
- "Describe the test pyramid for this kind of system."
- "Walk me through a CI/CD pipeline for a rack-management agent."
- "How do you safely roll out a new version to 10,000 nodes?" (canary + gates + rollback)
- "What gates a canary promotion, and what triggers a rollback?"
- "How is rolling out an agent like rolling out firmware?" (same discipline, Phase 08)
- "What do you look for in a code review?" (correctness, security, tests, the seam)