Warmup Guide — CI/CD, Testing & Release Engineering

Zero-to-expert primer for Phase 11. The twist for this role: you can't put a BMC/PDU/CDU/ accelerator in CI, so the standard test playbook needs adapting. This guide builds the hardware-aware test pyramid, emulator-based CI, and the release engineering (canary/rollback) that ships changes to a fleet without incidents.

Table of Contents

Chapter 1: The Special Problem — No Hardware in CI

A web service's dependencies (a DB, an API) can run in CI as containers. A rack-management agent's dependencies are physical: a BMC, a PDU, a CDU, a PCIe switch, an accelerator. You cannot spin those up in a CI runner. Yet the JD demands "continuous improvement of build, test, and deployment pipelines" — so the central question is how do you test thoroughly without the hardware?

The answer has three layers, and it's the spine of this phase:

  1. The seam (Phase 02 Ch. 9 / Phase 03 Ch. 8): program against a DeviceDriver/SensorSource interface so most logic is tested against fakes that simulate any behavior (failures, garbage, timeouts) deterministically — the wide base of the pyramid.
  2. Emulators in CI: real software implementations of the protocols (the Redfish mock, ipmi_sim, snmpsim) run as services in the pipeline, so integration tests exercise real protocol behavior with no physical hardware.
  3. HIL (hardware-in-the-loop): a small pool of real devices behind CI, for the tests only real hardware can validate (vendor quirks, firmware behavior, timing) — the thin top.

Get this right and you ship rack software with confidence; get it wrong and every change is a gamble on production hardware.

Chapter 2: The Test Pyramid for Rack Software

The pyramid (wide fast base → narrow slow top), mapped to this domain:

  • Unit (the wide base, ~70%): test pure logic against the fakes/seam — the budget calculator (Phase 01), the RAS parser (Phase 08), the reconcile loop (Phase 06), the retry/breaker (Phase 10). Fast (ms), deterministic (injected clock/RNG), no I/O. You can have thousands.
  • Integration (~20%): test against emulators — your Redfish client (Phase 03) against the Redfish mock, the exporter (Phase 07) scraped by a real Prometheus, the firmware orchestrator (Phase 08) against a mock UpdateService. Slower (seconds), exercises real wire behavior.
  • HIL / E2E (~10%): against real hardware in a lab — vendor conformance (Phase 03 Ch. 9), firmware flashes that must touch real flash, PCIe behavior under load. Slowest, scarcest, gated.
  • Soak/stress (alongside): run for hours/days to catch leaks, drift, and rare races (Phase 10).

The shape matters: most bugs are caught cheaply at the base; the expensive HIL tier is reserved for what only real hardware can prove. Inverting the pyramid (mostly E2E) is slow, flaky, and can't run on every commit — a common, costly anti-pattern.

The enabler, again: the seam. Without an interface at the hardware boundary you can't write the base, and you're forced up the pyramid into slow, hardware-needing tests. Testability is a design decision made in Phase 02, paid off here.

Chapter 3: Emulators and HIL

Emulators are the middle tier's superpower:

  • Redfish: the DMTF Redfish-Mockup-Server or OpenBMC sushy-emulator — a real Redfish service you run as a CI service container; your client/exporter/orchestrator test against it.
  • IPMI: ipmi_sim (OpenIPMI) simulates a BMC's IPMI interface.
  • SNMP: snmpsim serves recorded MIBs so your SNMP poller tests against realistic data.
  • These catch the bugs fakes miss (real serialization, status codes, link-following, error bodies) while staying hermetic and fast enough for CI.

HIL (hardware-in-the-loop): a managed pool of real devices (a few BMCs, a PDU, maybe an accelerator node) reachable from CI for a gated stage. Used for: per-vendor conformance suites (Phase 03 Ch. 9 — does this ODM's Redfish behave?), firmware flash/rollback that must touch real flash (Phase 08), and timing/perf the emulators can't model. HIL is scarce and slow, so it runs on merge-to-main or nightly, not every commit, with careful scheduling and reset-between-tests.

The Staff insight: invest in the emulator + HIL infrastructure as a product — it's what lets a distributed team ship hardware-adjacent code quickly and safely, and it's where a lot of an infra team's leverage lives.

Chapter 4: The CI Pipeline and Its Gates

A pipeline for a rack agent, stage by stage (each a gate that fails fast):

  1. Static: format (black/clang-format), lint (ruff/clang-tidy), types (mypy --strict) — Phase 02. Cheap, catches a lot.
  2. Build: compile (C++ with -Wall -Wextra -Werror), package (Python wheel), reproducibly.
  3. Unit: the wide base against fakes — fast, must be green to proceed.
  4. Sanitizers (C/C++): run tests under ASan/UBSan/TSan (Phase 02/10) — catches the memory/ race bugs that become production crashes.
  5. Integration: against emulators (Redfish mock, etc.) as service containers.
  6. Security scan: dependency audit (pip-audit), SAST, secret scanning (Phase 02 Ch. 8).
  7. Artifact: build + sign (Phase 08/09) + version the artifact; publish.
  8. Deploy (CD): to a staging/canary environment; then HIL/soak (gated, slower).

Principles: fail fast (cheap gates first), deterministic (no flaky tests — a flaky test is a broken test; quarantine and fix it), fast feedback (cache, parallelize; keep the per-commit path minutes not hours), and everything-as-code (the pipeline is reviewed code, Lab 01's ci.yml). Coverage is a signal, not a goal — chase meaningful coverage of failure paths, not a number.

Chapter 5: Versioning & Compatibility

A rack fleet runs many versions at once during a rollout, so compatibility is a first-class concern:

  • Semver (major.minor.patch): major = breaking, minor = additive, patch = fixes. Communicate intent; gate breaking changes.
  • The version matrix: the node agent, the control plane, and the firmware (Phase 08) all version independently and must interoperate across a window (you can't update everything atomically). Design APIs to be backward/forward compatible (additive changes, tolerate unknown fields — like Redfish, Phase 03) so an old agent works with a new control plane and vice versa during the rollout.
  • Pin versions for reproducibility and rollback (image digests, not floating tags).

The discipline mirrors the plugin-ABI/ schema-evolution problems in any platform: never break the contract a deployed component depends on without a migration path.

Chapter 6: Release Engineering — Canary, Rollout, Rollback

Shipping a change to one machine is easy; shipping to 10,000 without an incident is the skill (Lab 2, and the same discipline as Phase 08 firmware and Phase 06 k8s upgrades).

  • Canary: deploy the new version to a tiny slice (1 node, then 1%), bake (let it run long enough for problems to surface), and gate promotion on health/SLOs (Phase 07) — error rate, latency, crash count, hardware health. Only promote if the canary is healthy.
  • Progressive rollout: 1 → 1% → 10% → 50% → 100%, each wave gated, widening as confidence grows. A bad version caught at 1% is an annoyance; at 100% it's an outage.
  • Automatic rollback: if a wave's gates regress, halt and roll back to the pinned previous version automatically — fast, tested, image-pinned. The "can you undo it in 60 seconds?" question is the heart of safe releases.
  • Failure-domain-aware waves (Phase 01): don't put a whole PDU/ToR/rack in one wave — a bad release shouldn't down a correlated set; spread waves across domains.
  • Strategies: rolling (replace gradually — default for agents), blue/green (two environments, switch traffic — fast rollback, double resources), canary (the gated subset above). Feature flags decouple deploy from release (ship dark, enable gradually).
  • Drain-first (Phase 06): for anything requiring a node reboot/restart, cordon+drain tenant work first.

The mental model unifying Phases 06, 08, 11: every change to a fleet is a gated, canaried, reversible, domain-aware rollout — whether it's an agent, a config, firmware, or k8s. Master it once.

Chapter 7: Code Review & Continuous Improvement

The JD names code review and "continuous improvement" explicitly — the human + process layer (more in Phase 12).

Effective code review looks for, in priority order: correctness (does it do the right thing, including failure paths?), security (Phase 02 Ch. 8 — injection, validation, secrets), tests (is the change tested at the right tier? is there a seam?), operability (logs/metrics/ errors — Phase 07), and then style (let the linter handle most of it — don't bikeshed). Review is also knowledge-sharing across a distributed team and a place to mentor (Phase 12). Be kind, be specific, focus on the code not the person.

Continuous improvement of the pipeline is a never-done loop: reduce flakiness (the silent productivity killer — quarantine, root-cause, fix), speed it up (cache, parallelize), raise coverage of failure paths, and — crucially — turn every incident into a test (Phase 10 RCA → a regression test/gate so it can't recur). A pipeline that gets a new gate after each incident is a team that learns. That feedback loop (incident → RCA → test → gate) is the connective tissue between this phase and Phase 10.

Lab Walkthrough Guidance

Order: Lab 01 (test/CI) → Lab 02 (release). Build confidence in the change, then ship it safely.

Lab 01 (test pyramid + emulator CI):

  1. python3 run_tests.py — runs unit (fakes) + integration (against an in-process Redfish mock); read the per-tier results.
  2. See how the seam lets the base run with no hardware and how the integration tier uses a real (mock) Redfish service.
  3. Read .github/workflows/ci.yml — the real pipeline (gates + the Redfish mock as a service).
  4. Extensions: run the real DMTF mockup server as a CI service; add mypy/ASan gates; fuzz the RAS parser (Phase 08).

Lab 02 (canary + rollback):

  1. python3 solution.py — a healthy version rolls out 1→1%→10%→100%; a bad version is caught at the canary and auto-rolled-back.
  2. Read the wave/gate/rollback logic and the failure-domain-aware wave plan.
  3. Extensions: error-budget-based halting (Phase 07); bake time; integrate with Phase 08 firmware + Phase 06 drain.

Phase capstone question: "How do you test and ship a new rack-management agent to a 10,000-node fleet?" (Answer: a test pyramid — unit on fakes, integration on Redfish/IPMI emulators in CI, HIL on a real pool, soak for leaks — behind static/sanitizer/security gates; sign + version the artifact; roll out canary → progressive with health/SLO gates, automatic image-pinned rollback, failure-domain-aware waves, drain-first; and feed every incident back into a new test/gate.)

Success Criteria

You're done with this phase when — without notes:

  • You can explain why hardware can't go in CI and the seam+emulator+HIL answer (Ch. 1, 3)
  • You can draw the test pyramid and place this domain's tests on it (Ch. 2)
  • You can list the CI stages/gates and what each catches (Ch. 4)
  • You can explain semver and the agent/control-plane/firmware compatibility window (Ch. 5)
  • You can design a canary → progressive rollout with health gates and auto-rollback (Ch. 6)
  • You can explain failure-domain-aware waves and image-pinned rollback (Ch. 6, Phase 01)
  • You can connect this to firmware (Phase 08) and k8s (Phase 06) as one rollout discipline
  • You can state what you look for in code review and how incidents become gates (Ch. 7, Phase 10)

Interview Q&A

Q1: How do you test software that talks to hardware you don't have in CI? A: Three layers. (1) The seam: program against a device interface so most logic is unit-tested against fakes that simulate any behavior — failures, timeouts, garbage — deterministically (inject the clock too). That's the wide base. (2) Emulators in CI: run real protocol implementations as service containers — the DMTF Redfish mock, ipmi_sim, snmpsim — so integration tests exercise real wire behavior with no physical hardware. (3) HIL: a small pool of real devices behind a gated CI stage for what only hardware can prove — vendor conformance, firmware flashes, timing — run on merge/nightly, not every commit. The key realization is that testability is a design choice (the seam, Phase 02) made up front; without it you're forced into slow, hardware-needing end-to-end tests for everything.

Q2: Describe the test pyramid for rack-management software. A: Wide base of fast unit tests against fakes — the budget calculator, RAS parser, reconcile loop, retry/breaker — milliseconds, deterministic, no I/O, thousands of them. A middle integration layer against emulators — the Redfish client against the Redfish mock, the exporter scraped by real Prometheus — seconds, exercising real protocol behavior. A thin top of HIL/E2E on real hardware for vendor quirks, firmware, and PCIe-under-load — slow and scarce, gated. Plus soak tests running for hours to catch leaks and rare races. Most bugs die cheaply at the base; the expensive tiers are reserved for what only they can prove. The anti-pattern is inverting it — mostly E2E — which is slow, flaky, and can't run per-commit.

Q3: How do you safely roll out a new agent version to 10,000 nodes? A: As a gated, canaried, reversible, domain-aware rollout. Sign and version the artifact (semver, image-pinned). Deploy to a canary (1 node, then 1%), bake it, and gate promotion on health/ SLOs (Phase 07) — error rate, crashes, hardware health. Then progress 1% → 10% → 50% → 100%, each wave gated, automatically rolling back to the pinned previous version if a wave regresses (fast, because it's image-pinned). Make waves failure-domain-aware (Phase 01) so a bad release can't down a whole PDU/ToR/rack at once, and drain-first (Phase 06) for anything needing a restart. Design the agent/control-plane API to be backward/forward compatible since many versions run during the rollout. It's the same discipline as a firmware rollout (Phase 08) and a k8s upgrade (Phase 06) — a bad change caught at 1% is an annoyance; at 100% it's an outage.

Q4: What gates a canary promotion and what triggers a rollback? A: Promotion is gated on the canary being demonstrably healthy over a bake period: SLO/health signals (Phase 07) — error and crash rates at or below baseline, latency within budget, no new critical alerts, hardware health nominal (no new AER/MCE — Phase 08), and ideally error-budget not burning abnormally. A rollback triggers when any of those regress past a threshold during a wave: a spike in crashes/errors, an SLO breach, or a new critical alert. The rollback is automatic and fast because the previous version is image-pinned, and it halts the rollout so the blast radius stays at the current wave. The principle: never promote on hope, and always be able to undo in seconds.

Q5: What do you look for in a code review for this kind of software? A: In priority: correctness including the failure paths (timeouts, retries, partial failures — not just the happy path); security (no shell injection, input validation, no secrets in code/ logs — Phase 02); tests at the right tier and the presence of a seam so it's testable without hardware; operability (structured logs, metrics, sane errors — Phase 07); and only then style, which the linter should mostly handle so we don't bikeshed. I also use review to share knowledge across the distributed team and mentor (Phase 12). And I treat every production incident as a prompt to add a test or a CI gate (Phase 10 RCA → a regression test), so the pipeline gets stronger over time — that continuous improvement is literally in the JD.

References

  • Martin Fowler — Test Pyramid; "TestDouble"/mocks vs fakes — https://martinfowler.com/
  • DMTF Redfish-Mockup-Server; OpenBMC sushy-emulator; OpenIPMI ipmi_sim; snmpsim — emulators for CI
  • GitHub Actions / GitLab CI docs — pipelines, service containers, matrices — https://docs.github.com/actions
  • AddressSanitizer/UBSan/TSan in CI (Phase 02/10) — https://clang.llvm.org/docs/
  • Google — Software Engineering at Google (testing, CI, large-scale changes) — https://abseil.io/resources/swe-book
  • Continuous Delivery (Humble & Farley) — deployment pipelines, canary, blue/green
  • Google SRE Workbook — canarying releases, error budgets — https://sre.google/workbook/
  • Cross-track: Apache PMC/Committer — release engineering