Lab 02 — Release Engineering: Compatibility, Canary & Rollback

Phase: 10 — Observability | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–6 hours Language: Python (stdlib) | Hardware: none

Concept primer: ../WARMUP.md Ch. 8.

Run

python solution.py

0. The mission

Build the release-safety tooling that turns "ship the prototype" into "ship through a full release cycle" (the JD's requirement):

  1. Compatibility gate — reject incompatible driver/CUDA/engine pairs before deploy (Phase 03/04's skew failures are fleet-wide; catch them pre-deploy).
  2. Canary analyzer — compare a canary's SLIs to baseline statistically; decide PROMOTE / HOLD / ROLLBACK.
  3. Rollout state machine — progressive promotion with automatic rollback on SLO burn.

1. What the output shows

== compatibility gate ==
driver=535 (max CUDA 12.2), build needs CUDA 12.4  -> REJECTED (driver too old)
driver=550 (max CUDA 12.4), build needs CUDA 12.4  -> OK
engine vLLM 0.5 needs model arch 'llama3' -> OK

== canary analysis (p99 TTFT, ms) ==
baseline p99=240  canary p99=251   delta=+4.6%  within noise -> PROMOTE
baseline p99=240  canary p99=330   delta=+37.5% significant -> ROLLBACK

== rollout state machine ==
stage 5%   : canary healthy -> promote to 25%
stage 25%  : canary healthy -> promote to 50%
stage 50%  : SLO burn detected -> ROLLBACK to baseline
final state: ROLLED_BACK (budget protected)
  • Compatibility: the driver/CUDA gate rejects a build needing CUDA 12.4 on a 12.2 driver — the "driver insufficient for runtime" failure (Phase 04 Ch. 6), caught before it reaches a node.
  • Canary: a +4.6% p99 change is within noise (promote); a +37.5% regression is significant (rollback). Statistical, not eyeballed (WARMUP Ch. 8).
  • Rollout: progresses through stages, auto-rolls-back when a stage burns the error budget — image-pinned so rollback is fast.

2. Reading order (solution.py)

  1. Version parsing + check_compatibility — the matrix gate.
  2. analyze_canary — the statistical comparison (effect size + threshold).
  3. Rollout state machine — stages, gating, auto-rollback.

3. Extensions

  1. Progressive % rollout with configurable stages and per-stage gates.
  2. Error-budget freeze: track budget consumption; freeze releases when exhausted (WARMUP Ch. 5).
  3. Wire Lab 01's metrics as the canary signal (real exporter → canary decision).
  4. Fleet driver-upgrade plan: model the Phase 04 Q6 choreography (inventory → canary → drain → upgrade → health-check → progressive) as a state machine.
  5. Proper stats: replace the threshold with a Welch's t-test / Mann-Whitney on sampled latencies.

4. Common pitfalls

  1. Eyeballing canaries: a real p99 regression hides in noise; the comparison must be statistical with an effect-size threshold.
  2. Discovering incompatibility at runtime: the compatibility gate must be pre-deploy — a runtime "driver insufficient" is a fleet incident.
  3. Manual rollback: by the time a human reacts, the budget's gone; rollback on SLO burn must be automatic and fast (image-pinned).
  4. No semantic versioning: without major/minor/patch discipline you can't reason about compatibility at all.

5. What this lab proves about you

You can build the release machinery that makes a GPU platform safe to ship continuously — compatibility-gated, canary-analyzed, auto-rollback — which is exactly the "full release cycle, not just a prototype" the JD demands, and the operational backbone of the Phase 12 capstone.