Lab 03 — CI/CD Deployment Pipeline: Eval Gates, Canary & Rollback

Phase: 17 — MLOps Platform Difficulty: ⭐⭐⭐☆☆ (the control flow is small; the invariant — never ship a worse model — is ⭐⭐⭐⭐⭐) Time: 3–4 hours

The most expensive outage in ML is the one you cause yourself: a new model passes a notebook eval, gets pushed to 100% of traffic, and quietly tanks revenue for six hours before anyone notices. Safe deployment is the discipline that makes that impossible — gate on offline quality, canary a tiny slice of real traffic, watch its health, and roll back instantly if it misbehaves. This lab builds that machine from primitives: an EvalGate (min/max thresholds + no-regression-vs-champion), a CanaryController that ramps 5% → 25% → 50% → 100% and freezes traffic the moment a stage is unhealthy, rollback / blue_green_swap recovery, and a DeploymentPipeline orchestrator. Two invariants are the soul of the lab: the canary aborts and stays at the breaching stage when a health check fails midway, and the pipeline halts at the first failed gate and NEVER promotes a failing candidate.

What you build

• EvalGate — the offline bar. evaluate(metrics) -> GateResult checks min thresholds (accuracy>=, safety>=), max thresholds (latency_ms<=, cost<=), and a no-regression check vs the currently-shipping baseline (with absolute tolerance and lower_is_better direction). GateResult carries passed plus a per-criterion reasons map so a failure tells you exactly which bar it missed.
• CanaryController — progressive traffic shifting over stages = [5, 25, 50, 100]. step(health_metrics) advances if healthy, else aborts and freezes at the last good traffic. .status ∈ {in_progress, promoted, aborted}, .current_traffic.
• rollback / blue_green_swap — restore a previous version (no mutation); swap blue↔green only if the candidate is healthy, else keep active.
• DeploymentPipeline — ordered gated stages (eval_gate → canary → promote). run(candidate, context) -> PipelineResult records per-stage history, halts at the first failed gate, and rolls back if a failure occurs after the candidate already became active.

Key concepts

Files

Run

pip install -r requirements.txt
pytest test_lab.py -v                       # against your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # against the reference
python solution.py                          # the worked example

Success criteria

• All tests pass against your implementation (and LAB_MODULE=solution).
• You can explain why test_canary_aborts_and_freezes_at_breaching_stage and test_pipeline_halts_at_first_failed_gate_and_does_not_promote are the soul of the lab — and what production disaster each prevents.
• You can explain why an offline EvalGate is necessary but not sufficient, and what the canary catches that the gate can't.
• You can articulate the no-regression check: why "passes thresholds" is weaker than "beats the champion", and the tolerance boundary semantics.
• You can trace, for a candidate that fails the eval gate, exactly which stages run (only eval_gate), what active ends up as (unchanged), and why.
• You can explain when the pipeline rolls back vs simply halts (only when it had already promoted).

How this maps to the real stack

Limits of the miniature (own these in the interview): real canaries run for minutes to hours with statistical analysis (is the canary's error rate significantly worse? sequential tests, not a single boolean), traffic splitting is done by a service mesh / ingress you don't control from Python, and health signals are often delayed (the business metric you care about lags the deploy). Promotion is rarely a clean function call — it's a kubectl/registry API mutation with its own failure modes, and a real rollback must drain in-flight requests and warm caches. This lab models the control logic and invariants (gate → canary → promote, abort-and-freeze, halt-and-don't-promote, roll-back-if-past-promote); the orchestration substrate (Argo, Flagger, KServe) and the statistics (the sequential health test) are the extensions.

Extensions (build these toward a real rollout)

• Replace the boolean health_check with a statistical canary analysis: compare canary vs baseline error rates with a two-proportion test (or reuse lab-02's MetricMonitor / detect_regression) and abort on a significant degradation.
• Add a time/observation budget per stage (e.g. require N healthy observations before advancing) instead of a single boolean.
• Add automatic rollback with drain: model in-flight requests and only complete the rollback once the old version is fully reinstated.
• Emit the pipeline run as a structured audit record (stage, decision, metrics, timestamp source = injected, not wall-clock) suitable for a deployment log.
• Wire the EvalGate no-regression check to read two MLflow runs and the canary to emit an Argo Rollouts-style step plan.

Interview / resume

• Talking points: "Walk me through how you ship a new model to production safely." "Your candidate beats the offline metrics but the canary's error rate is up 0.5% — what do you do?" "What's the difference between blue/green and canary, and when do you pick each?" "How does your pipeline guarantee you never promote a model that fails a gate?"
• Resume bullet: Built a model-deployment pipeline with offline eval gates (min/max/no-regression vs champion), a progressive canary controller with abort-and-freeze semantics, and blue/green + rollback recovery — encoding the invariants (halt at first failed gate, never promote a failing candidate) behind Argo Rollouts / KServe-style progressive delivery.