Lab 02 — Model Registry & CI Promotion Gate

Phase 26 · Lab 02 · Phase README · Warmup

The problem

Lab 01's tracking store tells you what each run produced. Now you need to answer the operational question: which trained model is actually serving traffic, which is being tested, and how did it get there — safely? That is the job of a model registry: versioned models, each in a stage (None → Staging → Production → Archived), with the transitions between them governed by rules, and lineage back to the run that trained each version.

The part that turns "a registry" into "MLOps" is the promotion gate. In a mature program a model does not reach Production because someone clicked "deploy". It reaches Production because it provably beat the incumbent on a primary metric by a required margin, did not regress any guardrail metric beyond tolerance, and passed every required check (a bias/fairness audit, a latency budget). This is exactly the JD's "approve model sign-off packages" and "block promotion on metric regression" — and it is a deterministic function you can unit-test, which is what you build here.

What you build

File map

Run it

pytest test_lab.py -v                       # your lab.py (red until you implement)
LAB_MODULE=solution pytest test_lab.py -v   # the reference (green)
python solution.py                          # worked example

Success criteria

• register_model versions auto-increment per name; a new version starts in None.
• transition_stage enforces ALLOWED_TRANSITIONS; an illegal move raises InvalidTransitionError; a second Production without archive_existing=True raises.
• Promoting a new Production version auto-archives the incumbent (single-Production invariant).
• The gate blocks a candidate that regresses the primary metric (stays Staging, prod unchanged) and promotes one that beats it by the margin (old prod archived).
• A guardrail regression beyond tolerance blocks promotion even when the primary metric improves.
• A missing or false required check blocks promotion.
• The first model ever (no incumbent) passes the primary check and is promoted.
• rollback restores the prior Production model and archives the rolled-back one.
• evaluate is pure — it never mutates the registry.
• All 26 tests pass under both lab and solution.

How this maps to the real stack

• MLflow Model Registry has exactly these stages — historically None/Staging/Production/ Archived via transition_model_version_stage (and, in newer MLflow, the more general aliases + tags model, e.g. a champion alias you move between versions). Our transition_stage is that call; our single-Production invariant is the discipline teams wrap around it, and MLflow's archive_existing_versions=True flag is our archive_existing.
• SageMaker Model Registry models this as a Model Package Group with versioned Model Packages whose ModelApprovalStatus is PendingManualApproval → Approved / Rejected; a CI pipeline (SageMaker Pipelines / EventBridge) reacts to Approved and deploys. Our PromotionGate.promote is that approval-plus-deploy step, made into a testable function.
• Vertex AI Model Registry and W&B Model Registry offer the same version + alias/stage + lineage model; W&B's model-registry uses aliases like production you move between artifact versions, which is structurally our get_production + transition_stage.
• The gate itself is what a real CD pipeline runs in a "gate" job before the deploy step: pull the candidate's eval metrics (from the tracking store, Lab 01), pull the current Production model's metrics, assert improvement-with-margin, assert no guardrail regression, assert the required audits (fairness, latency) are green — and only then flip the alias / approval status. Cross-reference the eval side of this in Phase 11 — Agent Evaluation, Judge & Regression Gates: that phase builds the evaluation that produces the numbers; this gate consumes them to make the promote/block decision.

Limits of the miniature. The registry is in-memory with no auth, no audit log persistence, no concurrent writers — a production registry is a service with RBAC (who may approve?), an immutable audit trail, and webhook/event hooks that trigger the deploy. Our metrics are a static snapshot; real gates often re-run evaluation on a held-out set at gate time rather than trusting a stored number. Rollback here reverses one promotion; a real rollback also has to redeploy the previous artifact and drain in-flight traffic, which is a serving-layer concern this lab doesn't model.

Extensions (your own machine)

• Add an approval identity (approve(version, approver, note)) and an append-only audit log, so the GateResult plus the approver becomes a real sign-off package.
• Make the gate re-run a supplied evaluation function against a held-out dataset at gate time instead of trusting stored metrics — the "don't trust the number, recompute it" hardening.
• Add a canary stage between Staging and Production and a two-step gate (canary metrics must hold for N windows before full promotion), tying into Lab 03's monitoring.
• Wire the registry to Lab 01: register_model pulls metrics directly from a TrackingStore run by run_id, so lineage is a real link, not a copied dict.

Interview / resume signal

"Built a model registry with a stage machine (None/Staging/Production/Archived), a single-Production invariant with auto-archive on promote, and rollback — plus a configurable CI promotion gate that blocks a candidate unless it beats the incumbent on a primary metric by a margin, regresses no guardrail metric beyond tolerance, and passes required fairness/latency checks, emitting a structured sign-off package of exactly which checks passed and why. It's the deterministic core of 'approve model sign-off packages' and 'block promotion on metric regression.'"