Phase 17 — MLOps: The ML Platform (Tracking, Registry, Drift & CI/CD)

The phase where a model stops being a notebook artifact and becomes a production asset with a lifecycle. Software ships once and runs the same until you change it; an ML system rots in place while the code is untouched, because the world it learned drifts away underneath it. This phase builds the platform that makes ML operable: you can answer what model is in production, what data and params produced it, is it still good, and how do I ship the next one safely — on a whiteboard and in code. It is the difference between "we trained a model" and "we run a model business."

Why this phase exists

Every other phase in this track makes a model better — attention, fine-tuning, quantization, serving, agents, evals. This phase makes a model operable, and that is a different discipline. The famous result (Sculley et al., "Hidden Technical Debt in Machine Learning Systems") is that the ML code is a tiny box in the middle of a huge diagram of plumbing — configuration, data collection, feature extraction, serving infrastructure, monitoring, process management. MLOps is that plumbing, and senior AI engineers are hired to own it. Three facts make ML systems operationally unlike software:

Behavior is set by data, not just code. A model is code + weights + the data that made the weights. Reproducing a result needs all three versioned. This is why "it works on my machine" becomes "it worked on my data snapshot from three weeks ago."
They fail silently. A bug throws an exception; a stale model just gets quietly, expensively wrong — accuracy decays as the input distribution drifts, and nothing alarms because the service is still returning 200s. You must monitor the predictions, not just the uptime.
Training and serving are two different code paths that compute features two different ways, so they silently disagree — training-serving skew — and your offline AUC lies to you.

The platform answers these with experiment tracking + a model registry (know what you have and what made it), drift + performance monitoring (know when it's rotting), and CI/CD with eval gates (ship the next one without breaking prod). That is this phase, and those are its three labs.

Concept map — the ML lifecycle as a loop, not a line

                            ┌──────────────────────────────────────────────┐
                            │  MLOps = make this LOOP automated & observable │
                            └──────────────────────────────────────────────┘

   ┌────────┐   ┌─────────┐   ┌──────────┐   ┌──────────┐   ┌─────────┐   ┌───────────┐
   │  DATA  │──▶│  TRAIN  │──▶│  TRACK   │──▶│ REGISTER │──▶│ DEPLOY  │──▶│  MONITOR  │
   │ +feat. │   │ (sweep) │   │ runs:    │   │ versions │   │ canary  │   │ drift +   │
   │ store  │   │         │   │ params/  │   │ + stages │   │ +rollbk │   │ perf decay│
   └────────┘   └─────────┘   │ metrics/ │   │ + lineage│   └─────────┘   └───────────┘
       ▲                      │ artifacts│        │             ▲              │
       │                      └──────────┘   ┌────┴─────┐       │              │
       │                       (LAB 01)      │eval gate │   (LAB 03)       (LAB 02)
       │                                      │tests=evals│                    │
       │                                      └──────────┘                     │
       │                                                                       ▼
       │                          ┌───────────────────────────┐    drift/decay detected →
       └──────────────────────────│  RETRAIN  (feedback loop)  │◀───  trigger retrain
                                   └───────────────────────────┘
   Versioned everywhere:  CODE (git)  +  DATA (DVC/Delta)  +  MODEL (registry)

The loop is the whole idea. Software is a line (build → ship → done). ML is a closed loop that must keep turning, because the data keeps moving. Each lab owns one arc of it.

The labs

Lab	You build	Difficulty	Time
lab-01 — MLflow-style Tracking & Model Registry	a deterministic in-memory MLflow: runs (params/metrics-with-step-history/artifacts/tags), `search_runs`/`best_run`, and a model registry with auto-incrementing versions, a stage state machine (`None→Staging→Production→Archived`), atomic promote-and-archive, and version→run lineage	⭐⭐⭐☆☆	3–4 h
lab-02 — Drift Detection & Production Monitoring	the monitoring half: PSI (Population Stability Index) and the two-sample KS statistic over feature distributions, a binned reference-vs-live comparator, a performance-decay tracker on delayed labels, and an alerting rule that fires a retrain trigger when drift or decay crosses a threshold	⭐⭐⭐⭐☆	4–5 h
lab-03 — CI/CD Deployment Pipeline: Eval Gates, Canary & Rollback	the release half: a pipeline DAG (validate → eval-gate → canary → promote/rollback), an eval gate that blocks promotion unless the candidate clears a metric bar vs the incumbent ("tests are evals"), a canary that shifts traffic in steps and watches a guard metric, and an automatic rollback on guard breach	⭐⭐⭐⭐⭐	4–6 h

Each lab is a runnable, test-verified miniature — see the lab standard. Run it red (pytest test_lab.py), make it green, then read solution.py. All three are pure stdlib + pytest, offline, and deterministic (no real mlflow, no wall-clock, no network).

Integrated scenario ideas

The 2 a.m. "which model is in prod?" drill. Use lab-01's registry + lineage to answer, in ten seconds, what version is live, what run produced it, and what params/data it saw — then roll back to the previous Production version. This is the single most common MLOps fire.
Close the loop. Wire lab-02's drift alert to lab-01's "register a new version" and lab-03's "promote behind an eval gate + canary." Show the full data→train→track→register→deploy→monitor →retrain loop turning once, end to end, deterministically.
Catch a silent regression. Stand up lab-03's eval gate so a candidate that scores higher on one benchmark but lower on a slice (or a safety metric from Phase 16) is blocked from Production — "tests are evals" made real.
Defend tracking+registry over a platform purchase. Argue (with lab-01) that the boring core — reproducible runs and a registry — delivers 80% of MLOps value before any vendor platform, and name the maturity level you actually need.

Anti-patterns this phase kills

These are the recurring failures that "we have a model in production" papers over — each lab exists to make one of them impossible:

"Which model is in prod?" — an unversioned .pkl with no provenance, no history, no rollback. Killed by lab-01's registry + lineage.
The green-dashboard money-leak — uptime and latency are fine, accuracy has been quietly falling for a month because labels are delayed. Killed by lab-02's input-drift monitoring as a leading indicator.
The "it was better on the benchmark" regression — a candidate that wins overall but tanks a high-value slice ships on vibes. Killed by lab-03's eval gate ("tests are evals").
Training-serving skew — offline AUC is great, production is garbage, because the two feature paths disagree. Named here (Chapter 8 of the WARMUP) and fixed with a feature store when it bites.
Irreproducible results — you can't rebuild last quarter's model because the data leg was never versioned. Killed by versioning the triple (code + data + model) the whole phase assumes.

Deliverables checklist

lab-01 passes pytest test_lab.py -v (and LAB_MODULE=solution does too).
You can explain why params are immutable, metrics are a time-series, tags are mutable.
You can draw the registry stage state machine and name the illegal edges and why.
You can explain what archive_existing=True guarantees and why it's the registry soul test.
You can compute a PSI and a KS statistic by hand and say what each detects (lab-02).
You can describe an eval gate + canary + rollback release and what each defends against.
You can name training-serving skew and the role of a feature store in killing it.
You can place a real org on the MLOps maturity model (manual → automated → CI/CD/CT).

Key takeaways

ML systems rot in place. Code unchanged, accuracy falling, no exception thrown — because the data drifted. The job is to make that visible (monitoring) and recoverable (registry + rollback), not to assume "trained = done."
Reproducibility is a triple: code + data + model, all versioned, or you can't reproduce a result, debug a regression, or pass an audit. The registry + lineage is how you keep the triple.
The boring core beats the shiny platform. Experiment tracking and a model registry are ~80% of MLOps value; resist the resume-driven tool sprawl until you've earned the complexity.
Tests are evals. CI for ML can't be "the code runs" — it must be "the model still clears the quality and safety bar," gated before Production, with a canary and an automatic rollback.
Monitor predictions, not just uptime. A green dashboard with a drifting model is the most expensive kind of "working." Drift and performance-decay metrics are your real SLOs.

Next: Phase 18 — C++/CUDA & GPU Performance Engineering.

Senior AI Engineer — LLMs, VLMs & Agentic Systems