« Track Overview · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 26 — MLOps: Experiment Tracking, Model Registry, CI/CD & Drift Monitoring

Answers these JD lines: "Implement MLOps best practices including model versioning, monitoring, and CI/CD automation"; "Optimize model performance through ... continuous evaluation"; "Deploy and monitor ML models ... ensuring scalability, reliability" (JD1); "Model Testing & Validation ... Approve model sign-off packages"; "Performance Monitoring ... review daily/weekly drift reports and initiate retraining tickets when thresholds are breached" (JD3). This is the MLOps discipline layer — the lifecycle machinery around any model (classical ML or an LLM): how a training run becomes a tracked experiment, how a tracked experiment becomes a registered version, how a version earns its way into Production through a gate, and how a deployed model is watched for drift until the loop closes with retraining.

Why this phase exists

Everything else in this track treats the model as a given — you build loops, tools, retrieval, and platforms around an injected model. This phase asks the question none of those answer: where did the model come from, which one is actually serving, and how do you know it's still good? Software engineering has one axis of change (code); ML has three — code, data, and model — and a change on any axis can silently break the other two. That is why ML needs its own ops discipline, and why "we have CI" is not the same as "we have MLOps." Three ideas do most of the work:

1. Every model is a record, not a file. A training run is tracked (params, metric curves, artifacts, lineage to code + data); a model version points back at the run that made it. If you cannot answer "what produced the model in Production?", you don't have versioning, you have files.
2. Promotion is a gate, not a decision. A candidate reaches Production by provably beating the incumbent on a primary metric by a margin, regressing no guardrail metric, and passing required checks (fairness, latency budget) — a deterministic function CI can run, with rollback one call away. This is Phase 11's regression-gate discipline applied to the model artifact itself.
3. Deployment is the start of the job, not the end. The world drifts away from the training snapshot; you measure that gap continuously (PSI, KL divergence, KS, chi-square) across data, prediction, and concept drift, and a threshold breach opens a retraining ticket — continuous training (CT), the loop that only ML has.

This phase complements Phase 14 — Cost, Latency & Observability, it does not repeat it. Phase 14 watches the runtime of an agent system — tokens, dollars, latency, traces, "is the service healthy?" This phase watches the model lifecycle — versions, gates, distributions, "is the model still right?" A production system needs both dashboards, and they page different people.

The MLOps maturity model

The standard framing (Google's) — know where a team sits and what buys the next level:

The labs build the Level-2 machinery in miniature: tracking (Lab 01), registry + gate (Lab 02), drift → retraining trigger (Lab 03).

Concept map

• Experiment tracking — runs grouped under experiments; params, metric step history (learning curves), content-hashed artifacts, lineage tags (git sha, data version); search, compare, best-run selection (Lab 01).
• Model registry — versioned models with stages (None/Staging/Production/Archived), a transition state machine, a single-Production invariant, lineage to the training run (Lab 02).
• CI/CD/CT for ML — CI validates data + code + model; CD promotes through an evaluation gate (primary metric + margin, guardrail non-regression, required checks) with auto-archive and rollback; CT is retraining triggered by schedule or drift (Labs 02 + 03).
• Monitoring & drift — operational vs statistical monitoring; data vs prediction vs concept vs label drift; detectors (PSI, KL divergence, KS test, chi-square) with real thresholds; training-serving skew; cooldown/dedupe on triggers (Lab 03).
• Also: feature/data validation (schemas, ranges, expectations), reproducibility and determinism, and the tool landscape (MLflow, Weights & Biases, Kubeflow, SageMaker Pipelines, Vertex AI Model Monitoring, Evidently) — covered in the Warmup, not built as labs.

The labs

Integrated scenario (how this shows up at work)

Your churn model shipped eight months ago. Marketing changed the signup funnel in March; nobody told the ML team. The drift monitor (Lab 03) catches it first: the account_age feature's PSI crosses 0.2 in the weekly report, prediction drift follows a week later, and when the delayed churn labels arrive, the accuracy drop confirms concept drift — one deduped retraining ticket per signal, severity high, not three hundred pages. The retraining pipeline runs; the new run lands in the tracking store (Lab 01) with its params, learning curves, and a content-hashed model artifact, linked to the exact data snapshot. It's registered as churn v9 and staged. The promotion gate (Lab 02) compares it against the serving v7: +2.1 points on the primary AUC (clears the margin), latency within tolerance, fairness audit green — v7 auto-archives, v9 serves. Two days later a downstream bug surfaces; rollback restores v7 in one call while you investigate, and the registry's audit trail answers "who promoted what, when, and on what evidence" without a single Slack archaeology session. That whole loop — detect, retrain, track, gate, promote, roll back — is this phase, and it's what "MLOps best practices" means when a JD says it.

Deliverables checklist

• Lab 01 green under LAB_MODULE=solution pytest and your own lab.py (28 tests).
• Lab 02 green (26 tests); you can explain why a guardrail regression blocks promotion even when the primary metric improves.
• Lab 03 green (27 tests); you can write the PSI formula from memory and state the 0.1/0.2 bands.
• You can draw the Level 0 → 1 → 2 maturity model and say what each level adds.
• You can name the four drift types and which ones need labels.
• You can explain how this phase differs from Phase 14 in one sentence.

Key takeaways

• ML changes on three axes — code, data, model — and each needs versioning, testing, and monitoring. App CI covers one of the three; that gap is the entire reason MLOps exists.
• Tracking makes training queryable; the registry makes deployment governable. A run answers "what happened"; a version + stage answers "what's live and how it got there."
• The promotion gate is the sign-off package: beat the incumbent by a margin, regress no guardrail, pass required checks — deterministic, testable, reversible via rollback.
• Drift is measured, not noticed. PSI/KL/KS/chi-square against a reference, on a schedule, with thresholds — and a breach opens one ticket, because dedupe/cooldown is what separates monitoring from noise.
• CT — continuous training — is the loop that makes it MLOps: monitoring feeds retraining feeds tracking feeds the registry feeds the gate feeds serving feeds monitoring.
• The senior framing: "Phase 14 tells me the service is healthy; this phase tells me the model is still right. Different failure modes, different dashboards, both mandatory."