Hitchhiker's Guide — MLOps: Tracking, Registry, Drift & CI/CD

The compressed practitioner tour. If WARMUP.md is the professor, this is the senior who leans over and says "here's what you actually need to remember about running models in prod."

The 30-second mental model

A model is a perishable asset on a closed loop, not a binary you ship once. The loop: data → train → track → register → deploy → monitor → (drift) → retrain. MLOps is making that loop automated (it turns without heroics) and observable (you know which part is on fire). ML systems rot differently from software for three reasons: behavior is set by data (version code+data+model), they fail silently (monitor predictions, not uptime), and train/serve paths skew. The boring core — experiment tracking + a model registry — is ~80% of the value. Add monitoring when models hit prod, CI/CD/canary when releases get scary, a feature store only when skew bites.

The numbers / facts to tattoo on your arm

The commands / one-liners to keep in muscle memory

import mlflow
from mlflow import MlflowClient

mlflow.set_experiment("fraud-classifier")
with mlflow.start_run() as run:                     # a Run, auto-ended
    mlflow.log_param("lr", 0.05)                    # immutable
    mlflow.log_metric("val_auc", 0.91, step=3)      # step history
    mlflow.log_artifact("cm.png")                   # blob → artifact store
    mlflow.sklearn.log_model(model, "model")        # writes an MLmodel flavor

# register + promote, demoting the incumbent (the rollback-able move)
mv = mlflow.register_model(f"runs:/{run.info.run_id}/model", "fraud-model")
MlflowClient().transition_model_version_stage(
    "fraud-model", mv.version, "Production", archive_existing_versions=True)

# "what's in prod and what made it?"  (the 2 a.m. query)
prod = MlflowClient().get_latest_versions("fraud-model", stages=["Production"])[0]
src  = MlflowClient().get_run(prod.run_id)          # params/metrics lineage

# best run by a metric (the sweep payoff)
MlflowClient().search_runs(exp_id, order_by=["metrics.val_auc DESC"])[0]

mlflow ui                                           # the tracking UI (local)
mlflow server --backend-store-uri sqlite:///mlflow.db \
              --default-artifact-root ./mlruns      # the two-store split, explicit
mlflow models serve -m "models:/fraud-model/Production"   # serve the live version
dvc add data/train.parquet && git add data/train.parquet.dvc   # version the data leg

The tool cheat-table (what each is for)

War stories

• "We have no idea which model is in prod." The single most common MLOps fire. A model was serving bad predictions; nobody could say which version, what data trained it, or how to revert — because "the model" was a .pkl in S3 with no registry. The fix wasn't a platform; it was a registry + lineage. Build the boring core first.
• The silent decay. Service green, latency fine, revenue quietly down 8%. Labels were delayed three weeks, so accuracy looked fine until they landed. Input-distribution PSI had been screaming for a month — nobody was watching it. Monitor P(X) as the leading indicator.
• The training-serving skew that "passed every test." Offline AUC 0.92, prod garbage. The serving feature pipeline filled nulls with 0; training filled with the mean. Same model, different inputs. A feature store with one definition would have killed it. CACE in the flesh.
• The resume-driven platform. A two-engineer team stood up Kubeflow, a feature store, and three trackers before their second model. They spent six months operating tooling instead of shipping. Maturity is automation of the loop, not the size of the tool shelf.
• The eval gate that earned its keep. A "better" candidate (higher overall AUC) regressed badly on a small high-value customer slice. The eval gate blocked the promotion in CI. Without it, that ships Friday and pages you Saturday.

Vocabulary (rapid-fire)

• Run — one tracked training/eval trial (params + metrics + artifacts + tags).
• Lineage — version → run → the params/data/code that produced it.
• Stage / alias — registry lifecycle position (Production) / a named pointer (@champion).
• Drift — data (P(X)), concept (P(y|X)), or prediction (output) distribution shift.
• PSI / KS — drift detectors: bin-divergence / max-CDF-gap.
• Training-serving skew — train vs serve feature paths disagree.
• Point-in-time join — fetch a feature's value as of an event's timestamp (no leakage).
• Eval gate — block promotion unless the candidate clears the bar ("tests are evals").
• Canary — release to a small traffic slice first, watch a guard metric, then ramp.
• CT — continuous training: the loop automated end to end.
• Backend / artifact store — the DB (metrics) vs object store (blobs) split.

Beginner mistakes

• Treating "trained = done." Models rot; the loop must keep turning.
• Monitoring uptime/latency but not predictions/drift — the green-dashboard money-leak.
• Logging to a CSV/TensorBoard instead of a queryable, shared tracking store with a registry.
• No data versioning → "reproducible" results you can't reproduce (missing the data leg).
• Letting "the production model" be an unversioned .pkl with no lineage or rollback.
• CI that checks "the script runs" instead of "the model clears the eval bar."
• Adopting a feature store / Kubeflow before you've shipped a second model (tool sprawl).
• Forgetting labels are delayed — waiting on accuracy to drop instead of watching input drift.

The one thing to take away

Before you call a model "in production," you must be able to answer four questions in seconds: which version is live, what produced it, is it still good, and how do I roll it back. Tracking + registry + lineage answers the first two, drift monitoring the third, the registry the fourth. Build that boring core before anything shiny — it's 80% of MLOps and 100% of the 2 a.m. value.