Lab 02 — Data / Concept Drift Detection & Production Monitoring

Phase: 17 — MLOps Platform Difficulty: ⭐⭐⭐☆☆ (the statistics are small; knowing which detector for which failure is ⭐⭐⭐⭐⭐) Time: 3–4 hours

A model ships at 94% accuracy and is silently at 71% three months later — not because the code broke, but because the world did: the input distribution moved (data drift) or the input→label relationship moved (concept drift), and nobody was watching. This lab builds the watcher. You implement the four detectors a real monitoring stack runs on every batch of production traffic — PSI (the headline drift number on the dashboard), the two-sample Kolmogorov-Smirnov statistic (distribution-free continuous drift), chi-square (categorical drift), and a rolling-window MetricMonitor (the pager) — plus the regression gate the deployment pipeline in lab 03 consults before it promotes anything. The soul of the lab is one property: PSI ≈ 0 when the distribution is stable and crosses the significant threshold when it clearly shifts — that single number is what wakes an on-call engineer at 2 a.m.

What you build

• bin_edges / histogram — the binning substrate. equal_width (simple, outlier sensitive) vs quantile (equal-mass, robust to skew — what PSI prefers), and a histogram that clamps out-of-range values into the edge bins so drifted data is counted, never dropped.
• population_stability_index — PSI = Σ (a%−e%)·ln(a%/e%) over bins, with an EPSILON floor so empty bins keep the log finite. The number every ML dashboard leads with.
• psi_alert — the three-band verdict: no_drift (<0.10), moderate_drift (0.10–0.25), significant_drift (≥0.25).
• ks_statistic — two-sample KS D = max_x |F_a(x) − F_b(x)|, the distribution-free distance between two continuous samples (0 identical, →1 fully separated).
• chi_square_drift — Σ (O−E)²/E for categorical features, with expected rescaled to the actual total so it measures shape change, not sample size.
• MetricMonitor — a rolling-window guard: record, rolling_mean, is_alerting (mean crosses threshold, above/below mode), history. Mean-not-spike is what keeps it from paging on noise.
• detect_regression — the gate: is the candidate worse than what we already ship, beyond an absolute tolerance? Both directions (higher-is-better, lower-is-better).

Key concepts

Files

Run

pip install -r requirements.txt
pytest test_lab.py -v                       # against your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # against the reference
python solution.py                          # the worked example

Success criteria

• All tests pass against your implementation (and LAB_MODULE=solution).
• You can explain why test_psi_near_zero_for_identical_distribution and test_psi_crosses_significant_for_shifted_distribution are the soul of the lab — and what real-world failure each band maps to.
• You can compute PSI on paper for two 2-bin histograms and explain the EPSILON floor.
• You can state, for a given feature, whether PSI/KS (data drift) or a MetricMonitor on accuracy (concept drift) is the right detector — and why a stable PSI does not prove the model is fine.
• You can explain why chi_square_drift rescales expected to the actual total, and what bug you'd ship if it didn't.
• You can explain why the monitor fires on the rolling mean, and why the detect_regression tolerance boundary is </> and not <=/>=.

How this maps to the real stack

Limits of the miniature (say these in the interview, don't let them be exposed): PSI/KS/chi-square detect that inputs moved — they are blind to concept drift where inputs look identical but the right answer changed; for that you need labels (often delayed) and a MetricMonitor on a true performance metric, or a proxy like NannyML's performance estimation. PSI's 0.1/0.25 thresholds are conventions, not laws — calibrate per feature. KS gets over-powered at very large n (everything looks "significant"), so production tools report effect size, not just a p-value. Drift detection tells you something changed, not what to do — the runbook (retrain, roll back, alert a human) is the hard part this lab deliberately leaves to you and lab 03.

Extensions (build these toward a real monitor)

• Add PSI p-value / confidence via a bootstrap over the baseline (resample, recompute PSI, get the null band) so you can say "this PSI is beyond the 99th percentile of stable" instead of trusting a fixed cutoff.
• Add per-feature drift over a dict[str, list[float]] and emit a ranked drift report (the Evidently DataDriftTable shape).
• Add a Jensen-Shannon divergence detector and compare its sensitivity to PSI on the same shift.
• Add delayed-label concept-drift detection: buffer predictions, join late-arriving labels, and run the MetricMonitor on realized accuracy — the part that catches the failures PSI can't.
• Wire detect_regression to read two MLflow runs and gate a promotion, closing the loop with lab 03.

Interview / resume

• Talking points: "A model's accuracy is dropping but the inputs look unchanged — what's happening and how do you detect it?" (concept drift; you need labels, PSI won't see it). "What does PSI = 0.3 mean and what do you do?" "KS vs chi-square — when each?" "Why does your drift monitor use the rolling mean and not the last batch?"
• Resume bullet: Built a production drift-monitoring toolkit — PSI with banded alerting, two-sample KS and chi-square distribution tests, a rolling-window metric monitor, and a tolerance-aware regression gate — that distinguishes data drift from concept drift and feeds an automated promotion pipeline.