Lab 01 — Drift Monitor & Retraining Policy

Phase: 12 — Observability, Drift & Governance | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–5 hours

Deployment is the midpoint of a model's life. This lab builds what keeps shipped models honest: PSI/KS detection with attribution, a retraining policy that's a state machine instead of a vibe, and a model card generated from the live system.

What you build

  • psi — reference-quantile binning (the implementation detail that decides whether you can see shifts at all — a test plants the case that current-quantile binning hides), empty-bin smoothing, the standard thresholds
  • ks_statistic — two-sample D, validated against scipy to 1e-9
  • DriftMonitor.report — per-feature PSI+KS, importance-weighted ranking (drift in the top feature outranks drift in a minor one), prediction-distribution PSI as a first-class trigger
  • RetrainingPolicy — the FSM HEALTHY → RETRAINING → VALIDATING → PROMOTING, with two guards the tests enforce: broken data preempts retraining (laundering guard) and failed validation lands in INCIDENT (no auto-retry loop); append-only history as the audit trail
  • model_card — every governance section, generated from live monitor + policy state, not hand-maintained prose

Reference demo

feature drift (ranked):
  amount     PSI=0.499 KS=0.287 [action] weighted=0.299
  age_days   PSI=0.001 KS=0.007 [stable] weighted=0.000
prediction PSI: 0.180  any_breach=True

policy history:
  healthy -> retraining: drift breach on amount
  retraining -> validating: challenger trained
  validating -> promoting: challenger 0.8730 >= incumbent 0.8610
  promoting -> healthy: registry pointer moved

Run

pip install -r requirements.txt
pytest test_lab.py -v                       # your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # reference (17 tests)
python solution.py

Suggested TODO order

  1. psi — the reference-quantile test is the one to satisfy first
  2. ks_statistic — scipy agreement pins it
  3. FeatureDrift.level / DriftReport properties / DriftMonitor.report
  4. RetrainingPolicy — happy path, then the two guard tests
  5. model_card — the section checklist test

Success criteria

  • All 17 tests pass
  • You can explain why PSI bins on reference quantiles and what breaks otherwise
  • You can defend both FSM guards in one sentence each
  • You can state what drift detection does NOT tell you (impact), and how importance-weighting partially compensates

Extensions

  • Categorical PSI (chi-square / share-based with smoothing) for string features
  • Window management: rolling reference vs fixed reference, and when each is right
  • The label-delay simulator (phase README Extension A)
  • Per-slice drift: the same report computed per segment — drift that only hits one country is invisible in the aggregate

Interview Q&A

Q: PSI on your top feature is 0.4 but the model's trailing AUC is unchanged. What do you do? Investigate before retraining: PSI measures distribution movement, not damage. Check whether the shift is in a region where the model's prediction is flat (no impact), whether it's a data-quality artifact (null spike, unit change — contract layer), and what the prediction distribution did (unchanged predictions + shifted feature = likely benign or compensated). If genuinely benign, recalibrate the threshold for that feature; alert fatigue from impact-free PSI alarms is how monitoring dies.

Q: Why must the retraining validation gate compare against the incumbent on current data rather than the original test set? The original test set measures the old world — both models can score identically there while differing where it matters: the drifted region that triggered retraining. The decision being made is "which model serves today's traffic better," so the gate evaluates on the freshest labeled window (with the label-delay caveat documented). The original test set is still useful as a no-regression check on stable segments.

References

  • Rabanser et al., Failing Loudly (NeurIPS 2019) — what shift detectors actually detect
  • Evidently AI — the industrial version; map each lab piece to its reports
  • Mitchell et al., Model Cards for Model Reporting (2019)
  • Gama et al., A Survey on Concept Drift Adaptation (2014)