Phase 12 — Observability, Drift & Governance
Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 2 weeks (40–50 hours) Roles Supported: the JD's "observability … drift detection, retraining strategy, and responsible AI controls" and "model risk assessment"
Why This Phase Exists
Deployment is the midpoint of a model's life, not the end. The world drifts away from the training distribution, labels arrive late or never, silent feature breakage masquerades as gradual degradation, and someone — increasingly, a regulator — asks "who approved this model and what are its known limitations?" This phase builds the machinery that keeps shipped models honest: drift detection with statistical teeth, a retraining policy that's a state machine rather than a vibe, and the governance artifacts (model cards, risk tiers) that make ML auditable.
The flagship lab implements all three layers; the WARMUP supplies the monitoring taxonomy, the drift mathematics, and the responsible-AI frameworks the lab encodes.
Concepts
- The monitoring stack: system health → data quality → drift → model quality → business KPIs; why each layer catches what the previous can't
- Covariate / prior / concept drift — definitions, causes, and which monitors see which
- PSI (population stability index) and KS test — the workhorse detectors; binning discipline; thresholds and their abuse
- Proxy metrics under label delay: prediction-distribution monitoring, autoencoder/ density scores, delayed-label backfill joins
- Retraining strategy as policy: triggers (scheduled / drift / performance), validation gates, and the retraining-on-drifted-data trap
- Model risk: tiering by impact, model cards, fairness slices, human override paths
- Responsible AI in practice: documentation, monitoring of protected-group performance, the EU-AI-Act-shaped compliance reality
Labs
Lab 01 — Drift Monitor & Retraining Policy (flagship, implemented)
| Field | Value |
|---|---|
| Goal | Build the post-deployment layer: PSI + KS drift detectors validated on synthetic shifts, a per-feature drift report, a retraining policy state machine (healthy → drifting → retraining → validating, with gates), and a model-card/risk-report generator |
| Concepts | PSI math, KS statistics, binning, policy FSM with guarded transitions, model cards |
| How to Test | pytest test_lab.py — 15 tests: PSI hand-computed, detector ROC on planted shifts vs no-shift, report ranking, FSM transitions incl. the validation-failure path, card completeness |
| Talking Points | Why PSI bins on the reference quantiles? What does drift detection NOT tell you (impact)? Why must retraining validation compare against the incumbent on current data? |
| Resume Bullet | Built model observability: PSI/KS drift detection, per-feature drift attribution, and a gated retraining policy engine — caught a silent upstream feature break in staging via reference-quantile PSI before it reached production metrics |
→ Lab folder: lab-01-drift-monitor/
Extension Project A — Label-Delay Simulator (spec)
Simulate a fraud stream where labels arrive 30 days late; compare detection lag of (a) prediction-distribution drift, (b) PSI on features, (c) delayed ground-truth metrics; produce the "what monitoring buys you" timeline chart.
Extension Project B — Fairness Slice Monitor (spec)
Add protected-attribute slicing to the Phase 03 model's evaluation: per-slice precision/recall, demographic parity and equalized-odds gaps, with the small-slice-variance caveats handled (CIs per slice); wire as a Phase 08 CI gate.
Guides in This Phase
- WARMUP.md — the full zero-to-expert guide
Deliverables Checklist
- Lab 01 implemented; all 15 tests pass
- Extension A timeline chart
- Extension B fairness gate