Lab 03 — Drift Detection & the Retraining Trigger

Phase 26 · Lab 03 · Phase README · Warmup

The problem

A model is trained once, on a snapshot of the world — and then deployed into a world that keeps moving. Customer demographics shift, an upstream service changes a field's units, a competitor launches and your users start behaving differently. None of these throws an exception. The service returns 200s, latency is fine, and the model is quietly, increasingly wrong. That gap between the training distribution and live traffic is drift, and it is the reason the JD says "review daily/weekly drift reports and initiate retraining tickets when thresholds are breached."

Drift monitoring is a statistics problem with an ops problem stapled to it. The statistics: compare the live window's distribution against a reference (the training data) with a score — PSI (the industry default) or KL divergence — over deterministically binned features. The ops: not every wiggle deserves a retraining ticket. You need thresholds (PSI's classic 0.1/0.2 bands), severity, and a cooldown/dedupe so a drift that persists for two weeks produces one ticket, not one ticket per monitoring window paging someone every ten minutes.

What you build

Piece	What it does	The lesson
`bin_index` / `histogram`	deterministic binning against fixed edges	drift scores are computed over bins, and the bin edges are part of the monitoring contract
`normalize`	counts → smoothed probability distribution	Laplace smoothing: a zero bin would make `ln` explode — every real drift tool does this
`psi`	`Σ (p_live − p_ref) · ln(p_live / p_ref)`	≥ 0, zero iff identical, monotone in shift size; the `0.1`/`0.2` rule-of-thumb bands
`kl_divergence`	`Σ p_live · ln(p_live / p_ref)`	the information-theoretic cousin — asymmetric, same "distance from reference" idea
data drift	PSI per numeric feature vs the reference histogram	inputs changed — the earliest, cheapest warning, no labels needed
prediction drift	PSI over the model's output distribution	outputs changed — still label-free, closer to business impact
concept-drift proxy	accuracy drop on labeled feedback vs a baseline	the input→output relationship changed — needs labels, measures what you actually care about
`DriftMonitor` + `RetrainingTrigger`	thresholds → breach → ticket, with per-signal cooldown	closing the loop is the point; dedupe is what keeps it humane

File map

File	Role
`lab.py`	your implementation (fill the `# TODO`s)
`solution.py`	reference implementation + worked example (`python solution.py`)
`test_lab.py`	27 tests: binning, PSI properties, KL, all three drift types, cooldown, thresholds, determinism
`requirements.txt`	`pytest` only

Run it

pytest test_lab.py -v                       # your lab.py (red until you implement)
LAB_MODULE=solution pytest test_lab.py -v   # the reference (green)
python solution.py                          # worked example

Success criteria

PSI is ≈ 0 on identical distributions, < 0.1 on near-identical ones, > 0.2 on a real shift, and monotone: a bigger shift always scores a bigger PSI.
normalize returns a proper distribution (sums to 1) with no zero entries.
kl_divergence is 0 on identical inputs, positive on shifted ones; both statistics raise ValueError on mismatched bin counts.
The monitor reports no drift (and emits no ticket) on same-shape traffic and detects a shifted feature, a flipped prediction mix, and an accuracy drop on labeled feedback.
A breached signal emits a RetrainingTrigger with the signal name, severity, and window index — and the cooldown suppresses duplicate tickets while the drift persists, per signal (a different signal still fires during another's cooldown).
Changing a threshold changes the breach outcome without changing the score.
Trigger ids are sequential (RT-1, RT-2, ...); the same window sequence always produces the same reports.
All 27 tests pass under both lab and solution.

How this maps to the real stack

Evidently (the open-source standard for drift reports) computes exactly this: per-column drift with PSI, KL divergence, Kolmogorov–Smirnov (numeric), and chi-square (categorical) tests against a reference dataset, rolled into a report with per-feature drift flags and a dataset-level "share of drifted columns" — our DriftReport with its DriftSignal list is that report's skeleton. Evidently's default test selection varies by column type and size; the PSI path you built is its workhorse for tabular numeric features.
Vertex AI Model Monitoring runs training-serving skew detection (live vs training data) and prediction drift detection (live vs a previous window) on a schedule, with per-feature distance thresholds — categorical features use L-infinity or chi-square distance, numeric use Jensen-Shannon divergence; breaches raise alerts into Cloud Monitoring. Our psi_threshold-per-signal configuration is that per-feature threshold map.
SageMaker Model Monitor captures live inference data to S3, compares each window against a baseline (statistics + constraints JSON generated from the training set — our reference histograms), and emits CloudWatch violations on breach; teams wire those violations to an EventBridge rule that opens a ticket or starts a SageMaker Pipelines retraining execution — literally our RetrainingTrigger, with the cooldown implemented as alarm deduplication.
The three drift types are an industry-standard triage ladder: data drift (inputs moved — cheap, label-free, earliest), prediction drift (outputs moved — label-free, closer to impact), concept drift (the relationship moved — needs labels or delayed feedback, and is the one that directly measures model damage). Real programs monitor all three because each catches failures the others miss: a feature can drift without hurting accuracy (benign covariate shift), and accuracy can collapse with zero input drift (pure concept drift — e.g. fraud patterns changing behavior for the same-looking transactions).

Limits of the miniature. Real systems bin numeric features by reference quantiles (equal-mass bins), not hand-chosen edges — quantile binning makes PSI less sensitive to outliers; our fixed edges keep the math inspectable. We omit the KS test (needs sorting and a supremum over the empirical CDFs) and chi-square (a significance test with degrees of freedom) — the Warmup derives both, and the extension below builds KS. Real label feedback arrives late (fraud labels take weeks), so production concept-drift monitoring is always lagged; our synchronous labeled_feedback window hides that delay. And a real trigger opens a Jira ticket / starts a pipeline run; ours returns a dataclass, which is exactly the seam where that side effect would be injected.

Extensions (your own machine)

Add a KS test: sort both samples, walk the merged values computing the max gap between empirical CDFs, and compare against the critical value c(α)·sqrt((n+m)/(n·m)).
Add quantile binning: compute reference deciles and use them as edges, then show PSI over quantile bins is less outlier-sensitive than fixed-width bins.
Add a windowed baseline update policy: after a retraining trigger is resolved, re-baseline the monitor from the new training set — and prove the same traffic no longer drifts.
Wire the trigger to Lab 02: a RetrainingTrigger starts a "retrain" that registers a new model version, which must then pass the promotion gate — the full closed loop of this phase.

Interview / resume signal

"Built a drift monitor from first principles: PSI and KL divergence over deterministically binned features (with Laplace smoothing and the 0.1/0.2 PSI bands), covering data drift, prediction drift, and a concept-drift proxy via accuracy drop on labeled feedback — feeding a retraining trigger with per-signal severity and a cooldown/dedupe so sustained drift opens one ticket instead of paging every window. Same architecture as Evidently reports and SageMaker Model Monitor baselines, with the mechanism visible instead of behind a library call."

Agentic AI Engineer — Agent Infrastructure, Platforms & Production Systems