Lab 03 — Regression Detection with McNemar's Test
Phase: 09 — Model Accuracy Evaluation | Difficulty: ⭐⭐⭐⭐⭐ | Time: 3–4 hours
McNemar's test is the correct statistical test for detecting accuracy regressions between two models on the same dataset.
What you build
mcnemar_test— χ² statistic and p-value for paired categorical comparisonsRegressionDetector— stores baseline, compares new model per-taskRegressionPolicy— per-task delta gate + average delta gate for CI pass/failanalyze_slices— break down accuracy by data slice (domain, length, etc.)RegressionResult— full report with pass/fail status
Key concepts
| Concept | What to understand |
|---|---|
| McNemar's test | For paired binary outcomes: χ²=( |
| b, c cells | b = old correct, new wrong; c = old wrong, new correct |
| Why not t-test? | McNemar handles correlated binary pairs — t-test assumes independence |
| CI regression gate | p < 0.05 AND delta > threshold → block deployment |
Files
| File | Purpose |
|---|---|
lab.py | Stubs with TODO markers |
solution.py | Complete working implementation |
test_lab.py | pytest suite |
requirements.txt | Dependencies |
Run
pip install -r requirements.txt
python lab.py
pytest test_lab.py -v