Lab 01 — Backtesting Harness & ML Forecaster

Phase: 07 — Forecasting | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–6 hours

The forecasting lab is really a methodology lab: rolling-origin backtesting, scale-free metrics, baselines that must be beaten, and the lag-≥-horizon rule that keeps ML forecasters honest. The model is the easy part.

What you build

  • smape / mase — the two metrics worth carrying: bounded symmetric error and the beats-naive ratio (MASE < 1 = better than the naive baseline; > 1 = your model is worse than doing nothing clever)
  • naive_forecast / seasonal_naive_forecast — the baselines every forecast must clear, with the wrap-around indexing done correctly
  • rolling_origin_splits — expanding-window time-series CV (random K-fold on a time series is leakage by construction)
  • build_features — lag/rolling/calendar features under the lag ≥ horizon rule: when forecasting 7 days ahead, lag-1 doesn't exist at prediction time
  • GBMForecaster — a direct (non-recursive) gradient-boosting forecaster over those features, with a runtime assertion that no feature reached into the future
  • backtest — the harness scoring all methods on identical folds

Key concepts

ConceptWhat to understand
MASEMAE scaled by in-sample seasonal-naive MAE — comparable across series of different scales; the M-competitions' metric
Rolling originTrain on [0, t), test on [t, t+h), advance t — every fold respects time's arrow
Lag ≥ horizonA feature available at forecast time must be at least horizon old; violating this gives beautiful backtests and useless production forecasts
Direct vs recursiveOne model per horizon (direct, this lab) vs feeding predictions back (recursive, compounds errors)
Known-in-advance covariatesPromo flags / calendar ARE legal features at time t — they're planned, not observed

Run

pip install -r requirements.txt
pytest test_lab.py -v                       # your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # reference
python solution.py                          # the comparison table

Reference output on the synthetic demand series:

gbm             MASE = 0.476
seasonal_naive  MASE = 0.848
naive           MASE = 2.336

Suggested TODO order

  1. smape, mase — hand-verify against the worked numbers in the tests
  2. Baselines — the seasonal wrap-around indexing is the classic off-by-one
  3. rolling_origin_splits — check the no-leak property test first
  4. build_features — write the lag rule down before coding it
  5. GBMForecaster.fit/predict — the NaN assertion is your leak alarm
  6. backtest — same folds, same metric, all methods

Success criteria

  • All 12 tests pass
  • Your GBM beats seasonal-naive by ≥30% MASE on the demand series
  • You can explain why lag-1 is illegal at horizon 7, and what the backtest would (falsely) show if you used it anyway
  • You can say why MASE beats MAPE (zero/near-zero actuals; asymmetry) in one sentence

Extensions

  • Recursive forecasting + error-compounding comparison vs direct
  • Prediction intervals via quantile GBM (loss="quantile"); calibration check
  • Intermittent demand: Croston's method vs the GBM on a sparse series
  • Hierarchical reconciliation: forecast SKU and store level, reconcile bottom-up vs MinT, measure coherence

Interview Q&A

Q: Your model's backtest MASE is 0.4 but production forecasts are garbage. Top suspects? Leakage in features (a lag < horizon, or a covariate that's actually observed-after — audit build_features against the deployment timeline); backtest folds that don't match the production cadence (retraining weekly in backtest, monthly in prod); or the series regime-shifted and the expanding window is dominated by stale history. The lag-≥-horizon audit is always first.

Q: Why is random cross-validation invalid for time series? Training folds would contain the future of test folds — autocorrelation means the model effectively memorizes the neighborhood of each test point. Rolling-origin keeps every train set strictly before its test window, matching how the model will actually be used: trained on the past, scored on the future.

Q: When does seasonal-naive beat your ML model, and what do you do about it? Short histories (features can't be estimated), strong stable seasonality with little else (nothing left to learn), or regime changes the lags haven't caught up with. The response is not "more model" — it's ensembling toward the baseline (shrink predictions to seasonal-naive when fold-level MASE ≈ 1) and investing in covariates (promos, holidays) that carry genuinely new information.

References

  • Hyndman & Athanasopoulos, Forecasting: Principles and Practice (3rd ed., free online) — ch. 5 (the toolbox), ch. 13 (practical issues)
  • Hyndman & Koehler, Another look at measures of forecast accuracy (2006) — the MASE paper
  • Makridakis et al., The M5 Accuracy Competition (2022) — where GBMs-with-good-features beat everything
  • sktime and statsforecast — the industrial versions of this harness