Warmup Guide — Forecasting

Zero-to-expert primer for Phase 07: time series as a discipline of restraint — baselines that humble you, metrics that don't lie, backtesting that respects time, and the ML forecaster that earns its complexity.

Table of Contents

Chapter 1: What Makes Forecasting Different

Three structural facts separate forecasting from ordinary supervised learning:

  1. Time is not exchangeable: rows are ordered and autocorrelated; everything built on i.i.d. assumptions (random splits, standard CV) is invalid by default. The entire phase is consequences of this sentence.
  2. The future is genuinely uncertain, and the easy part of the future is carried by trivial models: tomorrow ≈ today, next Monday ≈ last Monday. Whatever your model adds must be measured above that floor (Ch. 3).
  3. The consumer is an operational decision: inventory orders, staffing levels, capacity buys — usually needing a quantile (order to P90 demand), a horizon (lead time), and a granularity (SKU × store × day) determined by the decision, not the data scientist. Asking "what decision does this forecast feed?" first is the senior reflex.

Chapter 2: The Anatomy of a Series

The decomposition vocabulary (the lens for looking at a series before modeling):

  • Trend: long-run level change. Seasonality: fixed-period patterns (weekly, yearly — multiple can coexist). Cyclical: irregular-period swings (business cycles). Irregular: what's left.
  • Additive ($y = T + S + R$) vs multiplicative ($y = T \cdot S \cdot R$) composition — retail demand is usually multiplicative (holiday lift is a percentage); log transform turns multiplicative into additive (and stabilizes variance — the workhorse preprocessing move).
  • The practical specs of a forecasting task: frequency (daily/weekly), horizon $h$ (how far ahead — set by the decision's lead time), history length, and the hierarchy (SKU → category → store → region: forecasts at every level that must reconcile — Ch. 8).
  • Calendar structure is signal, not noise: day-of-week, month, holidays (and holiday proximity), paydays, promotions — Ch. 7's features.

Chapter 3: Baselines — the Humbling Chapter

The mandatory floor, in ascending sophistication:

  • Naive: $\hat{y}_{t+h} = y_t$. Optimal for random walks — which many business series approximately are.
  • Seasonal-naive: $\hat{y}{t+h} = y{t+h-m}$ (same point last season, $m$ = season length). The one to beat for anything with weekly/yearly pattern; on M-competition-class data it routinely lands mid-pack against sophisticated entries.
  • Drift: naive + the average historical slope.

Why this chapter exists: decades of forecasting competitions (M1–M5) repeat one finding — simple methods are startlingly competitive, and complex methods win only with care (and in M5, with gradient boosting + good features rather than exotic architectures). The engineering posture that follows: every model claim in this phase is MASE vs seasonal-naive inside the backtest harness — your lab encodes the posture as a passing test.

Chapter 4: Classical Models at Working Level

What a senior MLE should know about, even when shipping GBMs:

  • Exponential smoothing (ETS): forecasts as exponentially-weighted recent history; Holt adds trend, Holt-Winters adds seasonality. State-space form gives prediction intervals. Strengths: tiny data appetite, robust, fast over millions of series — the default for the long tail.
  • ARIMA: model the series' autocorrelation structure — AR (regression on own lags), I (difference away trend), MA (regression on past errors); SARIMA adds seasonal terms. Strength: principled, interpretable correlation handling; weakness: per-series fitting/identification at scale, exogenous variables are bolt-ons.
  • The honest modern summary: ETS/ARIMA remain excellent per-series univariate tools; the ML approach (Ch. 7) wins when there are many related series (cross-learning), rich covariates (promos, prices, weather), or cold-start series — which is precisely the enterprise demand-forecasting regime. Neural forecasters (DeepAR/N-BEATS/TFT-class) extend the same cross-series logic; reach for them after GBMs plateau, not before.

Chapter 5: Metrics — MAPE's Traps, MASE's Fix

The metric chapter matters because the standard one is broken:

  • MAPE $= \text{mean},|y - \hat{y}| / |y|$: explodes at $y \to 0$ (intermittent demand → infinite/undefined), and is asymmetric — over-forecasting a small actual costs more than under-forecasting a large one, so optimizing MAPE systematically biases forecasts low (a real inventory incident, not a curiosity).
  • sMAPE (divide by the average of $|y|$ and $|\hat{y}|$): bounded, still awkward at zero, still not symmetric in the ways its name implies — know it as "the M-competition convention," not as a fix.
  • MASE — the one to internalize:

$$\text{MASE} = \frac{\text{MAE}(\text{forecast})}{\text{MAE}(\text{in-sample naive})}$$

The denominator is the one-step (or seasonal) naive error measured on the training history. So MASE reads as "error relative to the dumb method": < 1 beats naive, > 1 loses to it. It's defined at zero actuals, symmetric, and — the killer feature — comparable across series of different scales, which is what lets you average over a 10,000-SKU portfolio meaningfully (the seasonal variant scales by seasonal-naive in-sample error; your lab implements that one).

  • Probabilistic forecasts (Extension B): pinball/quantile loss per quantile; coverage checks (does the P90 cover ~90%?) — because Ch. 1's consumers need quantiles.

Chapter 6: Backtesting — Rolling-Origin Evaluation

One train/test split lies in time series: a single cutoff date is one draw from a nonstationary world (one holiday season, one promo calendar). The honest protocol — rolling-origin (a.k.a. time-series CV, expanding window):

fold 1: train [0 .. T1]      → forecast (T1, T1+h]
fold 2: train [0 .. T1+s]    → forecast (T1+s, T1+s+h]
fold 3: train [0 .. T1+2s]   → ...

Each fold's metrics are computed on its forecast window; report the distribution across folds (mean ± spread — the Phase 02-of-CV uncertainty ethic, time edition). Rules your lab's splitter enforces by test: train always ends strictly before the forecast window; windows never overlap a fold's training data; the feature builder runs inside the fold (computing features once on the full series then splitting is the subtle leak — rolling means computed over the future). Backtesting is the forecasting analog of the experiment ledger: the harness, not the notebook, is where claims live.

Chapter 7: The ML Forecaster — Features Without Leakage

The reduction: forecasting → tabular regression. For each (series, date), build features from the past only, target = value at date (+ horizon handling):

  • Lag features: $y_{t-1}, y_{t-7}, y_{t-14}, y_{t-28}$ (align lags to the seasonality), and for horizon h, lags must be ≥ h — predicting $t{+}7$ with lag-1 uses a value you won't have at forecast time; this is THE leak, and the lab's feature builder takes horizon as an argument to enforce it.
  • Rolling statistics: mean/std/min/max over trailing windows — shifted by the horizon for the same reason.
  • Calendar features: day-of-week, month, holiday flags — known in advance, always safe.
  • Known-future covariates (promo calendar, prices): safe if truly known ahead; observed-only covariates (weather): must use forecasts-of-the-covariate at inference, so train on forecast-vintage values or accept the optimism (a classic hidden leak — saying this in interviews lands).
  • One global model over all series (series id / category as features) is the cross-learning move that makes ML beat per-series classics — more data per parameter, cold-start handled by relatives (the M5-winning recipe: LightGBM + exactly these features).
  • Multi-horizon strategies: recursive (feed predictions back — error compounds), direct (a model per horizon, or horizon-as-feature — the lab's choice: simpler to keep honest).

Chapter 8: Production Forecasting Realities

  • Scale shape: forecasting is many-small-models/one-global-model over 10³–10⁶ series on a schedule (nightly/weekly) — an orchestration problem (Phase 08) more than a serving problem; "inference" is a batch job whose output lands in a database.
  • Hierarchy reconciliation: SKU forecasts must sum to category forecasts that planners also see; naive independent forecasts won't. Approaches: bottom-up, top-down, or optimal reconciliation (MinT) — know the problem exists and that "whose number is official" is as much governance as math.
  • Intermittent demand (mostly zeros, occasional spikes — spare parts, long-tail SKUs): standard metrics and models mislead; Croston's method / zero-inflated approaches; segment these series out in evaluation (Extension A) or they poison the averages.
  • Monitoring (Phase 12 hooks): forecast-vs-actual error tracked per series per cycle is the drift detector; structural breaks (COVID, a new competitor) demand intervention features or regime resets — a model quietly extrapolating a dead regime is the expensive failure.
  • The judgment layer: planners override forecasts; log overrides and measure their accuracy too — half the value of a forecasting platform is making the human-vs-model ledger visible.

Lab Walkthrough Guidance

Lab 01 — Backtesting Harness & ML Forecaster, suggested order:

  1. Metrics first, hand-verified: mase (seasonal in-sample scaling) and smape on tiny arrays you compute on paper (the tests carry the numbers).
  2. Baselines: naive_forecast, seasonal_naive_forecast — pure indexing; get the horizon arithmetic right.
  3. rolling_origin_splits(n, initial, step, horizon) — yields index pairs; the no-leak property is tested (train end < every test index).
  4. Feature builder: lags (≥ horizon!), trailing rolling mean (shifted), day-of-week — built per fold's training slice + frozen continuation, never on the full series.
  5. The GBM direct forecaster (sklearn GradientBoostingRegressor) and the harness loop producing the comparison table: on the synthetic trended-seasonal-with- covariate data, seasonal-naive MASE ≈ 1 by construction and the GBM must land clearly below.

Success Criteria

You are ready for Phase 08 when you can, from memory:

  1. Name the three structural differences from i.i.d. ML and their consequences.
  2. Define naive/seasonal-naive/drift and state the competition finding that makes them mandatory.
  3. Explain MAPE's two failures, what sMAPE is, and derive why MASE is cross-series comparable.
  4. Draw rolling-origin evaluation and name the two subtle leaks (features computed on the full series; lags < horizon).
  5. List the four feature families with their safety conditions.
  6. Sketch the production shape: batch orchestration, hierarchy reconciliation, intermittent segmentation, forecast-vs-actual monitoring.

Interview Q&A

Q: Your demand model shows 12% MAPE; the planner says the old spreadsheet was better. How do you investigate? First suspect the metric: MAPE is asymmetric and explodes near zero — check the intermittent SKUs' contribution and whether optimizing it biased forecasts low (stockouts would confirm). Re-evaluate both methods in the same rolling-origin harness with MASE per series; segment (fast-movers vs intermittent, promo vs regular weeks); and check the planner's complaint window for a structural break the model extrapolated through. The likely truth: both are right — different metrics, different segments — and the harness adjudicates.

Q: Why can't you use standard k-fold CV for time series, and what do you do instead? Random folds train on the future of their test points — autocorrelation makes neighboring points near-duplicates, so the model effectively memorizes the answer; offline scores inflate and production disappoints (temporal leakage, the same disease as Phase 03's feature-store joins). Instead: rolling-origin evaluation — expanding train windows, forecast the next h steps, repeat — with features rebuilt inside each fold. Bonus point: even hyperparameter tuning must respect this (nested rolling-origin), or the tuning itself leaks.

Q: When would you choose ETS over your gradient-boosting forecaster? Few series with decent individual history and no covariates (nothing for cross-learning to exploit); very short series (GBM features eat history); the long-tail regime where fitting cost per series matters (ETS is closed-form fast); and when calibrated prediction intervals are first-class (state-space form gives them natively). The deployment answer is usually a portfolio: ETS for the tail, the global GBM for covariate-rich heads, seasonal-naive as the floor everywhere — chosen per segment by the backtest harness, not by ideology.

References

  • Hyndman & Athanasopoulos, Forecasting: Principles and Practice (3rd ed., free at otexts.com/fpp3) — the field's textbook; ch. 5 (toolbox), 7–9 (ETS/ARIMA)
  • Hyndman & Koehler, Another look at measures of forecast accuracy (2006) — MASE's source
  • Makridakis et al., The M5 Accuracy Competition (2022) — what actually won (LightGBM + features) and why
  • Tashman, Out-of-sample tests of forecasting accuracy (2000) — rolling-origin methodology
  • Croston (1972) / Syntetos & Boylan — intermittent demand
  • Wickramasuriya et al., Optimal Forecast Reconciliation (MinT) (2019)
  • sktime and statsforecast — the libraries whose internals mirror this lab