Phase 07 — Forecasting
Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 2 weeks (35–45 hours) Roles Supported: the JD's "forecasting" — demand, capacity, and finance planning; the ML domain with the strictest temporal discipline and the strongest baselines
Why This Phase Exists
Forecasting is where ML engineers embarrass themselves most reliably: the baselines (naive, seasonal-naive) are brutally strong, temporal leakage is one careless feature away, and the standard accuracy metrics (MAPE!) are booby-trapped. It is also a core enterprise workload — demand planning, capacity, staffing, finance — and a standard senior-MLE interview domain precisely because it tests discipline over modeling flair.
The flagship lab builds the artifact every forecasting team actually lives in: a rolling-origin backtesting harness with honest metrics (MASE, sMAPE), seasonal- naive baselines, and a leakage-proof feature-based ML forecaster that must earn its complexity by beating the baseline inside the harness.
Concepts
- The forecasting data model: series, horizon, frequency, hierarchy; point vs probabilistic forecasts
- Baselines first: naive, seasonal-naive, drift — and why they're hard to beat
- Classical decomposition vocabulary: trend, seasonality, ETS/ARIMA at working level
- The ML approach: direct multi-horizon forecasting with lag/rolling/calendar features — and the leakage rules for constructing them
- Backtesting: rolling-origin (expanding window) evaluation; why one split lies
- Metrics: why MAPE fails (zeros, asymmetry), sMAPE's partial fix, MASE (scaled by in-sample naive error — the cross-series honest metric)
- Intermittent demand, cold-start series, hierarchy reconciliation (concept level)
Labs
Lab 01 — Backtesting Harness & ML Forecaster (flagship, implemented)
| Field | Value |
|---|---|
| Goal | Build rolling-origin backtesting with MASE/sMAPE; implement naive + seasonal-naive baselines; build a lag-feature GBM forecaster that beats seasonal-naive on trend+covariate data — with leakage-proof feature construction enforced by test |
| Concepts | Rolling origin, forecast horizon, MASE's scaling logic, lag/rolling features without future contamination |
| Steps | 1. Metrics (hand-verified); 2. baselines; 3. rolling-origin splitter (never leaks); 4. lag/rolling/calendar feature builder; 5. GBM direct forecaster; 6. the harness comparison table |
| How to Test | pytest test_lab.py — 12 tests incl. seasonal-naive MASE ≈ 1 on pure-seasonal data and GBM > seasonal-naive on trended data |
| Talking Points | Why is MASE scaled by in-sample naive error? Why does a single train/test split lie in time series? Where can lag features still leak? |
| Resume Bullet | Built a rolling-origin forecast backtesting harness (MASE/sMAPE, seasonal-naive baselines) and a leakage-proof lag-feature GBM forecaster that beat seasonal-naive by 30%+ MASE on trended seasonal demand data |
→ Lab folder: lab-01-backtesting-forecaster/
Extension Project A — M5/Retail Slice (spec)
Run your harness on a slice of the M5 (Walmart) dataset: seasonal-naive vs your GBM vs LightGBM with richer features; per-series MASE distribution (not just the mean); the intermittent-series segment called out separately (Croston's method as the reference there).
Extension Project B — Probabilistic Forecast Memo (spec)
Add quantile forecasts (GBM with quantile loss) to your harness; evaluate with pinball loss; write the one-pager on why the capacity-planning consumer needs P90, not the mean.
Guides in This Phase
Deliverables Checklist
- Lab 01 implemented; all 12 tests pass
- Extension A per-series distribution plots
- Extension B quantile harness + memo