Capstone 04 — Forecasting Platform with Backtest-Gated Promotion

Composes: Phase 07 (backtesting + forecaster) · Phase 08 (orchestration) · Phase 09 (batch serving pattern) · Phase 12 (monitoring + retraining FSM) Time: 2–4 weeks

The marketplace/supply-chain capstone: hundreds of series forecast nightly, every model promotion gated on rolling-origin backtests, accuracy monitored per-series in production.

What you ship

synthetic demand panel (200+ series: trend/seasonality/promo/intermittent mix,
                        planted regime change at a known date)
   │
nightly batch DAG (P08): validate ──► feature build ──► per-segment training
   │                                        │
   │                            rolling-origin backtest gate (P07):
   │                            challenger MASE must beat incumbent AND
   │                            seasonal-naive on the same folds
   │                                        │
batch scoring run (P09 batch pattern) ──► forecast store + accuracy ledger
   │
per-series monitoring (P12): trailing MASE control bands, drift on residuals,
retraining FSM with the regime change as the live-fire test

Stage list

The panel: heterogeneous series — smooth/seasonal, promo-driven, intermittent (zeros-heavy), short-history — plus one planted regime change. Heterogeneity is the point: one model config will not fit all.
Walking skeleton: seasonal-naive for every series, nightly DAG, forecast store, trailing-accuracy ledger. The baseline is the product on day one.
Models per segment: the Phase 07 GBM where history suffices; seasonal- naive where it doesn't; (extension: Croston's for intermittent). Segment assignment is itself backtested.
The gate: promotion requires beating the incumbent and seasonal-naive on identical rolling-origin folds (MASE, with the fold table archived as the approval artifact — Phase 08 lineage).
Operations: per-series trailing MASE with control bands; the planted regime change must (a) trip the per-series monitor, (b) trigger the FSM, (c) produce a challenger that passes the gate on post-change folds.
The aggregate report: fleet-level MASE distribution (median + p90 across series — a fleet is judged on its tail, not its average), per-segment breakdown, and cost of the nightly run (cached vs cold, Phase 08).

Money table (fill with your runs)

Metric	Seasonal-naive fleet	Segmented ML fleet
median MASE across series
p90 MASE (the tail!)
% series where ML beats naive
regime-change detection lag (days)
post-regime recovery (folds to gate-pass)
nightly run: tasks executed cached vs cold

Injected-failure demo (required)

The regime change plus a broken-promo-covariate injection on a different date: show the residual-drift monitor distinguishing them (regime → retrain path; broken covariate → data INCIDENT path). Two alarms, two different correct responses — that discrimination is the demo.

Honest-limitations paragraph (required)

Synthetic series vs real demand pathologies (stockouts censoring demand!); no hierarchical reconciliation; point forecasts only (quantile extension noted); single-process nightly scale.