Phase 11 — Online Experimentation
Difficulty: ⭐⭐⭐⭐⭐ Estimated Time: 2 weeks (40–50 hours) Roles Supported: the JD's "online experimentation" and "robust evaluation … strategies, including … online experimentation"
Why This Phase Exists
Offline metrics propose; online experiments dispose. A ranking model with +3% offline NDCG can lose revenue online (position effects, feedback loops, latency); the only ground truth for "did the model help" is a controlled experiment — and experiments are easy to run and very easy to run wrong. The failure modes are statistical (underpowered tests, peeking, multiple comparisons) and infrastructural (broken randomization, SRM), and they produce confident wrong answers.
The flagship lab builds the analyst's toolkit from scratch — power analysis, the two-proportion test, sample-ratio-mismatch detection, CUPED variance reduction, and a Thompson-sampling bandit — each pinned by hand-computed tests, so the statistics stop being incantations.
Concepts
- Randomized assignment: units (user vs session vs request), hash-based bucketing, stickiness; interference/network effects
- Hypothesis testing for experiments: two-proportion z-test, t-test for means, p-values honestly interpreted, confidence intervals over point estimates
- Power analysis: MDE, α/β, the sample-size formula — and what to do when traffic can't reach power
- SRM (sample ratio mismatch): the chi-square canary that catches broken randomization before it poisons conclusions
- Peeking and sequential testing: why repeated significance checks inflate false positives; group-sequential/always-valid approaches
- CUPED: variance reduction with pre-experiment covariates — the cheapest power multiplier in the industry
- Multiple comparisons across metrics/segments; guardrail vs goal metrics
- Bandits vs A/B: Thompson sampling, when regret minimization beats inference
- Interleaving for ranking (cross-ref Phase 05)
Labs
Lab 01 — Experiment Analyzer & Bandit (flagship, implemented)
| Field | Value |
|---|---|
| Goal | Build the experimentation toolkit: sample-size calculator, two-proportion z-test with CI, SRM detector, CUPED estimator (with measured variance reduction), and a Thompson-sampling bandit beating epsilon-greedy on regret |
| Concepts | Power/MDE, z-tests, chi-square SRM, control variates, Beta-Bernoulli posteriors, regret |
| How to Test | pytest test_lab.py — 15 tests: hand-computed statistics, false-positive calibration under A/A simulation, SRM sensitivity, CUPED variance reduction ≥ theory, bandit regret ordering |
| Talking Points | Why does peeking inflate false positives? What does SRM catch that significance tests can't? Why does CUPED's θ = Cov/Var? When is a bandit the wrong tool? |
| Resume Bullet | Built an A/B analysis toolkit (power, z-tests, SRM, CUPED) validated by simulation — CUPED cut required sample size 40% on covariate-correlated metrics; Thompson sampling halved exploration regret vs epsilon-greedy |
→ Lab folder: lab-01-ab-toolkit/
Extension Project A — Sequential Testing (spec)
Implement a group-sequential boundary (O'Brien-Fleming-style alpha spending) and an always-valid e-value test; simulate peeking with both vs naive repeated z-tests; produce the false-positive-rate comparison table.
Extension Project B — Interleaving Simulator (spec)
Team-draft interleaving for two rankers over simulated click models (position-biased); measure sensitivity vs A/B on the same traffic: how much less traffic does interleaving need to detect the same ranker difference?
Guides in This Phase
- WARMUP.md — the full zero-to-expert guide
Deliverables Checklist
- Lab 01 implemented; all 15 tests pass
- Extension A peeking comparison table
- Extension B interleaving sensitivity result