Lab 01 — Experiment Analyzer & Bandit

Phase: 11 — Online Experimentation | Difficulty: ⭐⭐⭐⭐⭐ | Time: 4–6 hours

Every number an experimentation platform shows you comes from five computations. Build all five from scratch — power, the z-test, SRM, CUPED, Thompson sampling — each pinned by hand-computed cases and simulation-based calibration checks.

What you build

  • required_sample_size — the power formula; the ~122k-per-arm anchor case
  • two_proportion_ztest — pooled-SE z, two-sided p, unpooled-SE CI — validated by an A/A calibration test (under no effect, rejects ≈ α — the test that tests your test)
  • srm_check — chi-square sample-ratio-mismatch detection at the strict 1e-3 threshold, including unequal design ratios
  • cuped_adjust — θ = Cov/Var, mean-zero adjustment; the test verifies measured variance reduction ≈ ρ² and that the mean (hence the effect estimate) is preserved
  • EpsilonGreedy / ThompsonSampling / run_bandit — regret ordering measured over seeds: TS < ε-greedy < uniform

Reference demo

n per arm to detect 5% → 5.25% at alpha=.05, power=.80: 122,121
A/B: effect=+0.0030 [+0.0011, +0.0049] p=0.0024 significant=True
SRM check 50600/49400: p=1.48e-04 detected=True
cumulative regret over 20k pulls: eps-greedy=35  thompson=31

Run

pip install -r requirements.txt
pytest test_lab.py -v                       # your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # reference (15 tests)
python solution.py

Suggested TODO order

  1. required_sample_size — anchor on 122k; check the inverse-square MDE scaling test
  2. two_proportion_ztest — hand-computed case, then run the A/A test repeatedly: it is the single most diagnostic test in the suite
  3. srm_check — note the three cases: scale makes 1.2% scream, n=200 stays quiet, design ratios respected
  4. cuped_adjust — the mean-preservation test is the unbiasedness property
  5. Bandits — seeded RNGs are injected everywhere (Phase 01's discipline); regret tests average over 8 seeds

Success criteria

  • All 15 tests pass
  • You can explain why the z-test uses pooled SE but the CI uses unpooled
  • You can state what an SRM detection means operationally (stop; debug plumbing; never reweight)
  • You can derive CUPED's variance-reduction factor (1−ρ²) and its one fatal misuse
  • You can explain why Thompson sampling's exploration anneals and ε-greedy's doesn't

Extensions

  • Welch's t-test for continuous metrics + winsorization for revenue-style tails
  • Group-sequential boundaries (O'Brien–Fleming) — simulate peeking, compare realized α
  • Always-valid e-value test; same peeking simulation
  • Bonferroni / Benjamini–Hochberg over a 20-metric scorecard; false-discovery demo
  • Contextual bandit (LinUCB) on synthetic user features

Interview Q&A

Q: The PM checked the dashboard daily and stopped the test on day 4 when p hit 0.03. What do you tell them? That p-value doesn't mean what it claims: each peek is another draw at α, and stopping-at-first-significance inflates the false-positive rate severalfold (simulate it — Extension A's table). Options going forward: pre-committed fixed horizon (full weeks), group-sequential boundaries if early stopping matters, or an always-valid method if the dashboard culture won't change. And re-run this test to the planned horizon before shipping.

Q: Why does an experimentation platform need A/A tests? They validate the machinery end-to-end: assignment uniformity (SRM at scale), metric-pipeline correctness, and statistical calibration (significance fires ≈ α with no effect). A platform that has never passed A/A tests produces numbers with unknown error rates — you're not measuring the product, you're measuring your bugs.

References

  • Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments (2020)
  • Deng et al., CUPED (WSDM 2013)
  • Fabijan et al., Diagnosing Sample Ratio Mismatch (KDD 2019)
  • Russo et al., A Tutorial on Thompson Sampling — arXiv:1707.02038