Warmup Guide — Online Experimentation

Zero-to-expert primer for Phase 11: how to learn the truth from live traffic — randomization, power, the tests, the traps (SRM, peeking, multiple comparisons), CUPED, and when bandits replace A/B tests.

Table of Contents

Chapter 1: Why Online Experiments Are Ground Truth

Offline evaluation (Phases 04–07) answers "does the model predict held-out data better?" The business question is "does shipping it cause better outcomes?" — and cause is the hard word. Correlations in logged data are poisoned by confounders (the old model chose what users saw — Phase 06's feedback loop); pre/post comparisons are poisoned by time (seasonality, marketing, news). The randomized controlled experiment is the one design where the difference between groups estimates the causal effect, because randomization makes the groups exchangeable in expectation — every confounder, known and unknown, is balanced.

The senior-engineer framing the JD implies: experimentation is infrastructure (assignment, logging, analysis pipelines, dashboards — this phase + Phase 08's patterns) plus statistical hygiene (this WARMUP) plus decision discipline (Ch. 8). Most experiment failures are not math failures; they're hygiene failures.

Chapter 2: Randomization — the Part That Does the Work

  • The unit: user (default — consistent experience, captures learning effects), session (more power, risks within-user contamination), request (max power, only for invisible backend changes). Choose the unit that matches where the effect lives and where interference is tolerable.
  • Hash-based assignment (Phase 09's canary router, generalized): bucket = hash(salt:unit_id) / MAX — deterministic (re-computable anywhere), sticky (same user, same arm, forever), and salt-independent across experiments (a fresh salt per experiment decorrelates assignments; reusing salts couples experiments — a real and nasty bug class).
  • Interference: when one user's treatment affects another's outcome (social features, marketplaces with shared inventory, shared caches!), the independence assumption breaks. Mitigations: cluster randomization (by region/ market), switchback designs (time-sliced for marketplaces). Know the names; recognize the situations.
  • The engineering invariant: assignment is logged at exposure time with the experiment version. Analyzing "everyone in the bucket" instead of "everyone exposed" dilutes effects with users who never saw the change.

Chapter 3: The Statistics You Actually Use

For conversion-style (binary) metrics — the two-proportion z-test (the lab's core):

$$\hat{p} = \frac{x_A + x_B}{n_A + n_B}, \qquad z = \frac{\hat{p}_B - \hat{p}_A}{\sqrt{\hat{p}(1-\hat{p})(\tfrac{1}{n_A} + \tfrac{1}{n_B})}}$$

with the two-sided p-value $2(1 - \Phi(|z|))$ and the CI on the difference using unpooled standard errors. For means (revenue, latency): Welch's t-test — and for heavy-tailed metrics like revenue, expect to need winsorization or much more traffic (a handful of whales dominates the variance).

Honest interpretation (the part interviews probe):

  • The p-value is P(data this extreme | no true effect) — not P(no effect | data).
  • A non-significant result is not "no effect" — it's "couldn't distinguish from zero at this power"; report the CI, which says what effect sizes remain plausible.
  • Always report the CI alongside the decision — "+0.4% [+0.1%, +0.7%]" carries the information; "significant ✓" does not.
  • Calibration check the lab performs: under A/A simulation (no true effect), your test must reject ≈ α of the time. A test that can't pass its A/A check is miscalibrated machinery, and so is an experimentation platform that never runs A/A tests.

Chapter 4: Power and the MDE Conversation

The pre-experiment math that prevents the most common waste (underpowered tests):

$$n \text{ per arm} \approx \frac{(z_{1-\alpha/2} + z_{1-\beta})^2 \cdot 2,\bar{p}(1-\bar{p})}{\delta^2}$$

for detecting an absolute difference δ at significance α and power 1−β (z's: 1.96 and 0.84 for the standard 0.05/0.80). The numbers that build intuition: baseline 5% conversion, MDE 0.25% absolute (5% relative) → ~120k users per arm. Small effects on small baselines are expensive.

The MDE conversation (the senior skill): before launch, compute the minimum detectable effect at the traffic you'll actually get. If the MDE exceeds any plausible effect size, the experiment cannot answer the question — the honest options are: a more sensitive metric (CUPED, Ch. 7; or a proxy metric closer to the mechanism), longer duration, larger traffic share, interleaving (for rankers), or don't run it and decide on other evidence. Running it anyway and reading the noise is the failure mode.

Chapter 5: SRM — the Canary in Every Experiment

Sample Ratio Mismatch: you configured 50/50; you observe 50.6/49.4 over a million users. Is that chance? The chi-square test on counts answers:

$$\chi^2 = \sum_{arms} \frac{(O_i - E_i)^2}{E_i} \sim \chi^2_{k-1}$$

If p < 0.001 (the conventional SRM threshold — deliberately strict), your assignment or logging is broken, and every other number in the experiment is untrustworthy — because whatever deleted users from one arm (a crash on the new code path, a redirect dropping trackers, a bot filter interacting with the treatment) almost certainly didn't delete them at random. Industry experience (Microsoft, booking.com report ~6–10% of experiments): SRM is common, automatic SRM checks are non-negotiable platform hygiene, and the response to SRM is stop and debug the plumbing, never "adjust for it."

Chapter 6: Peeking and Sequential Testing

The trap: check the p-value daily, stop when significant. Each look is another chance for a false positive; with daily peeks over a month, the realized false-positive rate can exceed 25% against a nominal α = 5%. The fixed-horizon test is only valid at its fixed horizon.

The legitimate options:

  1. Don't peek (for decisions): fix duration in advance — full weeks (weekly seasonality!), minimum two — and look once. Dashboards can display data; the decision rule fires once.
  2. Group-sequential: pre-plan k looks with adjusted boundaries (O'Brien–Fleming: very strict early, near-nominal late) — early stopping for clear winners/harms with controlled overall α (Extension A).
  3. Always-valid inference (mSPRT/e-values): p-values valid under continuous monitoring — costs some power, removes the discipline burden; increasingly the platform default.

The deeper point: peeking is one instance of garden-of-forking-paths problems (unplanned subgroups, metric switching, post-hoc exclusions). The cure is the same: pre-registration of the decision rule — written before launch (Ch. 8).

Chapter 7: CUPED — Variance Reduction for Free

The cheapest power multiplier in the industry. Idea: much of the variance in your metric is pre-existing user heterogeneity (heavy users convert more), which pre-experiment data already predicts — remove it.

With $X$ = the user's pre-experiment value of the (ideally same) metric:

$$Y_{cuped} = Y - \theta (X - \bar{X}), \qquad \theta = \frac{\text{Cov}(Y, X)}{\text{Var}(X)}$$

(θ is the OLS slope — the control-variates trick from simulation.) Properties that make it safe and effective: the adjustment is mean-zero, so the treatment-effect estimate is unbiased regardless of X's quality; variance shrinks by factor $(1 - \rho^2)$ where ρ = corr(X, Y) — at ρ = 0.6, variance drops 36%, equivalent to 36% more traffic for free; X must be pre-experiment (anything measured after assignment can be affected by treatment and biases the estimate — the one rule). New users have no X → use $\theta \cdot 0$ adjustment (their unadjusted Y), or stratify. The lab implements CUPED and measures the variance reduction against the $(1-\rho^2)$ theory.

Chapter 8: Multiple Metrics, Guardrails, and Decisions

  • Metric roles, declared up front: one goal metric (the decision metric — powering and α apply to it), a small set of guardrails (latency, error rates, unsubscribes, revenue-per-user — tested for harm, often one-sided), and diagnostics (unlimited, decision-irrelevant, exploration-only).
  • Multiple comparisons: 20 metrics × 5 segments at α=0.05 ≈ 5 false alarms per experiment. Corrections (Bonferroni, Benjamini–Hochberg FDR) for anything decision-relevant; the cleaner fix is the role discipline above — one goal metric needs no correction.
  • Segment "wins" ("it works for Android users in Egypt!") found post-hoc are hypotheses for the next experiment, not findings — forking paths again.
  • The decision document (pre-registered): goal metric + MDE + duration + guardrail thresholds + the action for each outcome (ship / iterate / abandon), agreed before launch. Experiments end in arguments exactly when this doesn't exist (the same pre-agreement discipline as Phase 09's canary thresholds).

Chapter 9: Bandits — When to Stop Splitting Traffic Evenly

A/B tests spend traffic on inference; bandits optimize traffic allocation while learning. Thompson sampling (the lab's algorithm) for Bernoulli rewards: keep a Beta(α=1+successes, β=1+failures) posterior per arm; each round, sample a value from each posterior and play the argmax. Exploration emerges from posterior uncertainty — wide posteriors win sometimes; as evidence accumulates, allocation concentrates on the best arm. It dominates epsilon-greedy (which explores at a fixed rate forever, paying linear regret on the exploration fraction — the lab measures this ordering).

Bandit vs A/B — the decision rule: bandits when the cost of serving the worse arm matters and the question is "which is best" (headlines, promos, creatives — short feedback, many variants); A/B when you need an unbiased effect estimate (ship decisions with guardrails, anything with slow/delayed outcomes, anything requiring segment analysis — bandit allocation confounds time with arm). Hybrid patterns exist (bandit for allocation + holdout for measurement). Delayed feedback and nonstationarity are the practical bandit killers — say so when proposing one.

Lab Walkthrough Guidance

Lab 01 — Experiment Analyzer & Bandit, suggested order:

  1. required_sample_size — pin against the worked example (120k figure).
  2. two_proportion_ztest — hand-computed case first, then the A/A calibration test (simulated no-effect experiments must reject ≈ α — if this fails, your test is broken, full stop).
  3. srm_check — chi-square with the strict threshold; the sensitivity test shows a 1% imbalance at scale screaming while a 1% imbalance at n=200 stays quiet — that's the statistics working.
  4. cuped_adjust — implement θ from the formula; the test checks measured variance reduction ≈ $(1-\rho^2)$ and that the effect estimate stays unbiased.
  5. The bandits — EpsilonGreedy then ThompsonSampling (seeded RNG injected — Phase 01's testability discipline); the regret test asserts the ordering TS < ε-greedy < uniform on a fixed scenario.

Success Criteria

You are ready for Phase 12 when you can, from memory:

  1. Explain what randomization buys (exchangeability → causal estimate) and choose a unit for three scenarios.
  2. Compute a required sample size and run the MDE conversation for an underpowered proposal.
  3. State precisely what a p-value is and is not; explain why CIs must accompany decisions.
  4. Define SRM, its test, its threshold, and the only correct response to it.
  5. Explain why peeking inflates false positives and name the three legitimate alternatives.
  6. Derive CUPED's θ, state its two safety properties, and the pre-experiment rule.
  7. Describe Thompson sampling in four sentences and give the bandit-vs-A/B decision rule.

Interview Q&A

Q: Your new ranker won the A/B (+2% CTR, significant) but revenue is flat. Ship? First check the pre-registered decision rule — if CTR was the goal metric and revenue a guardrail that didn't degrade, the rule says ship. But interrogate the gap: CTR up with flat revenue suggests clickbait-shaped ranking (clicks without purchases — check downstream funnel metrics), position effects, or revenue being underpowered relative to CTR (almost always true — compute revenue's MDE before calling it "flat"; the CI will likely include meaningful gains and losses). The senior answer is the decomposition, not a yes/no.

Q: An experiment shows SRM (p = 1e-7). The PM wants to "just reweight the arms." Respond. No — SRM means units were selectively lost, and selection is correlated with treatment by construction (the treatment caused the loss: a crash, a timeout, a tracker change). Reweighting fixes the counts, not the selection bias; every metric remains untrustworthy. The fix is forensic: segment the mismatch by browser/platform/date to localize what's dropping users, repair, relaunch. Quote the industry stat (≈6–10% of experiments hit SRM) to normalize stopping.

Q: When would you use CUPED and what's the one way to get it wrong? Any user-level metric with decent pre-period correlation (returning-user products: ρ 0.4–0.7 typical → 16–50% variance reduction → weeks of traffic saved). The fatal error: using a covariate measured after assignment — it can carry treatment effect, biasing the estimate. Everything else is safe: θ misestimated just reduces less variance; new users degrade gracefully to unadjusted.

References

  • Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments (2020) — the book; ch. 3 (SRM), 17 (statistics), 22 (CUPED context)
  • Deng, Xu, Kohavi & Walker, Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data (WSDM 2013) — the CUPED paper
  • Fabijan et al., Diagnosing Sample Ratio Mismatch (KDD 2019) — the SRM taxonomy
  • Johari et al., Always Valid Inference: Continuous Monitoring of A/B Tests (2017) — peeking, solved properly
  • Russo et al., A Tutorial on Thompson Sampling (2018) — arXiv:1707.02038
  • Chapelle & Li, An Empirical Evaluation of Thompson Sampling (NeurIPS 2011)
  • Spotify, Netflix, and booking.com engineering blogs on experimentation — platform war stories