P10 — Online Experimentation and A/B Testing Platform

Run it first. There is a companion page that builds this project's machinery as numbered, independently runnable blocks and then assembles them into one measured system: P10 hands-on — block by block (handson/h*.py). Every number on it was produced by running the code. Read it alongside the milestones below.

Small · 44 hours · Weeks 96–99 · Stage 4 · Python

Table of Contents

• The Loop, Instantiated
• Why This Project Matters
• Prerequisites
• Duration and Size
• Central Technical Questions
• Architecture
• The Four Statistical Facts
• Showcase — Build the A/A Test First
• Implementation Milestones
• Concepts To Study
• Primary-Source Readings
• Experiments
• Benchmarks and Metrics
• Correctness Tests
• Failure Tests
• Expected Difficulties
• Scope Boundaries
• Deliverables
• Exit Criteria
• Extension Ideas
• Connections
• References

The Loop, Instantiated

Why This Project Matters

This is the shortest project in the journey and possibly the highest-leverage one, because it is the only place where the thing you learn is a constraint on what questions can be answered at all.

The minimum detectable effect calculation tells you, before you write any code, whether a proposed experiment is even runnable at your traffic. Most teams discover this after building the variant and running for three weeks. Doing the arithmetic first changes which projects you propose.

The second reason is the offline/online gap. P08 measured offline metrics. P09 simulated online behaviour. This project is where you find out that they disagree, and quantifying that disagreement is the most professionally useful result in Stage 4 — it is the answer to "why did the model that won offline not win in the A/B test", which is the single most common frustrating conversation in applied ML.

Prerequisites

• P09 complete — the simulated population is what you assign and analyse
• From math.md: §Hypothesis Testing (2 h), §Power and Sample Size (2 h)
• tools/metrics.py already implements Welch's t, power sizing and SRM

Duration and Size

Small, 44 hours, 4 weeks. Short because it sits on P09's harness. If you have to cut scope in Stage 4, this is the project to fold into P09 as a module — see the cut table.

Central Technical Questions

1. How small an effect can you detect with your traffic in a reasonable time? Do the arithmetic before building anything.
2. Why is peeking so dangerous, and by how much? Quantify it — the number surprises people.
3. What does a sample-ratio mismatch mean, and why does it invalidate the experiment rather than merely warn you?
4. When do offline metrics predict online outcomes, and when do they systematically fail?
5. What is a guardrail metric for, and what should happen when one moves?
6. What is interference, and does it apply to a recommender? (It does — via shared inventory and popularity feedback.)

Architecture

   user_id ──► eligibility check ──► bucket = hash(user_id + experiment_salt) % 10000
                                          │
                    ┌─────────────────────┴──────────────────────┐
                    ▼                                            ▼
              control (0–4999)                           treatment (5000–9999)
                    │                                            │
                    └──────────── exposure log ──────────────────┘
                          (user, experiment, arm, timestamp)
                                          │
      metric events ─────────────────────►│
                                          ▼
                        ┌──── analysis pipeline ─────┐
                        │  SRM check   (gate)        │
                        │  primary metric + CI       │
                        │  guardrails                │
                        │  segments                  │
                        │  multiple-comparison adj.  │
                        │  power achieved            │
                        └────────────────────────────┘

Two design points that are easy to get wrong:

Bucket on a stable id with a per-experiment salt. Hashing user_id alone means every experiment splits the population the same way, so experiment 2 inherits experiment 1's imbalance. Salting per experiment decorrelates them. Bucketing to 10,000 buckets rather than 2 lets you run a 1% ramp without re-randomising.

Log exposure, not assignment. A user assigned to treatment who never saw a recommendation contributes noise, not signal. Analysing on assignment (intent-to-treat) is unbiased but low-powered; analysing on exposure is higher-powered and biased if exposure depends on the arm. Know which you are doing and say so. The standard safe choice is to trigger the experiment at the point where the arms first diverge.

The Four Statistical Facts

Everything else in this project follows from these. All four numbers below were produced by scripts in this repository.

1. Halving the detectable effect quadruples the sample

\[ n_{\text{per arm}} = \frac{2(z_{1-\alpha/2} + z_{\text{power}})^2 \sigma^2}{\delta^2} \]

At α=0.05, power=0.8, the bracket is \((1.96 + 0.8416)^2 = 7.849\), which is where the folklore "16σ²/δ²" comes from. Computed for σ=0.5 (tools/metrics.py):

Exactly 4× per halving. Do this calculation before proposing an experiment. If your realistic effect is 0.5% and your traffic gives you 20,000 users a week, the experiment needs a year and should not be run.

2. Peeking destroys your false-positive rate

Simulated A/A tests (4,000 trials, 500 users per look, α=0.05, stop at first significant result):

Checking a dashboard daily for a fortnight turns a 5% error rate into roughly 25%. One in four "wins" is noise. This is not a subtle statistical nicety; it is the single largest source of false results in industrial experimentation, and the fix is either a fixed sample size decided in advance or an explicitly sequential method.

3. Sample-ratio mismatch invalidates, it does not warn

Chi-square with 1 d.f.; the conventional alarm is p < 0.001, i.e. χ² > 10.83:

A 0.5% imbalance on 400k users is a five-sigma event. It means assignment, logging, or filtering differs between arms — so the two populations are not comparable and no analysis of the metric is valid. SRM is a gate before analysis, not a footnote after it.

4. Twenty metrics, one false positive

Testing \(m\) metrics at α=0.05, the probability of at least one false positive is \(1 - 0.95^m\): 22.6% at m=5, 40.1% at m=10, 64.2% at m=20. Declare one primary metric in advance; treat everything else as guardrails (one-sided, looking for harm) or as exploratory (reported without claims).

Showcase — Build the A/A Test First

Thirty minutes. The A/A test is the foundation of everything else in this project, and the defect it catches is the most common one in real experimentation platforms.

# P10 -- the A/A test. Build this before anything else; it calibrates the rest.
import random, math
Z = 1.959963984540054
def welch(a, b):
    na, nb = len(a), len(b)
    ma, mb = sum(a)/na, sum(b)/nb
    va = sum((x-ma)**2 for x in a)/(na-1); vb = sum((x-mb)**2 for x in b)/(nb-1)
    return abs(mb-ma)/math.sqrt(va/na + vb/nb)

def aa_rate(n_per_arm, trials, correlated=False, seed=0):
    rng = random.Random(seed); hits = 0
    for _ in range(trials):
        if correlated:      # BUG: randomise by session, analyse by user
            a = [v for _ in range(n_per_arm//5) for v in [rng.gauss(0,1)]*5]
            b = [v for _ in range(n_per_arm//5) for v in [rng.gauss(0,1)]*5]
        else:
            a = [rng.gauss(0,1) for _ in range(n_per_arm)]
            b = [rng.gauss(0,1) for _ in range(n_per_arm)]
        if welch(a,b) > Z: hits += 1
    return hits/trials

print(f"correct A/A (independent observations): {aa_rate(500, 3000)*100:5.2f}%  <- must be ~5%")
print(f"A/A with correlated observations:       {aa_rate(500, 3000, True)*100:5.2f}%  <- inflated")
print("\\nThe second row is the most common real defect: randomising by session but")
print("analysing by user. Each user contributes 5 correlated rows, the variance")
print("estimate is too small, and the test over-rejects.")
print("\\nIf your A/A does not calibrate at 5%, NOTHING downstream is trustworthy.")
print("Debug this before you build a single variant.")

correct A/A (independent observations):  5.37%  <- must be ~5%
A/A with correlated observations:       38.30%  <- inflated
\nThe second row is the most common real defect: randomising by session but
analysing by user. Each user contributes 5 correlated rows, the variance
estimate is too small, and the test over-rejects.
\nIf your A/A does not calibrate at 5%, NOTHING downstream is trustworthy.
Debug this before you build a single variant.

38% against a nominal 5%. Randomising by session and analysing by user is not an exotic mistake; it is what happens by default when the logging table has one row per event. An A/A test catches it in an afternoon, and nothing else will.

Implementation Milestones

Concepts To Study

• Randomisation unit: user, session, request — and why the wrong choice creates correlated observations that break the variance estimate
• Deterministic hashing and salt; ramping without re-randomisation
• Intent-to-treat vs triggered analysis
• Guardrail metrics: latency, coverage, error rate, revenue — things that must not get worse even if the primary metric improves
• Sample-ratio mismatch and its causes: bot filtering, arm-dependent crashes, logging loss
• Statistical power, MDE, and the 4× law
• The peeking problem; fixed-horizon vs sequential testing; α-spending
• Multiple comparisons: family-wise error vs false discovery rate
• Novelty and primacy effects: a change can win in week 1 and lose in week 4
• Interference / SUTVA violation: in recommenders, arms compete for the same inventory and the same popularity signals
• Variance reduction: CUPED, stratification, common random numbers
• Stopping rules and the pre-registration discipline

Primary-Source Readings

Budget: 9 hours.

Experiments

E12 is the headline. Take at least ten variants from P08 — different α, different diversity λ, different freshness τ — and rank them by offline NDCG and by simulated online engagement. Compute the rank correlation.

Predict it first. If ρ ≈ 1, offline evaluation is sufficient and your P08 work stands. If ρ ≈ 0.3, offline metrics are nearly useless for choosing between these variants, and that is a finding worth writing up carefully — it is the quantitative version of an argument that is usually had with anecdotes. Either result is publishable-adjacent; the interesting part is diagnosing the specific variants that flip and naming the mechanism for each.

Benchmarks and Metrics

This project's "benchmarks" are statistical properties, not performance numbers:

Also report the operational numbers: assignment latency (must be sub-millisecond — it is on the request path), exposure-log volume, and analysis runtime.

Correctness Tests

1. A/A test calibrates at 5%. The single most important test. If it does not, every other result is void.
2. Deterministic assignment: the same user always gets the same arm for a given experiment.
3. Salt independence: assignment in experiment A is uncorrelated with experiment B.
4. Allocation accuracy: a 90/10 split produces 90/10 within sampling error.
5. Eligibility respected: ineligible users never appear in the analysis.
6. Exposure logging is exactly-once per user per experiment per period.
7. SRM gate blocks analysis — it must be impossible to read the primary metric on an SRM'd experiment. Enforce in code, not in policy.
8. Metric computations verified against hand-computed examples.
9. Immutable config: changing allocation mid-experiment is rejected, or forces a restart with a new salt.
10. Power calculation verified by simulation with a known effect.

Failure Tests

Expected Difficulties

1. The A/A test will not calibrate at 5% on your first attempt. Usual causes: correlated observations (randomising by session but analysing by user), an unstable metric, or a bug in the variance estimate. Debug this before anything else — it is the foundation.
2. You will want to peek. Build the tooling to make peeking impossible rather than discouraged: analysis refuses to run before the pre-registered sample size.
3. Effects in the simulator may be enormous, making everything trivially significant. Deliberately tune to realistic effect sizes (0.5–2% relative) or the project teaches nothing.
4. Interference is genuinely hard to detect. Cluster randomisation is the standard approach; expect this milestone to be the roughest.
5. This is a Small project and the statistics are deep. Resist expanding it. The sequential-testing rabbit hole in particular is an extension, not a milestone.

Scope Boundaries

In scope: assignment, exposure, metrics, guardrails, SRM, power, multiple comparisons, novelty, interference, stopping rules, offline/online comparison — all against the simulated population.

Out of scope: real users; a production feature-flag service; a UI; Bayesian experimentation (mention only); bandits as an experimentation method; causal inference beyond randomised experiments; a metric warehouse.

Deliverables

1. abtest/ — assignment, logging, analysis, pre-registration generator
2. PRE-REGISTRATION.md template — hypothesis, primary metric, MDE, required n, horizon, stopping rule, guardrails. Signed before the experiment. This template is the most directly transferable artifact in the project
3. REPORT.md centred on E12 (offline vs online) and E4 (peeking)
4. Notebook entries for E4, E9, E12
5. The four statistical facts reproduced with your own code

Exit Criteria

• A/A test calibrates at 5% over ≥1,000 runs
• SRM gate blocks analysis, verified by injection
• E4 complete: peeking FPR table reproduced
• E5 complete: achieved power matches nominal at a known effect
• E8 complete: a guardrail catches an injected regression
• E12 complete: ≥10 variants ranked offline and online, rank correlation reported, divergences diagnosed individually
• Pre-registration template written and used for every experiment in the project
• REPORT.md written with a falsified prediction

Extension Ideas

• Sequential testing (mSPRT or always-valid p-values) so peeking becomes legitimate. Measure the sample-size cost of that legitimacy — it is not free.
• CUPED: use pre-experiment data to reduce variance; measure the reduction and translate it into experiment-days saved.
• Switchback experiments for interference-heavy surfaces.
• Heterogeneous treatment effects: which user segments respond differently, with honest multiple-comparison handling.

Connections

Backward: P09 supplies the population and the harness. P08 supplies the variants.

Forward:

• → P15: "can simulated users predict the relative performance of ranking algorithms?" is answered by E12 plus P09's E2. If you pick that research question, this project is half the paper

References

• Kohavi, R., Tang, D., Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020.
• Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y., Pohlmann, N. Online Controlled Experiments at Large Scale. KDD 2013.
• Deng, A., Xu, Y., Kohavi, R., Walker, T. Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data. WSDM 2013.
• Johari, R., Koomen, P., Pekelis, L., Walsh, D. Peeking at A/B Tests: Why it matters, and what to do about it. KDD 2017.
• Kohavi, R., Longbotham, R. Unexpected Results in Online Controlled Experiments. SIGKDD Explorations 12(2), 2010.
• Gupta, S. et al. Top Challenges from the first Practical Online Controlled Experiments Summit. SIGKDD Explorations 21(1), 2019.
• Benjamini, Y., Hochberg, Y. Controlling the False Discovery Rate. JRSS B 57(1), 1995.
• Fisher, R. A. The Design of Experiments. Oliver & Boyd, 1935. Still the clearest statement of why randomisation works.
• Imbens, G. W., Rubin, D. B. Causal Inference for Statistics, Social, and Biomedical Sciences. Cambridge, 2015. For SUTVA and interference.