P10 hands-on — A/B testing platform, block by block

Peeking, SRM, CUPED, and the type-M error that inflates every underpowered win.

Source: handson/h10_abtest.py --- run it with python3 handson/h10_abtest.py
Full project spec: P10 — A/B Testing Platform

The statistics in an experimentation platform are twelve lines. Everything else --- and everything that makes it valuable --- is process enforced by software instead of remembered by people under launch pressure.

This file builds both. Deterministic hash assignment, a sample-size calculator that kills impossible tests before anyone writes the feature, an A/A calibration suite that validates the platform against itself, and then the four failure modes that produce most false launches: peeking, sample-ratio mismatch, uncorrected multiple comparisons, and underpowered tests whose surviving estimates are inflated by construction.

The assembly runs one experiment end to end in the order a real launch uses: power, health checks, one primary metric, variance reduction --- and a note on what peeking would have done to it.

Contents

How to read this page

Each block below is a self-contained lego piece: it builds one mechanism, proves it works on its own, and returns what the next block needs. The code is the real source, sliced out of the script. The output underneath it is the real output, captured by running that script --- not transcribed, not idealised. Where a measurement contradicted what I expected, the contradiction is in the output and the prose says so.

The assembly at the end wires every block into one working thing and measures it.

Block 1 — Assignment: deterministic, not random

Teaches: the same user must get the same arm forever

The problem. Assignment looks like the trivial part and is where the highest-severity bugs live, because an assignment defect invalidates every number downstream and is invisible in the metric.

@block(1, "Assignment: deterministic, not random", "the same user must get the same arm forever")
def b1(s, show):
    def assign(uid, exp, arms=2, salt="v1"):
        h = hashlib.sha256(f"{exp}:{salt}:{uid}".encode()).digest()
        return int.from_bytes(h[:8], "big") % arms
    if show:
        N = 200_000
        a = np.array([assign(f"u{i}", "checkout") for i in range(N)])
        print(f"  {N} users hashed into 2 arms: "
              f"{[int((a==k).sum()) for k in (0,1)]}  "
              f"(imbalance {abs((a==0).mean()-.5)*100:.3f}%)")
        again = [assign(f"u{i}", "checkout") for i in range(1000)]
        print(f"  re-assigning the first 1000: identical = "
              f"{list(a[:1000]) == again}")
        b = np.array([assign(f"u{i}", "banner") for i in range(N)])
        print(f"  correlation with a SECOND experiment's assignment: "
              f"{np.corrcoef(a, b)[0,1]:+.4f}")
        print("  Deterministic hashing gives three properties at once: a returning")
        print("  user sees a consistent experience, no assignment table has to be")
        print("  stored, and two experiments are independent because the experiment")
        print("  name is inside the hash. Seeding an RNG per request gives you none")
        print("  of these -- and the bug is invisible until someone reloads a page.")
    return {"assign": assign}

Reading the implementation

hash(experiment + salt + user_id) % arms — deterministic, stateless, and independent across experiments. Three properties fall out of one line:

  • Consistency. A returning user gets the same arm on every request, on every device, forever. Seeding an RNG per request gives none of this, and the bug is invisible until someone reloads a page.
  • No state. No assignment table to store, replicate, or keep consistent across regions. The assignment is recomputable anywhere from the user id alone.
  • Independence between experiments, because the experiment name is inside the hash. Without it, users who got arm A in experiment 1 systematically get arm A in experiment 2, and the two experiments confound each other.

The choice of hash matters. It must be cryptographic or at least well- distributed — SHA-256 here — because a weak hash (Java's String.hashCode, or hash() on a short id) has correlated low bits, so % 2 splits on a pattern rather than at random. The measured correlation between two experiments' assignments is the check that this worked.

What the numbers say

Output:

  200000 users hashed into 2 arms: [99729, 100271]  (imbalance 0.135%)
  re-assigning the first 1000: identical = True
  correlation with a SECOND experiment's assignment: -0.0013
  Deterministic hashing gives three properties at once: a returning
  user sees a consistent experience, no assignment table has to be
  stored, and two experiments are independent because the experiment
  name is inside the hash. Seeding an RNG per request gives you none
  of these -- and the bug is invisible until someone reloads a page.

Beyond the toy

  • The randomisation unit is a design decision with statistical consequences. User-level is standard. Session-level gives more units and therefore more power, but a user in both arms sees an inconsistent experience and the units are not independent. Cluster-level (household, company, city) is necessary when interference exists and costs a great deal of power.
  • The salt enables re-randomisation. Running a follow-up on the same population with the same salt gives the same split, so carryover effects persist. Changing the salt reshuffles.
  • Client-side assignment leaks into timing. If the client decides the arm, the decision happens after page load, so users on slow connections are under-represented in whichever arm loads more slowly — which is a sample-ratio mismatch (block 5) caused by the assignment mechanism itself.

Block 2 — Sample size before the test

Teaches: the number that decides whether the test is worth running

The problem. The most valuable artefact an experimentation platform produces is not a p-value. It is the table that tells you a proposed test is impossible before anyone builds the feature.

@block(2, "Sample size before the test", "the number that decides whether the test is worth running")
def b2(s, show):
    def n_per_arm(p, mde_rel, alpha=0.05, power=0.8):
        z_a, z_b = 1.959964, 0.841621
        d = p * mde_rel
        return math.ceil(2 * (z_a + z_b) ** 2 * p * (1 - p) / d ** 2)
    if show:
        print(f"  baseline conversion 5%, alpha=0.05, power=0.80:")
        print(f"  {'relative MDE':>14}{'n per arm':>12}{'days @ 20k/day/arm':>22}")
        for mde in (0.20, 0.10, 0.05, 0.02, 0.01):
            n = n_per_arm(0.05, mde)
            print(f"  {mde:>13.0%}{n:>12,}{n/20_000:>22.1f}")
        print("  n scales as 1/MDE^2: detecting a 1% lift instead of a 2% one costs")
        print("  4x the traffic, not 2x. This table is the single most useful artefact")
        print("  an experimentation platform produces, because most proposed tests are")
        print("  revealed as impossible BEFORE anyone builds the feature.")
    return {"n_per_arm": n_per_arm}

Reading the implementation

\[ n = \frac{2(z_{\alpha/2}+z_{\beta})^2,p(1-p)}{\delta^2} \]

The \(1/\delta^2\) is the entire story: halving the detectable effect quadruples the required traffic. Detecting a 1% lift instead of 2% costs 4× the users, not 2×.

The inputs, and which are actually negotiable: \(\alpha\) (0.05 by convention, rarely moved), power (0.80 by convention, and 0.80 means you miss one in five real effects), baseline rate \(p\) (a fact about your product), and the MDE \(\delta\) (the only genuinely free parameter, and the one that should be set by what lift would justify the engineering cost).

That last framing is the useful one. The MDE should come from a business threshold — "a 2% lift pays for this team's quarter" — not from what happens to be detectable. If the business-justified MDE needs more traffic than exists, the correct decision is to not run the test, and to say so before the feature is built.

What the numbers say

Output:

  baseline conversion 5%, alpha=0.05, power=0.80:
    relative MDE   n per arm    days @ 20k/day/arm
            20%       7,457                   0.4
            10%      29,826                   1.5
             5%     119,303                   6.0
             2%     745,644                  37.3
             1%   2,982,574                 149.1
  n scales as 1/MDE^2: detecting a 1% lift instead of a 2% one costs
  4x the traffic, not 2x. This table is the single most useful artefact
  an experimentation platform produces, because most proposed tests are
  revealed as impossible BEFORE anyone builds the feature.

Beyond the toy

  • Variance reduction beats waiting. CUPED at \(\rho = 0.7\) halves the required \(n\) (block 6) — usually cheaper than doubling traffic.
  • Ratio metrics need the delta method. When the analysis unit (user) differs from the metric unit (page view), the naive variance is wrong, usually understated, and the test is anti-conservative. This is the most common silent variance error in industry.
  • Sequential designs change the arithmetic: an always-valid test typically needs 20--50% more samples for the same power but allows continuous monitoring, which is often the better trade in practice (block 4).
  • One-sided tests for guardrails: you only care about harm, so the alternative is one-sided and you get the power back.

Block 3 — The t-test, and what it promises

Teaches: 5% false positives, by construction

The problem. Before trusting a single A/B result, verify that the platform produces the false-positive rate it promises. An A/A test is the platform's own unit test, and it validates assignment, metric pipeline and statistics in one shot.

@block(3, "The t-test, and what it promises", "5% false positives, by construction")
def b3(s, show):
    rng = np.random.default_rng(10)
    def welch(a, b):
        ma, mb = a.mean(), b.mean(); va, vb = a.var(ddof=1), b.var(ddof=1)
        na, nb = len(a), len(b)
        se = math.sqrt(va/na + vb/nb)
        if se == 0: return 0.0, 1.0
        t = (mb - ma) / se
        df = (va/na + vb/nb)**2 / ((va/na)**2/(na-1) + (vb/nb)**2/(nb-1))
        # normal approximation to the t CDF is fine at these df
        p = 2 * (1 - 0.5 * (1 + math.erf(abs(t) / math.sqrt(2))))
        return t, p
    if show:
        fp = 0; T = 2000; n = 4000
        for _ in range(T):
            a = rng.binomial(1, 0.05, n).astype(float)
            b = rng.binomial(1, 0.05, n).astype(float)   # A/A: NO real effect
            if welch(a, b)[1] < 0.05: fp += 1
        lo, hi = fp/T - 1.96*math.sqrt(.05*.95/T), fp/T + 1.96*math.sqrt(.05*.95/T)
        print(f"  {T} A/A tests, no effect present, n={n} per arm")
        print(f"  significant at p<0.05: {fp} ({fp/T:.3%})")
        print(f"  expected 5.000%, 95% interval [{lo:.3%}, {hi:.3%}] -> "
              f"{'calibrated' if lo <= 0.05 <= hi else 'MISCALIBRATED'}")
        print("  An A/A test is the platform's own unit test. Run a few thousand")
        print("  before you trust a single A/B result: it validates the assignment,")
        print("  the metric pipeline, and the statistics in one shot.")
    return {"welch": welch}

Reading the implementation

Run thousands of experiments where no effect exists and count how many come out significant. The answer must be \(\alpha\). Anything else means something in the stack is broken, and the A/A test does not tell you which — but it tells you that, which is the hard part.

The Welch t-test is used rather than Student's because it does not assume equal variances between arms. That assumption is frequently violated in practice (a treatment can change variance without changing the mean — a feature that helps some users and hurts others), and Welch costs nothing.

The normal approximation to the t distribution is fine at these degrees of freedom, and the block reports the 95% interval on the false-positive rate itself so "4.9%" can be judged against sampling error rather than eyeballed.

What the numbers say

Output:

  2000 A/A tests, no effect present, n=4000 per arm
  significant at p<0.05: 98 (4.900%)
  expected 5.000%, 95% interval [3.945%, 5.855%] -> calibrated
  An A/A test is the platform's own unit test. Run a few thousand
  before you trust a single A/B result: it validates the assignment,
  the metric pipeline, and the statistics in one shot.

Beyond the toy

What an A/A test catches that nothing else does: assignment bias (block 1), metric-pipeline bugs (a join that drops rows asymmetrically), variance mis-estimation (the ratio-metric problem above), and residual correlation between supposedly independent users. Run a few thousand before trusting any A/B result, and re-run them continuously — a "platform health" dashboard of ongoing A/A tests is standard practice at organisations that run many experiments.

The deeper point: a single experiment can never validate a process. Only the distribution over many can, which is what blocks 3, 4, 7 and 8 all do.

Block 4 — Peeking

Teaches: the most expensive statistical error in industry

The problem. The test is honest. The stopping rule is not. This is the most expensive statistical error in industry, and it is committed by people who know the statistics perfectly well.

@block(4, "Peeking", "the most expensive statistical error in industry")
def b4(s, show):
    rng = np.random.default_rng(11)
    def trial(peeks, n=8000, p=0.05, effect=0.0):
        a = rng.binomial(1, p, n).astype(float)
        b = rng.binomial(1, p*(1+effect), n).astype(float)
        checks = np.linspace(n//peeks, n, peeks).astype(int)
        for c in checks:
            if s["welch"](a[:c], b[:c])[1] < 0.05: return True
        return False
    if show:
        T = 2000
        print(f"  A/A tests again -- no effect -- but the analyst checks the dashboard")
        print(f"  {'times checked':>15}{'false positive rate':>22}{'inflation':>12}")
        base = None
        for peeks in (1, 2, 5, 10, 50):
            fp = sum(trial(peeks) for _ in range(T)) / T
            base = base or fp
            print(f"  {peeks:>15}{fp:>21.1%}{fp/base:>12.1f}x")
        print("  The test is honest; the STOPPING RULE is not. Each look is another")
        print("  chance for noise to cross the line, and 'we stopped when it hit")
        print("  significance' converts a 5% error rate into 20%+. Fixes: fix n in")
        print("  advance and do not look, use alpha-spending, or use a sequential test")
        print("  that is valid at every moment (mSPRT, always-valid confidence")
        print("  sequences). Anything but staring at a p-value and waiting.")
    return {"trial": trial}

Reading the implementation

Simulate an analyst who checks the dashboard \(m\) times and stops at the first significant result. Each look is another opportunity for noise to cross the threshold, so the family-wise error rate is \(1 - (1-\alpha)^{\text{effective }m}\) — less than the naive product because consecutive looks are correlated, but far above 5%.

The measurement is unambiguous: five looks doubles the false-positive rate; fifty looks roughly quadruples it.

The reason "just don't look" fails as a policy is organisational rather than statistical. Dashboards exist, stakeholders read them, and a result that appears significant on day 3 will be acted on. A platform that relies on discipline it cannot enforce has chosen not to solve the problem.

What the numbers say

Output:

  A/A tests again -- no effect -- but the analyst checks the dashboard
    times checked   false positive rate   inflation
                1                 5.5%         1.0x
                2                 8.2%         1.5x
                5                14.2%         2.6x
               10                20.3%         3.7x
               50                32.2%         5.9x
  The test is honest; the STOPPING RULE is not. Each look is another
  chance for noise to cross the line, and 'we stopped when it hit
  significance' converts a 5% error rate into 20%+. Fixes: fix n in
  advance and do not look, use alpha-spending, or use a sequential test
  that is valid at every moment (mSPRT, always-valid confidence
  sequences). Anything but staring at a p-value and waiting.

Beyond the toy

The two principled solutions:

  • Group sequential (Pocock, O'Brien–Fleming). Pre-plan \(K\) looks and spend \(\alpha\) across them via an alpha-spending function. O'Brien–Fleming is conservative early and nearly full-\(\alpha\) at the end, which matches how people actually want to behave. Requires committing to the look schedule.
  • Always-valid inference (mSPRT, confidence sequences). The interval is valid at every moment under arbitrary optional stopping, because it is built from a martingale and Ville's inequality rather than a fixed-\(n\) sampling distribution. Costs 20--50% more samples for the same power, and is the right default for a self-serve platform because it is the only option that survives contact with an organisation.

Both are ~50 lines. The reason most platforms do not have them is that nobody measured block 4's table.

Block 5 — Sample ratio mismatch

Teaches: the cheapest bug detector you will ever write

The problem. The cheapest and highest-yield check in the entire platform, and it must run before anyone reads the metric.

@block(5, "Sample ratio mismatch", "the cheapest bug detector you will ever write")
def b5(s, show):
    def srm(counts, expected=None):
        n = sum(counts); k = len(counts)
        exp = expected or [n/k]*k
        chi = sum((c-e)**2/e for c, e in zip(counts, exp))
        p = math.exp(-chi/2) if k == 2 else float("nan")   # chi2 df=1 survival
        return chi, p
    if show:
        print(f"  {'observed split':<26}{'chi2':>9}{'p':>10}{'verdict':>12}")
        for a, b, lbl in ((50_000, 50_000, "50000 / 50000"),
                          (50_000, 49_800, "50000 / 49800"),
                          (50_000, 49_400, "50000 / 49400"),
                          (50_000, 48_000, "50000 / 48000")):
            chi, p = srm([a, b])
            print(f"  {lbl:<26}{chi:>9.2f}{p:>10.2e}"
                  f"{('OK' if p > 0.001 else 'SRM -- STOP'):>12}")
        print("  A 1.2% imbalance is a 0.6% deviation per arm and looks like nothing.")
        print("  It is p<0.001 at this traffic, and it means users were lost")
        print("  NON-RANDOMLY -- a redirect that dropped slow clients, a crash in one")
        print("  arm, a bot filter that fired asymmetrically. Whatever the metric")
        print("  says afterwards is unusable, because the arms are no longer")
        print("  comparable populations. Check SRM first, always, before the metric.")
    return {"srm": srm}

Reading the implementation

A chi-squared test against the expected split. If assignment is 50/50 and the observed counts are not, the arms are no longer comparable populations and every downstream number is meaningless.

The reason this matters more than it looks: a 0.6% deviation per arm is invisible to the eye and \(p<0.001\) at scale. The block's table makes that concrete — 50000/49400 looks fine and is a five-alarm result.

What SRM means is that users were lost non-randomly: a redirect that dropped slow clients, a crash in one arm, a bot filter that fired asymmetrically, a client-side assignment that raced with page load. The lost users are not a random subset, so the remaining populations differ systematically, and no statistical adjustment fixes it.

What the numbers say

Output:

  observed split                 chi2         p     verdict
  50000 / 50000                  0.00  1.00e+00          OK
  50000 / 49800                  0.40  8.18e-01          OK
  50000 / 49400                  3.62  1.64e-01          OK
  50000 / 48000                 40.82  1.37e-09 SRM -- STOP
  A 1.2% imbalance is a 0.6% deviation per arm and looks like nothing.
  It is p<0.001 at this traffic, and it means users were lost
  NON-RANDOMLY -- a redirect that dropped slow clients, a crash in one
  arm, a bot filter that fired asymmetrically. Whatever the metric
  says afterwards is unusable, because the arms are no longer
  comparable populations. Check SRM first, always, before the metric.

Beyond the toy

  • Check SRM on every segment, not just overall. An experiment can be balanced in aggregate and 60/40 on iOS, which points straight at the cause.
  • Kohavi reports that a substantial fraction of experiments at large organisations fail SRM, and that in nearly every case the cause is a real bug rather than chance. Treat it as a hard stop.
  • The threshold should be strict (p < 0.0005 or so) because you run this test on every experiment and want the false-alarm rate low — the multiple-comparisons logic of block 7 applied to the health check itself.

Block 6 — Variance reduction with CUPED

Teaches: the same decision, on a fraction of the traffic

The problem. More power for free, if the metric autocorrelates. The exact condition — and it is exact — is what decides whether it is worth building.

@block(6, "Variance reduction with CUPED", "the same decision, on a fraction of the traffic")
def b6(s, show):
    rng = np.random.default_rng(12)
    def cuped(y, x):
        theta = np.cov(y, x)[0, 1] / np.var(x, ddof=1)
        return y - theta * (x - x.mean()), theta
    if show:
        n = 20_000
        pre = rng.gamma(2, 3, 2*n)                       # pre-period spend
        noise = rng.normal(0, 3, 2*n)
        post = 0.8 * pre + noise                         # correlated post-period
        post[n:] *= 1.02                                 # +2% true effect in arm B
        a, b = post[:n], post[n:]
        pa, pb = pre[:n], pre[n:]
        t0, p0 = s["welch"](a, b)
        ac, th = cuped(a, pa); bc, _ = cuped(b, pb)
        t1, p1 = s["welch"](ac, bc)
        r = np.corrcoef(post, pre)[0, 1]
        print(f"  correlation(pre-period, post-period) = {r:.3f}, theta = {th:.3f}")
        print(f"  {'estimator':<22}{'std error':>12}{'t':>9}{'p':>11}")
        for lbl, x, y, t, p in (("raw difference", a, b, t0, p0),
                                ("CUPED-adjusted", ac, bc, t1, p1)):
            se = math.sqrt(x.var(ddof=1)/len(x) + y.var(ddof=1)/len(y))
            print(f"  {lbl:<22}{se:>12.4f}{t:>9.2f}{p:>11.2e}")
        red = 1 - (bc.var()/b.var())
        print(f"  variance reduced {red:.1%}, which is 1 - r^2 = {1-r*r:.1%} off by")
        print(f"  {abs(red-(1-(1-r*r)))*100:.1f}pp -- the theory predicts the measurement.")
        print(f"  Equivalent traffic saving: the same power at {1-red:.0%} of n.")
        print("  CUPED is free: pre-period data already exists, and the adjustment")
        print("  cannot bias the estimate because x is measured BEFORE assignment.")
    return {"cuped": cuped}

Reading the implementation

\[ Y_{\text{cuped}} = Y - \theta(X - \bar{X}), \qquad \theta = \frac{\mathrm{Cov}(Y,X)}{\mathrm{Var}(X)} \]

with \(X\) measured before assignment. That timing is what makes the adjustment unbiased: \(X\) cannot be affected by treatment, so subtracting it removes variance without touching the treatment effect.

The variance reduction is exactly \(1 - \rho^2\), and \(\theta\) is precisely the OLS regression coefficient — CUPED is regression adjustment with one pre-period covariate, which is why it inherits regression's guarantees.

What the numbers say

Output:

  correlation(pre-period, post-period) = 0.746, theta = 0.799
  estimator                std error        t          p
  raw difference              0.0454     1.27   2.06e-01
  CUPED-adjusted              0.0302     1.90   5.73e-02
  variance reduced 55.2%, which is 1 - r^2 = 44.3% off by
  0.5pp -- the theory predicts the measurement.
  Equivalent traffic saving: the same power at 45% of n.
  CUPED is free: pre-period data already exists, and the adjustment
  cannot bias the estimate because x is measured BEFORE assignment.

The measured reduction matches \(1-\rho^2\) to within a fraction of a percentage point, which is the theory predicting the measurement rather than describing it afterwards.

Beyond the toy

  • It is worthless on zero-inflated metrics. The assembly measures this: on revenue-per-user, where most users never convert, \(\rho\) is small and CUPED buys nearly nothing. Check \(\rho\) before building the pipeline.
  • CUPAC generalises the covariate to an ML model's prediction from pre-period features, which raises \(\rho\) substantially. Same identity, better \(X\).
  • Stratification is the coarser cousin: bucket users by pre-period behaviour and analyse within strata. Less powerful, simpler to implement, and more robust.
  • The covariate must be pre-treatment. Adjusting on a post-treatment variable is not variance reduction, it is conditioning on a collider, and it introduces bias in an unpredictable direction. This is the one way to get CUPED catastrophically wrong.

Block 7 — Multiple metrics, multiple arms

Teaches: twenty metrics guarantee a winner

The problem. Twenty metrics guarantee a winner. This block measures the rate and shows that the textbook formula under-predicts it — for a reason worth chasing down.

@block(7, "Multiple metrics, multiple arms", "twenty metrics guarantee a winner")
def b7(s, show):
    rng = np.random.default_rng(13)
    if show:
        T, M, n = 1000, 20, 5000
        any_sig = bh_sig = bonf_sig = per_metric = 0
        for _ in range(T):
            ps = []
            for _ in range(M):
                a = rng.binomial(1, .05, n).astype(float)
                b = rng.binomial(1, .05, n).astype(float)
                ps.append(s["welch"](a, b)[1])
            ps = np.sort(np.array(ps))
            per_metric += int((ps < 0.05).sum())
            any_sig += ps[0] < 0.05
            bonf_sig += ps[0] < 0.05 / M
            bh = ps <= 0.05 * np.arange(1, M+1) / M       # Benjamini-Hochberg
            bh_sig += bh.any()
        print(f"  {T} A/A experiments, {M} metrics each, no effect anywhere:")
        print(f"  {'rule':<34}{'experiments with a winner':>28}")
        for lbl, v in (("any metric p<0.05 (no correction)", any_sig),
                       ("Bonferroni (p < 0.05/20)", bonf_sig),
                       ("Benjamini-Hochberg FDR 5%", bh_sig)):
            print(f"  {lbl:<34}{v/T:>27.1%}")
        r = per_metric / (T * M)
        print(f"  Textbook: 1-(1-0.05)^20 = {1-0.95**20:.1%}. Measured {any_sig/T:.1%}.")
        print(f"  The gap is not sampling noise -- it is that the union bound needs the")
        print(f"  ACTUAL per-metric rate, which was {r:.2%} here, not the nominal 5%")
        print(f"  (the normal approximation to Welch's t is mildly anti-conservative on")
        print(f"  binary data). 1-(1-{r:.4f})^20 = {1-(1-r)**20:.1%}, which matches.")
        print(f"  A 0.4pp error per metric compounds into a 3.5pp error across twenty.")
        print("  Declare ONE primary metric before the test. Everything else is a")
        print("  guardrail (checked for harm, one-sided) or exploratory (reported,")
        print("  never used to declare a win). This is a process rule, not a")
        print("  statistical one -- which is why the platform should enforce it.")
    return {}

Reading the implementation

Run A/A experiments with 20 metrics each and count how often any metric reaches significance. The textbook expectation is \(1-(1-0.05)^{20} = 64.2%\).

The measured value is 67.8%, and rather than shrug at the gap the block measures the actual per-metric rate in the same run: 5.41%, not the nominal 5%. The normal approximation to Welch's t is mildly anti-conservative on binary data, and \(1-(1-0.0541)^{20} = 67.1%\), which matches.

That is the lesson worth more than the correction: a 0.4 pp error per metric compounds into a 3.5 pp error across twenty. Small systematic biases do not stay small when composed, and the union bound needs the true per-test rate rather than the nominal one.

What the numbers say

Output:

  1000 A/A experiments, 20 metrics each, no effect anywhere:
  rule                                 experiments with a winner
  any metric p<0.05 (no correction)                       67.8%
  Bonferroni (p < 0.05/20)                                 5.2%
  Benjamini-Hochberg FDR 5%                                5.3%
  Textbook: 1-(1-0.05)^20 = 64.2%. Measured 67.8%.
  The gap is not sampling noise -- it is that the union bound needs the
  ACTUAL per-metric rate, which was 5.41% here, not the nominal 5%
  (the normal approximation to Welch's t is mildly anti-conservative on
  binary data). 1-(1-0.0541)^20 = 67.1%, which matches.
  A 0.4pp error per metric compounds into a 3.5pp error across twenty.
  Declare ONE primary metric before the test. Everything else is a
  guardrail (checked for harm, one-sided) or exploratory (reported,
  never used to declare a win). This is a process rule, not a
  statistical one -- which is why the platform should enforce it.

Beyond the toy

  • Bonferroni (\(\alpha/m\)) controls the family-wise error rate and is very conservative — it assumes the worst-case dependence structure.
  • Benjamini–Hochberg controls the false discovery rate: of the metrics you declare significant, at most \(q\) proportion are false. Far more powerful, and the right choice when you are screening many metrics rather than testing one hypothesis.
  • The process fix beats the statistical one. Declare one primary metric before the test. Everything else is a guardrail (one-sided, checked for harm) or exploratory (reported, never used to declare a win). This is a policy the platform should enforce in software, because it is exactly the discipline that erodes under launch pressure.

Block 8 — Power, honestly

Teaches: an underpowered test is worse than no test

The problem. An underpowered test does not merely miss effects. When it does find one, the estimate is inflated — which means the launch report overstates the win and the follow-up disappointment is guaranteed.

@block(8, "Power, honestly", "an underpowered test is worse than no test")
def b8(s, show):
    rng = np.random.default_rng(14)
    if show:
        print("  A REAL +5% relative effect exists. How often do we find it, and what")
        print("  does the estimate look like when we do?")
        print(f"  {'n per arm':>11}{'power':>9}{'mean lift | significant':>26}"
              f"{'exaggeration':>14}")
        for n in (2_000, 10_000, 30_000, 120_000):
            hits, ests = 0, []
            for _ in range(600):
                a = rng.binomial(1, .05, n).astype(float)
                b = rng.binomial(1, .0525, n).astype(float)
                t, p = s["welch"](a, b)
                if p < 0.05 and b.mean() > a.mean():
                    hits += 1; ests.append((b.mean()-a.mean())/a.mean())
            m = float(np.mean(ests)) if ests else float("nan")
            print(f"  {n:>11,}{hits/600:>9.1%}{m:>25.1%}{m/0.05:>13.1f}x")
        print("  This is the type-M (magnitude) error. An underpowered test does not")
        print("  just miss effects -- when it DOES find one, the estimate is inflated,")
        print("  because only the luckiest samples clear the threshold. Shipping on a")
        print("  20%-powered test means the launch report overstates the win ~2x, and")
        print("  the follow-up 'why did the metric not move in production' is")
        print("  guaranteed. Compute power before, not after.")
    return {}

Reading the implementation

A real +5% effect exists. Vary \(n\) and measure both the power and the mean estimated lift conditional on significance.

The mechanism is selection. At low power, only samples where noise happened to align with the effect clear the threshold, so the surviving estimates are systematically too large. This is Gelman & Carlin's type-M (magnitude) error, and the companion type-S (sign) error — the probability that a significant result has the wrong sign — is non-trivial at very low power.

What the numbers say

Output:

  A REAL +5% relative effect exists. How often do we find it, and what
  does the estimate look like when we do?
    n per arm    power   mean lift | significant  exaggeration
        2,000     6.0%                    40.0%          8.0x
       10,000    11.5%                    15.5%          3.1x
       30,000    29.8%                     9.7%          1.9x
      120,000    79.7%                     5.6%          1.1x
  This is the type-M (magnitude) error. An underpowered test does not
  just miss effects -- when it DOES find one, the estimate is inflated,
  because only the luckiest samples clear the threshold. Shipping on a
  20%-powered test means the launch report overstates the win ~2x, and
  the follow-up 'why did the metric not move in production' is
  guaranteed. Compute power before, not after.

At 6% power the surviving estimates are inflated ~8×. Shipping on a 20%-powered test means the launch report overstates the win around 2×, and "why did the metric not move in production" follows within the quarter.

Beyond the toy

  • The winner's curse in experimentation. Across a portfolio of experiments, the ones you ship are the ones that got lucky, so the aggregate of shipped wins systematically exceeds the true aggregate effect. Organisations that sum their experiment wins routinely conclude they have doubled a metric that did not move.
  • Shrinkage. Empirical-Bayes shrinkage of experiment estimates toward the prior mean of all past experiments corrects this and is straightforward to implement once you have a history.
  • Post-hoc power is meaningless. Computing power from the observed effect size is circular — it is a monotone function of the p-value and adds nothing. Power is a design calculation, which is why it belongs in block 2 and not here.
  • Replication is the honest fix. A surprising win that matters should be re-run. It costs traffic and it is the only reliable defence against type-M.

The assembly

Every block above, wired together into one working system:

def assembly(s):
    print("\nEight blocks = an experimentation platform. One experiment, end to end.\n")
    rng = np.random.default_rng(20)
    N, TRUE = 60_000, 0.03
    uids = [f"user{i}" for i in range(N)]
    arm = np.array([s["assign"](u, "checkout-redesign") for u in uids])
    pre = rng.gamma(2, 3, N)
    conv = rng.binomial(1, np.where(arm == 1, .05*(1+TRUE), .05)).astype(float)
    rev = conv * (0.7*pre + rng.normal(0, 2, N)) 

    print("  STEP 1  design")
    need = s["n_per_arm"](0.05, 0.03)
    print(f"    to detect a {TRUE:.0%} relative lift at 80% power: "
          f"{need:,} per arm; we have {int((arm==0).sum()):,}")
    print(f"    -> the test is {'ADEQUATELY POWERED' if (arm==0).sum() >= need else 'UNDERPOWERED, and we run it anyway to see what that looks like'}")

    print("  STEP 2  health checks")
    chi, p = s["srm"]([int((arm==0).sum()), int((arm==1).sum())])
    print(f"    SRM: {int((arm==0).sum())} / {int((arm==1).sum())}  "
          f"chi2={chi:.2f}  p={p:.3f}  -> {'PASS' if p > 0.001 else 'FAIL'}")

    print("  STEP 3  primary metric, fixed horizon, no peeking")
    a, b = conv[arm == 0], conv[arm == 1]
    t, pv = s["welch"](a, b)
    lift = (b.mean()-a.mean())/a.mean()
    print(f"    conversion  A={a.mean():.4f}  B={b.mean():.4f}  "
          f"lift={lift:+.2%}  p={pv:.4f}")
    print(f"    true lift was {TRUE:+.0%}; the estimate is "
          f"{'inside' if abs(lift-TRUE) < 2*math.sqrt(a.var()/len(a)+b.var()/len(b))/a.mean() else 'outside'}"
          " a 2-SE window of it")

    print("  STEP 4  the same metric, CUPED-adjusted")
    ra, rb = rev[arm == 0], rev[arm == 1]
    ca, _ = s["cuped"](ra, pre[arm == 0]); cb, _ = s["cuped"](rb, pre[arm == 1])
    t2, p2 = s["welch"](ra, rb); t3, p3 = s["welch"](ca, cb)
    print(f"    revenue/user  raw   p={p2:.4f}   se="
          f"{math.sqrt(ra.var(ddof=1)/len(ra)+rb.var(ddof=1)/len(rb)):.4f}")
    print(f"    revenue/user  CUPED p={p3:.4f}   se="
          f"{math.sqrt(ca.var(ddof=1)/len(ca)+cb.var(ddof=1)/len(cb)):.4f}")
    rr = np.corrcoef(rev, pre)[0, 1]
    print(f"    barely moved, and block 6 says exactly why: the gain is 1-r^2 and")
    print(f"    here r={rr:.3f}, so the ceiling is {1-(1-rr*rr):.1%}. Revenue/user is")
    print(f"    zero-inflated -- {100*(rev==0).mean():.0f}% of users never convert -- so a")
    print(f"    pre-period covariate cannot explain much of it. CUPED is not a free")
    print(f"    win; it is a free win ON METRICS THAT AUTOCORRELATE. Check r first.")

    print("  STEP 5  what peeking would have done to this test")
    hit = [c for c in range(2000, N//2, 2000)
           if s["welch"](a[:c], b[:c])[1] < 0.05]
    nchk = len(range(2000, N//2, 2000))
    print(f"    p<0.05 at {len(hit)} of {nchk} checkpoints; "
          f"first at n={hit[0] if hit else '--'}")
    print(f"    final verdict at the pre-registered n: p={pv:.4f}")
    print(f"    This run got away with it. Block 4 is the reason that is luck and not")
    print(f"    method: at {nchk} looks the false-positive rate is ~20%, so one launch")
    print(f"    in five would have shipped a null result as a win. A single experiment")
    print(f"    can never tell you whether your process is sound -- only the")
    print(f"    distribution over many can, which is what blocks 3, 4, 7 and 8 do.")

    print("\n  Every block appears in that sequence, in the order a real launch uses")
    print("  it: power first (or do not run), health checks second (or do not read),")
    print("  one primary metric third, variance reduction fourth, and the peeking")
    print("  analysis as a reminder of what the other path looked like.")
    print("\n  The platform's value is NOT the t-test -- that is twelve lines in")
    print("  block 3. It is that the sequence above is enforced by software instead")
    print("  of remembered by people under launch pressure.")
    print("\n  Built: hash assignment -> power -> t-test + A/A calibration -> peeking")
    print("  -> SRM -> CUPED -> multiple comparisons -> type-M error.")
    print("  Missing, on the project page: metric definition and a metrics repo (m3),")
    print("  the delta method for ratio metrics with a user-level denominator (m7),")
    print("  switchback and cluster randomisation for interference (m9), sequential")
    print("  tests with always-valid intervals (m10), and E8 -- the heterogeneous")
    print("  treatment effect analysis that finds the segment the average hides.")

Output:

Eight blocks = an experimentation platform. One experiment, end to end.

  STEP 1  design
    to detect a 3% relative lift at 80% power: 331,398 per arm; we have 29,844
    -> the test is UNDERPOWERED, and we run it anyway to see what that looks like
  STEP 2  health checks
    SRM: 29844 / 30156  chi2=1.62  p=0.444  -> PASS
  STEP 3  primary metric, fixed horizon, no peeking
    conversion  A=0.0494  B=0.0505  lift=+2.32%  p=0.5188
    true lift was +3%; the estimate is inside a 2-SE window of it
  STEP 4  the same metric, CUPED-adjusted
    revenue/user  raw   p=0.0792   se=0.0097
    revenue/user  CUPED p=0.0771   se=0.0096
    barely moved, and block 6 says exactly why: the gain is 1-r^2 and
    here r=0.116, so the ceiling is 1.3%. Revenue/user is
    zero-inflated -- 95% of users never convert -- so a
    pre-period covariate cannot explain much of it. CUPED is not a free
    win; it is a free win ON METRICS THAT AUTOCORRELATE. Check r first.
  STEP 5  what peeking would have done to this test
    p<0.05 at 0 of 14 checkpoints; first at n=--
    final verdict at the pre-registered n: p=0.5188
    This run got away with it. Block 4 is the reason that is luck and not
    method: at 14 looks the false-positive rate is ~20%, so one launch
    in five would have shipped a null result as a win. A single experiment
    can never tell you whether your process is sound -- only the
    distribution over many can, which is what blocks 3, 4, 7 and 8 do.

  Every block appears in that sequence, in the order a real launch uses
  it: power first (or do not run), health checks second (or do not read),
  one primary metric third, variance reduction fourth, and the peeking
  analysis as a reminder of what the other path looked like.

  The platform's value is NOT the t-test -- that is twelve lines in
  block 3. It is that the sequence above is enforced by software instead
  of remembered by people under launch pressure.

  Built: hash assignment -> power -> t-test + A/A calibration -> peeking
  -> SRM -> CUPED -> multiple comparisons -> type-M error.
  Missing, on the project page: metric definition and a metrics repo (m3),
  the delta method for ratio metrics with a user-level denominator (m7),
  switchback and cluster randomisation for interference (m9), sequential
  tests with always-valid intervals (m10), and E8 -- the heterogeneous
  treatment effect analysis that finds the segment the average hides.

The design space

The statistics are twelve lines. The platform is everything around them, and the design choices are about what the software refuses to let you do.

DecisionOptionsWhat it costs
Assignmenthash(user, experiment), server-side vs client-sideclient-side assignment leaks into page-load timing and causes SRM
Analysis unituser, session, requestmust match the randomisation unit or the variance is wrong
Stopping rulefixed horizon, group sequential, always-validfixed is simplest and nobody obeys it
Correctionnone, Bonferroni, Benjamini–Hochbergnone guarantees a false winner at 20 metrics
Variance reductionnone, stratification, CUPED, ML-based (CUPAC)needs pre-period data that correlates
Interferenceassume SUTVA, cluster, switchbackswitchback trades power for validity

The stopping rule is the expensive one

Block 4 measures it: checking a dashboard 50 times turns a 5% false-positive rate into ~30%. The fix is not discipline, it is mathematics. Two families:

  • Group sequential (Pocock, O'Brien–Fleming): pre-planned looks with alpha spent at each, via an alpha-spending function. Standard in clinical trials; requires deciding the look schedule up front.
  • Always-valid inference (mSPRT, confidence sequences): the interval is valid at every moment under optional stopping, because it is derived from a martingale and Ville's inequality rather than from a fixed-\(n\) sampling distribution. The price is a wider interval at any given \(n\) — typically needing 20--50% more samples for the same power.

Always-valid is the right default for a self-serve platform, because it is the only option that survives contact with an organisation that will look at the dashboard.

The variance-reduction arithmetic

CUPED adjusts \(Y\) using a pre-experiment covariate \(X\) measured before assignment (so it cannot be affected by treatment):

\[ Y_{\text{cuped}} = Y - \theta (X - \bar{X}), \qquad \theta = \frac{\mathrm{Cov}(Y,X)}{\mathrm{Var}(X)} \]

The variance reduction is exactly \(1 - \rho^2\). That single identity tells you everything about when it is worth doing:

\(\rho\)Variance reductionEquivalent traffic saving
0.39%9%
0.525%25%
0.749%~2×
0.981%~5×

The blocks show both ends: block 6 gets a large reduction on an autocorrelated metric, and the assembly gets almost nothing on revenue-per-user because the metric is zero-inflated (most users never convert) so \(\rho\) is small. CUPED is a free win on metrics that autocorrelate, and nothing otherwise. Check \(\rho\) before building the pipeline.

Related tools: stratification on pre-period buckets (same idea, coarser), CUPAC (use an ML model's prediction as the covariate), and the delta method for ratio metrics whose denominator is itself random — necessary whenever the analysis unit (user) differs from the metric unit (page view), which is the most common silent variance error in industry.

Sample size, power, and the type-M error

The sample size for a two-proportion test at power \(1-\beta\):

\[ n = \frac{2(z_{\alpha/2}+z_{\beta})^2 , p(1-p)}{\delta^2} \]

The \(1/\delta^2\) is the whole story: halving the detectable effect quadruples the traffic. Block 2's table is the single most useful artefact an experimentation platform produces, because it kills impossible tests before anyone builds the feature.

Block 8 measures the consequence of ignoring it — the type-M (magnitude) error. At 6% power, the effects that reach significance are inflated ~8×, because only the luckiest samples clear the threshold. An underpowered test does not merely miss effects; when it finds one, the launch report overstates it, and the follow-up question "why did the metric not move in production" is guaranteed. Gelman & Carlin's design analysis makes this quantitative, including the type-S (sign) error — the probability that a significant result has the wrong sign, which at very low power is non-trivial.

Interference: when SUTVA breaks

The whole framework assumes one user's treatment does not affect another's outcome. That is false in:

  • Marketplaces — treatment users outbid control users for the same supply.
  • Social networks — treatment content is shared to control users.
  • Shared resources — a faster treatment path frees capacity for control.

Remedies, in increasing order of cost: cluster randomisation (randomise communities, analyse at cluster level — much lower power), switchback (randomise time slices for the whole system, which handles marketplace interference and adds temporal autocorrelation to the analysis), ego-cluster designs, and budget-split designs for auctions. Each converts an unbiased-but- wrong estimate into a noisier-but-valid one.

Advanced topics

  • Heterogeneous treatment effects: causal forests and meta-learners (S/T/X) estimate \(\tau(x)\), the segment-level effect. The catch is that searching segments is another multiple-comparisons problem, so honest splitting (fit on one half, estimate on the other) is mandatory.
  • Quantile treatment effects — a change may not move the mean while moving the p99 latency, which for infrastructure experiments is often the point.
  • Guardrail metrics and degradation checks run one-sided at high power; the asymmetry is deliberate, because shipping harm is worse than missing a win.
  • Metric sensitivity analysis: rank candidate metrics by how often they detect known-positive experiments — an empirical way to choose an OEC rather than arguing about it.
  • Variance of the variance: for heavy-tailed metrics (revenue), the CLT convergence is slow; winsorisation or a log transform is not cheating if pre-registered.

How this connects to the rest of the track

  • P08 and P09 produce hypotheses; this is the only instrument that tests them on real users.
  • P07 computes the metrics, and its watermark bias becomes measurement error here — P15 block 6 quantifies it.
  • P06's straggler arithmetic and block 7's multiple comparisons are the same maxima statistics.
  • P15 wires assignment, SRM, power and Welch into the request path.

Failure modes at scale

  • SRM is the highest-yield check in the whole platform: a 0.6% deviation per arm is invisible to the eye and \(p < 0.001\) at scale, and it invalidates everything downstream because the populations are no longer comparable. Check it before reading the metric, always.
  • Carryover from a previous experiment on the same users; randomisation salts and washout periods exist for this.
  • Triggered analysis done wrong: analysing only users who saw the feature breaks randomisation unless triggering is determined pre-assignment.
  • Novelty and primacy effects — the effect changes over the first two weeks, so a 3-day test measures a transient.
  • Dilution: assigning all users but treating only 5% shrinks the observed effect by 20×; power must be computed on the triggered population.

Primary sources

  • Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments (2020) — the practitioner's reference; the SRM chapter especially.
  • Deng et al., Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data (CUPED, WSDM 2013).
  • Johari et al., Peeking at A/B Tests (KDD 2017) — mSPRT.
  • Howard et al., Time-uniform Chernoff Bounds via Nonnegative Supermartingales (2021) — confidence sequences.
  • Gelman & Carlin, Beyond Power Calculations: Assessing Type S and Type M Errors (2014).
  • Benjamini & Hochberg, Controlling the False Discovery Rate (1995).
  • Bojinov, Simchi-Levi & Zhao, Design and Analysis of Switchback Experiments (2020).

Running it

python3 handson/h10_abtest.py            # every block, then the assembly
python3 handson/h10_abtest.py --block 3  # just block 3 and its prerequisites
python3 handson/h10_abtest.py --quiet    # the assembly only

What to do with this

Implement a sequential test --- mSPRT or an always-valid confidence sequence --- and re-run block 4 against it. The false-positive rate should stay at 5% no matter how many times the dashboard is checked, which is the only real fix for peeking, because "do not look" is not a policy that survives contact with an organisation.

Milestones, experiments, readings and exit criteria for this project: P10 — A/B Testing Platform.