Design 04 — Online Experimentation Platform

"Design the A/B testing system itself — assignment, metrics, analysis — for a company running 200 concurrent experiments." Phase 11 built the statistics; Phase 09 the routing; this design is the platform around them.

1. Requirements

Functional: define experiments (arms, traffic %, audience, metrics); deterministic sticky assignment; concurrent non-interfering experiments; automated analysis (the Phase 11 toolkit, productized); guardrail monitoring with auto-stop; the full audit trail.

Non-functional: assignment adds < 1 ms to any request path; analysis pipelines daily (and near-real-time for guardrails); the platform must be trustworthy — its own error rates measured (A/A tests as a first-class feature); self-serve for PMs with statistical guardrails built in (the platform encodes the expertise so users don't have to have it).

2. Architecture

experiment definitions (config service: arms, %, audience, metrics, dates,
                        pre-registered decision rules)
        │
assignment SDK/service:  bucket = hash(experiment_salt : unit_id) → arm
        │   per-experiment salts (decorrelation!), layers for mutual exclusion
        │   exposure logging AT THE MOMENT OF EXPOSURE (unit, exp, arm, ts)
        ▼
metric pipeline (P02/P08): events ⨝ exposures (event-time!), per-unit
        aggregation, metric definitions from a REGISTRY (one definition per
        metric, owned — or every team invents its own "conversion")
        ▼
analysis engine (P11 toolkit): power/MDE preflight · SRM check (every
        experiment, automatically) · z/t tests with CIs · CUPED (auto, where
        pre-period exists) · multiple-comparison policy by metric role
        ▼
results UI + guardrail monitor (near-real-time; auto-stop on breach)
        ▼
decision log: pre-registered rule + outcome + who decided (the audit trail)

3. Deep dives

Assignment correctness (the foundation; get this wrong and nothing above matters): hash-based (sha-class, not language hash()), salt per experiment — reused salts correlate assignments across experiments, silently confounding both; layers for mutual exclusion (experiments touching the same surface share a layer's traffic split; orthogonal layers stack); sticky across sessions/devices via identity resolution (state the limits honestly: logged-out ↔ logged-in stitching is approximate, and that approximation is a known SRM source).

Exposure semantics: analyze exposed units, not assigned ones — a user bucketed into a treatment they never saw dilutes the effect (triggered analysis). Exposure is logged where the experience diverges, not at assignment. This is the #1 analysis bug in homegrown platforms.

SRM as platform hygiene: chi-square on every experiment, every day, automatically (Phase 11 Ch. 5); detected → experiment flagged untrustworthy, analysis blocked, owner paged with the segment breakdown that localizes the loss (browser? geo? date?). Industry rate ~6–10% of experiments — quote it; it justifies the automation.

Metrics registry: every metric = owned definition (event spec, unit-level aggregation, winsorization policy) computed by one pipeline — the experiment platform and the BI dashboard must agree or trust evaporates. Metric roles declared per experiment (goal/guardrail/diagnostic) drive the multiple-comparison policy (Phase 11 Ch. 8): goal uncorrected, guardrails one-sided, diagnostics labeled exploratory.

The statistics engine: Phase 11's toolkit productized — auto-CUPED (pre-period auto-discovered from the registry; 30–50% variance reduction on returning-user metrics = weeks of runtime saved at platform scale), sequential/ always-valid options for monitored experiments (peeking is a platform problem to solve once, not a user-discipline problem to lecture about), and power preflight blocking launches whose MDE exceeds plausible effects ("this experiment cannot answer your question at your traffic — here's what could").

4. Trust machinery (what distinguishes platform design from script design)

• Continuous A/A tests: a slice of traffic always runs no-op experiments; the platform's realized false-positive rate is a published SLO. A platform that has never validated itself produces numbers with unknown error bars.
• Pre-registration enforced softly: decision rules captured at launch; post-hoc metric switches and segment findings are labeled as exploratory in the UI (forking-paths defense as product design).
• Auto-stop: guardrail breaches (error rates, latency, revenue floor) halt exposure without waiting for a human — the Phase 09 canary logic at experiment scale.
• The decision log: every conclusion linked to its pre-registered rule, data snapshot, and analysis version — experiments end in arguments precisely when this is missing.

5. Failure modes

6. Evolution

Always-valid inference as the default; interleaving as a first-class ranker- comparison mode (Phase 05); heterogeneous-effect exploration (uplift surfacing with exploratory labeling); experiment-aware feature flags unifying release engineering and experimentation (one assignment system, two use cases).