P09 — Recommendation Simulator and Experimentation Platform

Run it first. There is a companion page that builds this project's machinery as numbered, independently runnable blocks and then assembles them into one measured system: P09 hands-on — block by block (handson/h*.py). Every number on it was produced by running the code. Read it alongside the milestones below.

Medium · 66 hours · Weeks 90–95 · Stage 4 · Python

Table of Contents

• The Loop, Instantiated
• Why This Project Matters
• Prerequisites
• Duration and Size
• Central Technical Questions
• Architecture
• The User Model
• Showcase — The Check That Must Always Pass
• Implementation Milestones
• Scenarios
• Concepts To Study
• Primary-Source Readings
• Experiments
• Benchmarks and Metrics
• Correctness Tests
• Failure Tests
• Expected Difficulties
• Scope Boundaries
• Deliverables
• Exit Criteria
• Extension Ideas
• Connections
• References

The Loop, Instantiated

Why This Project Matters

P08 ended with an admission: offline evaluation is biased by the policy that produced the logs. Items your old system never showed have no recorded positives, so any new algorithm that would have surfaced them is penalised for it. You cannot fix this with better metrics, because the information is not in the data.

A simulator escapes the bias by generating the counterfactual: it can tell you what a user would have done with a list they were never shown. That is genuinely the only way to answer the question offline.

The price is that you have replaced a biased measurement with a model-dependent one. Everything the simulator says is conditional on assumptions you wrote down. So the project's real subject is not simulation — it is validity: knowing which of your conclusions are properties of recommender algorithms and which are properties of your click model.

That skill — separating a result from the harness that produced it — is the most research-like thing in this journey, and it is why this project comes after eight others rather than first.

Prerequisites

• P08 complete — the pipeline under test, with a stable swappable interface
• From math.md: §Probability Distributions (2 h), §Confidence Intervals and Bootstrap (2 h)
• The honesty to accept that a simulator that always confirms your preferred algorithm is broken

Duration and Size

Medium, 66 hours, 6 weeks.

Central Technical Questions

1. What must a simulated user do for algorithm comparisons to be meaningful? Not "be realistic" — be ordinally faithful: rank algorithms the way real users would.
2. How do you validate a simulator when the thing it simulates is what you lack?
3. Which of your conclusions are robust to the user model, and which are artifacts?
4. What does the feedback loop do? The recommender shapes the profile which shapes the recommender. Simulators make this visible; production hides it for months.
5. How many simulated users and sessions do you need for a given effect size? This is a power calculation and you should do it before running anything.
6. What can this simulator never tell you? Write the list before you build it, and again after.

Architecture

  ┌── content pool ──┐        ┌──── population ────┐
  │ sampled / synth  │        │ personas:          │
  │ + embeddings     │        │  topic prefs π_u   │
  │ + timestamps     │        │  drift rate        │
  │ + popularity     │        │  fatigue params    │
  └────────┬─────────┘        │  session behaviour │
           │                  │  activity level    │
           │                  └─────────┬──────────┘
           ▼                            ▼
      ┌───────────────── session loop ──────────────────┐
      │  for each user, for each session:               │
      │    algorithm.recommend(profile, pool, k)        │
      │      ──► user model examines position by        │
      │          position (with position bias)          │
      │      ──► click? dwell? skip? abandon session?   │
      │      ──► fatigue and exposure counters update   │
      │      ──► feedback emitted (possibly delayed)    │
      │      ──► profile updated from feedback          │
      │    drift preferences between sessions           │
      └───────────────────────┬─────────────────────────┘
                              ▼
       experiment harness: seeds · arms · trials · CIs · export

The User Model

This is the project. Every component below is a modelling decision that changes your conclusions, so each must be a named, swept parameter — never a hard-coded constant.

Latent preference. User \(u\) has a topic-preference vector \(\pi_u\) over the embedding space. Base affinity for item \(i\) is \(a_{ui} = \langle \pi_u, e_i \rangle\).

Position bias. Examination probability decays with rank. The standard form is \(P(\text{examine at rank } r) = r^{-\gamma}\) with \(\gamma \approx 0.7\text{–}1.0\) from eye-tracking studies. This single parameter dominates everything — with strong position bias, any algorithm that puts something plausible in slot 1 looks good.

Click. \(P(\text{click}) = P(\text{examine}) \cdot P(\text{attract} \mid \text{examine})\), the position-based model. The separation matters: a skip at rank 9 is weak evidence of dislike because the user probably never looked.

Dwell time. Log-normal, with median scaling in affinity. Gives you a graded signal and lets you model "clicked but bounced", which is a real and important negative.

Repeated exposure and fatigue. Attraction multiplied by \(\beta^{n_{ui}}\) where \(n_{ui}\) is prior exposures of item \(i\) — and a topic-level fatigue term that suppresses a topic after several consecutive items from it. Without topic fatigue, your simulator will conclude that diversity is worthless, because a simulated user with no fatigue is perfectly happy to read fifteen articles about the same thing.

Session boundaries and abandonment. \(P(\text{abandon after rank } r)\) rising with consecutive skips. Session length becomes an outcome, which makes it a metric — and a much better one than click count, because it responds to list quality rather than to a single good slot.

Drift. Between sessions, \(\pi_u \leftarrow \text{normalise}(\pi_u + \eta \epsilon)\) with \(\eta\) the drift rate. This is the parameter that decides P08's EMA-vs-mean question, and it is the reason the two projects are adjacent.

Delayed feedback. Some interactions arrive minutes or hours later. Directly tests whether an algorithm depending on immediate profile updates degrades.

Showcase — The Check That Must Always Pass

Twenty minutes, before you build a persona generator. Two things: the sanity check that runs on every change forever, and a first look at whether your conclusions survive the parameter you cannot measure.

# P09 -- the sanity check that must run on every change, and the parameter that
# quietly decides your conclusions.
import random, math
random.seed(3)
N_ITEMS, N_USERS, K = 500, 800, 10
topic = [random.randrange(6) for _ in range(N_ITEMS)]

def simulate(recommend, gamma):
    """gamma = position-bias exponent: P(examine at rank r) = r^-gamma."""
    clicks = 0
    for _ in range(N_USERS):
        fav = random.randrange(6)
        for rank, item in enumerate(recommend(fav)[:K], start=1):
            p_examine = rank ** (-gamma)
            p_attract = 0.8 if topic[item] == fav else 0.05
            if random.random() < p_examine * p_attract:
                clicks += 1
    return clicks / N_USERS

oracle = lambda fav: [i for i in range(N_ITEMS) if topic[i]==fav]
random_rec = lambda fav: random.sample(range(N_ITEMS), K)
half = lambda fav: ([i for i in range(N_ITEMS) if topic[i]==fav][:K//2]
                    + random.sample(range(N_ITEMS), K//2))

print(f"{'position bias':>14}{'oracle':>9}{'half-good':>11}{'random':>9}{'oracle/random':>15}")
for gamma in (0.0, 0.7, 1.5, 3.0):
    o, h, r = simulate(oracle,gamma), simulate(half,gamma), simulate(random_rec,gamma)
    print(f"{gamma:>14.1f}{o:>9.2f}{h:>11.2f}{r:>9.2f}{o/max(r,1e-9):>14.1f}x")

print("\\nCHECK 1: the oracle must beat random at every gamma. If it ever does not,")
print("the simulator is broken and every conclusion drawn from it is void.")
print("\\nCHECK 2, and it is good news. Absolute engagement collapses 8.7x as gamma")
print("goes 0 -> 3 (8.09 -> 0.93 clicks/user), but the oracle/random RATIO barely")
print("moves (4.5x -> 4.8x). The ORDINAL conclusion -- which recommender is better --")
print("is robust to a parameter you cannot measure precisely.")
print("\\nThat is exactly the property a simulator needs, and exactly the claim P09")
print("must verify rather than assume. Absolute simulated numbers are worthless;")
print("relative rankings may be trustworthy. E11 is where you find out which of your")
print("conclusions are in which category.")

 position bias   oracle  half-good   random  oracle/random
           0.0     8.09       4.91     1.81           4.5x
           0.7     3.27       2.39     0.72           4.6x
           1.5     1.65       1.47     0.34           4.8x
           3.0     0.93       0.91     0.21           4.5x
\nCHECK 1: the oracle must beat random at every gamma. If it ever does not,
the simulator is broken and every conclusion drawn from it is void.
\nCHECK 2, and it is good news. Absolute engagement collapses 8.7x as gamma
goes 0 -> 3 (8.09 -> 0.93 clicks/user), but the oracle/random RATIO barely
moves (4.5x -> 4.8x). The ORDINAL conclusion -- which recommender is better --
is robust to a parameter you cannot measure precisely.
\nThat is exactly the property a simulator needs, and exactly the claim P09
must verify rather than assume. Absolute simulated numbers are worthless;
relative rankings may be trustworthy. E11 is where you find out which of your
conclusions are in which category.

Run the oracle check on every commit. It is the cheapest possible detector of a simulator that has quietly started rewarding your preferred algorithm — which is this project's central hazard.

Implementation Milestones

Scenarios

Twelve, each a configuration plus a documented expectation. Write the expected outcome before running each one — a scenario whose result you cannot predict is testing your simulator, not your algorithm.

Concepts To Study

• Click models: position-based, cascade, dynamic Bayesian network; what each assumes
• Position bias and its measurement (interleaving, randomisation)
• Examination vs attraction — the decomposition that makes skips interpretable
• Simulation validity: face validity, construct validity, predictive validity (the only one that matters here)
• Feedback loops and algorithmic confounding
• Variance reduction: common random numbers across arms — the single highest-value technique in this project
• Bootstrap confidence intervals for non-normal metrics
• Statistical power and minimum detectable effect (implemented in tools/metrics.py)
• Sensitivity analysis and one-factor-at-a-time vs global methods
• Reproducibility: seed discipline and the difference between a global seed and per-component streams

Common random numbers

Use the same seeds for population, content, and noise across all arms, varying only the algorithm. Then a difference between arms cannot be caused by a different population. This routinely reduces variance by an order of magnitude and is the difference between needing 1,000 simulated users and needing 100,000. Implement it in milestone 10 as a hard requirement, not a nicety — separate numpy Generator streams per component, all derived from one root seed.

Primary-Source Readings

Budget: 9 hours.

Experiments

E2 is the project's headline result. You have two evaluation methods — P08's biased offline replay and this simulator — and they will disagree on some algorithm pairs. Neither is ground truth. The disagreement itself is the finding, and diagnosing which mechanism causes each disagreement is exactly the kind of analysis that makes research-quality work.

E11 is what separates a toy from a tool. For every conclusion in your report, vary each user-model parameter by ±50% and record whether the conclusion flips. Publish the table. Conclusions that survive are about recommenders; conclusions that flip are about your click model, and saying so plainly is the most credible thing in the report.

Benchmarks and Metrics

Because ground truth is known, you can compute something impossible in production: regret — the gap between the algorithm's achieved engagement and the oracle's. It is the cleanest single number the simulator produces and should be the headline metric.

Correctness Tests

1. Oracle wins. A recommender with access to true \(\pi_u\) must beat everything. If it does not, the simulator is broken. Run this on every change.
2. Random loses, by a predicted margin.
3. Determinism: same root seed → bit-identical outcomes, including every arm.
4. Seed independence: different seeds → different outcomes with overlapping CIs.
5. No information leakage: the algorithm cannot see \(\pi_u\), future items, or another user's state. Enforce by interface, not by discipline.
6. Click model calibration: with a uniform-random ranking, observed CTR by position matches the configured position-bias curve.
7. Fatigue monotonicity: repeated exposure never increases click probability.
8. Drift correctness: measured \(\pi_u\) displacement over n sessions matches \(\eta\sqrt{n}\).
9. Conservation: impressions = clicks + skips + unexamined. They must sum.
10. Power check: with a synthetic known effect, the harness detects it at approximately the predicted sample size.

Test 1 and test 6 are the two you run constantly. An oracle that stops winning is the single best signal that a change broke the simulator.

Failure Tests

Expected Difficulties

1. You will accidentally build a simulator that rewards your preferred algorithm. This is the central hazard. Mitigations: write scenario expectations before running; run E11; and have the oracle test always on.
2. Validating a simulator is genuinely unsolved. The honest position is: this simulator produces hypotheses and relative rankings under stated assumptions, not predictions of production lift. Say exactly that in the report.
3. Parameter count explodes. A dozen parameters is an unsearchable space. Mitigation: fix most from literature (position bias from Craswell et al.), sweep the three that matter, document every fixed value with its source.
4. Runtime becomes the constraint. Millions of sessions × a ranking pipeline is slow. Vectorise the user model; profile the harness at milestone 10.
5. Seeding is harder than it looks. One global seed means adding a parameter shifts every subsequent random draw and all your arms move. Use independent per-component streams from a root seed.
6. It is tempting to make the simulator realistic rather than useful. Realism is unbounded; ordinal faithfulness is the actual requirement. Every additional mechanism must earn its place by changing an algorithm ranking.

Scope Boundaries

In scope: persona-based simulation, click/dwell/skip/abandon models, drift, fatigue, delayed feedback, the twelve scenarios, an experiment harness with CIs, sensitivity analysis.

Out of scope: learned user models trained on real data (extension); RL agents as users; a UI beyond plots; real-time serving; multi-armed bandit algorithms as the subject (that is P08's extension); network effects between users; economic modelling.

Deliverables

1. recsim/ — simulator, personas, click models, scenarios, harness
2. USER-MODEL.md — every parameter, its default, its source, and which conclusions depend on it. This is the artifact that makes the simulator trustworthy
3. REPORT.md centred on E2 (simulator vs offline disagreement) and E11 (sensitivity)
4. A written list titled "What This Simulator Cannot Tell You" — required, scored
5. Notebook entries for E2, E3, E11
6. The experiment harness as a standalone reusable tool (P10 extends it)

Exit Criteria

• Oracle beats all real algorithms; random loses. Both by predicted margins
• Full determinism under a root seed, verified
• All twelve scenarios run, each with a pre-written expectation and a recorded outcome
• E2 complete: simulator vs P08 offline ranking compared, disagreements diagnosed
• E3 complete: drift × profile type, with P08's EMA question answered
• E9 complete: common random numbers implemented, variance reduction measured
• E11 complete: sensitivity analysis over all user-model parameters, with a table of which conclusions survive
• "What This Simulator Cannot Tell You" written and specific
• REPORT.md written with a falsified prediction

Extension Ideas

• Calibrate against real logs: fit the click model to real interaction data, then report how well the simulator predicts held-out behaviour. The strongest possible answer to the validity question.
• Learned user model (a sequence model as the user) compared against the hand-written one — does the conclusion change?
• Global sensitivity analysis with Sobol indices rather than one-factor-at-a-time.
• Multi-stakeholder simulation: publishers as agents responding to the recommender.

Connections

Backward: P08 is the system under test and supplies the offline ranking that E2 compares against.

Forward:

• → P10: the A/B platform assigns and analyses this population; the harness is the foundation
• → P15: "can simulated users predict the relative performance of ranking algorithms?" is one of the strongest candidate research questions, and this project is half of the answer

References

• Chaney, A. J. B., Stewart, B. M., Engelhardt, B. E. How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility. RecSys 2018.
• Ie, E. et al. RecSim: A Configurable Simulation Platform for Recommender Systems. arXiv:1909.04847, 2019.
• Chuklin, A., Markov, I., de Rijke, M. Click Models for Web Search. Morgan & Claypool, 2015.
• Craswell, N., Zoeter, O., Taylor, M., Ramsey, B. An Experimental Comparison of Click Position-Bias Models. WSDM 2008.
• Rohde, D., Bonner, S., Dunlop, T., Vasile, F., Karatzoglou, A. RecoGym. arXiv:1808.00720, 2018.
• Jeunen, O. Revisiting Offline Evaluation for Implicit-Feedback Recommender Systems. RecSys 2019.
• Law, A. M. Simulation Modeling and Analysis, 5th ed. McGraw-Hill, 2014. Chapters on validation and variance reduction — the standard reference for common random numbers.
• Saltelli, A. et al. Global Sensitivity Analysis: The Primer. Wiley, 2008.
• Efron, B., Tibshirani, R. An Introduction to the Bootstrap. Chapman & Hall, 1993.