P09 hands-on — Recsys simulator, block by block
Feedback loops, position bias, and a bandit that loses for a findable reason.
Source:
handson/h09_simulator.py--- run it withpython3 handson/h09_simulator.py
Full project spec: P09 — Recommendation Simulator
A simulator exists to answer the question an A/B test cannot: what would have happened under a policy nobody ran? Its answers are only as good as its user model, so the model has to be stated at the top and stress-tested at the bottom.
Between those two, this file demonstrates the feedback loop that gives recommender systems their characteristic pathology --- a greedy policy narrows its own catalogue from 599 items to 96 while every individual step is locally optimal --- and then measures what exploration costs to prevent it.
The best block is the one where the textbook answer fails. Thompson sampling loses badly to greedy here; the file forms a hypothesis about why, derives a prediction from it, tests the prediction with a modified policy, and confirms it --- and then block 7 independently confirms the same mechanism from a completely different direction.
Contents
- Block 1 — A user model you can interrogate
- Block 2 — Position bias makes logs lie
- Block 3 — Inverse propensity scoring
- Block 4 — Feedback loops
- Block 5 — Exploration as insurance
- Block 6 — When the bandit loses
- Block 7 — The counterfactual question
- The assembly
- The design space
- Position bias, and why logs cannot be read naively
- What the blocks found that the textbook does not say
- Advanced algorithms
- Calibration: the only thing that makes a simulator citable
- How this connects to the rest of the track
- Failure modes at scale
- Primary sources
- Running it
- What to do with this
How to read this page
Each block below is a self-contained lego piece: it builds one mechanism, proves it works on its own, and returns what the next block needs. The code is the real source, sliced out of the script. The output underneath it is the real output, captured by running that script --- not transcribed, not idealised. Where a measurement contradicted what I expected, the contradiction is in the output and the prose says so.
The assembly at the end wires every block into one working thing and measures it.
Block 1 — A user model you can interrogate
Teaches: the simulator's assumptions ARE its results
The problem. A simulator's assumptions are its results. State them in three lines at the top, where they can be argued with, rather than burying them in an appendix where they cannot.
@block(1, "A user model you can interrogate", "the simulator's assumptions ARE its results")
def b1(s, show):
rng = np.random.default_rng(9)
U = rng.normal(0, 1, (NU, D)); V = rng.normal(0, 1, (NI, D))
quality = rng.normal(0, 0.6, NI)
util = U @ V.T + quality # true utility of item i to user u
def click(u, ranked, rng, pos_bias=True):
"""Examine top-k with position-dependent probability; click if utility wins."""
clicks = []
for r, i in enumerate(ranked):
exam = 1.0 / (1 + r) ** 1.0 if pos_bias else 1.0
if rng.random() < exam and rng.random() < 1 / (1 + np.exp(-util[u, i])):
clicks.append(i)
return clicks
if show:
print(f" {NU} users x {NI} items, utility = <u,v> + item quality")
print(f" utility spread: p10={np.quantile(util,.1):+.2f} "
f"p50={np.quantile(util,.5):+.2f} p90={np.quantile(util,.9):+.2f}")
print(f" examination P(look at rank r) = 1/(1+r): "
f"rank0={1.0:.2f} rank4={1/5:.2f} rank19={1/20:.3f}")
print(" Every number this simulator later produces is a consequence of these")
print(" three lines. State them at the top of the report, not in an appendix:")
print(" a simulator is an argument, and these are its premises.")
return {"util": util, "click": click, "quality": quality}
Reading the implementation
The user model is exactly three statements:
- Utility is \(\langle u, v \rangle + \text{quality}_i\) — a latent taste term plus a global item-quality term.
- Examination probability at rank \(r\) is \(1/(1+r)\).
- A click happens if the item is examined and \(\sigma(\text{utility})\) fires.
That last line is the examination hypothesis: click = examine × relevant, with the two independent. It is the assumption underlying essentially all learning-to-rank debiasing, and it is falsifiable — real users' examination depends on what they have already seen (the cascade effect), which this model omits.
Separating quality from taste is not cosmetic. It creates items that are good for everyone (which popularity finds) and items that are good for specific people (which only personalisation finds), and that separation is what makes the exploration results in later blocks interpretable.
What the numbers say
Output:
400 users x 600 items, utility = <u,v> + item quality
utility spread: p10=-3.13 p50=+0.01 p90=+3.18
examination P(look at rank r) = 1/(1+r): rank0=1.00 rank4=0.20 rank19=0.050
Every number this simulator later produces is a consequence of these
three lines. State them at the top of the report, not in an appendix:
a simulator is an argument, and these are its premises.
Beyond the toy
The missing dynamics, roughly in order of how much they would change the conclusions: satiation (a user who has seen an item does not want it again), drift (preferences move over sixty days), arrival and churn (the user population is not fixed), and social influence (what others click changes what I click). Each is a few lines to add and each can reverse a policy comparison, which is precisely why block 7's sensitivity analysis is not optional.
Block 2 — Position bias makes logs lie
Teaches: the log measures the ranker, not the user
The problem. Click logs measure the ranker at least as much as they measure the user. This block quantifies it under the cleanest possible conditions — random serving, so item quality and position are independent by construction.
@block(2, "Position bias makes logs lie", "the log measures the ranker, not the user")
def b2(s, show):
rng = np.random.default_rng(10)
util = s["util"]
if show:
ctr_by_rank = np.zeros(20); shown = np.zeros(20)
for u in range(NU):
order = rng.permutation(NI)[:20] # RANDOM ranking: no confound
cl = set(s["click"](u, order, rng))
for r, i in enumerate(order):
shown[r] += 1; ctr_by_rank[r] += i in cl
obs = ctr_by_rank / shown
print(" Serving a RANDOM ranking, so item quality is independent of position:")
print(f" {'rank':>6}{'observed CTR':>15}{'1/(1+r) prediction':>21}")
for r in (0, 1, 4, 9, 19):
print(f" {r:>6}{obs[r]:>15.4f}{obs[0]/(1+r):>21.4f}")
print(f" CTR at rank 0 is {obs[0]/max(obs[19],1e-9):.1f}x rank 19 for items chosen")
print(" UNIFORMLY AT RANDOM. Naively training on click logs teaches the model")
print(" 'items at rank 0 are good', which is a fact about the old ranker.")
return {}
Reading the implementation
Serving a random ranking is the key experimental design. Under any real ranker, position and quality are confounded: good items are shown high, so high positions get more clicks for two reasons and you cannot separate them. Randomise, and the only remaining reason is examination.
This is exactly why result randomisation is the gold-standard method for estimating propensities in production — and why it is expensive: you are deliberately showing users worse results to learn how position affects them.
What the numbers say
Output:
Serving a RANDOM ranking, so item quality is independent of position:
rank observed CTR 1/(1+r) prediction
0 0.5100 0.5100
1 0.2425 0.2550
4 0.1025 0.1020
9 0.0550 0.0510
19 0.0250 0.0255
CTR at rank 0 is 20.4x rank 19 for items chosen
UNIFORMLY AT RANDOM. Naively training on click logs teaches the model
'items at rank 0 are good', which is a fact about the old ranker.
CTR at rank 0 is many times rank 19 for items chosen uniformly at random. Any model trained on raw clicks learns "items at rank 0 are good", which is a fact about the previous ranker. Deploy that model and it reinforces the previous ranker's choices — the feedback loop block 4 measures.
Beyond the toy
The click-model family, in increasing realism: position-based (this block — examination depends only on rank), cascade (the user scans top-down and stops at the first click, so items below a click are not examined), dependent click (cascade with a continuation probability), and dynamic Bayesian network (cascade plus a satisfaction probability after the click, which distinguishes "saw and liked" from "saw and bounced").
Estimating propensities without randomisation is a research area in itself: intervention harvesting exploits natural rank variation of the same item across queries, and regression-EM jointly fits relevance and examination. Both are cheaper than randomisation and both make assumptions that randomisation does not.
Block 3 — Inverse propensity scoring
Teaches: divide out the mechanism you know
The problem. If the mechanism producing the bias is known, you can divide it out. Inverse propensity scoring is that idea, and this block shows both that it works and why it is fragile.
@block(3, "Inverse propensity scoring", "divide out the mechanism you know")
def b3(s, show):
rng = np.random.default_rng(11)
if show:
true_rate = np.zeros(NI); naive = np.zeros(NI); ips = np.zeros(NI)
shown = np.zeros(NI)
for u in range(NU):
order = rng.permutation(NI)[:20]
cl = set(s["click"](u, order, rng))
for r, i in enumerate(order):
p = 1.0 / (1 + r) # KNOWN propensity
shown[i] += 1
naive[i] += i in cl
ips[i] += (i in cl) / p
true_rate[i] += 1 / (1 + np.exp(-s["util"][u, i]))
m = shown > 8
def corr(a, b): return float(np.corrcoef(a[m] / shown[m], b[m] / shown[m])[0, 1])
print(f" correlation with true click propensity, over {int(m.sum())} items:")
print(f" naive CTR estimate r = {corr(naive, true_rate):.4f}")
print(f" IPS-corrected estimate r = {corr(ips, true_rate):.4f}")
print(" IPS is unbiased when the propensity is known exactly -- which is true")
print(" in a simulator and never true in production. There you estimate the")
print(" propensity, and the variance of 1/p_hat at small p_hat is what")
print(" destroys the estimator. Clipping p at 0.01-0.1 is the standard trade:")
print(" accept a little bias to stop the variance from exploding.")
return {}
Reading the implementation
Weight each observation by \(1/p(\text{examined})\), so an event that was unlikely to be observed counts for more:
\[ \hat{V}_{\text{IPS}} = \frac{1}{n}\sum_i \frac{\mathbb{1}[\text{click}_i]}{p_i} \]
This is unbiased when the propensities are exact and non-zero everywhere. The proof is one line of expectation algebra, and the estimator is the foundation of counterfactual learning-to-rank.
The fragility is in the variance, not the bias. \(\mathrm{Var}(1/p)\) grows as \(p \to 0\), so a single observation at \(p = 0.001\) contributes weight 1000 and can dominate the entire estimate. In a simulator \(p\) is known exactly; in production it is estimated, and the variance of \(1/\hat{p}\) at small \(\hat{p}\) is what destroys the estimate in practice.
What the numbers say
Output:
correlation with true click propensity, over 551 items:
naive CTR estimate r = 0.3221
IPS-corrected estimate r = 0.2592
IPS is unbiased when the propensity is known exactly -- which is true
in a simulator and never true in production. There you estimate the
propensity, and the variance of 1/p_hat at small p_hat is what
destroys the estimator. Clipping p at 0.01-0.1 is the standard trade:
accept a little bias to stop the variance from exploding.
Beyond the toy
The standard repairs, and what each trades:
- Clipping weights at \(M\): bounded variance, introduces bias. Usual choice.
- Self-normalised IPS (SNIPS): divide by the sum of weights. Consistent rather than unbiased, and much lower variance — usually strictly better in practice.
- Doubly robust: combine a reward model \(\hat{r}\) with IPS on the residual. Unbiased if either the model or the propensities are correct, which is why it is the production default.
- Overlap / positivity: if the new policy puts mass where the logging policy put none, no amount of reweighting helps. This is a hard limit, not a variance problem, and it is why off-policy evaluation cannot assess a radically different policy — which is exactly the gap a simulator fills.
Block 4 — Feedback loops
Teaches: the ranker trains on data the ranker created
The problem. The ranker trains on data the ranker created. This block shows the loop closing, and the disturbing part is that every individual step is locally optimal.
@block(4, "Feedback loops", "the ranker trains on data the ranker created")
def b4(s, show):
def simulate(policy, days=T, explore=0.0, seed=12, k=10, warmup=2, order_fn=None):
"""warmup days of RANDOM serving seed the estimates; without it every
item ties at CTR 0 and 'greedy' just locks onto item ids 0..k-1, which
would make the feedback loop look like an artefact of argsort."""
rng = np.random.default_rng(seed)
clicks = np.zeros(NI); impr = np.ones(NI)
hist = []
for d in range(days):
served, got = np.zeros(NI), 0
for u in range(NU):
if d < warmup or (explore and rng.random() < explore):
order = rng.permutation(NI)[:k]
elif order_fn is not None:
order = order_fn(clicks, impr, rng, k)
else:
order = np.argsort(-policy(clicks, impr, rng))[:k]
cl = s["click"](u, order, rng)
for i in order: impr[i] += 1; served[i] += 1
for i in cl: clicks[i] += 1
got += len(cl)
hist.append((got / NU, int((served > 0).sum())))
return hist, clicks, impr
greedy = lambda c, im, rng: c / im
if show:
h, c, im = simulate(greedy)
print(f" 2 days of random serving, then greedy 'rank by observed CTR':")
print(f" {'day':>5}{'clicks/user':>13}{'distinct items shown':>22}")
for d in (0, 1, 2, 4, 19, 39, T-1):
tag = " <- random warm-up" if d < 2 else ""
print(f" {d:>5}{h[d][0]:>13.3f}{h[d][1]:>22}{tag}")
print(f" catalogue collapsed from {h[1][1]} items on day 2 to {h[-1][1]} "
f"on day {T} ({h[1][1]//max(h[-1][1],1)}x narrower)")
print(" Nothing broke. Every step was locally optimal: show what performed")
print(" well, observe it perform well, show it more. The feedback loop is")
print(" not a bug in the policy -- it is the policy, iterated.")
print(" Note the width is NOT monotone (185 -> 300 -> 313 -> 90). Straight")
print(" after warm-up, greedy chases items whose CTR was over-estimated by")
print(" noise; as they accumulate impressions their estimates regress and")
print(" other items overtake them, so the served set churns before it")
print(" freezes. That is the winner's curse, visible as a bump in a width")
print(" plot -- and it is why 'the ranking looks unstable' early in a launch")
print(" is expected rather than alarming.")
return {"simulate": simulate, "greedy": greedy}
Reading the implementation
The warm-up is the methodological point. Without two days of random serving, every
item has CTR 0, argsort returns the first \(k\) indices, and "greedy" locks
onto items 0--9 forever. That would be a property of argsort, not of the policy,
and reporting it as a feedback-loop result would be measuring the harness.
With the warm-up, the collapse is real: from 599 distinct items served on day 2 down to 96 on day 60. Nothing broke. Every step maximised expected clicks given current estimates. The feedback loop is not a bug in the policy — it is the policy, iterated.
What the numbers say
Output:
2 days of random serving, then greedy 'rank by observed CTR':
day clicks/user distinct items shown
0 1.393 600 <- random warm-up
1 1.407 599 <- random warm-up
2 2.007 185
4 1.905 300
19 2.005 313
39 2.100 90
59 2.007 96
catalogue collapsed from 599 items on day 2 to 96 on day 60 (6x narrower)
Nothing broke. Every step was locally optimal: show what performed
well, observe it perform well, show it more. The feedback loop is
not a bug in the policy -- it is the policy, iterated.
Note the width is NOT monotone (185 -> 300 -> 313 -> 90). Straight
after warm-up, greedy chases items whose CTR was over-estimated by
noise; as they accumulate impressions their estimates regress and
other items overtake them, so the served set churns before it
freezes. That is the winner's curse, visible as a bump in a width
plot -- and it is why 'the ranking looks unstable' early in a launch
is expected rather than alarming.
The non-monotonicity is worth pausing on: width goes 185 → 300 → 313 → 90. Straight after warm-up, greedy chases items whose CTR was over-estimated by noise; as those accumulate impressions their estimates regress toward truth and other items overtake them, so the served set churns before it freezes. That is the winner's curse, visible as a bump in a width plot — and it is why "the ranking looks unstable" early in a launch is expected rather than alarming.
Beyond the toy
The same loop appears wherever a model's outputs become its future training data: ad auctions (bids shape the data that trains the bidder), content moderation (enforcement shapes what is reported), credit scoring (denials mean no repayment data for the denied — the classic selective labels problem), and predictive policing. The general name is performativity, and the general defence is the same: keep a randomised holdout so you always have unbiased data about the actions your policy does not take.
Block 5 — Exploration as insurance
Teaches: epsilon buys catalogue coverage with clicks
The problem. Exploration is usually framed as a cost paid for future information. This block measures the curve, and the framing turns out to be wrong at the near end.
@block(5, "Exploration as insurance", "epsilon buys catalogue coverage with clicks")
def b5(s, show):
if show:
print(f" {'policy':<28}{'clicks/user d60':>17}{'distinct items':>16}"
f"{'cumulative':>12}")
rows = []
for lbl, ex in (("greedy (eps=0)", 0.0), ("eps=0.02", 0.02),
("eps=0.10", 0.10), ("eps=0.30", 0.30)):
h, c, im = s["simulate"](s["greedy"], explore=ex)
cum = sum(x for x, _ in h)
rows.append((lbl, h[-1][0], h[-1][1], cum))
print(f" {lbl:<28}{h[-1][0]:>17.3f}{h[-1][1]:>16}{cum:>12.1f}")
best = max(rows, key=lambda r: r[3])
print(f" highest cumulative clicks: {best[0]}")
print(" Exploration is NOT a pure cost here. eps=0.02 beats pure greedy on")
print(" BOTH axes -- more clicks (126.8 vs 120.5) and a live estimate for")
print(" items greedy abandoned. The cost only appears further along the")
print(" curve: eps=0.30 gives up 6% of clicks to keep 522 items alive.")
print(" The optimum is interior, so it has to be found by measurement; both")
print(" 'exploration is overhead' and 'more exploration is safer' are wrong.")
return {}
Reading the implementation
ε-greedy: with probability ε serve a random slate, otherwise serve the greedy one. Sweep ε and measure both cumulative clicks and catalogue coverage.
The cumulative column, not the day-60 column, is the one to optimise — the whole point of exploration is that it pays later, so a snapshot metric systematically undervalues it.
What the numbers say
Output:
policy clicks/user d60 distinct items cumulative
greedy (eps=0) 2.007 96 120.5
eps=0.02 2.277 88 126.8
eps=0.10 2.013 331 119.0
eps=0.30 1.857 522 113.3
highest cumulative clicks: eps=0.02
Exploration is NOT a pure cost here. eps=0.02 beats pure greedy on
BOTH axes -- more clicks (126.8 vs 120.5) and a live estimate for
items greedy abandoned. The cost only appears further along the
curve: eps=0.30 gives up 6% of clicks to keep 522 items alive.
The optimum is interior, so it has to be found by measurement; both
'exploration is overhead' and 'more exploration is safer' are wrong.
ε=0.02 beats pure greedy on both axes — more clicks and more of the catalogue alive. That is a strict improvement, available for free, and it contradicts the standard framing of exploration as a tax. The cost only appears further along the curve: ε=0.30 gives up ~6% of clicks to keep 522 items alive.
The optimum is interior, so it must be found by measurement. Both "exploration is overhead" and "more exploration is safer" are wrong.
Beyond the toy
Why a little exploration is free: greedy's estimates are wrong, and the items it abandoned early include some genuinely good ones (the winner's curse in reverse). A small ε corrects those errors cheaply. The marginal value of the next unit of exploration falls as estimates improve, while its marginal cost is constant — hence an interior optimum, and hence the standard practice of decaying ε over time.
Block 6 — When the bandit loses
Teaches: a prediction, a test, and a policy that fixes it
The problem. The textbook says Thompson sampling beats ε-greedy. Here it loses badly. This block is a full hypothesis-prediction-test loop on why, and the answer generalises to every slate-based system.
@block(6, "When the bandit loses", "a prediction, a test, and a policy that fixes it")
def b6(s, show):
def thompson(c, im, rng):
return rng.beta(1 + c, 1 + np.maximum(im - c, 0))
def ucb(c, im, rng):
return c / im + np.sqrt(2 * np.log(max(im.sum(), 2)) / im)
def slot_aware(c, im, rng, k):
"""Exploit the top slot; let Thompson have the rest."""
ts = np.argsort(-rng.beta(1 + c, 1 + np.maximum(im - c, 0)))
best = int(np.argmax(c / im))
return [best] + [int(i) for i in ts if i != best][:k - 1]
if show:
print(f" {'policy':<38}{'cum clicks/user':>17}{'distinct':>10}")
for lbl, pol, ex in (("greedy", s["greedy"], 0.0),
("eps-greedy 0.02", s["greedy"], 0.02),
("UCB1", ucb, 0.0),
("Thompson sampling", thompson, 0.0)):
h, _, _ = s["simulate"](pol, explore=ex)
print(f" {lbl:<38}{sum(x for x,_ in h):>17.1f}{h[-1][1]:>10}")
print(" The bandits LOSE, badly, and the textbook answer ('Thompson beats")
print(" epsilon-greedy') does not survive contact with this environment.\n")
ex = np.array([1 / (1 + r) for r in range(10)])
print(" HYPOTHESIS: examination is 1/(1+r), so attention is concentrated:")
print(" share by rank: " + " ".join(f"{x:.0%}" for x in ex / ex.sum()))
print(f" rank 0 alone carries {ex[0]/ex.sum():.0%} of all examination.")
print(" Thompson randomises ALL TEN slots, so it spends its most valuable")
print(" slot on an uncertain item every single impression.")
print(" PREDICTION: explore only in ranks 1-9 and most of the lost clicks")
print(" come back, while coverage stays near Thompson's.\n")
h, _, _ = s["simulate"](None, order_fn=slot_aware)
g, _, _ = s["simulate"](s["greedy"])
ts, _, _ = s["simulate"](thompson)
cg, ct, ch = (sum(x for x,_ in z) for z in (g, ts, h))
print(f" {'Thompson in ranks 1-9, greedy at rank 0':<38}{ch:>17.1f}"
f"{h[-1][1]:>10}")
print(f" VERDICT: confirmed. {ch/cg:.0%} of greedy's clicks "
f"(vs {ct/cg:.0%} for full Thompson),")
print(f" with {h[-1][1]} distinct items against Thompson's {ts[-1][1]}. The exploration")
print(" budget was never the problem -- WHERE it was spent was. This is why")
print(" production rankers explore in the tail of the slate and why a bandit")
print(" benchmarked without a position model reports the wrong winner.")
return {"thompson": thompson, "ucb": ucb, "slot_aware": slot_aware}
Reading the implementation
First the measurement: greedy 120.5 cumulative clicks, Thompson 97.8. The textbook result does not survive contact with this environment.
Hypothesis. Examination is \(1/(1+r)\), so attention is extremely concentrated — rank 0 alone carries 34% of all examination. Thompson randomises all ten slots, so it spends its single most valuable slot on an uncertain item on every impression.
Prediction. Explore only in ranks 1--9 and most of the lost clicks return while coverage stays near Thompson's.
Test. slot_aware puts the greedy best at rank 0 and lets Thompson have the
rest.
What the numbers say
Output:
policy cum clicks/user distinct
greedy 120.5 96
eps-greedy 0.02 126.8 88
UCB1 100.7 600
Thompson sampling 97.8 544
The bandits LOSE, badly, and the textbook answer ('Thompson beats
epsilon-greedy') does not survive contact with this environment.
HYPOTHESIS: examination is 1/(1+r), so attention is concentrated:
share by rank: 34% 17% 11% 9% 7% 6% 5% 4% 4% 3%
rank 0 alone carries 34% of all examination.
Thompson randomises ALL TEN slots, so it spends its most valuable
slot on an uncertain item every single impression.
PREDICTION: explore only in ranks 1-9 and most of the lost clicks
come back, while coverage stays near Thompson's.
Thompson in ranks 1-9, greedy at rank 0 112.0 532
VERDICT: confirmed. 93% of greedy's clicks (vs 81% for full Thompson),
with 532 distinct items against Thompson's 544. The exploration
budget was never the problem -- WHERE it was spent was. This is why
production rankers explore in the tail of the slate and why a bandit
benchmarked without a position model reports the wrong winner.
Verdict: confirmed. 93% of greedy's clicks (against 81% for full Thompson) with 532 distinct items against Thompson's 544. The exploration budget was never the problem — where it was spent was.
Beyond the toy
The generalisable statement: a bandit benchmarked without a position model reports the wrong winner. Standard bandit theory treats an impression as one action with one reward. A slate is ten actions with wildly unequal observation probabilities, and the regret analysis does not transfer.
This is why production rankers explore in the tail of the slate, why "exploration budget" is measured in attention rather than in impressions, and why slate bandits are a distinct research area (Swaminathan et al.'s pseudo-inverse estimator exists precisely because the combinatorial action space breaks naive off-policy evaluation).
Block 7 confirms the same mechanism from a completely different direction, which is the strongest form of evidence available here.
Block 7 — The counterfactual question
Teaches: a simulator's only real job
The problem. A simulator's only real job is the counterfactual — what would have happened under a policy nobody ran. That answer is only as good as the user model, so the last block attacks the user model.
@block(7, "The counterfactual question", "a simulator's only real job")
def b7(s, show):
if show:
print(" A/B tests answer 'which of these two shipped policies wins?'.")
print(" Simulators answer 'what would have happened under a policy nobody")
print(" ran?' -- and that answer is only as good as the user model.\n")
print(f" {'user model perturbation':<34}{'greedy':>9}{'Thompson':>11}"
f"{'winner':>10}{'margin':>9}")
base = None
for lbl, mult in (("as specified", 1.0), ("position bias 2x steeper", 2.0),
("position bias flat (no bias)", 0.0)):
orig = s["click"]
def click(u, ranked, rng, _m=mult):
out = []
for r, i in enumerate(ranked):
exam = 1.0 if _m == 0 else 1.0 / (1 + r) ** _m
if rng.random() < exam and rng.random() < 1/(1+np.exp(-s["util"][u,i])):
out.append(i)
return out
s["click"] = click
g = sum(x for x, _ in s["simulate"](s["greedy"])[0])
t = sum(x for x, _ in s["simulate"](s["thompson"])[0])
s["click"] = orig
print(f" {lbl:<34}{g:>9.1f}{t:>11.1f}"
f"{('Thompson' if t > g else 'greedy'):>10}"
f"{max(g,t)/min(g,t)-1:>8.0%}")
print(" The WINNER is stable across all three user models even though the")
print(" absolute numbers move by 7x. That is the claim a simulator can")
print(" support: ordinal, not cardinal. Quote 120.5 clicks/user to a")
print(" stakeholder and you are quoting your own assumptions back at them.")
print(" But read the margin column, because it is doing more work than the")
print(" winner column. Greedy's edge is 23% under the specified bias, 20%")
print(" when the bias doubles -- and 1% when it is removed entirely. That is")
print(" an INDEPENDENT confirmation of block 6: greedy wins here because")
print(" exploration is expensive at rank 0, so deleting position bias very")
print(" nearly deletes greedy's advantage. Two blocks, two methods, one")
print(" mechanism. A sensitivity table is not defensive paperwork; it is")
print(" where the causal claim actually gets tested.")
return {}
Reading the implementation
Three user models: as specified, position bias twice as steep, and position bias removed entirely. Same policies, same data-generating process otherwise.
The margin column is doing more work than the winner column, and adding it was the point of the revision: greedy's edge is 23% under the specified bias, 20% at double, and 1% with bias removed.
What the numbers say
Output:
A/B tests answer 'which of these two shipped policies wins?'.
Simulators answer 'what would have happened under a policy nobody
ran?' -- and that answer is only as good as the user model.
user model perturbation greedy Thompson winner margin
as specified 120.5 97.8 greedy 23%
position bias 2x steeper 61.9 51.5 greedy 20%
position bias flat (no bias) 419.2 415.3 greedy 1%
The WINNER is stable across all three user models even though the
absolute numbers move by 7x. That is the claim a simulator can
support: ordinal, not cardinal. Quote 120.5 clicks/user to a
stakeholder and you are quoting your own assumptions back at them.
But read the margin column, because it is doing more work than the
winner column. Greedy's edge is 23% under the specified bias, 20%
when the bias doubles -- and 1% when it is removed entirely. That is
an INDEPENDENT confirmation of block 6: greedy wins here because
exploration is expensive at rank 0, so deleting position bias very
nearly deletes greedy's advantage. Two blocks, two methods, one
mechanism. A sensitivity table is not defensive paperwork; it is
where the causal claim actually gets tested.
That is an independent confirmation of block 6. If greedy wins because exploration is expensive at rank 0, then deleting position bias should very nearly delete greedy's advantage — and it does. Two blocks, two methods, one mechanism. A sensitivity table is not defensive paperwork; it is where the causal claim gets tested.
Beyond the toy
What a simulator can and cannot support:
- Can: ordinal claims. "Policy A beats policy B across every plausible user model" is a defensible statement, and the sensitivity table is the evidence.
- Cannot: cardinal claims. The absolute numbers move 7× across these three models. Quoting "120.5 clicks per user" to a stakeholder is quoting your own assumptions back at them.
The calibration protocol that makes a simulator citable: state the model at the top, fit its free parameters to a real log, verify it reproduces a held-out period it was not fit on, and report the sensitivity sweep alongside every result. Without the held-out check, a simulator is a hypothesis-generating toy; with it, it is evidence.
The assembly
Every block above, wired together into one working system:
def assembly(s):
print("\nSeven blocks = a simulator. Run four policies for 60 days.\n")
print(f" {'policy':<24}{'cum clicks/user':>17}{'day-60 rate':>13}"
f"{'distinct items':>16}{'gini':>7}")
def gini(x):
x = np.sort(x[x >= 0]); n = len(x)
return float((2*np.arange(1, n+1) - n - 1) @ x / (n * max(x.sum(), 1e-9)))
for lbl, pol, ex in (("greedy", s["greedy"], 0.0),
("eps-greedy 0.02", s["greedy"], 0.02),
("eps-greedy 0.10", s["greedy"], 0.10),
("UCB1", s["ucb"], 0.0),
("Thompson", s["thompson"], 0.0)):
h, c, im = s["simulate"](pol, explore=ex)
print(f" {lbl:<24}{sum(x for x,_ in h):>17.1f}{h[-1][0]:>13.3f}"
f"{h[-1][1]:>16}{gini(im):>7.3f}")
h, c, im = s["simulate"](None, order_fn=s["slot_aware"])
print(f" {'Thompson, ranks 1-9':<24}{sum(x for x,_ in h):>17.1f}{h[-1][0]:>13.3f}"
f"{h[-1][1]:>16}{gini(im):>7.3f}")
print("\n The gini column is exposure inequality across the catalogue: 0 = every")
print(" item shown equally, 1 = one item takes everything. Read the two ends")
print(" first: greedy earns the most clicks of the pure policies and starves the")
print(" catalogue (gini 0.91, 96 items alive); UCB1 keeps all 600 items alive and")
print(" pays 16% of the clicks for it. There is no free lunch on that axis.")
print(" The interesting rows are the two in between. eps=0.02 beats greedy on")
print(" clicks AND on coverage -- a strict improvement, available for free. And")
print(" the last row buys 5.5x greedy's catalogue for 7% of its clicks, because")
print(" it explores in the slots nobody looks at. Exposure fairness is not")
print(" bought at a fixed exchange rate; the rate depends on where you spend.")
print(" None of this is measurable with an A/B test, because no one runs a")
print(" knowingly worse policy for sixty days to find the shape of a curve.")
print("\n Built: user model -> position bias -> IPS -> feedback loop -> epsilon ->")
print(" Thompson/UCB -> counterfactual sensitivity.")
print(" Missing, on the project page: user arrival and churn dynamics (m4),")
print(" a two-sided marketplace with supplier utility (m8), off-policy")
print(" evaluation against logged data with estimated propensities (m9-m10),")
print(" and E6 -- the calibration experiment where you fit the simulator to a")
print(" real log and check whether it reproduces a held-out week.")
Output:
Seven blocks = a simulator. Run four policies for 60 days.
policy cum clicks/user day-60 rate distinct items gini
greedy 120.5 2.007 96 0.913
eps-greedy 0.02 126.8 2.277 88 0.874
eps-greedy 0.10 119.0 2.013 331 0.740
UCB1 100.7 1.675 600 0.242
Thompson 97.8 1.667 544 0.415
Thompson, ranks 1-9 112.0 1.960 532 0.448
The gini column is exposure inequality across the catalogue: 0 = every
item shown equally, 1 = one item takes everything. Read the two ends
first: greedy earns the most clicks of the pure policies and starves the
catalogue (gini 0.91, 96 items alive); UCB1 keeps all 600 items alive and
pays 16% of the clicks for it. There is no free lunch on that axis.
The interesting rows are the two in between. eps=0.02 beats greedy on
clicks AND on coverage -- a strict improvement, available for free. And
the last row buys 5.5x greedy's catalogue for 7% of its clicks, because
it explores in the slots nobody looks at. Exposure fairness is not
bought at a fixed exchange rate; the rate depends on where you spend.
None of this is measurable with an A/B test, because no one runs a
knowingly worse policy for sixty days to find the shape of a curve.
Built: user model -> position bias -> IPS -> feedback loop -> epsilon ->
Thompson/UCB -> counterfactual sensitivity.
Missing, on the project page: user arrival and churn dynamics (m4),
a two-sided marketplace with supplier utility (m8), off-policy
evaluation against logged data with estimated propensities (m9-m10),
and E6 -- the calibration experiment where you fit the simulator to a
real log and check whether it reproduces a held-out week.
The design space
A simulator is a claim about counterfactuals, and there are only three ways to make one. They differ in what they assume and what they can be wrong about.
| Approach | Assumes | Answers | Fails when |
|---|---|---|---|
| Off-policy evaluation (IPS, SNIPS, DR) | logging policy's propensities are known and non-zero everywhere | "what would policy \(\pi\) have scored on logged data?" | \(\pi\) differs a lot from the logger (variance explodes) |
| Simulation | a generative user model | "what happens over months under \(\pi\)?" | the user model is wrong in a way that matters |
| A/B test (P10) | nothing about users | "did \(\pi\) beat \(\pi_0\) for real users?" | you need to run it, on real users, for real time |
The three are complementary and ordered by cost and by trustworthiness in opposite directions. Simulation is the only one that can answer long-horizon questions — feedback loops, catalogue collapse, supplier churn — because no one will run a knowingly worse policy on real traffic for sixty days.
Off-policy estimators, precisely
With logged data \((x, a, r)\) collected under \(\mu\), the IPS estimate of policy \(\pi\) is
\[ \hat{V}_{\text{IPS}}(\pi) = \frac{1}{n}\sum_i \frac{\pi(a_i|x_i)}{\mu(a_i|x_i)} r_i \]
which is unbiased when \(\mu > 0\) wherever \(\pi > 0\). Its variance scales with the squared importance ratio, so a single action with \(\mu = 0.001\) and \(\pi = 1\) contributes a weight of 1000 and dominates the estimate. The standard repairs:
- Clipping / capping weights at some \(M\): trades bias for variance.
- Self-normalised IPS (SNIPS): divide by the sum of weights; consistent rather than unbiased, and much lower variance.
- Doubly robust: combine a reward model \(\hat{r}\) with IPS on the residual. Unbiased if either the model or the propensities are right — which is why it is the default in practice.
Block 3 measures the estimator with known propensities, which is the situation that exists only in a simulator. In production you estimate \(\hat{\mu}\), and the variance of \(1/\hat{\mu}\) at small \(\hat{\mu}\) is what destroys the estimate.
Position bias, and why logs cannot be read naively
Block 2's measurement — CTR at rank 0 is many times rank 19 for items chosen uniformly at random — is the cleanest statement of the problem. The click model zoo formalises it:
| Model | Assumption |
|---|---|
| Position-based (PBM) | \(P(\text{click}) = P(\text{examine} \mid \text{rank}) \cdot P(\text{relevant})\) |
| Cascade | user scans top-down, stops at first click |
| Dynamic Bayesian Network | cascade + a satisfaction probability after the click |
| Click chain / UBM | examination depends on rank and distance from last click |
Estimating examination probabilities without random serving requires either result randomisation (expensive, hurts users) or intervention harvesting (exploiting natural rank variation of the same item across queries), or a jointly estimated model like regression-EM. Every one of these is an attempt to recover the propensity that block 3 simply knows.
What the blocks found that the textbook does not say
Thompson sampling loses here, and the mechanism is measurable: under \(1/(1+r)\) examination, rank 0 carries 34% of all attention, so a policy that randomises all ten slots spends its most valuable slot on an uncertain item every impression. Exploring only in ranks 1--9 recovers 93% of greedy's clicks while keeping 532 of 544 items alive.
Block 7 then confirms the same mechanism from a different direction: greedy's margin is 23% under the specified bias, 20% at double the bias, and 1% with bias removed. Two methods, one mechanism.
The generalisable rule: a bandit benchmarked without a position model reports the wrong winner, because the cost of exploration is not uniform across the slate. This is why production systems explore in the tail of the slate, and it is invisible in any bandit formulation that treats an impression as a single action.
Advanced algorithms
- Contextual bandits: LinUCB and Thompson sampling with linear payoffs give \(\tilde{O}(d\sqrt{T})\) regret; the practical obstacle is that the reward model must be updated online, which conflicts with batch training infrastructure.
- Slate bandits and combinatorial actions. The action space is \(\binom{N}{k}\) ordered slates; pseudo-inverse estimators (Swaminathan et al.) exploit linearity assumptions to make off-policy evaluation of slates tractable.
- Counterfactual risk minimisation (POEM) optimises a variance-regularised IPS objective directly, rather than evaluating a policy after the fact.
- Two-sided marketplaces: adding supplier utility turns exposure inequality (the gini column in the assembly) from an aesthetic concern into a churn model. Fairness-of-exposure formulations (Singh & Joachims) make the exposure allocation an explicit constrained optimisation.
- Agent-based / LLM-driven user simulators are the current research frontier; the calibration problem — does the simulator reproduce a held-out week? — is unchanged and is the only thing that makes such a simulator credible.
Calibration: the only thing that makes a simulator citable
A simulator's assumptions are its results, so the sensitivity table in block 7 is not defensive paperwork — it is where the causal claim gets tested. The discipline:
- State the user model at the top, in three lines, as block 1 does.
- Fit the free parameters to a real log.
- Check that the simulator reproduces a held-out period it was not fit on.
- Report ordinal conclusions (which policy wins) rather than cardinal ones (how many clicks), because the absolute numbers move 7× across plausible user models while the ranking is stable.
How this connects to the rest of the track
- P08 is the system whose feedback loop this simulates.
- P10 is the ground truth this is a cheap approximation of; the two answer different questions and neither substitutes for the other.
- P07's event-time discipline is the same clock reasoning applied to logs.
- P15 closes the loop: offline hypothesis → online experiment → production monitor.
Failure modes at scale
- Simulator overfitting: tuning the user model until the policy you like wins. The defence is pre-registration of the sensitivity axes.
- Ignoring the feedback loop in the training data — the log you fit the simulator on was itself generated under a policy.
- Assuming stationarity over the horizon simulated; user preferences, catalogue and competition all move over sixty days.
- Exposure collapse presented as success: greedy maximises clicks and produces gini 0.91. A metric that does not include catalogue health will recommend catalogue destruction.
Primary sources
- Chapelle & Zhang, A Dynamic Bayesian Network Click Model (WWW 2009).
- Joachims, Swaminathan & Schnabel, Unbiased Learning-to-Rank with Biased Feedback (WSDM 2017).
- Dudík, Langford & Li, Doubly Robust Policy Evaluation and Learning (ICML 2011).
- Swaminathan et al., Off-policy Evaluation for Slate Recommendation (NIPS 2017).
- Chapelle & Li, An Empirical Evaluation of Thompson Sampling (NIPS 2011).
- Ie et al., RecSim (2019).
- Singh & Joachims, Fairness of Exposure in Rankings (KDD 2018).
Running it
python3 handson/h09_simulator.py # every block, then the assembly
python3 handson/h09_simulator.py --block 3 # just block 3 and its prerequisites
python3 handson/h09_simulator.py --quiet # the assembly only
What to do with this
Add supplier-side utility and re-run every policy. A marketplace has two populations whose interests do not coincide, and exposure inequality --- the gini column that greedy maximises --- stops being an aesthetic concern and becomes the thing that drives sellers off the platform. That is a question no offline metric can answer and a simulator can.
Milestones, experiments, readings and exit criteria for this project: P09 — Recommendation Simulator.