Warmup Guide — Recommendation Systems

Zero-to-expert primer for Phase 06: implicit feedback's epistemology, matrix factorization and BPR derived, the production candidate-gen→rank architecture, cold start, feedback loops, and evaluation that doesn't fool you.

Table of Contents

Chapter 1: The Recommendation Problem, Honestly Stated

Given a user's interaction history, rank a catalog by predicted value of showing — which already contains the field's three honest difficulties: the label is behavioral (what users did under what they were shown — Ch. 7), the objective is contested (clicks? watch time? long-term retention? — metric choice is product strategy), and the catalog is huge while attention is ~10 slots (ranking economics, Phase 05, at their sharpest). The senior posture from the first sentence: a recommender is not a model but a system with a loop in it — model → exposure → behavior → training data → model — and most production pathologies live in the loop, not the matrix algebra.

Chapter 2: Implicit Feedback and What Absence Means

Explicit ratings (1–5 stars) are scarce and biased toward the opinionated; production runs on implicit signals: views, clicks, purchases, dwell. Two structural facts:

  • Positive-only: you observe engagement, never dislike. An unobserved (user, item) pair conflates "unaware" with "uninterested" — missing not at random, since exposure was itself chosen by the previous model (Ch. 7's loop, already present in the data).
  • Consequence for objectives: treating unobserved as 0 and regressing (the naive port of ratings MF) optimizes the wrong thing. The two classic responses: weighted regression with confidence weights (implicit-ALS — observed pairs get high confidence, unobserved low) or — cleaner — stop predicting values and rank (BPR, Ch. 4): only claim that seen things should rank above unseen things, the weakest statement the data actually supports.

Chapter 3: Matrix Factorization — Embeddings Before Deep Learning

The model: user $u$ and item $i$ get $d$-dimensional vectors; affinity is their dot product, $\hat{x}_{ui} = p_u^\top q_i$ (+ biases in the ratings era). Stack them and you've claimed the interaction matrix is approximately low-rank — preferences are combinations of a few latent factors (genre-ness, price-tier, novelty appetite…) discovered, not designed.

Why this 2009-vintage idea is still the right first model: it is representation learning (the original embedding model — today's two-tower nets are MF with feature towers, the LLM track's word2vec is MF of a co-occurrence matrix [its Phase 02 chapter says so explicitly], and your trained $q_i$ vectors drop straight into Phase 04's dense index for candidate generation); it's data-efficient, trains on a laptop, and remains an embarrassing-to-lose-to baseline. Training: alternating least squares (closed-form per row, parallelizes — the implicit-ALS classic) or SGD on sampled entries (your lab, because it composes with Ch. 4).

Chapter 4: BPR — Ranking, Not Rating

The objective: for user $u$, an interacted item $i$ should outscore a non-interacted item $j$:

$$\mathcal{L}{BPR} = -\sum{(u,i,j)} \ln \sigma\big(\hat{x}{ui} - \hat{x}{uj}\big) + \lambda |\Theta|^2$$

— Phase 05's RankNet logistic pair loss, with recsys's pair source: $(u, i, j)$ triples sampled as (user, positive, random-unseen-negative). The gradients (your lab implements exactly these):

$$\delta = 1 - \sigma(\hat{x}{ui} - \hat{x}{uj}); \quad p_u \mathrel{+}= \eta,(\delta,(q_i - q_j) - \lambda p_u); \quad q_i \mathrel{+}= \eta,(\delta,p_u - \lambda q_i); \quad q_j \mathrel{-}= \eta,(\delta,p_u + \lambda q_j)$$

Notes with production weight: random negatives are mostly easy (popular-item or hard-negative sampling speeds learning — same lesson as DPR's hard negatives in the LLM track); sampled pairwise ranking optimizes AUC-like objectives, not top-of-list specifically (WARP and lambda-weighting exist for that — Phase 05's idea returning); and the output that matters is the embedding spaces, reusable far beyond the scoring rule.

Chapter 5: The Production Architecture

No system scores the whole catalog per request. The canonical four stages:

catalog (10^6–10^9)
  → CANDIDATE GENERATION (10^2–10^3): ANN over two-tower/MF embeddings (Phase 04's
    indexes!), plus heuristic sources (co-visitation, same-author, trending, recency)
  → FILTERING: availability, already-seen/bought, policy, region
  → RANKING (10^1–10^2): the heavy model — GBDT/neural with full features
    (user, item, context, *freshness-sensitive* — Phase 03's online store serves this)
  → RE-RANKING: diversity (MMR-style), business rules, exploration slots

Design notes that interviews probe: multiple candidate sources are the norm (union + dedup — robustness over elegance; each source covers another's blind spot); the ranker's features are where Phase 03's feature store earns its keep (counts, recencies, CTRs — all freshness-budgeted); seen-item filtering is a product decision (re-recommending consumables is correct; re-recommending a watched movie isn't); and exploration slots (Ch. 7's antidote) are reserved here, in re-ranking, where their cost is controlled.

Chapter 6: Cold Start and the Content Bridge

Collaborative signals need history; new users and new items have none.

  • New items: content features are the bridge — text/image embeddings (the other tracks' encoders) place a zero-interaction item near similar items in candidate space; engagement data then refines. Two-tower models do this natively (item tower consumes content features); pure-ID MF cannot (its $q_i$ is untrained) — the architectural reason two-tower replaced ID-MF at companies with item churn.
  • New users: popularity-by-context (region/device/hour) as the fallback ranker, onboarding preference capture, and fast session-based adaptation (recommend from the last 3 clicks — sequence models shine here).
  • The operational must: cold-start is a measured segment — recall@k sliced by user/item age (Ch. 8), never just the global average, because the global average is dominated by warm entities and will hide a broken cold path entirely.

Chapter 7: Feedback Loops and Popularity Bias

The loop stated plainly: the model decides exposure; users can only engage with what they see; engagement becomes training data; the next model amplifies the last one's choices. Consequences, each with a name:

  • Popularity bias: popular items gather more interactions regardless of per-exposure quality; naive training compounds it; catalogs effectively shrink (the Gini concentration your Extension B measures rising over simulated days).
  • Filter bubbles / interest narrowing: per-user loops — show sci-fi → clicks are sci-fi → show more sci-fi. Diversity re-ranking and exploration are the levers.
  • Degenerate counterfactuals: you never learn what users would have done with what you never showed — the recsys twin of Phase 05's position bias, with the same family of answers: exploration budgets (small ε of traffic on non-greedy slates — the data they generate is what keeps future models honest; frame it as paying the data-collection bill, not losing revenue), propensity logging (record why each item was shown and with what probability — Phase 11's off-policy evaluation needs it), and popularity-debiasing in training (downweight popular positives / popularity-matched negatives).

Chapter 8: Evaluating Recommenders

  • Split by time, not randomly: leave-last-out (each user's final interaction is the test target) or a global time cutoff — random splits leak the future (Phase 03's lesson; recsys edition) and overstate everything.
  • Metrics beyond accuracy (recall@k/NDCG@k on held-out interactions): coverage (share of catalog ever recommended — the health metric for Ch. 7's concentration), novelty/popularity-slices (recall on tail items), per-segment slices (cold users!). A model that wins recall by recommending bestsellers to everyone loses coverage catastrophically — the lab measures both because the trade is the point.
  • The baseline discipline: popularity (global, or per-segment) is the mandatory baseline; it's embarrassingly strong on accuracy metrics (Ch. 2's bias guarantees it) and embarrassingly bad on coverage — quoting both numbers for both models is the honest comparison.
  • Offline is a weak proxy, weaker than in search: held-out clicks were generated under the old policy (the loop again); offline wins select A/B candidates (Phase 11), and long-term metrics (retention, not session clicks) are the real objective — the gap between them is where recsys product judgment lives.

Lab Walkthrough Guidance

Lab 01 — BPR Recommender, suggested order:

  1. Data plumbing: interactions → per-user sets; leave-last-out split (each user's last interaction by timestamp is test).
  2. Baselines first (the discipline): random and popularity top-k with seen-item masking; record their recall@10 and coverage.
  3. BPR: embeddings ~ Normal(0, 0.1/√d); triple sampling (user with ≥2 train interactions, one of their positives, one unseen negative); the four-line gradient update; λ≈0.01, η≈0.05, a few epochs over n_interactions samples.
  4. recommend(u, k): scores = $p_u Q^\top$, mask seen, top-k. Cold users (no train history) → popularity fallback (the test checks you don't crash and don't return garbage).
  5. Eval: recall@10 (BPR must beat popularity on the synthetic structured data — the test's bar) and coverage (BPR must beat popularity's by construction; notice why).

Success Criteria

You are ready for Phase 07 when you can, from memory:

  1. Explain missing-not-at-random and why it kills naive regression on implicit data.
  2. Write the BPR loss and all four gradient updates.
  3. Draw the four-stage architecture with cardinalities and say where Phases 03/04/05 each plug in.
  4. Give the cold-start answer for items (content bridge / two-tower) and users (context popularity + session adaptation), plus the sliced-metrics must.
  5. Tell the feedback-loop story with its three named consequences and three levers.
  6. Defend leave-last-out + coverage + popularity-baseline as the minimum honest eval.

Interview Q&A

Q: Design a recommender for an e-commerce marketplace. (The classic.) Structure beats brilliance: clarify the objective (purchase? GMV? repeat rate?) and constraints (catalog size, item churn — determines cold-start weight). Then the four stages with numbers; candidate sources (two-tower ANN + co-purchase + trending); ranker features by freshness tier (feature store); business re-ranking; the loop-hygiene paragraph (exploration slots, propensity logging) — and the evaluation plan: offline leave-last-out for iteration, A/B on the true metric for truth, coverage and cold-segment dashboards for health. Interviewers grade the loop awareness and the eval plan more than the model choice.

Q: Your new model beats popularity by 8% recall@10 offline but the A/B shows nothing. Most likely explanations? (1) The offline gain was concentrated where it doesn't matter online — held-out clicks under the old policy reward predicting the old policy's exposure (the loop); (2) recall@10 isn't the product metric — position-weighted engagement may be flat while raw recall rises; (3) serving-path divergence — feature freshness differs between offline eval and online serving (Phase 03's temporal skew, the recsys incident); (4) the test lacked power for the realized effect (Phase 11's math). The investigation order: parity-check features, slice the A/B by segment, check the offline metric's correlation with the online one historically.

Q: Why does everyone use a two-stage (or four-stage) architecture instead of one great model? Economics and physics: scoring 10⁸ items with a 50-feature ranker per request is impossible at latency; candidate generation buys recall cheaply (ANN dot products), ranking buys precision expensively on 500 items. The split also decouples concerns — candidate sources are robust, recall-oriented, and individually simple; the ranker absorbs the business objective and feature complexity. The cost: the ranker can't recover what candidates missed (first-stage recall bounds all — Phase 04's law), so each stage gets its own metric and its own iteration loop. That sentence is the architecture, understood.

References

  • Rendle et al., BPR: Bayesian Personalized Ranking from Implicit Feedback (UAI 2009)
  • Hu, Koren, Volinsky, Collaborative Filtering for Implicit Feedback Datasets (ICDM 2008) — implicit-ALS, the confidence-weighting alternative
  • Koren, Bell, Volinsky, Matrix Factorization Techniques for Recommender Systems (IEEE Computer 2009) — the Netflix-era summary
  • Covington, Adams, Sargin, Deep Neural Networks for YouTube Recommendations (RecSys 2016) — the canonical two-stage production paper; read with Ch. 5 open
  • Yi et al., Sampling-Bias-Corrected Neural Modeling for Recommendations (RecSys 2019) — two-tower with in-batch negatives, corrected
  • Chaney et al., How Algorithmic Confounding in Recommendation Systems Increases Homogeneity (RecSys 2018) — Ch. 7, formalized
  • Microsoft Recommenders repo — reference implementations to read after building yours