P08 — End-to-End Recommendation System

Run it first. There is a companion page that builds this project's machinery as numbered, independently runnable blocks and then assembles them into one measured system: P08 hands-on — block by block (handson/h*.py). Every number on it was produced by running the code. Read it alongside the milestones below.

Medium · 66 hours · Weeks 84–89 · Stage 4 · Python

The bar is higher here. This is your professional domain. A project that would be a strong result in Stage 1 is a mediocre one in Stage 4. Score yourself against what a specialist would expect, not against what a newcomer would achieve.

Table of Contents

• The Loop, Instantiated
• Why This Project Matters
• Prerequisites
• Duration and Size
• Central Technical Questions
• Architecture
• Showcase — Do This Before You Start
• Implementation Milestones
• Concepts To Study
• Primary-Source Readings
• Experiments
• Benchmarks and Metrics
• Correctness Tests
• Failure Tests
• Expected Difficulties
• Scope Boundaries
• Deliverables
• Exit Criteria
• Extension Ideas
• Connections
• References

The Loop, Instantiated

Why This Project Matters

You build these professionally. So the value here is not "learn recommenders" — it is learn to evaluate one honestly, which is a genuinely different and rarer skill.

The central hazard: almost every accuracy metric can be improved by recommending popular items, because popular items are popular. A system that quietly collapses onto the head of the catalogue will show a rising NDCG and a falling product. The only defence is a metric suite that includes coverage, novelty, and exposure concentration alongside accuracy, and the discipline to report all of them every time.

The second reason is specific to news, and to you: a news recommender is a system where the catalogue turns over faster than the user model converges. That makes cold start the normal case rather than the exception, and it makes freshness a first-class metric rather than a tie-breaker. Most recommender literature assumes a stable catalogue and does not transfer.

Prerequisites

• P02 complete — retrieval calls your index, not a library's
• P01/P13 helpful — you can generate embeddings and reason about their geometry
• From math.md: §Ranking Metrics (2 h), §Implicit Feedback and Bias (2 h)

Duration and Size

Medium, 66 hours, 6 weeks. Shorter than comparable projects because you start with domain knowledge. That is deliberate, and the exit bar compensates.

Central Technical Questions

1. What is a user's interest, as a mathematical object? A point, a distribution, a set of points, a trajectory? Each choice makes different failures possible.
2. How fast should a profile forget? Derive the relationship between decay rate and effective memory before you sweep it.
3. Where does recall actually get lost — retrieval or ranking? Most teams optimise the wrong stage because they never separate them.
4. What does diversity cost in accuracy, and is the trade favourable? Measure it; do not assume.
5. How do you evaluate a recommender offline when the logs were produced by a different policy? This is the hardest question in the project and the one with no clean answer.
6. Which of your improvements is just popularity bias in disguise?

The EMA profile, derived

An exponentially-weighted profile updates as \(u_t = \alpha x_t + (1-\alpha) u_{t-1}\), giving item \(i\) interactions ago a weight \(\alpha(1-\alpha)^i\). The half-life — interactions until an item's weight halves — is \(h = \ln(0.5)/\ln(1-\alpha)\):

At α=0.5 the profile is essentially "the last two things you clicked". At α=0.02 it is a slow-moving average that will not notice a genuine interest change for a month. The mean profile is the α→0 limit with equal weights, and it can never adapt at all.

Choose α from a stated assumption about how fast interests drift, then test that assumption. In P09 you will simulate populations with known drift rates and can finally check whether your chosen α was right — which is the single best reason to build the simulator.

Recall propagation through the pipeline

Retrieval-then-ranking is a funnel, and the funnel has a ceiling:

\[ \text{recall}_{\text{end-to-end}}@k \le \text{recall}_{\text{retrieval}}@K \]

If retrieval recall@K is 0.90, no ranker — however good — can exceed 0.90 end-to-end. This is obvious once stated and routinely ignored: teams spend quarters on ranking models while the retrieval stage silently caps them.

Your ANN index's recall from P02 is therefore a hard ceiling on your recommender's quality, and E5 measures exactly how much of it you are losing. That linkage between two of your own projects is one of the most satisfying measurements in the journey.

Popularity concentration

If item popularity follows a Zipf distribution with exponent α, the head's share of total engagement mass is:

At α=1.0, recommending only the top 1% of the catalogue captures 53% of all engagement. A trivial bestseller list will beat a mediocre personalised model on accuracy metrics, and it will do so while covering 1% of the catalogue. This is why the popularity baseline is mandatory and why coverage is reported alongside NDCG, always. tools/metrics.py computes both.

Architecture

  interactions ──► profile builder ──┬─ mean vector
                                     ├─ EMA vector (α swept)
                                     └─ multi-interest: cluster history, keep top-c centroids
                                              │
   catalogue ──► embeddings ──► ANN index (P02)│
                                              ▼
                            ┌──── retrieval (candidates, K≈500) ────┐
                            │  ANN by profile · ANN per interest    │
                            │  recency pool · popularity pool       │
                            └──────────────┬────────────────────────┘
                                           │  union, dedupe by id
                                           ▼
                            filters: seen · blocked · region · age
                                           ▼
                            ranking: score = w1·sim + w2·freshness
                                             + w3·quality − w4·redundancy
                                           ▼
                            diversity pass (MMR) ──► near-duplicate collapse ──► top-k

Freshness for news is not a tie-breaker. Model it explicitly, e.g. an exponential decay \(f(a) = e^{-a/\tau}\) in article age \(a\), with \(\tau\) a swept parameter. A relevance-only ranker on a news corpus surfaces last month's best article over today's good one, forever.

MMR (maximal marginal relevance) selects greedily: \(\arg\max_i [\lambda \cdot \text{rel}(i) - (1-\lambda)\max_{j \in S}\text{sim}(i,j)]\). One parameter, one line, and it produces the accuracy/diversity frontier of E7.

Showcase — Do This Before You Start

W6 · walkthroughs/w6_popularity.py · ~40 minutes

A working miniature of this project: a bestseller list beating every personalised recommender on NDCG by 2.5x while covering 0.5% of the catalogue.

cd walkthroughs && python3 w6_popularity.py

It is 80-ish lines and it surfaces this project's central surprise in an evening rather than in week six. Run it before committing the weeks.

Implementation Milestones

Concepts To Study

• Implicit feedback: clicks are not ratings; absence of a click is not a negative
• Position and presentation bias: the strongest predictor of a click is where the item was shown
• Two-stage retrieval/ranking and the recall ceiling
• User representation: mean, EMA, multi-interest clustering, sequence models
• Content-based vs collaborative vs hybrid, and why news is content-heavy
• Cold start: item cold start dominates in news; user cold start dominates in acquisition
• Freshness modelling: decay functions, and half-life as a product decision
• Diversity: MMR, determinantal point processes (know they exist), intra-list distance
• Filter bubbles and feedback loops: the recommender shapes the data that trains its successor
• Ranking metrics: NDCG's discount as a modelling assumption, MAP, MRR — and which fits a feed
• Beyond-accuracy metrics: coverage, novelty, serendipity, Gini
• Calibration: does the topic mix of recommendations match the user's history mix?
• Off-policy evaluation: IPS, capped IPS, doubly robust — enough to know why naive offline replay is biased

Primary-Source Readings

Budget: 10 hours.

Experiments

E11 is the experiment that makes this a Stage 4 project. For every configuration you evaluate, record coverage and Gini next to NDCG. Then find at least one config where NDCG improved and coverage collapsed, and write it up. That is the honest- evaluation skill the project exists to build.

E5 is the linkage experiment. Sweep your ANN's efSearch, measure retrieval recall and end-to-end NDCG at each point, and plot them together. You will find a knee beyond which better retrieval buys nothing — that knee is your correct operating point, and almost nobody computes it.

Benchmarks and Metrics

All computed by tools/metrics.py. Report the whole suite for every configuration. A table with only NDCG is a rejected result.

Segmented reporting is not optional. An aggregate metric on a Zipf-distributed user population is dominated by heavy users. A change that helps the top decile and harms everyone else looks like a win in aggregate, and it is the most common way a recommender gets worse while its dashboard improves.

Correctness Tests

1. Never recommend a seen item. Zero, across the full evaluation.
2. No duplicate ids within one list.
3. Every returned item exists and satisfies every active filter.
4. Exactly k items returned, or fewer with an explicit reason logged.
5. Determinism: same user state + same seed → same list.
6. Metric implementations verified against hand-computed examples — the worked cases in metrics.py's demo.
7. No future leakage: an item published after the request time can never appear. This is the label-leakage bug of recommenders and it inflates offline metrics enormously.
8. Empty history produces a sensible cold-start list, not a crash.
9. Empty candidate set after filtering degrades gracefully.
10. Profile update is order-dependent for EMA and order-independent for mean. Test both properties — getting this backwards is a real bug.

Failure Tests

Expected Difficulties

1. You will beat the baselines by less than you expect, or not at all. See the Zipf table. This is the expected outcome and it is a legitimate result; report it and diagnose it rather than tuning until the number looks better.
2. Offline evaluation is biased by the logging policy. Your logs record what the old system showed. Items never shown have no positives and score as failures. Read the off-policy material, state the bias in the report, and note that P09 exists precisely to escape it.
3. Metric selection will tempt you. Decide the primary metric before running the sweep and write it down.
4. Segment or be fooled. Always.
5. Freshness and accuracy fight, and the accuracy metric always wins offline because offline data cannot express "this was stale when shown". Note the limitation explicitly; it is another P09 motivation.
6. The domain-expertise trap: you will be tempted to jump to the sophisticated design. Build the naive one and measure it first — the whole method depends on it, and knowing the answer in advance is exactly when the discipline matters most.

Scope Boundaries

In scope: content-based and hybrid retrieval, profile construction, ranking, diversity, freshness, dedup, cold start, offline evaluation, a full metric suite.

Out of scope: training a large neural ranker; real user traffic; a production serving stack; multi-objective optimisation beyond a weighted sum; sequence models (extension); a feature store; real-time profile updates (P07 does that, and P15 integrates it).

Deliverables

1. recsys/ — pipeline, four baselines, metric suite, sweep runner
2. REPORT.md centred on E11 (the popularity-bias audit) and E5 (the recall ceiling)
3. A comparison matrix: every configuration × every metric, one table. This is the portfolio artifact
4. Notebook entries for E2, E5, E11
5. A written statement of the offline-evaluation bias, and what P09 will do about it

Exit Criteria

• All four baselines implemented and scored on the full suite
• Your system compared against all four, with an honest verdict per metric
• E2 + E3 complete: EMA vs mean, swept, segmented by history length
• E5 complete: retrieval recall ceiling measured, knee identified
• E11 complete: at least one configuration documented where accuracy rose and coverage fell
• All metrics reported segmented, not only aggregate
• Cold-start paths measured separately for new users and new items
• Latency budget measured per stage
• REPORT.md written with a falsified prediction and the offline-bias limitation stated

Extension Ideas

• Two-tower retrieval trained on your interactions, compared against content embeddings at equal latency.
• Bandit exploration (Thompson sampling or LinUCB) with measured regret and coverage effects.
• Sequence model (GRU4Rec-style) as the profile, compared against EMA. The direct test of "is a learned sequence better than an exponential average", which is a question with a real answer in your domain.
• Off-policy evaluation with capped IPS, compared against naive replay — quantifying the bias you flagged.

Connections

Backward: P02 is retrieval and sets the recall ceiling. P03 provides filtered search and the selectivity regime. P01/P13 provide embeddings.

Forward:

• → P09: the simulator evaluates this pipeline under known ground truth, escaping offline bias. Keep the pipeline's interface stable and swappable
• → P10: A/B testing over the same pipeline
• → P15: the serving path of the integrated system

References

• Covington, P., Adams, J., Sargin, E. Deep Neural Networks for YouTube Recommendations. RecSys 2016.
• Steck, H. Calibrated Recommendations. RecSys 2018.
• Carbonell, J., Goldstein, J. The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries. SIGIR 1998.
• Chaney, A. J. B., Stewart, B. M., Engelhardt, B. E. How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility. RecSys 2018.
• Cañamares, R., Castells, P. Should I Follow the Crowd? SIGIR 2018.
• Hu, Y., Koren, Y., Volinsky, C. Collaborative Filtering for Implicit Feedback Datasets. ICDM 2008.
• Wu, F. et al. MIND: A Large-scale Dataset for News Recommendation. ACL 2020.
• Joachims, T., Swaminathan, A., Schnabel, T. Unbiased Learning-to-Rank with Biased Feedback. WSDM 2017.
• Ricci, F., Rokach, L., Shapira, B. (eds.) Recommender Systems Handbook, 3rd ed. Springer, 2022.
• Kunaver, M., Požrl, T. Diversity in recommender systems — A survey. Knowledge-Based Systems 123, 2017.