Design 02 — Recommendation System End-to-End

"Design product recommendations for an e-commerce site: 10M users, 1M items, 5k QPS on the homepage, 100 ms budget." Phases 03–06, 09, 11 built the parts; capstone 01 assembled them.

1. Requirements & metrics

Clarify: surface (homepage feed), feedback (implicit: views/carts/purchases, weighted), 100 ms p99 end-to-end, business metric = conversion-weighted engagement with revenue and diversity guardrails. Cold start matters (20% of traffic is new users; items churn weekly).

Metric stack: offline recall@k / NDCG@k against held-out future interactions (temporal split — random splits leak); online CTR/conversion via A/B; the baseline to beat is popularity-by-segment — and it's embarrassingly strong; say so.

2. Architecture — two-stage (the canonical shape and why)

Scoring 1M items per request with a heavy model at 5k QPS is ~5B scores/sec — impossible. The architecture is the funnel:

request ──► candidate generation (~1M → ~500, cheap, recall-oriented)
              ├─ two-tower / MF (BPR) via ANN index          (P06)
              ├─ popularity-by-segment (cold start floor)
              ├─ item-item neighbors of recent history
              └─ exploration slice (2–5%, propensity-logged)
                         │ union, dedup
                         ▼
            ranking (~500 → ~50, precision-oriented)          (P05)
            GBM/LTR over [user × item × context] features     (P03 store)
                         │
                         ▼
            business layer: diversity (MMR), exclusions,
            merchandising rules, position assignment
                         │
                         ▼
            serving: batched scoring (P09), feature cache,
            response cache for anonymous traffic

3. Deep dives

Candidate generation: two-tower (user tower from history/profile, item tower from content+id) trained with BPR/in-batch negatives (Phase 06); item embeddings refreshed nightly into an ANN index (HNSW: recall@500 ≥ 95% vs exact, ~1–2 ms — quote the tradeoff); user embedding computed at request time from the online store's recent-history features. Multiple sources because no single one covers head, tail, and cold start — the union is the design.

Ranking: LambdaMART-style LTR (Phase 05) with NDCG-weighted pairwise training on impression logs. Features: user (long/short-term aggregates), item (popularity, price band, quality), cross (user-category affinity, co-view counts), context (hour, device, session position) — all from the feature store with point-in-time training frames (Design 01's machinery; mention and move on).

Position bias (they will probe): clicks are biased by position; train with IPS-weighted clicks or click-model debiasing; estimate propensities from a small randomization slice; use interleaving (Phase 05) for cheap ranker iterations — 10×+ sample efficiency vs A/B — reserving A/B for end-to-end ship decisions.

The feedback loop (volunteer it): the system trains on exposures it chose — popularity compounds, the tail starves, offline eval on logged data drifts from counterfactual truth. The exploration slice is the answer and a cost; its propensity logs are what make IPS evaluation possible at all.

Latency budget (do the arithmetic aloud): 100 ms p99 = network 10 + candidate gen 15 (ANN + sources in parallel) + feature fetch 20 (one batched online-store read for 500 candidates) + ranking 25 (batched GBM) + business layer 10 + headroom 20. Each component gets its number; the fan-out tail math (Phase 09 Ch. 2) is why headroom exists.

4. Training & operations

Nightly: two-tower retrain + index rebuild; LTR retrain on trailing impression window — both as orchestrated DAGs (Phase 08) with offline gates (recall/NDCG vs incumbent on temporal holdout) and registry promotion. Online: shadow → canary (engagement + latency + revenue guardrails) → A/B with CUPED on pre-period engagement (Phase 11 — engagement metrics have high pre-period correlation; expect 30–50% variance reduction). Monitoring (Phase 12): prediction-score PSI, per-feature drift importance-weighted, coverage metrics (catalog share recommended — the feedback loop's dashboard), and freshness of embeddings/index as first-class alerts.

5. Failure modes

6. Evolution

Sequence models (SASRec-class) for session-aware candidates; real-time feature streaming (Phase 02) for in-session adaptation; multi-objective ranking (engagement + revenue + diversity scalarization → Pareto conversation with product, Phase 09 model-accuracy style); LLM-augmented item understanding for cold-start content features.