Phase 05 — Learning to Rank
Difficulty: ⭐⭐⭐⭐⭐ Estimated Time: 2 weeks (35–45 hours) Roles Supported: the JD's "ranking" — the precision stage on top of Phase 04's retrieval, and the modeling core of search, recommendations, and ads
Why This Phase Exists
Ranking is the highest-leverage ML problem in industry — small NDCG gains move revenue at companies whose product is an ordered list — and it is methodologically its own discipline: the unit of prediction is a list, the metrics are position-discounted, the labels come from biased clicks, and the objectives (pointwise/pairwise/listwise) differ in what they actually optimize. Senior MLE interviews probe ranking depth specifically because it can't be faked from classification knowledge.
The flagship lab implements NDCG from scratch (an interview classic) and a pairwise ranker with LambdaRank-style NDCG-weighted gradients — the idea that powers LambdaMART, still the strongest tabular ranker in production twenty years on.
Concepts
- The LTR data model: queries, candidate lists, graded relevance, query groups
- Metrics: DCG/NDCG (derived, not recited), MRR, MAP; why position discounts
- Objectives: pointwise (regression on labels — ignores list structure), pairwise (order pairs correctly — RankNet's logistic loss), listwise (LambdaRank/LambdaMART — pairwise gradients weighted by metric impact |ΔNDCG|)
- Features for ranking: query, document, and query-document interaction features (BM25 score as a feature — Phase 04 composes in)
- Position bias: clicks are biased labels; examination hypothesis, counterfactual LTR / inverse propensity weighting at concept level
- Evaluation discipline: query-grouped splits (leakage otherwise!), per-query metric distributions
Labs
Lab 01 — Learning to Rank from Scratch (flagship, implemented)
| Field | Value |
|---|---|
| Goal | Implement DCG/NDCG exactly; build a pairwise logistic ranker on numpy with plain RankNet gradients and LambdaRank ( |
| Concepts | NDCG mechanics (gains, discounts, ideal ordering), pairwise loss, lambda weighting, query groups |
| Steps | 1. dcg/ndcg with the 2^rel−1 gain and log2 discount, hand-verified; 2. pair generation within query groups; 3. RankNet gradient on a linear scorer; 4. LambdaRank weighting; 5. synthetic data where lambda beats plain pairwise on NDCG@5; 6. query-grouped train/test split |
| How to Test | pytest test_lab.py — 12 tests: NDCG hand-cases + invariances, pair generation, training improves NDCG, lambda ≥ plain on the constructed task |
| Talking Points | Why is NDCG@k not differentiable and how does LambdaRank sidestep that? Why must splits be by query? What does the log2 discount encode? |
| Resume Bullet | Implemented NDCG and a LambdaRank-style pairwise ranker from scratch; demonstrated metric-weighted gradients improving NDCG@5 over plain pairwise loss on top-heavy ranking tasks |
→ Lab folder: lab-01-ltr-ndcg/
Extension Project A — LambdaMART on a Public LTR Set (spec)
Train LightGBM's lambdarank on MSLR-WEB10K (or the small LETOR sets): proper
query-grouped CV, NDCG@10 vs your linear ranker, feature importance reading, and the
"which features carry ranking" memo.
Extension Project B — Position-Bias Simulation (spec)
Simulate click logs with an examination-probability model over a known-relevance corpus; show that naive click-trained ranking learns the bias; implement inverse propensity weighting and recover the true ordering. The counterfactual-LTR intuition, built by hand.
Guides in This Phase
Deliverables Checklist
- Lab 01 implemented; all 12 tests pass
- Extension A numbers + memo
- Extension B simulation notebook