Lab 01 — Learning to Rank from Scratch

Phase: 05 — Learning to Rank | Difficulty: ⭐⭐⭐⭐⭐ | Time: 6–8 hours

"Implement NDCG" is a real interview question; "why does LambdaRank work" is its senior follow-up. This lab does both: NDCG to the letter, then RankNet and LambdaRank gradients on a linear scorer — with a constructed dataset where the |ΔNDCG| weighting measurably wins (≈0.52 → ≈0.74 NDCG@5 on the reference run).

What you build

dcg_at_k / ndcg_at_k / ndcg_of_scores — exact conventions ($2^{rel}-1$ gains, $\log_2(i{+}1)$ discounts, 0/0→0), pinned by hand-computed tests and invariance tests (downward swap strictly decreases; below-k changes are invisible)
make_pairs — within-group cross-label pairs only (the cross-group pair is LTR's classic leakage bug; a test hunts it)
train_ranker — RankNet gradients ($-\sigma(-(s_i - s_j))$), then LambdaRank ($\times|\Delta NDCG@k|$, computed via the two-term swap shortcut you derive)
grouped_train_test_split — query-grouped, never row-level

Key concepts

Concept	What to understand
NDCG's three parts	exponential gains (top-heavy values), log discounts (attention decay), ideal normalization (cross-query comparability)
The zero-gradient obstruction	ranks come from sorting; sorting is flat almost everywhere — you can't backprop a sort
LambdaRank's move	skip the loss, define the gradients: pairwise force × metric impact
The top-heavy dataset	two features, two objectives: plain pairwise chases the many mediocre pairs; lambda chases the metric
Query groups	pairs, splits, and metrics all respect them — or the numbers lie

Files

File	Purpose
`lab.py`	Skeleton with `# TODO` markers (dataset generator provided)
`solution.py`	Complete reference; `python solution.py` prints the ranknet-vs-lambda gap
`test_lab.py`	12 tests: hand-computed NDCG, invariances, pair discipline, training improvement, the lambda-wins construction, split discipline
`requirements.txt`	numpy, pytest

Run

pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v
python solution.py     # see the gap

Success criteria

All 12 tests pass — test_lambdarank_beats_plain_pairwise_on_topheavy_task is the thesis
You can compute NDCG@3 for a 4-doc list on paper in under two minutes
You can explain why the constructed dataset separates the two objectives (read make_topheavy_dataset until the trap is obvious)

Extensions

Replace the linear scorer with a 2-layer MLP (the gradients don't change — that's the point of the lambda formulation)
LightGBM lambdarank on the same data — verify it finds f0 too
NDCG with alternative gain (linear) and discount (1/rank) — measure how convention choices move the metric (they're choices, not laws)

References

Burges, From RankNet to LambdaRank to LambdaMART (MSR-TR-2010-82)
Järvelin & Kekäläinen (2002) — NDCG
WARMUP.md chapters 3–5

Senior ML Engineer