ML Fundamentals Questions — Ranking, RecSys, Forecasting, Tabular

Complete answers at the senior bar. Each maps to a track lab — if you built it, re-derive rather than re-read.

Ranking & Retrieval (Phases 04–05)

Q: Implement NDCG and explain every design choice. (This is a real coding screen — Phase 05 lab 01.) DCG@k = Σ (2^rel − 1)/log₂(i+1) over ranked positions; NDCG = DCG/IDCG where IDCG is the DCG of the ideal ordering. Choices to narrate: the log discount (smooth position decay matching attention data; base-2 convention), gain 2^rel−1 (emphasizes high grades; linear gain is the alternative for binary labels), normalization per-query (makes queries with different label counts comparable — and means NDCG can't compare absolute quality across query sets), and the edge case IDCG = 0 (no relevant docs → define 0 or skip query; say which and why it matters for averaging).

Q: Pointwise vs pairwise vs listwise LTR? Pointwise regresses absolute relevance — ignores that ranking only needs order per query and wastes capacity on calibration across queries. Pairwise (RankNet family) optimizes P(i ≻ j) on within-query pairs — matches the task, but counts all inversions equally. Listwise/LambdaRank weights each pair's gradient by |ΔNDCG| of swapping them — inversions at the top matter more, directly optimizing the position-discounted metric. That ΔNDCG weighting is the entire LambdaRank trick (Phase 05 lab implements it); LambdaMART = those gradients in a GBM, still the tabular-LTR production default.

Q: Why hybrid (BM25 + dense) retrieval instead of dense-only? Complementary failure modes: BM25 is exact on terms — wins on identifiers, rare words, exact phrases; embeds nothing it hasn't seen. Dense wins on paraphrase and semantics; fails on out-of-vocabulary precision (part numbers, names) and domain shift from its training data. RRF fusion (1/(k+rank) sums) combines them without score-scale calibration — rank-based, hence robust to incomparable score distributions (Phase 04 lab measures all three on the same eval set; hybrid wins recall@10 on mixed query sets).

Q: What is position bias and how do you train on clicks despite it? Users click what they see; position causally inflates clicks independent of relevance — naive click-trained rankers learn "whatever the old ranker put first." Mitigations: model the bias (click models — examination hypothesis), inverse-propensity weighting (weight clicks by 1/P(examined at position)), randomization slices to estimate propensities, and interleaving for unbiased ranker comparison. Volunteering this question's existence is itself the senior signal.

Recommendation (Phase 06)

Q: Why BPR rather than rating regression for implicit feedback? Implicit data has no negatives — only observed interactions and unobserved items (which mix "disliked" with "never seen"). BPR sidesteps labeling unobserveds by optimizing a ranking assumption: observed ≻ unobserved per user, via σ(score_i − score_j) on sampled triples. The loss matches the product need (rank items) rather than inventing a regression target. (Phase 06 lab: BPR recall@10 ≈ 4–5× popularity baseline on the synthetic set.)

Q: Cold start — new users and new items? New items: content features (text/image embeddings) into the candidate generators so similarity works pre-interaction; exploration boost bounded by guardrails. New users: popularity-within-context (geo, channel), session-based signals from the first clicks (sequence models shine here), onboarding preference capture. Architecturally: cold-start paths are separate candidate sources in the two-stage design, not patches on the MF model — the ranker blends them.

Q: What feedback loop does a deployed recommender create? It trains on clicks over items it chose to show — exposure bias compounds: popular gets popular, the catalog tail starves, and offline metrics computed on logged data stop reflecting counterfactual quality. Mitigations: exploration traffic (an ε slice, logged with propensities), IPS-weighted offline evaluation, diversity injection, and — the honest one — online experiments as the only unconfounded readout (Phase 11).

Forecasting (Phase 07)

Q: Why MASE over MAPE? MAPE divides by actuals: undefined/exploding near zero (intermittent demand!), and asymmetric (over-forecasting a small actual costs more than under). MASE scales MAE by the in-sample seasonal-naive MAE — defined everywhere, symmetric, scale-free across series (fleet-comparable), and interpretable: <1 beats the naive baseline. The M-competitions standardized it for these reasons.

Q: Why can't you use random K-fold CV on time series, and what replaces it? Folds would train on the future of their test points; autocorrelation lets the model memorize the neighborhood — beautiful CV scores, useless deployment. Rolling-origin replaces it: expanding train window, test on the next horizon, advance, repeat — every fold respects time's arrow and matches the deployment cadence. Corollary (the Phase 07 lab's leak alarm): every feature must be lag ≥ horizon or known-in-advance.

Q: When does ML lose to seasonal-naive, and what do you do? Short histories (features can't be estimated), stable pure seasonality (nothing left to learn), and immediately after regime changes (lags stale). Response: keep the baseline as a floor in the fleet (per-segment method assignment), shrink toward it when backtest MASE ≈ 1, and invest in covariates (promos, holidays) — new information beats more model.

Tabular & General (Phases 03, 11)

Q: Why do GBMs still beat deep learning on most tabular problems? Tabular features are heterogeneous, non-smooth, and often axis-aligned — trees' inductive bias (axis splits, scale invariance, native missing-value handling) matches; NNs' smoothness prior doesn't, and typical tabular n is too small to learn the equivalent from scratch. Empirically (Grinsztajn et al.) the gap persists at n < ~1M. NNs win when modality crosses in (text/image columns) or for embedding-sharing across tasks.

Q: What is training-serving skew and the three mechanisms that prevent it? The same feature computed differently at training vs serving time — the model learned f(x) but receives x′. Mechanisms: (1) single-definition features (one codepath/registry feeding both stores — Phase 03's design), (2) point-in-time training joins (training sees what serving would have seen), (3) continuous parity checks (sample online requests, recompute offline, alert on divergence) plus logging served features for training reuse ("log and wait" — the strongest form).

Q: Your model's offline AUC improved but the A/B shows nothing. Enumerate. (1) Offline eval distribution ≠ live distribution (logged-data bias, the recommender loop); (2) the metric gap — AUC moved where decisions don't change (thresholded actions, top-k cutoffs); (3) the experiment is underpowered for the effect size AUC predicts (do the Phase 11 math before concluding "nothing"); (4) serving-path divergence — skew, latency-induced fallbacks eating the gain; (5) the improvement is real but on a segment too small to move the aggregate. Each has a check; run them in cost order.