ML System Design Questions

The method, then four worked questions in compressed form. Full-depth walkthroughs of five designs live in system-design/; these are the interview-time versions.

The method (use it every time)

1. Clarify (3–4 min): who uses it, what decision it makes, scale (users, QPS, items, data volume), latency budget, what "good" means (the metric — and who owns it), constraints (labels available? cold start? regulatory?).
2. Metric & baseline (2 min): name the offline metric, the online metric, and the dumb baseline you must beat. Saying "seasonal-naive" or "popularity" out loud is a senior signal.
3. Data & features (5 min): sources, labels (and their delay!), the feature groups, point-in-time discipline, the feature store split (online/offline).
4. Model(s) (5 min): start simple, justify each escalation; for retrieval- shaped problems, the two-stage candidate-gen → rank architecture.
5. Serving (5 min): pattern (online/batch/streaming), latency budget decomposition, caching, batching.
6. Evaluation & rollout (5 min): offline harness → shadow → canary → A/B with guardrails; the decision rule pre-registered.
7. Operations (5 min): monitoring stack, drift, retraining policy, the feedback loop you're creating and its mitigation.
8. Failure modes & evolution (remaining): what breaks first at 10×, what you'd build next.

The seven seams to volunteer unprompted: training-serving skew · leakage · label delay · feedback loops · cold start · SRM/experiment hygiene · drift laundering. Each maps to a phase of this track; each has a one-line mitigation.

Q1: Design a product-recommendation system (e-commerce, 10M users, 1M items)

Clarify: homepage feed vs item-page "similar items"? (Assume homepage.) Implicit feedback only. 100 ms budget. Online metric: conversion-weighted CTR with revenue guardrail.

Skeleton: two-stage. Candidate gen — BPR/two-tower retrieving ~500 from ANN index (HNSW), unioned with popularity (cold start) and recent-history neighbors; ranker — GBM/LTR over (user, item, context) features from the feature store with point-in-time training frames; business layer — diversity (MMR), exclusions, exploration slice (~2–5% — the feedback-loop tax, stated as such).

Numbers to drop: two-tower trains nightly (items churn), user embeddings computed online from session; ANN recall@500 ≥ 95% vs exact; ranker p99 ≤ 30 ms at batch 500 → feature fetch budget 20 ms → online store required.

Operations: interleaving for ranker iterations (10× sample efficiency), A/B for end-to-end with CUPED on pre-period engagement; per-feature drift with importance weighting; weekly retrain + drift-accelerated.

Evolution: sequence models for candidate gen; real-time features (session intent) — streaming pipeline (Phase 02) is the prerequisite, say so.

Q2: Design a fraud-detection system for payments (5k TPS, decision in 150 ms)

Clarify: block vs review vs allow (three-way, asymmetric costs); labels from chargebacks (30–90 days!) plus analyst reviews (fast, biased); base rate ~0.1%.

Skeleton: streaming features (velocity counters: txn count/amount per card/merchant/device over 5 min–30 d windows — the Phase 02 windowed aggregations, exactly); GBM scorer (calibrated — thresholds are cost decisions); rules layer for hard constraints and analyst agility; review queue as the human-in-the-loop tier.

The label-delay answer (they will probe): proxy monitoring meanwhile — score-distribution drift, fast-label canary cohort (analyst-reviewed sample), backfill joins computing true metrics with honest lag (event-time! predictions joined to their moment); selective-labels bias named: blocked transactions never get labels → propensity-aware evaluation or exploration slice of borderline approvals.

Operations: adversarial drift is the norm — fraud adapts to the model; retraining cadence aggressive (daily-to-weekly), with the Phase 12 guard: data-quality gate before retrain. Asymmetric rollout (Phase 09): shadow for weeks, conservative canary, rollback-biased guardrails (decline rate, review queue depth).

Q3: Design demand forecasting for 50k SKUs × 500 stores

Clarify: horizon (1–14 days), granularity (SKU-store-day), downstream consumer (replenishment — over/under costs asymmetric → quantile forecasts, not point), history depth varies, promos known in advance.

Skeleton: this is a fleet problem — segment series (smooth / seasonal / intermittent / short) and assign methods per segment: seasonal-naive floor everywhere, GBM with lag-≥-horizon features (Phase 07's rule, stated) for rich series, Croston-style for intermittent. Hierarchical reconciliation if planners need coherent totals (bottom-up first; MinT if justified).

The discipline: rolling-origin backtesting as the promotion gate — MASE vs both incumbent and seasonal-naive on identical folds; fleet judged on median AND p90 MASE (the tail is where stockouts live). Nightly batch DAG (Phase 08) with content-hash caching — most series don't change most nights.

The trap to name: stockouts censor demand (sales ≠ demand) — uncensoring or flagging is a data decision that dominates model choice. Naming it usually ends the question.

Q4: Design the ML platform itself (feature store + training + serving for 30 teams)

Clarify: this is a product question — users are ML engineers; the metric is time-to-production and incident rate, not AUC.

Skeleton: golden path over mandates — a paved road (feature definitions as code → registry; orchestrated training with lineage; registry-gated promotion; serving templates with batching/canary built in; monitoring auto-wired from the registry). Each component is a phase of this track (03, 08, 09, 12) — design by walking the lifecycle.

The senior content: what you don't centralize (modeling choices, feature logic) vs what you do (point-in-time join engine, lineage, gates, monitoring plumbing); migration strategy for the 30 teams' existing pipelines (strangler, not rewrite); the two metrics that prove the platform: median days-to-production and % of models with passing monitoring coverage.

Rehearsal protocol

For each question: 25 minutes against a wall-clock, out loud, whiteboard or paper. Then grade against the method's eight steps and the seven seams — which did you skip? The capstone you built (Phase 13) is your worked example for at least one of these; rehearse pivoting from the abstract design to "I built exactly this seam, here's what broke."