Warmup Guide — ML Fundamentals

Zero-to-expert primer for Phase 02: the machine-learning method itself — generalization, the data discipline, evaluation that doesn't lie, and the sklearn machinery that encodes all of it — before any deep learning.

Table of Contents

Chapter 1: What Learning Is — Generalization, Not Memorization

Supervised learning: from examples $(x_i, y_i)$ drawn from some distribution, find $f$ minimizing loss on new draws from the same distribution. The italics carry the whole field: training loss is what you optimize, generalization is what you want, and the gap between them is the central object of study. A lookup table achieves zero training loss and learns nothing; the entire apparatus of this phase — splits, validation, regularization, cross-validation — exists to estimate and control that gap honestly.

Corollary that bites in practice: the i.i.d. assumption is a claim about deployment. A model is a contract with a distribution; when production drifts from training (new camera, new season, new user mix), the contract is void and metrics rot silently — which is why monitoring (Phase 07) is part of ML, not an afterthought.

Chapter 2: The Bias-Variance Lens

Expected error decomposes (conceptually always, exactly for squared loss) into:

  • Bias: error from the model family being too simple for the truth — underfitting. Signature: train and validation error both high and close.
  • Variance: error from sensitivity to the particular training sample — overfitting. Signature: train error low, validation error much higher.
  • Irreducible noise: label noise and unmeasured factors — the floor.

The knobs and their direction: more model capacity (deeper trees, higher polynomial, more features) trades bias down for variance up; more data reduces variance at fixed capacity (why deep learning works at scale); regularization (L2 ridge, L1 lasso, tree depth limits, early stopping) buys variance reduction at a bias price. The diagnostic instrument is the learning curve (error vs training-set size): high bias = both curves plateau high together (more data won't help — change the model); high variance = persistent gap (more data or regularization will help). Drawing and reading these curves is a lab deliverable and a career-long reflex.

Chapter 3: Data Splits and Leakage — The Cardinal Sins

  • Three-way split: train (fit parameters) / validation (choose hyperparameters, models) / test (touched once, at the end, as the unbiased estimate). The moment you iterate against the test set, it becomes a validation set and its number becomes optimistic fiction — the sin every leaderboard-chasing team commits.
  • Leakage: information from outside the training fold influencing the model. The canonical forms, all of which the lab makes you commit and then catch:
    • Preprocessing leakage: fitting the scaler/imputer/feature-selector on all data before splitting (test statistics leaked into training). The fix is structural — Chapter 8's Pipeline.
    • Group leakage: correlated samples straddling the split — multiple images of the same patient/object/session; random splits inflate scores wildly (use GroupKFold).
    • Temporal leakage: shuffling time series — training on the future (use time-based splits).
    • Target leakage: features that encode the label by construction (the "filled by the outcome process" column).
  • The professional reflex: a score that looks too good is a bug until proven otherwise — leakage hunting is debugging, with the split design as the prime suspect.

Chapter 4: Preprocessing as Part of the Model

Transformations are learned from training data and therefore part of the model:

  • Scaling: standardization (z-score) for SVMs/linear models/k-NN (anything distance- or gradient-based — features on different scales otherwise dominate by unit choice); trees don't care (split points are scale-invariant) — knowing who needs what beats applying everything everywhere.
  • Categorical encoding: one-hot for low cardinality; ordinal only when order is real; target encoding is powerful and a leakage minefield (must be fit within folds).
  • Imputation: median/most-frequent as defaults; add missingness-indicator columns — missingness is often signal.
  • Class imbalance: class_weight first (cheap, principled), resampling (SMOTE et al.) with care — and never resample the validation/test folds (evaluate on the true distribution you'll face).
  • Every one of these has a fit/transform split — fit on train folds only — which is exactly the contract sklearn's API encodes (Ch. 8).

Chapter 5: The Classical Model Zoo and When Each Wins

The models worth genuinely knowing, each with its niche and its failure:

  • Linear/logistic regression: the baseline that's embarrassingly hard to beat on wide, noisy data; coefficients are interpretable (with correlated-features caveats); regularized variants (ridge/lasso) are the defaults. Logistic outputs are probabilities by construction (calibration still worth checking).
  • k-NN: no training, all inference; the curse of dimensionality is its biography — in high-D, everything is equidistant (Phase 00 Ch. 4's geometry). Great as a sanity baseline and for "similar items" features.
  • SVM: maximum-margin linear classifier; the kernel trick buys nonlinearity without explicit features. Strong on small/medium clean data; scales poorly past ~10⁵ samples.
  • Decision trees: axis-aligned recursive splits — interpretable, scale-free, high-variance alone.
  • Random forests: bagging (bootstrap + feature subsampling) averages away tree variance — the robust default for tabular data; near-zero tuning, built-in feature importances (with bias toward high-cardinality features — caveat).
  • Gradient boosting (XGBoost/LightGBM): trees fit sequentially to residuals — the tabular state of the art when tuned; the answer to "why not deep learning on tabular data" is usually "boosting wins."
  • The meta-skill: always fit the dumb baseline first (majority class, linear model) — the delta over baseline is your actual contribution, and half of impressive numbers evaporate against it.

Chapter 6: Evaluation Metrics — Beyond Accuracy

Accuracy fails exactly when it matters (imbalance): 99% accuracy on 1% prevalence is the all-negative classifier. The toolkit:

  • The confusion matrix is the ground truth; everything else is a summary of it.
  • Precision (of predicted positives, how many real) vs recall (of real positives, how many found) — the trade is set by the decision threshold, and choosing the threshold is a business decision (cost of FP vs FN), not a default 0.5. F1 is their harmonic mean when you must have one number; F-β when costs are asymmetric.
  • PR curves vs ROC curves: ROC-AUC is threshold-free but flatters under heavy imbalance (the FPR denominator is huge); PR-AUC is the honest summary for rare positives — the distinction is a standard interview probe and a real reporting decision.
  • Calibration: does predicted 0.8 mean 80%? Reliability diagrams; Platt/isotonic recalibration — matters whenever scores feed downstream decisions (it almost always does).
  • And — from the other tracks' shared discipline — error bars: scores on finite test sets carry sampling variance; cross-validation's per-fold spread (Ch. 7) is the cheapest honest uncertainty estimate.

Chapter 7: Cross-Validation and Honest Model Selection

  • k-fold CV (k=5/10): every sample serves in validation once; report mean ± std — the spread is your uncertainty and your overfitting alarm. Stratify for classification (preserve class ratios per fold); group/time variants per Ch. 3's leakage modes.
  • Hyperparameter search: grid (small spaces), random (better per budget in high-D — the counterintuitive classic result), then Bayesian/successive-halving when runs are expensive. All searched inside the CV loop on train data only.
  • Nested CV for the strict question "what's the generalization of my whole selection procedure": outer folds estimate, inner folds select — expensive, and the honest answer when the dataset is small and the claim matters.
  • The result of selection is a procedure; the final model refits the winning recipe on all training data, and the untouched test set speaks once.

Chapter 8: Pipelines — The Engineering Pattern

sklearn's deepest design contribution is the API contract: everything is an estimator (fit), transformers add transform, predictors add predict — and Pipeline chains them into one estimator:

pipe = Pipeline([("prep", ColumnTransformer([...])), ("clf", RandomForestClassifier())])
cross_val_score(pipe, X, y, cv=5)        # preprocessing fit *inside* each fold
GridSearchCV(pipe, {"clf__max_depth": [3, 10, None]}, cv=5)

What the pattern buys, structurally: preprocessing leakage becomes impossible (fit happens per-fold automatically — Ch. 3's worst sin engineered away), hyperparameter search spans preprocessing and model in one space, and the deployed artifact is a single serializable object whose train-time and serve-time transforms cannot drift apart (the training-serving skew bug, pre-empted — Phase 07 returns to this). ColumnTransformer routes numeric/categorical columns; custom transformers implement fit/transform and slot in. This composition discipline is the phase's lasting engineering lesson — the deep-learning equivalents (torchvision transforms, tf.data) are the same idea with tensors.

Lab Walkthrough Guidance

Order: labs 01→04 as numbered.

  • Lab 01 (sklearn pipeline): build the full Pipeline + ColumnTransformer + CV + GridSearch stack on a tabular problem; then deliberately commit preprocessing leakage (scaler fit on all data, manually) and measure the score inflation — the delta is the lesson.
  • Lab 02 (preprocessing): per-model scaling experiments (SVM vs forest with/without standardization); target-encoding leakage demo; imbalance handling with the evaluate-on-true-distribution rule.
  • Lab 03 (model evaluation): imbalanced dataset; produce confusion matrix, PR and ROC curves, threshold sweep with a cost function, calibration plot — and write the one-paragraph "which metric and threshold I'd ship and why."
  • Lab 04 (pandas/sklearn deep dive): the EDA-to-model workflow — groupby aggregations as features, learning curves (Ch. 2's instrument), and the baseline- first discipline with a results table.

Success Criteria

You are ready for Phase 03 when you can, from memory:

  1. Define generalization gap and explain why the test set speaks once.
  2. Diagnose bias-vs-variance from learning curves and prescribe per diagnosis.
  3. Name four leakage modes with their split-design fixes.
  4. Say which models need scaling and why trees don't.
  5. Choose PR-AUC vs ROC-AUC under imbalance and defend a threshold from costs.
  6. Explain how Pipeline makes preprocessing leakage structurally impossible.

Interview Q&A

Q: Your colleague reports 99.2% accuracy on fraud detection. What do you ask? Base rate first (at 0.5% fraud prevalence, all-negative scores 99.5% — accuracy is the wrong metric; show me the confusion matrix and PR curve). Then split design (group leakage across the same card/customer? temporal leakage — trained on future?), then preprocessing leakage (encoders fit on full data), then target leakage (post-hoc features). The ordered skepticism — metric, split, leakage — is the answer; "great, ship it" is the failing one.

Q: Random forest vs gradient boosting — how do you choose? Forest: parallel bagging, variance reduction, nearly tuning-free, robust to noise — the fast strong baseline. Boosting: sequential bias reduction, higher ceiling when tuned (LR, depth, regularization), more overfit-prone on noisy/small data, and sensitive to label noise. Practice: forest first as the honest baseline; boosting when the delta matters and you'll pay the tuning cost; both lose to "fix the features" most of the time.

Q: Why does cross-validation give you uncertainty "for free," and what doesn't it cover? Per-fold scores are k estimates of generalization from k different train/val draws — their spread estimates sampling variance of the score. What it misses: folds share most training data (estimates correlated — the spread understates), it says nothing about distribution shift (i.i.d. within the dataset only), and selection bias if you report the best of many CV runs (multiple comparisons — the other tracks' Phase 08/09 statistics apply here verbatim).

References

  • Hastie, Tibshirani, Friedman, The Elements of Statistical Learning (free PDF) — ch. 2, 7 for bias-variance and model assessment
  • scikit-learn User Guide — especially "Common pitfalls and recommended practices" (the leakage doc)
  • Kaufman et al., Leakage in Data Mining (2012) — the leakage taxonomy's source
  • Bergstra & Bengio, Random Search for Hyper-Parameter Optimization (2012)
  • Davis & Goadrich, The Relationship Between Precision-Recall and ROC Curves (2006)
  • Niculescu-Mizil & Caruana, Predicting Good Probabilities with Supervised Learning (2005) — calibration
  • Hands-On Machine Learning (Géron, 3rd ed.) — ch. 2–7 pair with the labs