« Phase 32 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor

Phase 32 — Staff Engineer Notes: ML/DL Foundations

Anyone can type from sklearn.ensemble import RandomForestClassifier and call .fit(). The gap this phase closes is between the person who runs a model and the person a team trusts to own one in production. That gap is almost entirely judgment: framing the problem, refusing to fool yourself in evaluation, and knowing which failures don't announce themselves. This doc is about that judgment and the exact signal it sends in an interview or an architecture review.

The decision framework: which quadrant, and do you even need ML

The first move on any ML problem is not choosing an algorithm — it is naming the signal you actually have. Walk it in order:

  1. Do you have labeled outcomes? If yes, it is supervised — and the follow-up is discrete (classification) or continuous (regression). Fraud, churn, credit risk, CTR, demand: almost every "ML" line in a business is here.
  2. No labels, but you want structure? Unsupervised — clustering for segmentation, PCA for compression/decorrelation, density for anomalies. The trap: you have no ground truth, so you owe the team a story for how you'll validate it (silhouette, elbow, and domain sense) before you build it.
  3. Supervision manufactured from the data itself? Self-supervised — how every foundation model this track builds on was pre-trained. You rarely build this; you consume its output.
  4. No dataset, an agent acting for delayed reward? Reinforcement — and be honest that most teams never train an RL agent. The exception that ships constantly is the bandit: any "adaptive A/B test instead of a fixed 50/50 split" is a multi-armed bandit, and naming it is a genuinely senior move.

Then the meta-question a staff engineer asks that a junior doesn't: does this need ML at all, and if so, does it need a neural net? A gradient-boosted tree on tabular features beats a transformer on tabular data, trains in seconds on a CPU, and serves in microseconds. The senior default is a baseline — logistic regression, then a GBM — before anything fancy, because it sets the bar every expensive model must clear and frequently turns out to be the answer. Reaching for a neural net on 500 rows of tabular data is the tell of someone who confuses novelty with fit.

Reading a loss curve is the single highest-leverage skill

Every "my model isn't working" conversation resolves to one question: is the training error high too, or just the validation error? Two curves, one plot:

  • Both high → underfitting, high bias. The model can't fit what it already sees. "Collect more data" does nothing. Add capacity, add features, reduce regularization, train longer.
  • Train low, val high (and rising) → overfitting, high variance. The model is memorizing. Now more data helps, as do stronger regularization, a simpler model, and early stopping.
  • Val tracks train, both plateau low → you're done; ship it.
  • Loss goes NaN → learning rate first, almost every time. Then check for a log(0) (clamp probabilities) or exploding gradients (clip them).
  • Loss dead flat from step one → gradients aren't flowing: wrong sign, a forgotten zero_grad, a detached tensor, or a dead-activation regime.

Interviewers pose this as "your model has 96% train and 71% val accuracy — what do you do?" precisely because the answer instantly reveals whether you understand the machine or just operate it. The 25-point gap is the variance; the response is deterministic once you read the regime.

Code-review red flags that should stop a merge

These are the lines a staff engineer flags on sight, because each is a silent production failure:

  • Any statistic fit before the split. A scaler, imputer, target-encoder, feature-selector, or PCA basis fit on data that includes validation/test rows. The test set has now influenced training. Fix: fit inside the training fold — an sklearn Pipeline inside cross-validation makes it automatic.
  • A random split on timestamped data. The model trains on the future and predicts the past. Anything with a timestamp gets a time-based split, full stop.
  • A suspiciously perfect feature or metric. AUC 0.99 offline is guilty of leakage until audited, not a win to celebrate. The classic is a feature that is a consequence of the label.
  • Accuracy reported on an imbalanced dataset. 99% accuracy on 1% positives is the all-negative predictor. Demand precision/recall/F1 and PR-AUC.
  • A number quoted from the test set mid-development. One look equals one use; iterate against it and it is a validation set with better branding, and the "final" number is biased.
  • No held-out set at all, or random_state unset so nobody can reproduce the run.
  • Near-duplicate or same-user rows straddling the split. Memorization scores as generalization; split by group (GroupKFold).

Production war stories (and the lesson each encodes)

  • The 0.99-AUC churn model that flatlined. A feature was days_since_account_closed — a consequence of churning, absent at prediction time. Target leakage. Nobody caught it in review because everyone was admiring the AUC. Lesson: be the person suspicious of the beautiful number.
  • The scaler fit on the whole dataset. Standardization computed over all rows before the split; test statistics bled into training; offline great, production worse, and no one could explain the gap. Lesson: fit transforms inside the fold, always.
  • "We're getting NaN — lower the batch size, add clipping, swap optimizers." The learning rate was 10× too high. Lesson: diverging loss is learning rate first, before the elaborate theories.
  • The clustering that found the units, not the clusters. k-means on unscaled features put 99% of the distance into annual_income (range 0–200,000). Lesson: standardize before any distance-based method — k-means, kNN, SVM, PCA.
  • The RL agent that "wouldn't learn." ε was fixed at 0.01 in a sparse-reward grid, so it barely explored. Lesson: exploration is a knob you tune and anneal, not a default you accept.

The signal an interviewer or review is actually listening for

They are not checking whether you can recite the Adam update. They are listening for whether you refuse to fool yourself. The candidates who get senior offers say, unprompted: "I'd split before I look, fit every transform inside the training fold, pick PR-AUC because the positives are rare, tune on validation and open the test set exactly once, and treat a too-good result as a leak to hunt." That sentence signals ownership. Equally, on frameworks: don't have a favorite, have a reason — "the concepts transfer, so it's an ecosystem decision" is the senior answer; naming a tribe is not. And on scope: knowing when the answer is a GBM on tabular features, not a foundation model is the judgment that gets you handed the ambiguous "figure out if ML even helps here" problem instead of the "implement this spec" ticket.

Closing takeaways

  1. Frame before you fit. Name the signal you actually have (label / structure / reward), then ask whether the problem needs ML — and whether it needs a neural net — before writing a line.
  2. The loss curve is your diagnostic. Train-vs-validation gap tells you bias vs variance and dictates the lever; "more data" only ever fixes variance.
  3. Leakage is the most expensive bug in ML because it doesn't crash — it makes offline numbers beautiful and production numbers a disaster with no stack trace. Split first, fit in-fold, audit the beautiful number.
  4. Evaluation lies unless you force it honest. Honest metric for the imbalance, test set opened once, mean ± std over seeds rather than one lucky run.
  5. Baselines earn their keep. Logistic regression then a GBM before anything fancy; they set the bar and often are the answer.
  6. Ownership is a discipline, not an algorithm. The person trusted with the important model is the one who is suspicious of good news and rigorous about the split — every time.