« Phase 32 README · Warmup · Hitchhiker's · Deep Dive · Core Contributor · Staff Notes

Phase 32 — Principal Deep Dive: ML/DL Foundations

The training loop of Lab 01 is a 30-line function. In production it is the smallest box on the architecture diagram, and almost none of the failures happen inside it. This doc looks at where these foundations actually sit — under a real ML platform — and where the bodies are buried when a model that scored beautifully offline degrades a business metric in production.

The reference architecture: five stages, one contract

A serving ML system is a pipeline: data → features → training → evaluation → serving, with a feedback loop from serving back to data (labels arrive late, distributions drift). This phase owns the middle three; the neighbors own the ends — Phase 27 for features at scale, Phase 25 for training jobs, registries, and endpoints, Phase 26 for tracking, CI/CD, and drift. The one contract that binds all five: the transformation applied at training must be bit-identical to the transformation applied at serving. Violate it and you get train/serve skew — the model learns on features it will never see in the shape it saw them. This is why Lab 01's train_val_split asserts a no-row-overlap invariant that looks trivial at lab scale: the discipline it encodes — fit every statistic (scaler mean, imputation median, target encoding, PCA basis) inside the training fold and freeze it — is the single most violated rule in applied ML, and the violation is invisible until production.

Scaling the loop: the performance envelope

The mini-batch loop scales along axes that trade off against each other, and knowing the couplings is the principal-level skill.

• Batch size shrinks gradient-estimate variance as 1/√B and saturates hardware parallelism, but it also changes the optimization: large batches take fewer, smoother steps and empirically find sharper minima that generalize slightly worse, so the standard compensation is the linear scaling rule — scale the learning rate with the batch size — plus a warmup (start η tiny, ramp up over the first few hundred steps) because a large-batch, large-η start is the classic early-divergence trap. Warmup looks like a superstition until you have watched a transformer's loss explode on step 50 without it.
• Learning-rate schedules (step decay, cosine annealing) matter more than the optimizer choice. Adam gets you 95% of the way with no tuning, but the global η still sets the scale and is still searched on a log axis — the number-one interview misconception is that "Adam means I don't tune the learning rate."
• Distributed data-parallel (PyTorch DDP, tf.distribute.MirroredStrategy) replicates the model per device and all-reduces gradients each step, so the effective batch is B × num_devices — which silently pushes you back into the large-batch regime and its LR consequences. Model too big for one device? FSDP / sharding splits parameters and optimizer state across devices.
• Mixed precision (bf16/fp16 compute, fp32 master weights) roughly halves memory and doubles throughput on modern accelerators; the sharp edge is that fp16 needs loss scaling to keep small gradients from flushing to zero, while bf16's wider exponent usually does not.
• Memory at training time is dominated not by parameters but by the cached activations the backward pass needs (the same intermediates Lab 01 caches, at scale). Gradient checkpointing trades compute for memory by recomputing them. This is why "how big a model fits" is an activation-memory question, not a parameter-count question.

Where the bodies are buried

Leakage is the most expensive failure mode in the stack because it doesn't crash — it makes offline metrics beautiful and production numbers a disaster, weeks later, with no stack trace. Four forms, each with a mechanical fix: target leakage (a feature that is a consequence of the label — days_since_account_closed in a churn model — audit any suspiciously predictive feature); preprocessing leakage (fitting a scaler on data that includes the test rows — fit inside the fold); temporal leakage (random splits on time-ordered data let the model train on the future — split by time); tuning leakage (choosing hyperparameters against the test set — open it once). The blast radius is total: every downstream decision, capacity plan, and go/no-go was made on a number that was fiction.

Non-stationarity is the RL and drift tax. Lab 03's GridWorld is stationary; production is not. Bandits and RL policies act in an environment that changes because they acted (an ad allocator shifts traffic, which shifts the reward distribution). Supervised models face the slower version: the feature distribution drifts, the label relationship drifts (concept drift), and a model frozen at training time decays. This is why Phase 26's drift monitoring is not optional infrastructure — it is the smoke detector for a silent-degradation failure whose only other symptom is a slowly-falling business metric no one attributes to the model.

Reproducibility as a system property

An unreproducible result is a rumor, and reproducibility is a system property, not a seed=42 line. The labs model the real pattern: a random.Random(seed) passed in, never the global random — because a shared global generator couples every stochastic component (init, shuffle, exploration) into one entangled state that a later refactor silently reorders. In production the same discipline scales to: seed every generator (torch.manual_seed, tf.random.set_seed, NumPy Generator), and know its limits — some CUDA kernels are non-deterministic by default for speed, so bit-exact replay needs opt-in flags (torch.use_deterministic_algorithms(True)) that cost performance. Floating-point addition is not associative, so even "the same math" in a different reduction order drifts in the last bits — which is why every lab test compares with a tolerance (pytest.approx) and never == on floats. The full lineage — which code + which data snapshot + which config produced which model artifact — is the actual reproducibility contract, and it is what experiment trackers and model registries exist to enforce.

"Looks wrong but is intentional"

• Gradients accumulate in PyTorch and you must zero_grad() every step. This looks like a footgun (and is the #2 beginner bug). It is a deliberate design that enables gradient accumulation across micro-batches — simulating a large batch on small memory by summing several backward passes before one step.
• Dropout and batch-norm behave differently at inference (model.eval() / training=False). This is not a bug; dropout must be off and batch-norm must use running statistics at serve time, and forgetting the mode switch is the #1 dropout bug.
• k-means empty-cluster repair relocates a centroid to a far point mid-iteration — which transiently raises inertia. Intentional: the alternative (a dead cluster forever) is worse, and the objective resumes decreasing afterward.
• The bias term is unregularized in Lab 01's L2. Intentional: penalizing the intercept just biases the model's baseline output, which you never want to shrink toward zero.

Cost, observability, and the honest defaults

The cost lever most teams miss is model selection itself: a gradient-boosted tree on tabular features trains in seconds on a CPU and serves in microseconds, while the reflexive neural net costs GPUs to train, an accelerator to serve, and usually loses on tabular data anyway. The senior default is a baseline (logistic regression, then a GBM) before anything fancy — it sets the bar every expensive model must clear and often ships as the final answer. Observability for a model is not request logs; it is: the train/val loss curves (TrainHistory productionized), the live feature distributions versus training, the prediction distribution, and the deferred metric that arrives when labels land. Under class imbalance the metric must be honest — PR-AUC over ROC-AUC when positives are rare (ROC's huge negative denominator hides thousands of false positives) — because a dashboard reporting 99% accuracy on 1% fraud is a dashboard reporting the class ratio.

The capacity math you should be able to do live

Bias-variance is a budget you can reason about numerically. If train accuracy is 96% and validation 71%, the 25-point gap is the variance, and the levers are ordered: more data, stronger regularization, simpler model, early stopping. If both are 71%, the gap is zero and the problem is bias — more data does nothing, and the levers invert (more capacity, more features, less regularization). The diagnostic is one plot of two curves; the response is deterministic once you read which regime you are in. That is the whole discipline, and it is what separates a platform that improves under load from one that just accumulates models nobody trusts.