« Phase 32 README · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor · Staff Notes

Phase 32 Warmup — ML & Deep Learning Foundations

Who this is for: someone who can build agent infrastructure on top of foundation models (the rest of this track) and now needs the classic ML competency an AI/ML Engineer JD leads with: place a problem in the supervised/unsupervised/RL taxonomy, build and train a model with TensorFlow or PyTorch, engineer features without leaking the future, evaluate without fooling yourself, and tune hyperparameters systematically. Everything here is first-principles and mechanism-level; the deep-transformer internals (attention math, autograd engines, LLM training) live in the sibling Senior AI Engineer track and are only surveyed here. No GPU, no framework install — the labs implement every numeric routine in pure Python.

Table of Contents

  1. The ML problem taxonomy: what supervises the learning
  2. Generalization: the bias-variance tradeoff
  3. Regularization: L1, L2, dropout, early stopping
  4. The supervised family and its loss functions
  5. Gradient descent: batch, SGD, mini-batch, momentum, Adam
  6. The training loop: forward, loss, backward, step
  7. The unsupervised family: k-means, hierarchical, GMM, PCA
  8. Reinforcement learning: MDPs, Q-learning, bandits, policy gradients
  9. TensorFlow vs PyTorch: the deep comparison
  10. Deep learning survey: MLP, CNN, RNN, Transformer
  11. Deep-learning plumbing: activations, initialization, normalization, backprop
  12. Feature engineering: encoding, scaling, imputation, selection
  13. Data leakage: the silent killer
  14. Evaluation: splits, cross-validation, metrics, imbalance
  15. Hyperparameter tuning: grid, random, Bayesian, Hyperband
  16. Experiment reproducibility
  17. Common misconceptions
  18. Lab walkthrough
  19. Success criteria
  20. Interview Q&A
  21. References

1. The ML problem taxonomy: what supervises the learning

Machine learning is function approximation from data; the taxonomy is organized by what signal supervises the approximation. Getting a problem into the right box is the first decision an ML engineer makes, and the first thing an interviewer checks.

  • Supervised learning — the dataset contains the answer: pairs \((x, y)\), and the model learns \(f: x \mapsto y\) by minimizing a loss between \(f(x)\) and \(y\). Two sub-shapes: classification (discrete \(y\): spam/not-spam, churner/loyal) and regression (continuous \(y\): price, demand, latency). Almost every "ML" line in a business — fraud, churn, credit risk, CTR — is supervised.
  • Unsupervised learning — no \(y\) at all. The model finds structure in \(X\) alone: clustering (which points belong together — k-means, hierarchical, GMM), dimensionality reduction (which directions carry the variance — PCA, autoencoders, UMAP for visualization), density/anomaly modeling (which points are improbable). The hard part is that there is no ground truth to score against; §7 covers how you evaluate anyway.
  • Self-supervised learning — supervision manufactured from the data itself: mask a word and predict it, delete a patch and inpaint it, contrast two augmentations of the same image. No human labels, yet the training is mechanically supervised (there is a target — it just came from the input). This is how every foundation model this track builds on was pre-trained; next-token prediction is self-supervision at internet scale.
  • Reinforcement learning — no dataset at all at the start: an agent acts in an environment, receives rewards (often delayed), and must learn a policy that maximizes cumulative reward. The two problems that define it — credit assignment across time and the exploration-exploitation dilemma — exist in no other quadrant (§8, Lab 03).

Two placements interviewers love because they straddle boxes: recommendation (supervised on clicks, but exploration-sensitive — the bandit view often wins in production), and anomaly detection (unsupervised density modeling when you have no labeled anomalies, supervised the moment you accumulate confirmed incidents). The senior move is naming the signal you actually have, not the algorithm you'd like to use.

2. Generalization: the bias-variance tradeoff

The job is never to fit the training set — a lookup table does that perfectly. The job is to predict on data you haven't seen, and the bias-variance decomposition explains the fundamental tension. For squared error, the expected error of a learner at a point decomposes as:

\[ \mathbb{E}\big[(y - \hat f(x))^2\big] \;=\; \underbrace{\big(\mathbb{E}[\hat f(x)] - f(x)\big)^2}{\text{bias}^2} \;+\; \underbrace{\mathbb{E}\big[(\hat f(x) - \mathbb{E}[\hat f(x)])^2\big]}{\text{variance}} \;+\; \underbrace{\sigma^2}_{\text{irreducible noise}} \]

Bias is the error from the model family being too rigid to represent the truth (a line fit to a parabola: wrong no matter how much data you feed it). Variance is the error from the model being so flexible it fits the sampling noise of this particular training set (a depth-30 tree memorizing 500 rows: retrain on a new sample and get a completely different tree). The noise term is the floor no model beats.

The observable symptoms are underfitting (high bias: training error and validation error both high — the model can't even fit the data it sees) and overfitting (high variance: training error low, validation error high and often rising while training error keeps falling — the model is memorizing). The diagnostic is always the same picture: plot training and validation loss against epochs or model capacity, and read which regime you're in. Lab 01's TrainHistory records exactly these two curves, and its early-stopping test catches the moment the curves diverge.

The levers: more bias-reduction (bigger model, more features, less regularization, longer training) when underfitting; more variance-reduction (more data, stronger regularization, simpler model, early stopping, ensembling — bagging exists purely to average variance away) when overfitting. Modern deep learning complicates the clean U-shaped picture (very large networks can "double-descend" back to good generalization), but the diagnostic — compare train vs validation curves, respond to the gap — remains exactly how practitioners work.

3. Regularization: L1, L2, dropout, early stopping

Regularization is any technique that trades a little training fit for better generalization — deliberately adding bias to remove variance.

L2 (ridge / weight decay) adds \(\frac{\lambda}{2}\lVert w \rVert_2^2\) to the loss; the gradient contribution is \(\lambda w\), so every step shrinks each weight proportionally to its size — "weight decay." Large weights mean sharp, confident decision surfaces that swing wildly with small input changes; L2 keeps the surface smooth. It shrinks weights toward zero but almost never to zero. Lab 01 implements this exactly and its test proves the trained \(\lVert w \rVert\) drops when \(\lambda\) rises. (In Adam, naive L2-in-the-loss and true weight decay differ — that is the entire point of AdamW, §5.)

L1 (lasso) adds \(\lambda\lVert w \rVert_1\); its gradient is a constant-magnitude \(\pm\lambda\) regardless of the weight's size, so small weights get pushed through zero and stay there. Result: sparse models — L1 is feature selection performed by the optimizer, and the reason lasso is the classic "which of my 10,000 features matter" tool. Elastic net mixes both.

Dropout (Srivastava et al., 2014): during training, independently zero each hidden unit with probability \(p\) (and rescale the survivors by \(1/(1-p)\), the "inverted dropout" convention, so activation magnitudes match at inference when dropout is off). No unit can rely on a specific co-adapted partner, so the network learns redundant, robust features; equivalently, you are cheaply training an exponential ensemble of subnetworks that share weights. It is a train-time-only behavior — the number one dropout bug in practice is forgetting to switch the model to eval mode (model.eval() in PyTorch, training=False in Keras) at inference.

Early stopping: monitor validation loss during training; when it has failed to improve by min_delta for patience consecutive epochs, stop and restore the best-seen weights. It regularizes because training-set optimization explores increasingly training-specific minima over time; stopping at the validation optimum picks the point where generalization peaked. It is the cheapest regularizer (costs nothing, requires no retraining to tune) and the most used. Lab 01 implements Keras-EarlyStopping semantics exactly, restore_best_weights included.

Also in the toolbox: data augmentation (enlarge the effective dataset with label-preserving transforms), label smoothing, and ensembling. The interview trap: "which regularizer is best?" has no answer — L1 when you need sparsity/selection, L2 as the default smoother, dropout for wide deep nets, early stopping always, because it's free.

4. The supervised family and its loss functions

Each classic supervised model is a hypothesis class plus a loss; knowing the pairs is the difference between name-dropping and understanding.

Linear regression — \(\hat y = w \cdot x + b\), trained by minimizing mean squared error \(\frac{1}{n}\sum (\hat y_i - y_i)^2\). MSE is not arbitrary: it is the maximum- likelihood estimator under Gaussian noise, it makes the objective convex with a closed-form solution (\(w = (X^TX)^{-1}X^Ty\), the normal equations), and its quadratic growth is also its weakness — outliers dominate. MAE (mean absolute error) is the robust alternative (median regressor); Huber loss interpolates between them.

Logistic regression — the linear score squashed through the sigmoid \(\sigma(z) = 1/(1+e^{-z})\) to give \(P(y{=}1\mid x)\), trained with binary cross-entropy (log loss): \(-[y\log p + (1-y)\log(1-p)]\). BCE is the negative log-likelihood of a Bernoulli model — and the reason everyone uses it with sigmoid is the gradient identity Lab 01 makes you derive and verify: the messy chain rule collapses to

\[ \frac{\partial L}{\partial z} = p - y \]

— the residual. Prediction error is the gradient signal. (Why not MSE on probabilities? The sigmoid's flat tails make MSE's gradient vanish exactly when the model is confidently wrong; BCE's log punishes confident wrongness without bound.) Multi-class: softmax + categorical cross-entropy, the same structure.

Decision trees — recursive axis-aligned splits chosen greedily to maximize impurity reduction: Gini \(\sum_k p_k(1-p_k)\) or entropy \(-\sum_k p_k \log p_k\) for classification, variance for regression. Interpretable, scale-invariant, natively handle mixed feature types — and high-variance memorizers when grown deep, which is why they ship inside ensembles: random forests (bagging + random feature subsets: average many decorrelated trees to kill variance) and gradient-boosted machines (XGBoost, LightGBM, CatBoost: fit each new shallow tree to the gradient of the loss w.r.t. the current ensemble's predictions — gradient descent in function space). GBMs remain the strongest default on tabular data, a fact every honest deep-learning practitioner concedes.

SVMs — find the maximum-margin separator by minimizing hinge loss \(\max(0, 1 - y \cdot f(x))\) (labels ±1) plus L2. Points beyond the margin contribute zero loss — only the support vectors on or inside the margin define the boundary. The kernel trick computes inner products in an implicit high-dimensional feature space (RBF kernel = infinite-dimensional) without materializing it, buying nonlinear boundaries with convex optimization. Costs scale poorly with dataset size, which is why SVMs faded from big-data production but remain an interview staple for margin/kernel reasoning.

k-nearest neighbors — no training at all: predict the majority label (or mean) of the k closest training points. No loss function; all the modeling lives in the distance metric and the choice of k (small k = low bias/high variance; large k = the reverse — the cleanest bias-variance dial in ML). Requires feature scaling (§12) or the largest-scaled feature silently owns the distance, and inference costs grow with the training set — the exact tradeoff inverted by parametric models. kNN over embedding vectors is also, mechanically, what every vector-store retrieval in this track's RAG phases does.

5. Gradient descent: batch, SGD, mini-batch, momentum, Adam

Everything trainable in this phase and beyond is trained by some member of one family: follow the negative gradient of the loss.

\[ w_{t+1} = w_t - \eta \, \nabla_w L(w_t) \]

with learning rate \(\eta\) — the single most important hyperparameter in ML. Too large: divergence or oscillation. Too small: geological convergence, and a real risk of parking in a poor region. Search it on a log scale (§15).

Batch gradient descent computes the exact gradient over the full dataset per step — exact but expensive, and one step per epoch. Stochastic gradient descent (SGD) uses one example per step: cheap, noisy (the noise usefully kicks the iterate out of sharp minima), but hardware-hostile. Mini-batch SGD — the universal practice — uses batches of ~32–1024: the gradient estimate's variance shrinks with batch size while the matrix math saturates the GPU/CPU's parallelism. Lab 01's loop is mini-batch SGD with a seeded shuffle each epoch (the shuffle matters: a fixed order correlates consecutive gradients and can bias the trajectory).

Momentum treats the iterate as a heavy ball: keep a velocity that exponentially averages past gradients,

\[ v_{t+1} = \mu v_t - \eta \nabla L(w_t), \qquad w_{t+1} = w_t + v_{t+1} \]

(\(\mu \approx 0.9\)). Components of the gradient that persist across steps (the true downhill) accumulate; components that flip sign (ravine walls, batch noise) cancel. Result: faster progress along valleys, damped oscillation across them.

Adam (Kingma & Ba, 2014) adds per-parameter adaptive scaling: track the exponential moving average of gradients \(m_t\) (first moment ≈ momentum) and of squared gradients \(v_t\) (second moment), bias-correct both (\(\hat m_t = m_t/(1-\beta_1^t)\), likewise \(\hat v_t\) — early EMAs are biased toward zero), then step each parameter by

\[ w_{t+1} = w_t - \eta \, \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon} \]

Parameters with consistently large gradients get small effective steps; rarely-updated or small-gradient parameters get large ones. Defaults (\(\beta_1{=}0.9, \beta_2{=}0.999\)) work almost everywhere, which is why Adam is the default optimizer of deep learning. AdamW fixes a subtle wrong: L2 added to the loss gets divided by \(\sqrt{\hat v_t}\) like every other gradient, so heavily-updated weights are barely decayed; AdamW applies weight decay directly to the weights, outside the adaptive scaling — decoupled weight decay, now the standard for training transformers.

Complete the picture with learning-rate schedules — step decay, cosine annealing, and warmup (start tiny, ramp up — essential for transformer training stability) — and the rule of thumb: SGD+momentum can generalize slightly better with careful tuning; Adam/AdamW gets you 95% of the way with none.

6. The training loop: forward, loss, backward, step

The loop is one shape everywhere; internalize it once and every framework becomes syntax.

  1. Forward: run the batch through the model to get predictions.
  2. Loss: compare predictions to targets with the loss function — one scalar.
  3. Backward: compute the gradient of that scalar w.r.t. every parameter. In Lab 01 you write the analytic gradient by hand (possible because logistic regression is one layer); in a deep network, autograd applies the chain rule mechanically through the recorded computation graph. Same quantity, automated derivation. (Building the autograd engine itself is the Senior AI Engineer track's business.)
  4. Step: let the optimizer update parameters from the gradients; zero/reset the gradients.

Wrapped around it: iterate batches (with a seeded shuffle) into epochs; after each epoch compute validation loss; early-stop on its plateau (§3). The canonical PyTorch and Keras skeletons — worth knowing cold, and printed in the Hitchhiker's Guide — are this list verbatim: optimizer.zero_grad(); loss = criterion(model(x), y); loss.backward(); optimizer.step() and model.compile(...); model.fit(train, validation_data=val, callbacks=[EarlyStopping(...)]).

Debugging the loop is a skill interviewers probe directly. Loss NaN: learning rate too high, a log/likelihood fed 0 (Lab 01 clamps probabilities before the log for exactly this reason), or exploding gradients (clip them). Loss flat from step one: gradients not flowing — wrong sign, forgotten zero_grad (gradients accumulate in PyTorch by design), a detached tensor, or a dead activation regime (§11). Training falls but validation rises: overfitting — you are watching the bias-variance picture of §2 live. The finite-difference gradient check — perturb one parameter by \(\pm\epsilon\), difference the losses, compare to the analytic gradient — is the ground-truth test for a hand-written backward pass; Lab 01's test suite runs it, and torch.autograd.gradcheck is the same idea industrialized.

7. The unsupervised family: k-means, hierarchical, GMM, PCA

k-means partitions \(n\) points into \(k\) clusters minimizing inertia, the within-cluster sum of squared distances \(\sum_i \lVert x_i - \mu_{c(i)} \rVert^2\). Lloyd's algorithm alternates two steps, each of which provably does not increase inertia — assign every point to its nearest centroid, update every centroid to its cluster's mean (the mean is the point minimizing summed squared distance; that's why it converges) — until nothing changes. It converges only to a local optimum, so initialization decides everything: k-means++ (Arthur & Vassilvitskii, 2007) picks the first centroid uniformly, then each next with probability proportional to \(D(x)^2\), the squared distance to the nearest chosen centroid — spreading the seeds, with a proven \(O(\log k)\) expected-approximation bound. Production practice adds restarts (n_init in sklearn) and handles the classic silent failure — an empty cluster — by relocating its centroid to a far point. Lab 02 builds all of it, including the empty-cluster repair, plus the two evaluation tools that work without labels: inertia's elbow across k, and the silhouette score \(s = (b-a)/\max(a,b)\) (cohesion \(a\) vs separation \(b\), per point, in \([-1, 1]\)). Know k-means' assumptions: roughly spherical, similar-sized clusters, a k you must choose, and feature scaling required (it's distance-based).

Hierarchical (agglomerative) clustering starts with every point as its own cluster and repeatedly merges the closest pair, producing a dendrogram you cut at any height — no upfront k, and the merge history is itself informative. The linkage — single (min pairwise distance; chains), complete (max; compact), average, or Ward (minimum inertia increase; the k-means-like default) — determines cluster shape. Cost is \(O(n^2)\) memory for the distance matrix, which is what rules it out at scale.

Gaussian mixture models are k-means with honesty about uncertainty: model the data as a weighted sum of \(K\) Gaussians and fit by Expectation-Maximization — E-step: compute each point's soft responsibility under each component (a probability, not a hard label); M-step: re-fit each component's mean, covariance, and weight from responsibility-weighted points. k-means is exactly the degenerate GMM with identical spherical covariances and hard assignments. GMMs give ellipsoidal clusters, per-point membership probabilities, and a density you can threshold for anomaly detection; they cost more and can degenerate (a component collapsing onto one point).

PCA finds the orthogonal directions of maximum variance: the eigenvectors of the covariance matrix \(C = \frac{1}{n-1} X_c^T X_c\) (data centered — uncentered "PCA" finds the mean, not the variance), with eigenvalues equal to the variance each direction explains. Uses: compression (keep the components covering 95% of variance), decorrelation, visualization (top 2), and noise reduction (small components are often noise). Lab 02 computes them with power iteration — repeatedly multiply a random vector by \(C\) and normalize; every multiply amplifies the dominant eigendirection most, so the iterate converges to it, at a rate governed by the eigenvalue ratio \(\lambda_2/\lambda_1\) — then deflates (\(C' = C - \lambda_1 v_1 v_1^T\)) and repeats for the next component. Real libraries use SVD of \(X_c\) directly (numerically superior, same subspace), but power-iteration-family methods (randomized SVD, Lanczos) are exactly what scalable PCA implementations run — and PageRank is power iteration on the web graph. Reconstruction error — project down, map back, measure MSE — equals the discarded eigenvalues' variance and doubles as an anomaly score.

8. Reinforcement learning: MDPs, Q-learning, bandits, policy gradients

The formalism. An RL problem is a Markov Decision Process: states \(S\), actions \(A\), transition dynamics \(P(s' \mid s, a)\), reward function \(R\), and discount factor \(\gamma \in [0, 1)\). The Markov property — the future depends only on the current state, not the path to it — is what makes the math tractable. The agent seeks a policy \(\pi(a \mid s)\) maximizing expected return \(\sum_t \gamma^t r_t\); \(\gamma\) encodes time preference (and keeps infinite-horizon sums finite): \(\gamma{=}0\) is myopic, \(\gamma \to 1\) far-sighted. Lab 03's test shows this concretely — the same GridWorld start state is worth more to a \(\gamma{=}0.95\) agent than a \(\gamma{=}0.5\) one, because the distant goal reward survives discounting.

Value functions and Bellman. \(V^\pi(s)\) is the expected return from \(s\) under \(\pi\); \(Q^\pi(s,a)\) the same after committing to action \(a\) first. The optimal Q-function satisfies the Bellman optimality equation:

\[ Q^(s,a) = \mathbb{E}\big[ r + \gamma \max_{a'} Q^(s', a') \big] \]

— the value of the best behavior decomposes into the immediate reward plus the discounted value of behaving optimally afterwards. If you know the MDP's dynamics you can solve this directly (value iteration — model-based, dynamic programming). RL's defining constraint is that you usually don't: you learn from sampled transitions.

Q-learning (model-free, value-based) turns the Bellman equation into a stochastic update applied to each experienced transition \((s, a, r, s')\):

\[ Q(s,a) \leftarrow Q(s,a) + \alpha \big[ r + \gamma \max_{a'} Q(s',a') - Q(s,a) \big] \]

The bracketed quantity is the temporal-difference (TD) error. The crucial detail is the \(\max\): the target uses the best next action, not the action the exploring policy will actually take — Q-learning is off-policy, learning the optimal greedy policy's values while behaving ε-greedily (or even fully randomly: Lab 03's test trains with \(\epsilon{=}1\) and still extracts the optimal path, residual ≈ 0). SARSA replaces the \(\max\) with the actually-taken next action \(Q(s', a')\) — on-policy, learning the value of the policy it runs, exploration cost included; near a cliff of penalties SARSA learns the safe path, Q-learning the risky-optimal one. The off-policy property is not academic: it is why DQN's replay buffer (training on transitions collected by stale policies) is legal.

Exploration-exploitation is the dilemma with no supervised analogue: act on current estimates (exploit) or gather information (explore)? ε-greedy is the workhorse: explore uniformly with probability ε, else exploit; anneal ε over training. The multi-armed bandit isolates the dilemma completely (one state, K arms, unknown reward distributions): ε-greedy solves it but pays a fixed exploration tax forever; UCB1 (Auer et al., 2002) instead pulls the arm maximizing

\[ \bar x_a + c \sqrt{\frac{\ln t}{n_a}} \]

— mean estimate plus a confidence bonus that is large for undersampled arms and shrinks as evidence accumulates. "Optimism in the face of uncertainty," deterministic, with provably logarithmic regret; the same idea drives Monte-Carlo Tree Search (AlphaGo's UCT). Bandits ship constantly in production — adaptive A/B tests, ad and recommendation allocation — often as the correct replacement for a static split. Lab 03 builds both agents and shows UCB wasting fewer pulls.

The deep and policy-based ladder (survey level — deep-RL internals are beyond this phase): DQN (Mnih et al., 2015) replaces the Q-table with a neural network, made stable by the replay buffer and a slowly-updated target network; it played Atari from pixels. Policy-gradient methods skip values-then-argmax and optimize the policy directly: REINFORCE ascends \(\nabla_\theta \, \mathbb{E}[\text{return}] = \mathbb{E}[\nabla_\theta \log \pi_\theta(a \mid s) \cdot \text{return}]\) — increase the log-probability of actions in proportion to how well things went. Actor-critic variants subtract a learned baseline (the critic) to cut variance; PPO (Schulman et al., 2017) adds a clipped objective that forbids destructively large policy updates, making it the robust default — and the algorithm classically used in RLHF to fine-tune LLMs against a learned reward model. That last sentence is why an agentic-track engineer should care about this section at all: the "RL" in RLHF is exactly this machinery, one abstraction level up.

9. TensorFlow vs PyTorch: the deep comparison

The decision table is in the Phase README; this is the mechanism underneath each row.

Execution model. PyTorch is define-by-run: every tensor op executes immediately, and the computation graph autograd will differentiate is recorded as your Python runs — if, for, recursion all just work, and debugging is print() and pdb on live tensors. TensorFlow 1.x was the opposite — define-then-run: build a static graph, then feed it through a session — which enabled whole-graph optimization and easy serialization but made debugging notoriously indirect. TF2 made eager execution the default and moved the graph behind tf.function: decorate a Python function and TF traces it — runs it once with symbolic tensors, records the ops into a graph, then compiles and reuses that graph. The tracing model has real sharp edges you must know: Python side effects (prints, appends) happen only at trace time; each new input signature (shape/dtype) triggers a retrace (a classic performance bug); and data-dependent Python branching must become graph ops (tf.cond) to be captured. PyTorch's answer from the other direction is torch.compile (TorchDynamo): keep eager semantics, JIT-capture and fuse graphs where possible, fall back to Python where not. The two frameworks converged on the same endpoint — eager for development, captured graphs for speed — from opposite starting points.

Autograd surface. PyTorch: tensors with requires_grad=True are tracked everywhere, loss.backward() walks the recorded tape, gradients land in param.grad, and you must optimizer.zero_grad() because gradients accumulate by design (a feature for gradient accumulation across micro-batches, a bug when forgotten). TensorFlow: recording is scoped — ops are only taped inside a with tf.GradientTape() as tape: block, and you pull gradients explicitly with tape.gradient(loss, model.trainable_variables) then optimizer.apply_gradients(...). Same chain-rule machinery; opt-out vs opt-in recording.

Model API. PyTorch's nn.Module: subclass, define layers in __init__, computation in forward, and write the training loop yourself — maximal control, zero magic, the reason research lives here. TensorFlow's Keras: Sequential/functional API plus compile(optimizer, loss, metrics) and fit(train, validation_data, callbacks) — the loop, metrics, checkpointing, and early stopping are built in and production-hardened. Keras 3 now runs on TF, PyTorch, or JAX backends, blurring the line further. The honest summary: PyTorch gives you the loop; Keras gives you the trainer; both let you drop to the other level when needed (Keras train_step override; PyTorch Lightning adds the trainer layer).

Data pipelines. tf.data: a declarative op graph — Dataset.map(...).shuffle(buffer, seed).batch(...).prefetch(tf.data.AUTOTUNE) — that the runtime parallelizes and overlaps with training. PyTorch: Dataset (indexable) + DataLoader(batch_size, shuffle, num_workers, collate_fn) — imperative Python fed by worker processes. Same goals (never starve the accelerator; shuffle reproducibly — both take seeds); declarative-graph vs multiprocess-Python mechanics.

Distributed. PyTorch: DistributedDataParallel (one process per GPU, gradient all-reduce — the standard), FSDP for sharding models too big for one device, launched via torchrun. TensorFlow: tf.distribute.Strategy objects (MirroredStrategy single-node, MultiWorkerMirroredStrategy, TPUStrategy) that wrap existing Keras code with minimal changes — and first-class TPU support, historically TF's exclusive edge, now shared with JAX.

Deployment. TensorFlow's story is the most mature: TF Serving (a C++ gRPC/REST server with versioned model hot-swap, built for SavedModel), TFLite → LiteRT for mobile/edge (quantization toolchain included), TF.js for browsers, and TFX for full pipelines. PyTorch: TorchServe, torch.export/ONNX export into any ONNX-compatible runtime, and ExecuTorch for mobile/edge — a stack that closed most of the gap, while LLM serving largely bypassed both with specialized engines (vLLM et al., PyTorch-ecosystem). JAX, the honorable mention: NumPy semantics plus composable function transforms — grad, jit (XLA compilation), vmap (auto-batching), pmap/sharding (distribution) — over pure functions with explicit PRNG keys; the functional purity is why large-scale distributed research (and Google's frontier models) favor it.

How to answer "which one?": concepts transfer (tensor, autograd, module, loader, loop); choose by ecosystem and serving path — PyTorch for research velocity and the open-model/HF ecosystem, TF/Keras for hardened serving-to-mobile pipelines and TFX shops, JAX for functional-style research at scale. Naming that tradeoff is the senior answer; naming a favorite is not.

10. Deep learning survey: MLP, CNN, RNN, Transformer

This section is deliberately a survey: what each architecture is, and the inductive bias that justifies it — which is what the AI/ML Engineer interview actually probes. The internals (attention math, backprop through these structures, building them from scratch) are the sibling Senior AI Engineer track's territory; this phase does not duplicate it.

MLP (multi-layer perceptron) — stacked fully-connected layers with nonlinear activations: \(h = \phi(Wx + b)\), repeated. The universal approximation theorem says one wide hidden layer can approximate any continuous function; depth makes representation efficient (features composing into higher-level features), not merely possible. No structural assumption about the input — every unit sees every input — which is its flexibility and its sample-inefficiency. Baseline for tabular data (where GBMs still usually win, §4).

CNN (convolutional network) — layers of small learned filters slid across the input, plus pooling/striding for downsampling. Two inductive biases: locality (nearby pixels relate) and translation equivariance (a feature detector works anywhere in the image, via weight sharing), which slashes parameters versus an MLP on raw pixels and matches image structure. Stacked convolutions grow the receptive field: edges → textures → parts → objects. The default for vision for a decade (LeNet → AlexNet → ResNet, whose skip connections made 100+-layer training possible by giving gradients a highway), and still the efficiency choice even where vision transformers now compete.

RNN (recurrent network) — process a sequence step by step, carrying a hidden state: \(h_t = \phi(W_h h_{t-1} + W_x x_t)\). The inductive bias is sequential/temporal structure with shared dynamics across time. Trained by backpropagation through time, plain RNNs suffer vanishing/exploding gradients across long sequences (repeated multiplication by the same Jacobian); LSTM/GRU fix this with gates — learned sigmoid valves controlling what enters, persists in, and leaves an additive cell state, giving gradients an unbroken path. Their fundamental limit is sequential computation: step t needs step t−1, so training cannot parallelize across the sequence.

Transformer (Vaswani et al., 2017) — replaces recurrence with self-attention: every position computes weighted combinations of every other position's representation, with weights from learned query/key similarity; positional encodings restore order information that attention alone ignores. Two consequences made it the dominant architecture: all positions compute in parallel (training scales on modern hardware in a way RNNs never could), and any two positions interact in one hop (no long-path gradient decay). Cost: attention is \(O(n^2)\) in sequence length. Every foundation model this track orchestrates is this architecture; when you need its internals — multi-head attention, KV caches, the full training stack — that is, again, the Senior AI Engineer track.

Choosing: tabular → GBM first, MLP maybe; images → CNN (or fine-tune a pretrained vision model); sequences/text → transformer (fine-tune, don't pretrain); tiny data → classic ML, not deep anything. "Fine-tune a pretrained model" beats "architect from scratch" in almost every applied setting — knowing that is also a foundations competency.

11. Deep-learning plumbing: activations, initialization, normalization, backprop

Activations supply the nonlinearity without which any depth of linear layers collapses to one linear map. Sigmoid \(\sigma(z) = 1/(1+e^{-z})\): output layer for binary probability; avoided in hidden layers because its gradient \(\sigma(1-\sigma) \le 0.25\) shrinks signals layer after layer (vanishing gradients). tanh: zero-centered sigmoid, same saturation issue. ReLU \(\max(0, z)\): gradient exactly 1 when active — no saturation on the positive side, cheap, the modern default; its failure mode is dying ReLU (a unit pushed permanently negative never recovers, since its gradient is 0), mitigated by Leaky ReLU or GELU (the smooth default in transformers). Softmax turns a logit vector into a distribution; compute it (and cross-entropy) with the max-subtraction/log-sum-exp trick or large logits overflow — the same stability discipline as Lab 01's branch-stable sigmoid.

Initialization exists because both symmetry and scale can kill training before it starts. All-equal weights make every unit in a layer compute identical gradients forever (symmetry never breaks) — hence random init; Lab 01's seeded uniform(-0.01, 0.01) is the miniature. Scale: too-large weights saturate activations, too-small ones shrink signals to nothing across depth. Xavier/Glorot init sizes variance as \(\sim 1/n\) (fan-in/fan-out average) to keep activation variance constant layer to layer for sigmoid/tanh; He/Kaiming init doubles it (\(2/n_{in}\)) to compensate for ReLU zeroing half its inputs. Frameworks default to these; knowing why is the interview point.

Normalization. Batch norm (Ioffe & Szegedy, 2015) standardizes each feature over the current mini-batch, then re-scales/shifts with learned \(\gamma, \beta\); it smooths the optimization landscape, permits larger learning rates, and regularizes slightly — but it couples examples in a batch, behaves differently at inference (running statistics), and degrades with tiny batches. Layer norm (Ba et al., 2016) normalizes across features within each example — batch-size independent, identical at train and inference, and therefore the norm of transformers and RNNs. Rule of thumb: batch norm for CNNs with healthy batch sizes; layer norm for sequence models.

Backprop intuition, without the full derivation: the chain rule, organized. The forward pass caches every intermediate value; the backward pass walks the computation graph in reverse, multiplying local derivatives, so the loss gradient w.r.t. every parameter arrives in one backward sweep costing about as much as a forward pass. Lab 01's hand-derived \(\partial L / \partial w = (p - y) \, x\) is backprop through one layer — sigmoid and BCE's local derivatives chained and collapsed. The vanishing/exploding phenomena of §10–11 are just this multiplication chain compounding factors below or above 1 — which is precisely what skip connections, gates, careful init, and normalization exist to counter. The full mechanized story — building the autograd tape — is, one more time, the Senior AI Engineer track.

12. Feature engineering: encoding, scaling, imputation, selection

Models consume numbers; features are the numbers you choose to show them, and on tabular problems feature quality beats model choice with monotonous regularity. (This section is the modeling-side view; building these transformations at scale — feature stores, pipelines, training/serving skew — is Phase 27.)

Encoding categoricals. One-hot for low-cardinality: one binary column per category — faithful but explodes with cardinality. Ordinal/label encoding only when a true order exists (S < M < L); imposing fake order on nominal categories misleads linear models and kNN (trees survive it since they only split). High cardinality (user IDs, zip codes): target encoding (replace category with the target's mean for that category — powerful and dangerously leaky unless computed out-of-fold, §13), hashing, or learned embeddings (the deep-learning answer; also how LLMs see tokens). Frequency encoding is a cheap, often-effective fallback.

Scaling. Distance- and gradient-based models (kNN, SVM, k-means, PCA, linear models, all neural nets) need comparable feature scales: standardization \((x - \mu)/\sigma\) is the default; min-max to \([0,1]\) when bounded ranges matter; robust scaling (median/IQR) under outliers. Unscaled features hand PCA and k-means to whichever column has the biggest units — a classic silent failure. Trees/GBMs are scale-invariant. The cardinal rule: fit the scaler on training data only, apply to val/test (§13).

Imputation. Missing values: mean/median (simple, distribution-distorting), constant-plus-missingness-indicator column (often best: "was missing" is frequently signal — a blank income field correlates with outcomes), model-based/kNN imputation (better, costlier). GBM libraries handle missing values natively (learning which branch missing goes to), one more reason they dominate messy tabular data. Never impute using statistics computed over the full dataset — that's leakage.

Feature selection. Three families: filter (rank by mutual information/correlation with the target — fast, ignores interactions), wrapper (add/remove features, re-fit, keep what helps — thorough, expensive, easy to overfit the validation set), embedded (the model selects while training: L1 zeroing coefficients, tree/GBM feature importances). Modern practice leans on embedded importances plus permutation importance (shuffle one column, measure the metric drop — model-agnostic and honest) and SHAP for attribution. Fewer, better features: faster training, less overfitting, cheaper serving, easier debugging.

13. Data leakage: the silent killer

Leakage is any information available at training time that will not be available at prediction time. It is the most expensive failure mode in applied ML because it inflates every offline metric and fails silently — the model looks brilliant until it meets production. The forms to know on sight:

  • Target leakage: a feature that is a consequence of the label — account_closed_date in a churn model, amount_refunded in a fraud model. Offline AUC 0.99; production, nothing.
  • Preprocessing leakage: fitting any statistic — scaler mean, imputation median, target encoding, feature selection, even PCA — on data that includes the validation/test rows. The test set has now influenced training. The fix is mechanical: fit transforms inside the training fold only — sklearn's Pipeline inside cross-validation exists precisely to make this automatic.
  • Temporal leakage: random splits on time-dependent data let the model train on the future and predict the past. Anything with a timestamp gets a time-based split (train on the past, validate on the future), full stop.
  • Duplicate/group leakage: near-duplicate rows (or rows from the same user/patient/session) straddling the split — memorization scores as generalization. Split by group (GroupKFold), not by row.
  • Tuning leakage: hyperparameters chosen against the test set (§15) — subtler, universal, and the reason the test set is opened once.

The discipline: split first (respecting time and groups), fit everything inside the training fold, and treat any too-good-to-be-true result as leakage until proven otherwise — the suspiciously perfect feature is guilty until audited. Lab 01's train_val_split test asserts the no-row-overlap invariant; trivial at lab scale, worth a code review comment at every real one.

14. Evaluation: splits, cross-validation, metrics, imbalance

Splits. Three sets, three jobs: train fits parameters; validation drives decisions (early stopping, hyperparameters, model choice); test is opened once, at the end, to report generalization. The validation set stops being an unbiased estimate the moment you optimize against it — that's why the test set exists and why touching it during development is self-deception (§15). Typical ratios 60/20/20 or 80/10/10; time-series data splits by time (§13).

Cross-validation. Small data makes one split noisy: k-fold CV (k=5 or 10) trains k times, each fold serving once as validation, and reports mean ± std — the std is information (a huge spread means your estimate is luck). Stratified k-fold preserves class ratios per fold (essential under imbalance); GroupKFold prevents group leakage; TimeSeriesSplit does expanding-window validation for temporal data. Everything learned from data — scaling, encoding, selection — must happen inside each fold (§13).

Classification metrics all derive from the confusion matrix — TP, FP, TN, FN (Lab 01 builds them from these counts):

  • Accuracy \((TP{+}TN)/N\): fine when classes are balanced and errors cost the same; worthless under imbalance — 99% accuracy on 1% fraud is the all-negative predictor.
  • Precision \(TP/(TP{+}FP)\): of what I flagged, how much was real — the metric when false alarms are expensive (spam filter eating real mail).
  • Recall \(TP/(TP{+}FN)\): of what was real, how much I caught — the metric when misses are expensive (cancer screening, fraud).
  • F1 \(= 2PR/(P{+}R)\): their harmonic mean — punishes imbalance between the two; use when you need one number and both error types matter.
  • ROC-AUC: TPR vs FPR across all thresholds — threshold-independent ranking quality ("probability a random positive scores above a random negative"). Deceptively rosy under heavy imbalance because FPR's huge negative denominator hides thousands of false positives.
  • PR-AUC: precision vs recall across thresholds — the honest curve for rare positives; prefer it whenever the positive class is scarce.

The threshold itself is a business decision, not 0.5 by divine right: sweep it, price the two error types, pick deliberately. Under class imbalance, beyond metric choice: class weights in the loss (misclassifying a rare positive costs more), resampling (oversample minority — SMOTE interpolates synthetic positives — or undersample majority, training set only), and always compare against the majority-class baseline first.

Regression metrics: MSE/RMSE (squared error — outlier-sensitive, RMSE in target units), MAE (robust, linear penalty), R² \(= 1 - SS_{res}/SS_{tot}\) (fraction of variance explained: 1 perfect, 0 no better than predicting the mean, negative worse than the mean — possible and meaningful on a test set). MAPE for percentage errors (beware zeros).

15. Hyperparameter tuning: grid, random, Bayesian, Hyperband

Parameters are learned by training; hyperparameters — learning rate, regularization strength, depth, batch size, ε, γ — are chosen outside the loop, and their choice routinely moves metrics more than model choice does.

Grid search evaluates a Cartesian product of values. It is exhaustive, embarrassingly parallel, and exponentially wasteful: with d hyperparameters and k values each, \(k^d\) runs — and if one dimension barely matters, entire slices of the grid are redundant. Random search (Bergstra & Bengio, 2012) samples configurations at random from the same space and wins for a crisp reason: with a budget of N runs, random search tests N distinct values of every hyperparameter, while a grid tests only \(k = N^{1/d}\) per dimension. Since a few hyperparameters usually dominate, coverage of each dimension beats coverage of the grid. It's the sane default.

Search spaces matter as much as the algorithm. Scale-type hyperparameters — learning rate, regularization λ — are searched on a log scale (10^uniform(-5, -1)): the difference between 0.001 and 0.01 is a regime change; between 0.41 and 0.42, nothing. Integers (depth, layers) uniformly in a plausible range; conditional parameters (dropout rate only if dropout is on) need tree-structured spaces — one reason dedicated tools (Optuna, Ray Tune, KerasTuner) exist.

Bayesian optimization treats tuning as itself a learning problem: fit a cheap surrogate model (Gaussian process or TPE) mapping configuration → observed score, then choose the next configuration by maximizing an acquisition function (e.g., expected improvement) that balances exploring uncertain regions against exploiting promising ones — the bandit dilemma of §8 in tuning clothes. Sample-efficient when each training run is expensive; sequential by nature (each result informs the next pick). Hyperband (Li et al., 2018) attacks the budget instead: successive halving — start many configurations on tiny budgets (few epochs), keep the top fraction, multiply their budget, repeat — so bad configs die cheap and only survivors get full training; it exploits the fact that learning curves reveal losers early. Optuna's default combination (TPE sampling + successive-halving/Hyperband pruning) is the current practical sweet spot.

The cardinal sin: tuning on the test set. Every configuration you evaluate against a dataset leaks information about it into your choices; tune against the test set and the "final" number is an optimistically-biased fiction — you have overfit the selection process to it. The protocol: tune on validation (or CV folds), then evaluate the single chosen configuration once on the untouched test set, and report that. If you must iterate after seeing test results, you need a new test set. This is §13's tuning leakage, and it is everywhere — including leaderboard-chasing public benchmarks.

16. Experiment reproducibility

An unreproducible result is a rumor. The discipline, in increasing order of rigor:

  • Seed everything, explicitly. Every stochastic component — init, shuffles, dropout, augmentation, ε-greedy exploration — draws from a seeded generator. The labs' pattern (a random.Random(seed) passed in, never the global random) is the real-world pattern too: frameworks provide torch.manual_seed / tf.random.set_seed / NumPy Generators, and DataLoader workers and CUDA need their own seeding care. Same seed → same weights, bit for bit; Lab 01–03 all assert exactly this.
  • Know determinism's limits on GPUs. Some CUDA kernels (atomics, cuDNN autotuning) are non-deterministic by default for speed; full determinism needs opt-in flags (torch.use_deterministic_algorithms(True), disabling cuDNN benchmark mode) and costs performance. Floating-point addition isn't associative, so even "the same math" in a different reduction order drifts in the last bits — which is why tests compare with tolerances (pytest.approx in every lab), never == on floats.
  • Version the whole experiment: code (a commit hash), data (a snapshot/hash — data changing under you is the most common "irreproducible" cause), environment (pinned dependencies, container image), configuration (every hyperparameter, in a file, not a shell history), and results (metrics + the seed that produced them). Run-to-artifact lineage — which code + data + config produced which model — is exactly what experiment trackers and model registries exist for; that machinery is Phase 26.
  • Report distributions, not lucky runs. Deep training is seed-sensitive; a serious comparison reports mean ± std over several seeds. A method that only wins on one seed doesn't win.

17. Common misconceptions

  • "More data always beats a better model." More data cures variance, not bias — a model too simple for the signal stays wrong at any scale (§2). Diagnose first, then choose the lever.
  • "Accuracy tells you how good the classifier is." Under imbalance it mostly tells you the class ratio (§14). The all-negative predictor scores 99% on 1% fraud.
  • "Deep learning has made classic ML obsolete." GBMs remain the tabular-data default in practice and in competition results; kNN runs your vector store; logistic regression runs more production scoring than transformers do. The taxonomy question is "what structure does the data have," not "what year is it."
  • "The test set is for checking progress during development." One look = one use. Iterate against it and it is a validation set with better branding, and your reported number is biased (§15).
  • "Q-learning and SARSA are basically the same." One symbol — max vs taken-action — flips off-policy to on-policy, changes what the values mean, and decides whether replay buffers are legal (§8).
  • "k-means finds the clusters." It finds a local optimum of inertia for your chosen k, conditioned on init — which is why k-means++ and restarts exist, and why validation (silhouette, elbow, domain sense) is your job (§7).
  • "PCA selects the important features." PCA finds high-variance directions — linear combinations of all features — and variance is not importance for your target (that's supervised feature selection, §12). It also silently requires centering and scaling.
  • "Adam means I don't need to tune the learning rate." Adam adapts per-parameter relative step sizes; the global η still sets the scale, still matters, and is still searched on a log axis (§5, §15).
  • "TensorFlow is graphs, PyTorch is eager." Ten years stale: TF2 is eager-first with tf.function graph capture; PyTorch has torch.compile. Both converged on eager-development/compiled-production (§9).
  • "A fixed random seed makes the experiment reproducible." Necessary, nowhere near sufficient — data versions, dependency versions, and GPU nondeterminism all break replays (§16).

18. Lab walkthrough

Build the three miniatures in order; every numeric routine is pure Python stdlib, seeded and deterministic, per the Lab Standard.

  1. Lab 01 — Supervised Training Loop. Logistic regression by mini-batch gradient descent: numerically stable sigmoid, bce_loss with L2, the analytic (p − y) gradient proven against a finite-difference check, a seeded shuffle-per-epoch loop, early stopping with best-weight restore, and confusion-matrix metrics. 27 tests.
  2. Lab 02 — Unsupervised: k-means + PCA. k-means++ initialization (seeded weighted draw), Lloyd's assign/update loop with monotone inertia and empty-cluster repair, silhouette scoring; PCA as center → covariance → power iteration + deflation, with projection and reconstruction error. 28 tests.
  3. Lab 03 — Reinforcement Learning. A GridWorld MDP, tabular Q-learning with ε-greedy exploration, convergence verified against the Bellman optimality equation and the known optimal path (including the off-policy proof: optimal values from ε=1 random behavior), plus ε-greedy and UCB1 bandits. 28 tests.

Run each with LAB_MODULE=solution python3 -m pytest test_lab.py -q first (green reference), then fill your lab.py until the default run matches, then read solution.py's main() output.

19. Success criteria

  • You can place any named business problem into the supervised / unsupervised / self-supervised / RL taxonomy and say what signal supervises it.
  • You can sketch the bias-variance decomposition and diagnose over- vs underfitting from a pair of loss curves.
  • You can name the loss function for linear regression, logistic regression, trees, SVMs — and derive the p − y gradient for logistic regression.
  • You can write the SGD, momentum, and Adam update rules and say what each term buys.
  • You can explain k-means++'s weighting, why Lloyd's loop converges, and how power iteration finds an eigenvector.
  • You can write the Q-learning update, state what makes it off-policy vs SARSA, and compare ε-greedy with UCB.
  • You can fill the TensorFlow-vs-PyTorch table from memory — including tf.function tracing semantics and zero_grad's reason for existing.
  • You can list four kinds of data leakage and the mechanical fix for each.
  • You can defend PR-AUC over ROC-AUC for rare positives, and log-scale search for learning rates.
  • All three labs green under both lab and solution (83 tests total).

20. Interview Q&A

Q: Explain the difference between supervised, unsupervised, and reinforcement learning — with a production example of each. A: Supervised learns a mapping from labeled pairs — fraud classification from confirmed chargebacks. Unsupervised finds structure without labels — k-means customer segmentation, PCA compression. RL learns a policy from delayed rewards through interaction — a bandit allocating traffic between retention offers. The distinguishing question is what signal supervises learning: labels, the data's own structure, or reward. I'd add self-supervised — supervision manufactured from the data itself, like next-token prediction — because it's how every foundation model is pre-trained.

Q: Your model has 96% training accuracy and 71% validation accuracy. What's happening and what do you do? A: Classic overfitting — high variance. In rough order: get more data if possible; strengthen regularization (raise L2/weight decay, add dropout for a net); simplify the model; early-stop on validation loss; for tabular models, prune features. Then re-diagnose — if both accuracies had been low, that's underfitting and the levers point the other way (more capacity, more features, less regularization). The train/val gap is the diagnostic, not either number alone.

Q: Why does logistic regression use cross-entropy instead of squared error? A: BCE is the negative log-likelihood of the Bernoulli model, so it's the statistically principled choice — but the practical reason is gradients. With sigmoid + BCE, the gradient w.r.t. the linear score collapses to p − y: large exactly when the model is badly wrong. With sigmoid + MSE, the gradient carries a σ′(z) factor that vanishes in the sigmoid's saturated tails — so a confidently wrong model learns slowest precisely when it should learn fastest. BCE also keeps the objective convex for logistic regression.

Q: Walk me through what happens in one iteration of a PyTorch training loop. A: optimizer.zero_grad() clears accumulated gradients (PyTorch accumulates by design, to support gradient accumulation). Forward: pred = model(x) runs the computation and records the dynamic graph. loss = criterion(pred, y) produces the scalar. loss.backward() walks the recorded graph in reverse applying the chain rule, depositing gradients in each parameter's .grad. optimizer.step() updates parameters from those gradients per the optimizer's rule — plain SGD subtracts lr * grad; Adam scales by its running moment estimates. Periodically: evaluate on validation in no_grad + eval mode, and early-stop on its plateau.

Q: SGD vs momentum vs Adam — when and why? A: SGD takes raw noisy gradient steps. Momentum accumulates an EMA of gradients — persistent directions add up, oscillating ones cancel — faster through ravines. Adam adds per-parameter scaling by the EMA of squared gradients with bias correction, so step sizes adapt to each parameter's gradient history; it's the robust default that mostly works out of the box. SGD+momentum with a tuned schedule can generalize marginally better on some vision tasks; AdamW (decoupled weight decay) is the transformer standard. Whatever the optimizer, learning rate remains the hyperparameter that matters most.

Q: How does k-means work, and what are its failure modes? A: Alternate assigning points to nearest centroids and moving centroids to cluster means — both steps monotonically lower inertia, so it converges, but only to a local optimum. Failures: bad init (fixed by k-means++'s D²-weighted seeding plus restarts), wrong k (elbow/silhouette to choose), non-spherical or differently-sized clusters (its implicit assumption — use GMM or density methods), unscaled features (distance-based, so the biggest-unit feature dominates), and empty clusters during iteration (relocate the centroid — my lab relocates to the farthest point, sklearn does the inertia-weighted equivalent).

Q: Explain Q-learning to someone who knows supervised learning. A: You want Q(state, action) = long-run value of an action. There's no labeled target, so you build one from the Bellman equation: after experiencing (s, a, r, s′), the target is r + γ·max_a′ Q(s′, a′) — observed reward plus the discounted value of the best next action — and you nudge Q(s,a) toward it by the TD error times a learning rate. It's like regression where the labels are bootstrapped from your own improving estimates. The max makes it off-policy: it learns the optimal policy's values even while behaving exploratorily — I've verified that in a lab by training with 100% random actions and still extracting the optimal path. SARSA swaps the max for the action actually taken: on-policy, values include exploration cost.

Q: What's the exploration-exploitation tradeoff and how do you handle it? A: Acting on current estimates versus gathering information — exploit too early and you lock onto a suboptimal choice; explore too long and you burn reward on known-bad options. ε-greedy explores uniformly with probability ε (simple, but pays a fixed tax forever unless annealed). UCB adds a confidence bonus c·sqrt(ln t / n_a) to each estimate and picks the argmax — optimism under uncertainty, deterministic, logarithmic regret. Thompson sampling samples from posteriors. In production this is adaptive A/B testing: a bandit shifts traffic toward winners while still probing, instead of a fixed 50/50 split running to significance.

Q: TensorFlow or PyTorch — how do you choose? A: The concepts transfer — tensors, autograd, modules, data pipelines, the same training loop — so it's an ecosystem decision. PyTorch: define-by-run debugging, research and open-model/Hugging Face gravity, torch.compile for speed, TorchServe/ONNX/ExecuTorch for serving. TensorFlow: Keras's built-in trainer, tf.function graph capture, and the most hardened deploy path — TF Serving, LiteRT for mobile, TFX pipelines. I'd pick PyTorch for research/LLM-adjacent work, TF for a mobile-heavy or TFX shop, and note both converged on eager-development-plus-compiled-production. JAX deserves a mention for functional-style, large-scale research. Then I'd say the honest thing: the team's existing stack usually decides.

Q: What is data leakage? Give three forms and the fix. A: Information available in training that won't exist at prediction time — inflates offline metrics, fails silently in production. Target leakage: a feature caused by the label (refund amount in fraud) — audit suspiciously-predictive features. Preprocessing leakage: fitting scalers/encoders/selection on data including validation rows — fit transforms inside the training fold only, e.g. sklearn Pipelines inside CV. Temporal leakage: random splits on time-ordered data — split by time. Fourth for free: tuning against the test set, which is why the test set is opened exactly once.

Q: Your fraud dataset is 0.5% positive. How do you evaluate a classifier on it? A: Not accuracy — all-negative scores 99.5%. Report the confusion matrix and precision/recall/F1; prefer PR-AUC over ROC-AUC because FPR's enormous negative denominator makes ROC look rosy while precision collapses. Use stratified splits/CV, class weights or resampling (training set only), and pick the operating threshold from business costs — the cost of a missed fraud versus a false alarm — not from 0.5 by default.

Q: Why is random search usually better than grid search? A: With a budget of N runs across d hyperparameters, grid search tests only N^(1/d) distinct values per dimension; random search tests N distinct values of every dimension. Since performance is usually dominated by one or two hyperparameters, per-dimension coverage wins — Bergstra & Bengio's result. Also: search scale-type parameters (learning rate, λ) on a log scale, use Bayesian optimization when each run is expensive, and Hyperband/successive-halving to kill bad configurations on partial budgets. And never tune against the test set — that's selection overfitting with a good conscience.

21. References

  • Ian Goodfellow, Yoshua Bengio, Aaron Courville — Deep Learning (MIT Press, 2016). https://www.deeplearningbook.org/
  • Richard S. Sutton, Andrew G. Barto — Reinforcement Learning: An Introduction, 2nd ed. (MIT Press, 2018). http://incompleteideas.net/book/the-book.html
  • Trevor Hastie, Robert Tibshirani, Jerome Friedman — The Elements of Statistical Learning, 2nd ed. (Springer). https://hastie.su.domains/ElemStatLearn/
  • Diederik P. Kingma, Jimmy Ba — "Adam: A Method for Stochastic Optimization" (2014). https://arxiv.org/abs/1412.6980
  • Nitish Srivastava et al. — "Dropout: A Simple Way to Prevent Neural Networks from Overfitting" (JMLR, 2014). https://jmlr.org/papers/v15/srivastava14a.html
  • Sergey Ioffe, Christian Szegedy — "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift" (2015). https://arxiv.org/abs/1502.03167
  • Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton — "Layer Normalization" (2016). https://arxiv.org/abs/1607.06450
  • Xavier Glorot, Yoshua Bengio — "Understanding the difficulty of training deep feedforward neural networks" (AISTATS 2010). https://proceedings.mlr.press/v9/glorot10a.html
  • Kaiming He et al. — "Delving Deep into Rectifiers" (He initialization, 2015). https://arxiv.org/abs/1502.01852
  • David Arthur, Sergei Vassilvitskii — "k-means++: The Advantages of Careful Seeding" (SODA 2007). https://theory.stanford.edu/~sergei/papers/kMeansPP-soda.pdf
  • Peter Auer, Nicolò Cesa-Bianchi, Paul Fischer — "Finite-time Analysis of the Multiarmed Bandit Problem" (Machine Learning, 2002). https://link.springer.com/article/10.1023/A:1013689704352
  • Volodymyr Mnih et al. — "Human-level control through deep reinforcement learning" (DQN, Nature 2015). https://www.nature.com/articles/nature14236
  • John Schulman et al. — "Proximal Policy Optimization Algorithms" (2017). https://arxiv.org/abs/1707.06347
  • Ashish Vaswani et al. — "Attention Is All You Need" (2017). https://arxiv.org/abs/1706.03762
  • James Bergstra, Yoshua Bengio — "Random Search for Hyper-Parameter Optimization" (JMLR, 2012). https://www.jmlr.org/papers/v13/bergstra12a.html
  • Lisha Li et al. — "Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization" (JMLR, 2018). https://arxiv.org/abs/1603.06560
  • PyTorch documentation — Autograd mechanics; torch.optim; torch.utils.data; Reproducibility notes; torch.compile. https://pytorch.org/docs/stable/index.html
  • TorchServe. https://pytorch.org/serve/ · ExecuTorch. https://pytorch.org/executorch-overview
  • TensorFlow documentation — "Better performance with tf.function" (https://www.tensorflow.org/guide/function); tf.GradientTape (https://www.tensorflow.org/guide/autodiff); tf.data (https://www.tensorflow.org/guide/data); Keras training APIs (https://www.tensorflow.org/guide/keras).
  • TensorFlow Serving. https://www.tensorflow.org/tfx/guide/serving · LiteRT (formerly TensorFlow Lite). https://ai.google.dev/edge/litert
  • JAX documentation. https://docs.jax.dev/
  • scikit-learn User Guide — "Common pitfalls and recommended practices" (leakage) (https://scikit-learn.org/stable/common_pitfalls.html); "Cross-validation" (https://scikit-learn.org/stable/modules/cross_validation.html); "Clustering" (https://scikit-learn.org/stable/modules/clustering.html); "Model evaluation" (https://scikit-learn.org/stable/modules/model_evaluation.html).