« Phase 32 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Staff Notes

Phase 32 — Core Contributor Notes: ML/DL Foundations

Our labs are miniatures of three real systems: PyTorch's autograd, scikit-learn's estimators, and the Gym/Gymnasium RL environment API. This doc is the maintainer's-eye view of how those systems actually implement the mechanisms Lab 01–03 hand-roll, the source-level decisions that are not obvious from the docs, the API evolution and why it changed, and precisely what our stdlib version simplifies away.

PyTorch autograd: the tape is the graph

Lab 01 writes compute_gradients by hand because logistic regression is one layer. PyTorch's real answer is reverse-mode autodiff over a dynamically-built graph. Every Tensor created with requires_grad=True, and every tensor derived from one, carries a grad_fn attribute — a pointer to a Function node such as MulBackward0, AddBackward0, SigmoidBackward0. As the forward pass runs your ordinary Python, each op appends its backward node to a DAG (this is what "define-by-run" means: the graph is a side effect of execution, not a pre-declared structure). Calling loss.backward() does a reverse topological traversal from the scalar root, invoking each node's saved backward closure, multiplying local Jacobians, and accumulating the result into each leaf tensor's .grad. Non-obvious details a committer knows: nodes stash a saved_tensors list of the forward intermediates they need (the activation-memory cost of §Principal), and reusing a saved tensor after an in-place op that mutated it is exactly what triggers the infamous "a variable needed for gradient computation has been modified by an inplace operation" error.

Why zero_grad() exists is a genuine design decision, not an oversight: .grad is accumulated (+=) across backward calls so that gradient accumulation across micro-batches — several backward passes summed before one optimizer step — is expressible without special API. The cost is that forgetting to clear it silently sums gradients across iterations. Modern zero_grad(set_to_none=True) (now the default) sets .grad to None rather than a zeroed tensor, which frees memory and skips a kernel, at the cost that code reading .grad must handle None.

API evolution worth knowing. Pre-0.4 PyTorch had a separate Variable class that wrapped a Tensor to carry autograd metadata; the 0.4 release merged them, so a plain Tensor now holds requires_grad/grad/grad_fn directly and Variable is a deprecated alias. That merge is why old tutorials wrap tensors in Variable(...) and current code never does. nn.Module registration is the other piece of magic: assigning an nn.Parameter or sub-Module in __init__ is intercepted by Module.__setattr__, which files it into _parameters / _modules ordered dicts; .parameters() recurses those, which is how optimizer = AdamW(model.parameters(), ...) sees every weight, and how state_dict() / load_state_dict() serialize them (exactly what Lab 01's best-weight-restore does by hand with a dict copy). torch.no_grad() / inference_mode() suppress graph construction for eval; torch.compile (TorchDynamo) captures and fuses the graph where it can while falling back to eager where it cannot. Our miniature simplifies all of this to: one analytic gradient, plain Python floats, no graph, no GPU, no saved-tensor bookkeeping.

scikit-learn: the estimator contract and the KMeans/PCA internals

Lab 02's kmeans and fit_pca mimic sklearn's estimator API, and its conventions are load- bearing enough that violating them breaks GridSearchCV and Pipeline. The rules: constructor __init__ stores hyperparameters and does no validation and no computation (so estimators are cheap to clone); fit(X, y=None) does the work and stores learned attributes with a trailing underscore (cluster_centers_, labels_, inertia_, n_iter_, components_, explained_variance_ratio_); transform / predict use them; get_params / set_params enable cloning and search. This is why our KMeansResult and PCAResult are separate value objects — the lab splits what sklearn packs onto the fitted estimator.

KMeans internals the lab compresses. Real KMeans defaults to init='k-means++' (the same D²-weighted seeding Lab 02 implements in kmeans_pp_init) and runs it n_init times, keeping the lowest-inertia result — and the default of n_init changed from 10 to 'auto' in a recent release ('auto' = 1 for k-means++, 10 for random), a source of "my results changed after an upgrade" surprises. The algorithm parameter selects Lloyd (what our lab does — full assign/ update sweeps) or Elkan, which uses the triangle inequality to cache lower/upper distance bounds and skip distance computations that cannot change the assignment; Elkan is much faster on low-dimensional dense data but uses more memory, and the older 'full'/'auto' names were deprecated in favor of 'lloyd'/'elkan'. Empty clusters: sklearn relocates a stranded centroid to the point contributing most to inertia (the farthest-from-its-centroid point), which is the same intent as our lab's "relocate to the farthest point," implemented at production polish. MiniBatchKMeans exists for when the dataset does not fit the O(n·k·d) full-sweep budget.

PCA internals differ from the lab in a way worth stating precisely. Lab 02 forms the covariance matrix and runs power iteration + deflation — pedagogically transparent, and genuinely what scalable methods (randomized SVD, Lanczos) do under the hood. But sklearn's PCA does not eigendecompose the covariance matrix; it runs an SVD of the centered data matrix directly, because forming XᵀX squares the condition number and loses precision. The svd_solver parameter chooses among 'full' (LAPACK), 'randomized' (the Halko-Martinsson-Tropp randomized SVD, for many features and few components), 'arpack' (truncated, ARPACK), and a newer covariance-based path for the tall-skinny case; 'auto' picks by shape. PCA also centers automatically (our lab's center_data is manual) and applies sign-flipping (svd_flip) so component signs are deterministic — because an eigenvector and its negation both maximize variance, and without a convention the signs are arbitrary run to run. TruncatedSVD is the variant that skips centering, which is what you want on sparse term-document matrices where centering would destroy sparsity.

The Pipeline is a correctness tool, not sugar. Chaining a scaler and an estimator in a Pipeline and passing it to cross-validation is what makes "fit transforms inside the training fold only" automatic — the anti-leakage discipline the Warmup hammers, enforced by construction rather than by remembering.

Gym / Gymnasium and the RL libraries

Lab 03's GridWorld is a hand-built MDP; the real interface it mirrors is the Gym environment API, now maintained as Gymnasium under the Farama Foundation after OpenAI stopped maintaining Gym. The API a contributor must get right: an env exposes action_space and observation_space (the Spaces typing system — Discrete, Box, etc.), reset() returns the initial observation, and step(action) advances one tick. The signature change everyone hits: old Gym's step returned a 4-tuple (obs, reward, done, info), and reset() returned just obs. Gymnasium returns a 5-tuple (obs, reward, terminated, truncated, info) and reset() returns (obs, info). That split of done into terminated (the MDP reached a real terminal state) versus truncated (a time/step limit cut the episode off) matters because bootstrapping is only correct on true terminals — you must not zero the future value when an episode was merely truncated. Lab 03's single done boolean deliberately conflates the two, which is fine for a GridWorld with genuine terminal states but is exactly the subtlety a real Q-learning implementation gets wrong.

stable-baselines3 (the PyTorch RL library) is where the DQN/PPO machinery of Warmup §8 lives as production code: model = PPO("MlpPolicy", env); model.learn(total_timesteps=...). A contributor's notes on it: it standardizes on vectorized environments (VecEnv) running many env copies in parallel to feed the batch a policy-gradient step needs; off-policy algorithms (DQN, SAC) carry a replay buffer — legal precisely because of the off-policy max Lab 03 proves — while on-policy ones (PPO, A2C) collect fresh rollouts each update and discard them. Our tabular QLearningAgent with its dict Q-table is the pre-function-approximation ancestor: replace the dict with a neural network and add a target network plus replay buffer and you have DQN.

What the miniatures deliberately omit

Every lab trades production concerns for legibility: no GPU or vectorization, plain Python floats instead of typed tensor buffers, hand-written gradients instead of a tape, Lloyd-only instead of Elkan, covariance power-iteration instead of a numerically-hardened SVD, a single scalar done instead of terminated/truncated, and value objects instead of the trailing-underscore estimator contract. The point of the miniatures is that when you next read the real source — torch.autograd, sklearn.cluster._kmeans, gymnasium.Env — the shapes are already familiar and the production complexity reads as hardening of a mechanism you have already implemented, not as magic.