Lab 01 — A From-Scratch Supervised Training Loop

Phase 32 · Lab 01 · Phase README · Warmup

The problem

Every deep-learning framework hides the same four-step loop behind model.fit() or loss.backward(); optimizer.step(): forward → loss → backward → step. If you have only ever called that method, you cannot answer the interview question "what does .backward() actually compute?" or debug a model whose loss goes to NaN, sits flat, or diverges. So you build the loop yourself, for the simplest model that still has every moving part: binary logistic regression.

Logistic regression is the honest floor of supervised learning — a single linear layer plus a sigmoid, trained by gradient descent on a convex loss. It has a loss function (binary cross-entropy), an analytic gradient (the famous residual p − y), a regularizer (L2), a training loop (epochs × mini-batches with a shuffle), an overfitting control (early stopping on a validation set), and an evaluation story (a confusion matrix and its metrics). Get all of that right in pure Python and the same skeleton scales — unchanged in shape — to a 100-layer network; only the gradient computation gets automated (that is what autograd is, and the sibling Senior AI Engineer track builds it from scratch).

What you build

File map

Run it

python3 -m pytest test_lab.py -q                     # your lab.py (red until you implement)
LAB_MODULE=solution python3 -m pytest test_lab.py -q # the reference (green)
python3 solution.py                                   # worked example + learning curve

Success criteria

• sigmoid(1000) and sigmoid(-1000) return 1.0/0.0 without overflowing.
• bce_loss is small for confident-correct weights, large for confident-wrong, and its L2 term adds exactly (l2/2)‖w‖².
• compute_gradients matches a central finite-difference estimate of bce_loss to 1e-5 for every weight and the bias — the proof your backward pass is correct.
• Training loss decreases over epochs and reaches >90% accuracy on separable data.
• A larger l2 produces a smaller learned weight norm.
• Early stopping fires before the epoch cap and restores the best-validation weights.
• Metrics match a hand-checked confusion matrix, and precision/recall/F1 return 0.0 (not a ZeroDivisionError) when a denominator is empty.
• Same seed → identical trained weights; different seed → a different trajectory.
• All 27 tests pass under both lab and solution.

How this maps to the real stack

• compute_gradients is what loss.backward() does in PyTorch and what GradientTape records in TensorFlow — for one linear layer we can write the gradient by hand because it has a closed form; a deep network has the autograd engine apply the chain rule for you, but the quantity computed is identical (the partial derivative of the loss w.r.t. every parameter). The finite-difference check here is exactly torch.autograd.gradcheck in miniature.
• LogisticRegression.fit is the canonical PyTorch training loop — for epoch: for batch in loader: optimizer.zero_grad(); loss = criterion(model(x), y); loss.backward(); optimizer.step() — and Keras's model.fit(...) with validation_data and an EarlyStopping callback. Our per-batch update w -= lr * grad is plain SGD; Adam and momentum (Warmup §5) change how the step is computed from the gradient, not the loop's shape.
• The seeded shuffle is DataLoader(shuffle=True, generator=torch.Generator().manual_seed(s)) / tf.data.Dataset.shuffle(..., seed=s): reproducible epoch order is a real requirement, not a test convenience.
• Early stopping with restore_best_weights is keras.callbacks.EarlyStopping and the standard hand-rolled PyTorch pattern — the single most common regularizer in production, and the min_delta/patience semantics here match Keras's exactly.
• The metrics are sklearn.metrics.confusion_matrix / precision_recall_fscore_support; computing them from the four cells is what those library calls do under the hood.

Limits of the miniature. Real optimizers use momentum/Adam and learning-rate schedules; ours is fixed-LR SGD. Real tokenized/standardized features come from a preprocessing pipeline (Phase 27); ours is a clean synthetic Gaussian. Real autograd handles arbitrary computation graphs; our gradient is hand-derived for one convex model. The shape — forward, loss, backward, step, validate, early-stop, evaluate — is the production shape.

Extensions (your own machine)

• Add momentum (v = μv − lr·g; w += v) and then Adam (Warmup §5) and compare convergence speed on the same seed.
• Swap BCE for hinge loss to turn this into a linear SVM; note how the gradient changes.
• Add k-fold cross-validation (Warmup §14) around fit and report mean ± std of val F1.
• Extend to multi-class with a softmax head and categorical cross-entropy.
• Add a learning-rate schedule (step decay) and watch the loss curve smooth out near the end.

Interview / resume signal

"Implemented logistic regression end-to-end in pure Python: numerically stable sigmoid, binary cross-entropy with L2, the analytic p−y gradient verified against a finite-difference check, a mini-batch SGD training loop with a seeded shuffle, early stopping with best-weight restoration, and confusion-matrix metrics (precision/recall/F1) — the same forward→loss→backward→step loop every framework hides behind .fit(), built so I can debug a flat, diverging, or overfitting loss instead of guessing."