Warmup Guide — PyTorch

Zero-to-expert primer for Phase 03: tensors and autograd, the training loop and its failure modes, CNNs through ResNet, transfer learning's regimes, and a working mental model of distributed training.

Table of Contents

Chapter 1: Tensors — NumPy with Two Superpowers

A torch.Tensor is Phase 00's ndarray (same buffer/shape/strides/dtype model, same broadcasting rule, same views-vs-copies semantics) plus: device placement (.to("cuda") — and the rule that all operands of an op must share a device; the Expected all tensors to be on the same device error is your introduction) and gradient tracking (Ch. 2).

The transfer costs that shape code structure: host↔device copies are expensive and asynchronous-able (pin_memory + non_blocking=True in loaders); .item()/.cpu() synchronize — calling them per step in the loop serializes the GPU (the bug that the model-accuracy track's profiling phase hunts with traces; here, just don't write it). Image layout is (N, C, H, W) in PyTorch — the transpose from OpenCV/NumPy's HWC is a permanent border-crossing ritual (permute(2, 0, 1)), and contiguous() after permute when an op demands it.

Chapter 2: Autograd — The Tape Behind the Curtain

Every operation on a requires_grad tensor records a node in a dynamic computation graph; loss.backward() walks it in reverse applying the chain rule (Phase 00 Ch. 5, automated). The working rules:

  • Gradients accumulate into .grad — hence optimizer.zero_grad() every step (forgetting it = silently summed gradients = effective LR drift; bug bestiary #1).
  • The graph is rebuilt every forward (define-by-run) — Python control flow just works; the price is per-op overhead (what torch.compile later removes — the model-accuracy track's Phase 04 is the deep dive; here, know it exists).
  • with torch.no_grad(): for inference/eval — skips graph construction (memory and speed); .detach() cuts a tensor out of the graph (the stop-gradient tool — you saw its load-bearing uses in DINO and DPO if you've toured other tracks).
  • Leaf vs non-leaf: only leaves (parameters, inputs you created) get .grad by default; intermediate gradients need retain_grad() — relevant when debugging.
  • Verify understanding once: implement a custom torch.autograd.Function with explicit forward/backward and gradcheck it (the lab does) — after that, autograd is bookkeeping you trust.

Chapter 3: Modules, Losses, Optimizers — The Object Model

  • nn.Module: parameters (registered automatically by attribute assignment) + forward(). Composition is nesting; model.parameters() walks the tree — understanding registration explains the classic silent bug (modules stored in a plain Python list don't register — use nn.ModuleList).
  • train() vs eval() mode: flips Dropout and BatchNorm behavior. Forgetting model.eval() at validation (or train() after) is bug bestiary #2 — symptoms: noisy val metrics or frozen-stats training.
  • Losses: CrossEntropyLoss takes logits, not probabilities (it fuses log-softmax + NLL for numerical stability — bestiary #3 is applying softmax first); class weights live here for imbalance.
  • Optimizers: SGD+momentum (the CNN classic — pairs with stepped/cosine LR decay) and AdamW (the adaptive default; the LLM track's Phase 05 derives it fully). LR schedulers step per epoch or per iteration — know which yours expects. The LR is the single most important hyperparameter; everything else is second-order.

Chapter 4: The Training Loop and Its Bug Bestiary

The canonical loop you'll write hundreds of times — and its known failure modes, each of which the lab makes you commit deliberately:

for x, y in loader:
    x, y = x.to(dev, non_blocking=True), y.to(dev, non_blocking=True)
    optimizer.zero_grad()
    loss = criterion(model(x), y)
    loss.backward()
    optimizer.step()
BugSymptom
missing zero_gradloss decreases then oscillates; effective LR wrong
missing model.eval() / no_grad at valval noisy, OOM at eval, slow eval
softmax before CrossEntropyLosstrains, but worse — doubly-squashed gradients
accumulating loss (not loss.item()) into a listmemory leak — the whole graph retained per step
labels/logits shape or dtype mismatchsometimes an error; sometimes silent broadcasting garbage (Phase 00 Ch. 3's warning, realized)
transforms with stale normalization statsaccuracy mysteriously capped

And the two pre-flight rituals from the shared curriculum discipline: overfit one batch to ~zero loss before any long run (wiring test), and fixed-seed determinism so changes are attributable.

Chapter 5: Data Loading — The Underrated Bottleneck

Dataset (__getitem__/__len__ — Phase 00's dunder protocol, productized) + DataLoader (batching, shuffling, parallel workers). The performance model: workers decode/augment on CPU while the GPU trains — if the GPU waits (low utilization, jagged step times), raise num_workers, use pin_memory=True, move heavy augmentation to GPU (or pre-resize the dataset on disk — JPEG decode of full-resolution photos is the classic hidden cost). Augmentation policy lives here too: train-time randomness (flips, crops, color jitter) is regularization; val/test transforms are deterministic (resize + center-crop + normalize) — train transforms on val data is bestiary #7. Normalization uses the pretraining dataset's stats when fine-tuning (ImageNet means/stds) — a contract with the checkpoint, same species as the LLM track's chat-template contract.

Chapter 6: Convolutions to ResNet

  • Convolution layers are Phase 01 Ch. 3 with learned kernels: local connectivity + weight sharing = translation equivariance and parameter economy (a 3×3×C_in×C_out kernel vs a dense layer over pixels). Output size arithmetic ($\lfloor (n + 2p - k)/s \rfloor + 1$) and receptive field growth with depth are the two calculations to be fluent in.
  • BatchNorm: normalize each channel over the batch, then learnable scale/shift — stabilizes and accelerates CNN training; its batch-statistics dependence is also its curse (small batches → noisy stats; train/eval mode split; the running-stats machinery). Conv-BN-ReLU is the CNN unit (and BN folds into conv at inference — the model-accuracy track's FX lab does it mechanically).
  • The ResNet insight: deeper plain nets trained worse (not just overfit — higher training error: a degradation problem). Residual blocks ($x + F(x)$) make identity the default and let layers learn corrections — gradient highways (the same idea you've now met as LSTM cell states and transformer residuals; deep learning has one good trick and it keeps wearing new clothes). Bottleneck blocks (1×1 reduce → 3×3 → 1×1 expand) buy depth cheaply; stride-2 stages halve resolution while doubling channels.
  • Building ResNet-18 from scratch (the lab) and matching torchvision's parameter count exactly is the rite of passage — the count check catches every wiring mistake.

Chapter 7: Transfer Learning — The Default Strategy

Training from scratch is the exception; the default is starting from ImageNet-pretrained (or CLIP-class) weights:

  • Why it works: early conv layers learn universal features (edges, textures — Gabor-like filters emerge regardless of task); only later layers are task-specific. Pretraining is, in effect, a better initialization plus a feature library.
  • The regimes, by data size and domain gap:
    1. Linear probe (freeze all, train a new head): tiny data, similar domain.
    2. Partial fine-tune (unfreeze last block(s) + head): the workhorse middle.
    3. Full fine-tune at low LR (often with discriminative LRs — smaller for early layers): enough data, or large domain gap.
    • Larger domain gap (medical, satellite, industrial) shifts you toward later regimes — early features transfer less.
  • The mechanics that matter: replace the classifier head to your class count; match the pretraining normalization stats; consider BN-stats behavior when freezing (frozen-but-train-mode BN is a subtle gotcha); start with 10–100× lower LR than scratch training. The lab runs regimes 1–3 on the same dataset — the comparison table is the deliverable.

Chapter 8: Distributed Training in One Chapter

The working mental model (the LLM track's Phase 10 carries the full depth):

  • DDP (the 90% case): N processes, each a GPU, each a model replica and a data shard; gradients all-reduced (averaged) each step so replicas stay identical. Launch via torchrun; wrap with DistributedDataParallel; use DistributedSampler (and set_epoch for reshuffling — the classic omission). Effective batch = per-GPU batch × N: adjust LR accordingly (linear-scaling heuristic + warmup).
  • The practical rules: rank-0-only logging/checkpointing; barrier() around shared filesystem moments; seeds offset per rank for augmentation diversity; and measurement before scaling — a dataloader-bound job scales GPUs into idleness (Ch. 5 first!).
  • Beyond DDP — when the model doesn't fit — lives sharding (FSDP/ZeRO) and tensor/pipeline parallelism: pointers in the LLM track's Phase 10 WARMUP; for CV-scale models, DDP is almost always the whole story.

Lab Walkthrough Guidance

Order: labs 01→05 as numbered.

  • Lab 01 (tensors/autograd): port a Phase 00 NumPy exercise to tensors; build and gradcheck a custom autograd Function; demonstrate the .grad accumulation behavior explicitly.
  • Lab 02 (training loop): write the loop bare (no Lightning-style wrappers); commit each bestiary bug deliberately and document its symptom; finish with the overfit-one-batch ritual and a clean CIFAR-10 run with curves.
  • Lab 03 (ResNet from scratch): build ResNet-18; assert parameter-count parity with torchvision; train on CIFAR-10 (with the stem adaptation for 32×32) and compare against your plain-CNN baseline — the residual delta, personally measured.
  • Lab 04 (transfer learning): the three regimes on a small fine-grained dataset; the comparison table + learning curves; note where each regime's curve says bias vs variance (Phase 02's lens, reused).
  • Lab 05 (distributed): convert lab 02's script to DDP via torchrun on 2 processes (GPU or even CPU-Gloo to learn the mechanics); verify same-loss-as-single-process at matched effective batch; measure scaling efficiency and name what limits it.

Success Criteria

You are ready for Phase 04 when you can, from memory:

  1. Explain device semantics, the .item() sync trap, and NCHW vs HWC.
  2. Describe graph construction/backward, why zero_grad, and what detach does.
  3. Recite five bestiary bugs with symptoms.
  4. Do conv output-size and receptive-field arithmetic; explain BN's train/eval split.
  5. Tell the ResNet degradation story and connect residuals to the other gradient highways you know.
  6. Choose a transfer regime from data size × domain gap and list the three mechanical musts.
  7. Sketch DDP's replicate-shard-allreduce loop and the LR/batch adjustment.

Interview Q&A

Q: Training loss decreases but validation accuracy is stuck at chance. What's your checklist? Wiring first: labels shuffled relative to inputs in the val loader (or a transform mismatch — train augmentations applied to val); model.eval() missing (BN/Dropout chaos); metric computed on logits vs probabilities with a wrong argmax axis; normalization stats mismatched between train and val. The tell is chance-level — that's a broken pipe, not underfitting; bisect by evaluating on the training set (if that's also chance under the val code path, the eval code is the bug).

Q: Why does CrossEntropyLoss take logits instead of probabilities? Numerical stability: it computes log-softmax internally via the log-sum-exp trick; feeding probabilities means an external softmax → log of small numbers → precision loss, and gradients through two softmaxes are doubly squashed. It also fuses two kernels. The deeper point: API contracts encode numerics — same reason binary_cross_entropy_with_logits exists and is preferred.

Q: You move from 1 GPU to 4 with DDP and accuracy drops slightly. Why might that be correct behavior? Effective batch ×4 changes the optimization trajectory: fewer steps per epoch, less gradient noise (which regularizes), and the LR heuristic (linear scaling) is an approximation — some recipes need warmup retuning. Also check DistributedSampler without set_epoch (same shard order every epoch) and per-rank seed handling. The point being probed: DDP is mathematically not identical to the single-GPU run at the same nominal hyperparameters — it's identical to a 4×-batch run.

References