Warmup Guide — Foundations

Zero-to-expert primer for Phase 00: the Python, NumPy, and math substrate every later phase silently assumes. This is the only concept guide for this phase — read it before the three labs.

Table of Contents

Chapter 1: The Python That ML Code Actually Uses

The subset of "advanced Python" that ML codebases lean on daily — learn these as reading skills first:

  • Mutability and references: assignment never copies; b = a aliases. The classic ML bugs: a mutable default argument (def f(x, history=[])) accumulating across calls, and in-place tensor/array mutation visible through another name. Know copy vs deepcopy and when neither is needed.
  • Comprehensions and generators: dataset pipelines are generator chains — lazy, memory-bounded, single-pass (a consumed generator is empty; the "why is my second epoch empty" bug).
  • Decorators: @torch.no_grad(), @property, @lru_cache, @dataclass — a decorator is just f = wrapper(f); being able to write one demystifies half the framework magic you'll meet.
  • Context managers: with = guaranteed setup/teardown — files, locks, autocast(), no_grad(). Implementing one (__enter__/__exit__ or @contextmanager) is a lab exercise because you'll use twenty of them.
  • Dunder protocol: __getitem__/__len__ is the Dataset interface; __call__ is why modules are callable; __repr__ is your debugging UX.
  • Typing: modern ML code is type-hinted (def forward(x: Tensor) -> Tensor); hints are documentation that tools check — write them from day one.

Chapter 2: NumPy — Arrays as a Computational Model

The mental model that transfers to PyTorch/TF unchanged:

  • An ndarray is a flat memory buffer + shape + strides + dtype. Strides (bytes to step per dimension) explain everything "weird": transpose is free (swap strides, no data movement), slicing returns views (mutate the slice, mutate the original — intentional and bug-prone), and some operations need contiguous memory (a copy) before they can run.
  • Vectorization is not optional: a Python loop over pixels runs ~100–1000× slower than the equivalent array expression, because the loop pays interpreter+dispatch cost per element while NumPy pays it per array. The phase's core skill is re-expressing loops as array operations (elementwise ops, reductions with axis=, fancy indexing, where).
  • dtype discipline: images arrive as uint8 (0–255); doing math in uint8 overflows silently (200 + 100 = 44); convert to float32 for processing, clip+convert back for display. This single habit prevents the most common beginner-CV bug family.
  • Axis fluency: an image batch is (N, H, W, C) (or (N, C, H, W) in PyTorch); mean(axis=(1,2)) is per-image-per-channel mean. Reading shapes aloud and tracking them through operations is the discipline; assert x.shape == (...) liberally in lab code.

Chapter 3: Broadcasting — The Rule Everything Relies On

The rule, exactly: align shapes right-to-left; dimensions are compatible if equal or if either is 1 (the 1 stretches, without copying); missing leading dimensions are treated as 1.

(N, H, W, C) op (C,)      → per-channel (mean subtraction!)
(N, H, W, C) op (H, W, 1) → per-pixel mask across channels
(N, 1, D) op (1, M, D)    → (N, M, D): all-pairs — the distance-matrix trick

Broadcasting is how normalization, masking, distance matrices, and attention scores are written without loops — and mis-broadcasting is the classic silent bug: an (N,) vector where (N, 1) was needed produces an (N, N) matrix and no error, just wrong numbers downstream. Defense: keep dims explicit (keepdims=True, x[:, None]), and assert shapes. The lab drills exactly this.

Chapter 4: Linear Algebra for Vision

What you need, with its CV meaning attached:

  • Matrix multiply as transformation: rotation/scale/shear are 2×2 matrices on pixel coordinates; with homogeneous coordinates (append a 1), translation and perspective join — a 3×3 homography warps one image plane onto another (Phase 01's calibration and stitching live here).
  • Dot product as similarity: $u \cdot v = |u||v|\cos\theta$ — embeddings, template matching, attention — one geometric idea everywhere (the LLM track's Phase 02 builds its whole field on it).
  • Eigenvectors/SVD as structure: PCA (top eigenvectors of the covariance = directions of variance) for dimensionality reduction and whitening; SVD's low-rank approximation underlies compression and LoRA-style ideas later. Compute one PCA by hand-ish (cov → eig → project) in the lab; never again.
  • Norms and distances: L2 (Euclidean — sensitive to scale: standardize first!), L1 (robust), cosine (scale-invariant). Choosing the metric is choosing what "similar" means — a recurring design decision from k-NN to retrieval.

Chapter 5: Calculus — Gradients Without Mysticism

  • The gradient $\nabla f$ points uphill; learning is "step downhill": $\theta \mathrel{-}= \eta \nabla_\theta \mathcal{L}$. All of deep learning's optimization is this sentence plus engineering.
  • The chain rule is the entire mechanism of backpropagation: $\frac{\partial \mathcal{L}}{\partial x} = \frac{\partial \mathcal{L}}{\partial y}\frac{\partial y}{\partial x}$ composed along the computation graph. In the lab you compute a two-layer network's gradients by hand once — after which autograd (Phase 03) is bookkeeping, not magic.
  • Numerical gradient checking — $(f(x{+}\epsilon) - f(x{-}\epsilon))/2\epsilon$ — is the verification tool for every hand-derived gradient and a professional habit (every custom op in every later track gets gradchecked).
  • Functions to know by derivative: sigmoid ($\sigma' = \sigma(1{-}\sigma)$ — saturates: the vanishing-gradient seed), ReLU (kills negatives — dead-neuron caveat), softmax+ cross-entropy (combined derivative is the beautifully simple $p - y$ — derive it once in the lab; it explains why that pairing is universal).

Chapter 6: Probability for ML Decisions

  • Distributions as models of uncertainty: Gaussian (noise, init), Bernoulli/ categorical (labels, predictions). A classifier's softmax output is a categorical distribution — treating it as calibrated truth is a known mistake (Phase 02's evaluation chapter picks this up).
  • Bayes' rule read as belief update: posterior ∝ likelihood × prior — the lens for understanding class imbalance (a 99%-accurate detector on a 1%-prevalence class may be worthless: precision/recall, Phase 02), and for generative-vs-discriminative framing.
  • Expectation and variance as the vocabulary of estimators: training loss is an expectation estimated on mini-batches (hence gradient noise, hence batch-size effects later); evaluation scores carry sampling variance (hence error bars — a theme this curriculum hammers in every track).
  • Sampling: np.random.Generator with explicit seeds; reproducibility is a Phase-00 habit, not an afterthought.

Chapter 7: Plotting as a Debugging Instrument

Matplotlib here is not for papers — it's an instrument:

  • The four debugging plots you'll make a thousand times: the image itself (imshow — with correct dtype/range and cmap='gray' when single-channel; half of "my preprocessing is broken" is diagnosed by looking), histograms (pixel/ activation/gradient distributions — saturation and dead units are visible), loss curves (train vs val on one axes, log-scale y when appropriate), and scatter/embedding plots (PCA projections — Ch. 4).
  • Mechanics worth automating early: subplot grids for batch samples, consistent figure-saving (savefig with dpi and tight bbox) into a run directory — artifacts outlive terminal scrollback.
  • The habit: plot at every pipeline stage when something is wrong — raw → decoded → resized → normalized → augmented. The bug is visible at exactly one boundary.

Lab Walkthrough Guidance

Order: Lab 01 (Python) → Lab 02 (NumPy/Matplotlib) → Lab 03 (Math).

  • Lab 01: implement the decorator, context manager, and a Dataset-shaped class (__getitem__/__len__) — these three patterns are the rest of the curriculum's plumbing. Test the mutable-default and generator-exhaustion traps deliberately.
  • Lab 02: the vectorization drill — take provided loop implementations (normalization, distance matrix, image tiling) and re-express them array-wise; benchmark each (×100+ speedups make the point viscerally); finish with the broadcasting-bug hunt (find the (N,) vs (N,1) defect by shape-asserting).
  • Lab 03: hand-derive and gradcheck the two-layer network; derive softmax+CE to $p - y$; PCA from covariance on a real small dataset with the scatter plot.

Success Criteria

You are ready for Phase 01 when you can, from memory:

  1. Explain views vs copies via strides, and the uint8-overflow habit.
  2. State the broadcasting rule exactly and produce the all-pairs distance trick.
  3. Write a decorator and a context manager without reference.
  4. Hand-compute a chain-rule gradient and verify it numerically.
  5. Derive softmax+cross-entropy's $p - y$ gradient.
  6. Name the four debugging plots and what each reveals.

Interview Q&A

Q: Why is the NumPy version 500× faster than your loop — and when would the loop win? The loop pays Python interpreter dispatch, boxing, and dynamic typing per element; NumPy pays it once per array, executing a compiled C kernel over contiguous memory with SIMD. The loop "wins" only when the work is inherently scalar/sequential (early-exit searches, complex per-element control flow) — at which point the real answer is numba/Cython or restructuring the algorithm, not the loop.

Q: a = b[2:5]; a[0] = 99 — what happened to b, and why is it designed that way? b[2] is now 99: basic slicing returns a view (same buffer, adjusted offset/strides) — zero-copy by design, because copying every slice of large arrays would be ruinous. Fancy indexing (b[[2,3,4]]) returns a copy — knowing which operations view vs copy is the difference between fast code and haunted code.

Q: Your loss is NaN at step 3. Walk through it with Phase-00 tools only. Bisect with plots and asserts: histogram the inputs (uint8 not normalized? inf from a bad decode?), check the forward intermediates for overflow (float32 ranges, log of zero — clip/eps), gradcheck the hand-written layer if any, and shrink LR to test divergence-vs-bug (NaN at step 3 with sane data is usually log(0)/div-by-zero in the loss, not optimization). The method — plot distributions at each stage, assert shapes and ranges — is exactly what Phase 00 trains.

References