Warmup Guide — Classical CV & OpenCV

Zero-to-expert primer for Phase 01: images as arrays, filtering and morphology, features and matching, optical flow, and camera geometry — the pre-deep-learning toolkit that still does half the work in production vision systems.

Table of Contents

Chapter 1: What an Image Is

A digital image is a sampled, quantized measurement of light: an (H, W) array of intensities (grayscale) or (H, W, 3) of color samples, almost always uint8 (0–255). Phase 00's dtype discipline applies with force: process in float, store and display in uint8, and remember that OpenCV indexes (row, col) = (y, x) — the coordinate-order bug is the field's hazing ritual.

Resolution, sampling, and aliasing: downsample without first low-pass filtering and high frequencies fold into artifacts (jagged edges, moiré) — which is why cv2.resize's interpolation choice matters (INTER_AREA for shrinking, LINEAR/CUBIC for enlarging), and why "thumbnail then detect" pipelines quietly destroy small objects.

Chapter 2: Color Spaces and Why BGR Will Bite You

  • OpenCV loads BGR, everything else expects RGB — the red-faces-look-blue bug that every practitioner ships exactly once. cv2.cvtColor(img, cv2.COLOR_BGR2RGB) at the boundary, always.
  • HSV separates what color (hue) from how colorful (saturation) and how bright (value) — color-based segmentation thresholds in HSV survive lighting changes that destroy RGB thresholds. The classic "track the orange ball" is an HSV inRange + morphology pipeline.
  • Grayscale is a weighted sum (≈ 0.299R + 0.587G + 0.114B — human luminance sensitivity), and most classical algorithms run on it: features, edges, and flow are geometry, not color.
  • Lighting robustness in general — histogram equalization and CLAHE (local, contrast-limited — the practical one) — is a preprocessing decision you make looking at histograms (Phase 00 Ch. 7).

Chapter 3: Convolution and Filtering

The single most important operation in vision, classical or deep: slide a small kernel over the image; each output pixel is the kernel-weighted sum of its neighborhood.

  • Gaussian blur: kernel = sampled 2-D Gaussian — the principled smoother (separable into two 1-D passes: O(k) not O(k²) per pixel — the optimization to know). σ controls scale; blur-before-downsample is anti-aliasing (Ch. 1).
  • Median filter: not a convolution (order statistic) — removes salt-and-pepper noise while preserving edges, which Gaussian can't.
  • Bilateral filter: Gaussian in space × Gaussian in intensity difference — smooths regions, preserves edges; the "beauty filter" primitive.
  • Border handling (replicate/reflect/constant) changes results at the edges — an explicit parameter, not a footnote, when outputs feed measurement.
  • The bridge to Phase 03+: a CNN is this exact operation with learned kernels stacked deep. Hand-designing kernels here (blur, sharpen, Sobel) is what makes "the network learns its own Sobel" meaningful later.

Chapter 4: Edges, Gradients, and Canny

  • Image gradient = intensity change per pixel: Sobel kernels approximate $\partial I/\partial x, \partial I/\partial y$; magnitude $\sqrt{G_x^2 + G_y^2}$ and orientation $\arctan(G_y/G_x)$ — the raw material of edges, HOG, and SIFT alike.
  • Canny is the principled pipeline, worth knowing stage-by-stage: Gaussian smooth → Sobel gradients → non-maximum suppression along gradient direction (thin ridges to 1-px edges) → hysteresis double-threshold (strong edges seed; weak edges kept only if connected to strong) — the two-threshold idea (high precision seeds + recall by connectivity) is a design pattern you'll reuse outside vision.
  • Tuning reality: Canny's thresholds depend on image contrast — auto-tuning from median intensity (the 0.66×/1.33× heuristic) is the practical move.

Chapter 5: Thresholding and Morphology

The workhorse segmentation stack for controlled imaging (documents, industrial inspection, microscopy):

  • Otsu's threshold: choose the global threshold minimizing intra-class variance of the resulting histogram split — optimal for bimodal histograms, automatic, and the reason you look at the histogram first.
  • Adaptive thresholding: per-neighborhood thresholds — survives illumination gradients that kill any global value (uneven document lighting).
  • Morphology on the resulting binary masks: erosion (shrink — remove specks), dilation (grow — fill pinholes), opening (erode→dilate: remove small noise, keep shape), closing (dilate→erode: seal gaps). With a structuring element whose shape means something (horizontal kernel → extract table lines).
  • Then connected components / contours (findContours) → per-object measurements (area, centroid, bounding box) — a complete measurement pipeline with no learning, which is exactly what you deploy when the camera and lighting are yours to control. Knowing when this stack beats a neural network is a senior judgment call (it's faster, explainable, and data-free — and brittle to scene change; say both).

Chapter 6: Features — Detection, Description, Matching

The local-feature pipeline: find repeatable points → describe their neighborhoods → match across images. It powers stitching, SLAM, and registration to this day.

  • Detection: corners are points where the local gradient structure is strong in two directions — Harris formalizes this via the structure tensor's eigenvalues; Shi-Tomasi's goodFeaturesToTrack is the practical variant; FAST is the speed-optimized comparison-based one.
  • Description: SIFT's neighborhood histogram-of-gradients descriptor, normalized for rotation (dominant orientation) and scale (detected in scale space) — 128-D float, the gold standard; ORB (FAST + rotated BRIEF binary descriptor) — 256-bit, Hamming-matchable, ~100× faster, patent-free — the production default when SIFT is too slow.
  • Matching: brute-force nearest descriptor + Lowe's ratio test (best/second-best < 0.75 — reject ambiguous matches; this one heuristic removes most garbage), then RANSAC to fit the geometric model (homography/fundamental matrix) while rejecting outliers: sample minimal sets, fit, count inliers, keep the consensus. RANSAC's pattern — hypothesize from minimal data, verify by consensus — is another exportable design idea.

Chapter 7: Optical Flow and Tracking

  • The brightness-constancy assumption: a moving pixel keeps its intensity; first-order expansion gives one equation per pixel with two unknowns (u, v) — the aperture problem (through a small window, only motion along the gradient is observable).
  • Lucas-Kanade resolves it by assuming constant flow in a window and solving the overdetermined system — valid where the structure tensor is well-conditioned, i.e., at corners (Ch. 6's detector is LK's natural partner: goodFeaturesToTrack → calcOpticalFlowPyrLK). Pyramidal LK handles large motions coarse-to-fine.
  • Dense flow (Farnebäck classically; RAFT-class networks today) computes flow everywhere — for motion segmentation and video features, at real cost.
  • Tracking as the system built on top: detect → associate across frames (IoU or appearance) → smooth with a motion model (Kalman filter: predict-correct under Gaussian assumptions) → handle birth/death of tracks. SORT is exactly this composition and remains the baseline tracker; the lab's tracking exercise is its reduced form. Flow tells you pixel motion; tracking maintains object identity — different problems, frequently confused.

Chapter 8: Camera Geometry and Calibration

The math that turns pixels into measurements:

  • Pinhole model: a 3-D point projects through the intrinsics matrix K (focal lengths fx, fy; principal point cx, cy) after the extrinsics [R|t] place it in camera coordinates: $x \sim K[R|t]X$. Homogeneous coordinates make projection linear; the ~ hides the perspective divide.
  • Lens distortion (radial k1, k2…, tangential p1, p2) bends the ideal model — straight lines bow; undistortion applies the inverse mapping.
  • Calibration (Zhang's method, cv2.calibrateCamera): images of a checkerboard of known geometry → solve for K, distortion, and per-view extrinsics; reprojection error (~<0.5 px when healthy) is the quality metric — and the lab's acceptance test.
  • What calibration buys: metric measurement (pixels → angles/sizes), undistorted inputs for geometry-sensitive downstream (lane detection, AR), stereo rectification → depth, and homography-based plane measurements (bird's-eye view). Cameras are sensors; calibration is their datasheet — uncalibrated "measurement" pipelines are vibes.

Lab Walkthrough Guidance

Order: labs 01→05 as numbered — each consumes the previous chapter's tools.

  • Lab 01 (image basics): drill the BGR/RGB and (y, x) conventions deliberately; build the float-process/uint8-display habit; histograms before and after every operation.
  • Lab 02 (filtering/morphology): implement convolution yourself once (then use cv2.filter2D); verify Gaussian separability numerically; build the threshold→morphology→contours measurement pipeline on a document or coins image — count and measure objects, end to end.
  • Lab 03 (features): ORB + ratio test + RANSAC homography → stitch two photos of your desk. Inspect rejected matches — the ratio test's victims teach more than its survivors.
  • Lab 04 (flow/tracking): corners + pyramidal LK on a video; then the Kalman-smoothed track on a moving object; break brightness constancy (lighting change) and watch flow fail — knowing the assumption by its failure.
  • Lab 05 (calibration): print a checkerboard, calibrate your webcam, report reprojection error, undistort, and measure a known object on a plane via homography — closing the pixels-to-millimeters loop personally.

Success Criteria

You are ready for Phase 02 when you can, from memory:

  1. State the uint8/float, BGR/RGB, and (y, x) conventions and the bug each causes.
  2. Explain Gaussian separability's O(k²)→O(k) and when median/bilateral beat Gaussian.
  3. Walk Canny's four stages and the hysteresis design pattern.
  4. Build (on paper) the Otsu→morphology→contours measurement pipeline and say when it beats deep learning.
  5. Explain ratio test + RANSAC and the hypothesize-verify pattern.
  6. State brightness constancy, the aperture problem, and why LK tracks corners.
  7. Write the pinhole projection and define reprojection error.

Interview Q&A

Q: When would you choose classical CV over a neural network today? Controlled imaging (fixed camera, lighting, parts): threshold+morphology+contours is faster (sub-ms on CPU), explainable, tunable without data collection, and certifiable — industrial inspection runs on it. Also: geometry tasks (calibration, stitching, AR-plane math) are solved classically, and hybrid systems are the real answer — a detector finds the part, classical geometry measures it. The wrong choice is classical methods on in-the-wild imagery — that's what learned features fixed.

Q: Your feature matcher works on photos but fails between a photo and a rendering of the same scene. Why? Descriptors encode local appearance (gradient statistics); rendering changes texture, noise, and shading statistics even with identical geometry — descriptors land far apart despite corresponding points. Classical fixes: match on structure (edges/lines), heavier normalization; the modern fix: learned descriptors/matchers (SuperPoint/ SuperGlue-class) trained for cross-domain invariance. The diagnosis being tested: detection repeatability vs descriptor invariance are separate failure axes.

Q: Explain RANSAC to a junior, including its knobs. Fit a model when most data is good but some is garbage: repeatedly sample the minimum points to define the model (4 correspondences for a homography), fit, count how many other points agree within a tolerance (inliers), keep the best consensus, refit on its inliers. Knobs: the inlier tolerance (your noise model), iterations (from the outlier rate: enough trials that ≥1 all-inlier sample is near-certain), and the minimal-set size (the model's freedom). It fails when outliers are structured (a second valid motion) — at which point you want robust multi-model methods, or to ask why your data has two stories.

References

  • Szeliski, Computer Vision: Algorithms and Applications (2nd ed., free online) — ch. 2–9 are this phase
  • OpenCV-Python tutorials — work alongside each lab
  • Canny, A Computational Approach to Edge Detection (1986)
  • Lowe, Distinctive Image Features from Scale-Invariant Keypoints (2004) — SIFT + the ratio test
  • Rublee et al., ORB: an efficient alternative to SIFT or SURF (2011)
  • Fischler & Bolles, RANSAC (1981)
  • Lucas & Kanade (1981) and Bouguet, Pyramidal Implementation of the Lucas-Kanade Feature Tracker
  • Zhang, A Flexible New Technique for Camera Calibration (2000)
  • Bewley et al., Simple Online and Realtime Tracking (SORT) (2016) — arXiv:1602.00763