Warmup Guide — SOTA Models: From ViT, CLIP, DINO to the 2026 Stack

Zero-to-expert primer for Phase 06: the transformer era of vision — ViT's patches-as-tokens move, CLIP's language-supervised embeddings, and DINO's self-supervised features — then the 2024–2026 layer built on top of them: promptable segmentation (SAM/SAM2), open-vocabulary detection, NMS-free set-prediction detectors (DETR/RT-DETR), frozen DINOv2/v3 backbones, and vision-language models — and what each changes about how you build CV systems.

Table of Contents

Chapter 1: Why Vision Went Transformer

CNNs encode inductive biases — locality (3×3 kernels) and translation equivariance (weight sharing) — that act as built-in assumptions about images. Assumptions are training wheels: they help enormously when data is scarce and constrain when data is abundant. The transformer (Phase 04 of the LLM track derives it fully) makes almost no spatial assumptions: every patch can attend to every patch from layer 1.

The empirical law that decided the era (the ViT paper's core finding): below ~ImageNet-1k-scale data, CNNs win; with large-scale pretraining (JFT/LAION-class), ViTs match then exceed them — the bias was worth less than the flexibility once data paid for the difference. Combined with the transformer's scaling behavior and its unification bonus (one architecture for text, vision, audio → multimodal models become plumbing rather than research — Ch. 4), the field moved. Your CNN knowledge (Phase 03/05) is not obsolete: it remains the right answer at small data and tight latency — Chapter 13 is the decision framework.

Chapter 2: ViT — The Architecture, Honestly Assessed

The whole trick is the input adapter: an image becomes a token sequence.

  1. Split the image into non-overlapping P×P patches (16×16 standard → 224² image = 196 tokens); linearly project each flattened patch to d dimensions — implemented as a Conv2d with kernel = stride = P (one fused op, mathematically identical).
  2. Prepend a learnable [CLS] token (the sequence-level summary slot); add learned position embeddings (without them, attention is permutation-invariant and the image is a bag of patches).
  3. Run a standard pre-norm transformer encoder (bidirectional attention — no causal mask: vision has no autoregressive order); classify from the CLS token (or mean of patch tokens — both work).

What to know beyond the recipe:

  • Resolution changes break position embeddings (more patches than trained positions) — handled by 2-D interpolation of the embedding grid; fine-tuning at higher resolution than pretraining is standard and this is the mechanism.
  • Cost scales quadratically in patch count: halving P quadruples tokens and ~16×'s attention cost — patch size is the resolution/compute dial. Hierarchical variants (Swin: local windowed attention with shifts, producing CNN-like multi-scale maps) exist precisely to feed dense-prediction heads (Phase 05's FPN world) efficiently.
  • DeiT's contribution: with strong augmentation + distillation, ViTs train respectably on ImageNet-1k alone — the data-hunger story softened by recipe.

Chapter 3: Attention for CV Engineers — the Minimum Core

The self-attention mechanics in one paragraph (full derivation in the LLM track's Phase 04 WARMUP — read it if any step feels unfamiliar): each token emits query/key/value projections; attention weights = softmax(QKᵀ/√d); output = weights · V; multiple heads run in parallel subspaces and concatenate. For vision specifically: attention maps are interpretable spatial objects — the CLS token's attention over patch tokens is a saliency map you will actually plot (DINO's famous result, Ch. 6), and "which heads look where" (local vs global specialization across depth) is a real diagnostic. The engineering note: 196–1024 tokens is small by LLM standards — vision attention is rarely the memory crisis it is in language; resolution (token count) is your knob when it becomes one.

Chapter 4: CLIP — Supervision from Language

The supervision bottleneck: ImageNet's 1.28M images took years to label over 1000 fixed classes. The internet has billions of (image, caption) pairs — noisy, free, and unboundedly descriptive. CLIP's move: train image and text encoders to agree.

  • Architecture: an image encoder (ViT) and a text encoder (transformer) each map to a shared embedding space; normalize; similarity = cosine.
  • The contrastive objective (InfoNCE): in a batch of N pairs, the N×N similarity matrix's diagonal entries (true pairs) should beat their row and column competitors — symmetric cross-entropy over the matrix, with a learned temperature scaling the logits. Large batches matter (more negatives = harder discrimination = better geometry) — this is word2vec's negative-sampling idea (LLM track Phase 02) industrialized.
  • Zero-shot classification, the headline capability: embed prompts ("a photo of a {class}") for your label set; classify an image by nearest text embedding. No training on your classes at all — competitive with supervised baselines on many distributions, and prompt wording measurably matters (prompt ensembling is the standard boost).

Chapter 5: What CLIP Changes in Practice

The capability list a working CV engineer actually uses:

  • Open-vocabulary everything: classes defined at inference time by text — classification without retraining (Ch. 4), and open-vocabulary detection/ segmentation when grafted onto Phase 05 architectures (OWL-ViT, Grounding-DINO-class systems — Chapter 8 opens up their mechanisms).
  • Image-text retrieval: both directions, one index — the engine of semantic photo search and dataset curation (find label errors by searching "mislabeled-looking X").
  • The vision tower of multimodal LLMs: LLaVA-class assistants are CLIP's image encoder + a projector + an LLM (built hands-on in the model-accuracy track's Phase 02 lab-05) — your CLIP understanding is the entry ticket to VLM work (Chapter 11 dissects the anatomy).
  • Honest limitations: CLIP is weak at counting, spatial relations ("left of"), fine-grained text rendering, and inherits web-data biases (evaluate before deploying in anything consequential — the Phase 02/08 evaluation ethics apply); zero-shot underperforms a fine-tuned model on specialized domains (medical, industrial) — it's a starting point and a fallback, not a universal answer.

Chapter 6: DINO — Supervision from Agreement

The third supervision source: none at all. Self-supervised learning makes the data its own teacher; DINO is the vision instance whose features became the field's workhorse:

  • Mechanism in brief (the full anatomy — centering vs sharpening collapse analysis, EMA teacher, multi-crop — is implemented and tested in the model-accuracy track's lab-04; this track's lab focuses on using and probing it): a student ViT sees local+global crops, a momentum-EMA teacher sees global crops, and the student learns to predict the teacher's output distribution — agreement across views forces representations that capture what the object is rather than where the crop was.
  • Why CV engineers care — the emergent properties: (1) CLS attention maps segment objects with zero segmentation labels (the famous visualization, and a real weak-segmentation tool); (2) frozen DINO/DINOv2 features are superb for k-NN and linear-probe classification (label-efficient transfer: tiny labeled sets go far); (3) dense features match across images (correspondence, retrieval-by-part) — DINOv2 features are the current default "just give me good visual features" answer (Chapter 10: what v2 changed, and why v3 doubles down).
  • The practical pattern this enables: pretrain-free pipelines — frozen foundation features + a small head trained on dozens-to-hundreds of labels, evaluated honestly (Phase 02 discipline) — often beating full fine-tunes of smaller supervised backbones at a fraction of the cost. This is the lab's experiment.

Chapter 7: SAM and SAM2 — Promptable Segmentation

Phase 05's segmentation models answer one fixed question — "which pixels belong to class k?" — and only for classes they were trained on. The Segment Anything Model (SAM, 2023) reframes segmentation as a promptable task: given an image and a prompt — a click, a box, a rough mask — return the mask of whatever the prompt indicates. Recognize the move: it is CLIP's move applied to segmentation. CLIP freed classification from a fixed label set by putting classes into text; SAM frees segmentation from a fixed label set by putting what to segment into an interaction.

The architecture is an asymmetric split, and the split IS the engineering:

  • Image encoder (heavy): a ViT-H (~0.6B params) run once per image, producing a dense 64×64 embedding grid from a 1024² input. 95%+ of total compute lives here — and it amortizes over every prompt on that image.
  • Prompt encoder (trivial): points and boxes become positional encodings plus learned type embeddings (foreground point / background point / box corner); rough masks pass through a small conv stack. Microseconds of work.
  • Mask decoder (light): two transformer layers of two-way attention over a handful of tokens — the prompt tokens plus a few learned output-mask tokens (CLS-style summary slots, one per candidate mask). Each layer runs three moves: (1) self-attention among the tokens; (2) tokens→image cross-attention — the tokens read the image ("what is under and around my click?"); (3) image→tokens cross-attention — the image embedding is updated with prompt information ("pixels, here is what is being asked of you"). The reverse direction is the "two-way": image features become prompt-conditioned instead of static, which is why a two-layer decoder suffices where a one-way design would need depth. Mask tokens are then dotted against upscaled image features to produce masks — fast enough (~ms-scale) for interactive dragging.

Ambiguity is handled structurally, not ignored. A click on a shirt could mean shirt, person, or person-plus-backpack. SAM emits three masks per prompt (roughly whole/part/subpart granularities) plus an IoU-prediction head — a small MLP regressing each candidate's expected IoU with the (unknown) intended mask — whose scores rank the candidates. During training, only the lowest-loss candidate receives gradients, so the three heads specialize instead of averaging into mush. Downstream code keeps the top-ranked mask, or all three when a human is choosing.

The data engine is the deep lesson. SA-1B (11M images, 1.1B masks) was not labeled and then a model trained — model and dataset grew in a flywheel: (1) assisted-manual: annotators correct an early SAM's proposals, ~6× faster than freehand polygons; (2) semi-automatic: SAM pre-fills confident masks, humans add what it missed (buys diversity); (3) fully automatic: prompt a grid of points, keep masks that are confident and stable under threshold perturbation — the stage that produced 99%+ of the final masks. Retrain between stages. The architecture is a good paper; model-assisted annotation loops are the transferable asset — you will rebuild this flywheel at small scale on your own data (Ch. 8 shows the standard stack for it).

SAM2 (2024) extends the same interface to video: prompt on one frame, get the object segmented and tracked through the clip. The new mechanism is streaming memory: each frame is encoded once; before decoding, a memory-attention module cross-attends the current frame's features to a memory bank holding compact encodings of recent frames' predictions plus every prompted frame. Object identity is carried by attention over stored (feature, mask) memories rather than a bolted-on tracker — and re-prompting any frame corrects drift and refreshes the bank. Images are treated as one-frame videos (one model for both), ~6× faster than SAM per frame.

Production reality:

  • Where it earns its keep: interactive annotation tooling (one click replaces a 20-vertex polygon — labeling cost drops an order of magnitude, which compounds with the data-engine lesson above); auto-labeling with box prompts fed from a detector (Ch. 8); zero-shot mask generation feeding classical pipelines — mask → measure/crop/remove-background, Phase 01 geometry meeting 2026 features.
  • Honest limits: SAM has no semantics — it produces masks without labels; pairing it with something that names things (Ch. 8, Ch. 11) is your job. The encoder is heavy: interactivity only works because of the run-once split, and dense grid-prompt mask generation over large datasets is real GPU money. Edge deployment goes through distilled variants: MobileSAM (distill the ViT-H encoder into a tiny ViT by regressing SAM's image embeddings; decoder reused unchanged), EfficientSAM (pretrain a light encoder to reconstruct SAM features via masked-image modeling, then fine-tune on SA-1B).

Chapter 8: Open-Vocabulary Detection — Text as the Label Space

A Phase-05 detector's classification head is a fixed matrix: N classes chosen at training time, scores over exactly those N forever after. A new class means relabel, retrain, redeploy. Plenty of 2026 products cannot live with that: retail catalogs with weekly SKUs, content moderation chasing novel objects, robotics following "pick up the charger cable". Open-vocabulary detection makes text the label space — classes are strings supplied at inference.

OWL-ViT is the minimal graft: take a CLIP-style dual encoder, remove the image encoder's pooling so every output patch token survives, and put two small heads on each token — one predicting a box, one an embedding for whatever object that token represents. Classification is cosine similarity between each object embedding and the text embeddings of your runtime queries — the zero-shot trick of Ch. 4 pushed down from image level to per-object level. Detection-data fine-tuning teaches localization while CLIP's pretraining supplies the open vocabulary. Clean and simple; its localization is the weaker half versus DETR-family designs.

Grounding DINO is the heavyweight: a DETR-family detector (Ch. 9 derives object queries from first principles; short version — a fixed set of learned query vectors, each responsible for finding one object) with text wired into every stage. What talks to what: (1) a feature enhancer cross-attends image features and text features in both directions early — the picture learns what is being asked, the words learn what is visible; (2) language-guided query selection initializes decoder queries from the image locations most similar to the text, so queries start on plausible objects; (3) a cross-modality decoder has every query attend to both image features and text features in each layer, and final classification is query–text similarity. The payoff over OWL-ViT is phrase grounding: "the leftmost red mug", not just "mug".

The composition pattern — Grounded-SAM: text → Grounding DINO boxes → boxes as SAM prompts → labeled instance masks, with zero task-specific training. This is the de-facto 2024–2026 auto-labeling stack: run it offline over your unlabeled pool with your ontology as prompts, QA-sample the output (foundation models make confident mistakes — Phase 02 evaluation discipline is non-negotiable here), then distill: train a YOLO/RT-DETR-class small model on the generated labels. The deployed artifact stays small, fast, and measurable; the foundation models were labeling leverage, not the product. The cost framing that sells it: pay the big models once per dataset offline, instead of forever per frame online.

Chapter 9: DETR and RT-DETR — Detection as Set Prediction

Phase-05 detectors treat detection as dense classification: score thousands of anchors/grid cells, then deduplicate with NMS. DETR (2020) restates the problem as set prediction — the model outputs a fixed-size set of detections directly:

  • Object queries: N learned vectors (N=100 in the paper), each a slot whose job is "find one object or declare nothing". A CNN backbone + transformer encoder produce image features; in the decoder, queries self-attend (coordinating — "you take that one, I'll take this one") and cross-attend the image features; small MLPs map each query to a class and a box. No anchors, no grid — boxes are direct regressions.
  • Hungarian (bipartite) matching: predictions form a set, ground truth forms a set. Compute a matching cost for every (prediction, GT) pair — class probability plus L1 and generalized-IoU box terms — and solve for the one-to-one assignment minimizing total cost (the Hungarian algorithm: optimal assignment in polynomial time). The loss is applied only through that matching; unmatched predictions are pushed to a "no object" class.
  • Why NMS disappears: one-to-one matching means two queries claiming the same object guarantees one of them eats loss — duplicates are penalized during training, so the model learns not to produce them. NMS existed because dense detectors are trained many-to-one and need a heuristic to dedup afterwards. The operational wins are concrete: no confidence-threshold/IoU-threshold pair to tune per dataset (a quiet production failure source — the thresholds that win on one distribution lose on another, and crowded scenes force painful trade-offs); a fully end-to-end differentiable graph with no non-differentiable post-op (clean ONNX/TensorRT export, no NMS plugin); and deterministic latency — NMS runtime varies with box count, a set head is fixed-shape every frame.

The catch was convergence: original DETR needed ~500 epochs against a Faster R-CNN's ~12–36. Two root causes: every query cross-attends all pixels uniformly at init and takes ages to learn to focus; and bipartite matching is unstable early — assignments flip between epochs, so gradients thrash. The fixes that made DETR practical:

  • Deformable attention: each query attends to only K (e.g., 4) sample points whose offsets and weights are predicted from the query itself, around a learned reference point — attention becomes sparse and spatially localized. ~10× fewer epochs, and multi-scale feature maps become affordable (dense global attention over high-res maps was prohibitive).
  • Denoising training (DN-DETR-style, one line): feed noised copies of ground-truth boxes as extra decoder queries whose task is reconstructing the originals — a matching-free auxiliary task that stabilizes and accelerates training.

RT-DETR (2023) is the first DETR to beat YOLO-class models on the real-time accuracy/latency curve. Two moves: a hybrid encoder — full self-attention only on the highest-level (smallest, most semantic) feature map where token count is tiny, with cheap CNN-style top-down/bottom-up fusion handling the other scales — removing the encoder bottleneck that made previous DETRs slow; and IoU-aware query selection — decoder queries are initialized from encoder tokens scored for both classification and localization quality, so the decoder starts near real objects instead of from blank slates. Result: ~53 AP at 100+ FPS (T4, TensorRT) at R50 scale, ahead of the contemporaneous YOLOs.

Positioned honestly against YOLO: RT-DETR wins mostly at S/M/L scales on GPU with TensorRT and wherever NMS-free matters (crowded scenes, export simplicity, latency determinism). YOLO keeps the deeper ecosystem (edge/tiny variants, export tooling, community recipes), and convolutions still quantize to INT8 more gracefully than attention. And note the convergent evolution: YOLOv10 dropped NMS too, via a one-to-one matching head trained alongside the classic one-to-many head — the set-prediction insight arriving in CNN land. The deep YOLO lineage belongs to phase-05's WARMUP; the short decision rule: edge/CPU/tiny → YOLO-class; GPU serving where NMS is a liability → RT-DETR; then measure both on your data, not the COCO table.

Chapter 10: DINOv2, DINOv3 and the Frozen-Backbone Default

DINO proved agreement-across-views yields good features (Ch. 6). DINOv2 (2023) is what turned that result into infrastructure — three changes:

  • Curated data (LVD-142M): not "scrape more" but curate — start from ~1.2B web images, deduplicate, then build the corpus by retrieval: embed everything and pull nearest neighbors of curated seed datasets. Self-supervised learning on data selected to resemble the evaluation domains. Same lesson as SA-1B from the other direction: the data pipeline, not the loss function, is where the win lives.
  • Two objectives combined: the image-level DINO objective (student matches an EMA teacher's output distribution across global/local crops — Ch. 6's mechanism) plus a patch-level iBOT-style masked-token objective: mask random patches in the student's input and train it to predict the teacher's representation of those patches — BERT's masked-LM move with teacher features as targets instead of a word vocabulary. Image-level buys global semantics; patch-level buys dense features that are locally correct — what segmentation, depth, and correspondence heads actually consume.
  • KoLeo regularizer (one line): penalizes tiny nearest-neighbor distances within a batch, spreading embeddings uniformly over the sphere — retrieval quality's quiet friend.

The register-tokens story — a genuinely instructive internals find. Large ViTs trained long enough develop attention artifacts: a few background patch tokens with outlier norms that soak up attention, polluting the pretty maps of Ch. 3 and degrading dense features. The 2023 investigation ("Vision Transformers Need Registers") found the cause: the model needs somewhere to perform and store global computation, and with every token tied to a patch, it hijacks low-information patches as scratch space — overwriting their local content (hence broken dense features there) and marking them with huge norms. The fix is almost comic: append a few extra learned tokens — registers — that participate in attention but are discarded at output. The freeloading computation moves into them; artifacts vanish; dense predictions and attention maps clean up. The lesson underneath: architectures silently repurpose capacity, and the attention-map-plotting habit is how you catch them doing it.

DINOv3 (2025) pushes scale (ViT-7B, ~1.7B curated images) and fixes the failure mode scale exposed: over long training, global features keep improving while dense features quietly degrade. Its answer is Gram anchoring: constrain the Gram matrix of the student's patch features — the patch-to-patch similarity structure — to stay close to that of an earlier checkpoint whose dense features were still clean. Global semantics keep improving; local geometry stops eroding. Result: one frozen backbone competitive with specialized models across dense tasks.

The practical consequence — frozen is the default starting point. For classification, segmentation, and depth: frozen DINOv2/v3 features + a linear head or light decoder, before anything else. It is the cheapest experiment you can run — embeddings cached once, heads train in minutes on a laptop — and one backbone serves many heads, amortizing the encoder across products. Fine-tuning the backbone is the escalation you take when the frozen baseline measurably falls short (extreme domain gap: medical, thermal, microscopy), not the reflex you start with.

Chapter 11: Vision-Language Models — When Perception Needs Reasoning

Everything above emits fixed types: labels, boxes, masks, embeddings. Some products need answers: "read this invoice and return the line items", "why does this listing photo violate policy?". A vision-language model (VLM) is a vision encoder married to an LLM so that outputs are text conditioned on pixels.

LLaVA-class anatomy from first principles: (1) a vision tower — a CLIP-class ViT producing patch embeddings; (2) a projector — an MLP mapping each patch embedding into the LLM's token-embedding space, so visual patches become pseudo-words the LLM ingests alongside real ones; (3) the LLM decodes the answer autoregressively, attending to visual tokens and prompt tokens alike. Training is two cheap-ish stages: projector alignment on image–caption pairs (tower and LLM frozen; only the projector learns to speak the LLM's embedding dialect), then instruction tuning on (image, instruction, response) data — projector plus LLM — teaching it to follow tasks rather than merely caption. The 2023 surprise was that this blunt gluing works at all; everything since is refinement.

What changed 2024–2026:

  • Native/any-resolution tiling: a fixed 336² input turns a document into a postage stamp — OCR and small objects are unreadable because the information is gone before the model starts. Fix: split the image into tiles at (near-)native resolution, encode each tile, concatenate the visual tokens, plus one downscaled global thumbnail for overall layout. Token count now scales with resolution — you pay for legibility in context length — and this, more than anything, is why modern VLMs actually read documents.
  • Grounding: VLMs trained to emit boxes/points as text tokens (coordinates in the output vocabulary), so "where is the leak?" returns a localized answer — a bridge between chat and detection; verify against GT before trusting it.
  • Video: sampled frames become visual tokens (with compression tricks); long video is fundamentally a token-budget problem.
  • Open-weight families at production quality — one line each: Qwen2.5-VL-class shows open weights matching closed models on OCR, document parsing, and grounding, with native dynamic resolution; PaliGemma-class shows the transfer-first recipe — a small (~3B) base built to be fine-tuned, beating bigger generalists on your specific task; Llama-vision-class shows vision grafted onto a flagship LLM via cross-attention adapter layers, leaving the text weights untouched.

The decision framework — VLM vs classical pipeline (the actual CV-engineer skill):

  • Per-image cost/latency: a detector runs in milliseconds at fractions of a cent per thousand images; a VLM pushes hundreds-to-thousands of tokens through a multi-billion-parameter decoder — orders of magnitude more per image, seconds not milliseconds.
  • Verifiability: boxes and confidences are measurable with mAP against ground truth (Phase 02 machinery); free text is far harder to evaluate — and hallucination is worst precisely at counting and precise localization: a VLM will fluently report "4 bolts" over an image containing 5.
  • Structured output: production VLM usage means forcing JSON (schema-constrained decoding) so downstream code can parse — and still validating fields against reality, because well-formed is not the same as true.
  • Where the VLM wins: open-ended reasoning, reading, long-tail queries no label set anticipated, low-volume workloads, and prototyping speed — a prompt this afternoon versus a labeling project this quarter.
  • The hybrid pattern, often the right answer: a cheap detector/classifier filters the stream; the VLM runs only on the hits. Detector flags 0.1% of frames, VLM explains those — the system costs like a detector and reasons like a VLM.

Chapter 12: Generative Models in a Perception Stack

Diffusion in one honest paragraph: train a network to denoise — corrupt training images with Gaussian noise at randomly sampled severity levels and learn to predict the noise (equivalently, the clean image) at each level; generation runs the process backwards, from pure noise to an image over many iterative denoising steps (hence the cost: one image = many forward passes). Latent diffusion (Stable-Diffusion-class) runs the whole loop inside a VAE's compressed latent space instead of pixel space — the same mechanism at a fraction of the spatial cost, which is what made it run on consumer GPUs; text conditioning enters through cross-attention to text-encoder embeddings.

What a perception engineer actually uses it for:

  • Synthetic training data for rare classes — the defect that appears twice a year, the near-collision you cannot stage. Useful when the gap is appearance-level; the standing caveat is distribution shift: synthetic ≠ real, and a model can learn to detect the generator's artifacts instead of the object. Treat synthetic as augmentation, never as the eval set — validation stays on real held-out data.
  • Restoration/super-resolution pre-processing — diffusion-based SR/deblur ahead of a recognition stage; caveat: generative SR invents plausible detail, which is disqualifying in evidence-grade pipelines (forensics, medical, metrology).

This track stays discriminative. What you need is the judgment call: when diffusion is a cheap data tool, and when its hallucinated pixels are a liability.

Chapter 13: Choosing a Backbone in the Foundation-Model Era

The decision framework that replaces "use ResNet":

SituationDefault answer
Tight edge latency, fixed classes, enough labelsCNN (or small hierarchical ViT) fine-tune — Phase 03/05 playbook
Few labels, common visual domainfrozen DINOv2 features + linear/k-NN head
Classes change at runtime / long-tail vocabularyCLIP zero-shot or open-vocab detector (Ch. 8)
Masks needed, no segmentation labelsSAM/SAM2 prompted (distilled variant at the edge) — Ch. 7
Dense prediction at scalehierarchical ViT (Swin-class) or ConvNeXt + FPN — or frozen DINOv2/v3 + light decoder (Ch. 10)
Multimodal (image+text reasoning)CLIP tower + LLM (VLM stack)
Specialized domain (medical/industrial), enough datafine-tune the strongest available backbone — domain gap beats zero-shot

Plus the operational lens (every prior phase's ethics): measure on your distribution, account deployment cost (ViT-L is not an edge model — distill or choose smaller), and prefer the boring sufficient option. The era's real change is that features are now infrastructure — the engineering moves to heads, data, and evaluation.

Lab Walkthrough Guidance

Order: Lab 01 (ViT) → Lab 02 (CLIP) → Lab 03 (DINO) → Lab 04 (mini-SAM).

  • Lab 01: implement ViT's patch embedding + encoder from your Phase 03 toolkit (or port the model-accuracy lab's ViT); verify the conv-as-patchify equivalence numerically; fine-tune pretrained ViT-B on a small dataset vs your ResNet — the comparison at this data scale should favor… find out, and explain via Ch. 1. Plot attention maps across depth.
  • Lab 02: build zero-shot classification with open CLIP weights on your Phase 05 dataset: prompt engineering ablation (3 templates + ensemble), then linear probe on CLIP features vs zero-shot vs your fine-tuned CNN — the three-way table is the deliverable. Build a tiny text→image search demo over your own photo folder.
  • Lab 03: load DINOv2; k-NN and linear-probe classification at 10/50/250 labels per class (the label-efficiency curve is the money plot); visualize CLS attention segmentation on non-iconic images; try dense-feature correspondence between two views of an object.
  • Lab 04 (lab-04-promptable-segmentation/): a mini-SAM built from scratch in pure PyTorch — synthetic shapes dataset, point-prompt encoder, two-way-attention mask decoder, multi-mask output with an IoU-prediction head. Do it after Lab 03: it assumes the ViT/attention fluency of Labs 01–03 and turns Ch. 7 from prose into tensors. What to look for: click a point where shapes overlap and watch the three candidate masks diverge (whole vs part — ambiguity resolved by architecture, not averaging); confirm the min-loss-only gradient routing is what makes them specialize; and check that the IoU head's ranking actually correlates with each mask's true IoU — that ranking is what makes multi-mask output usable downstream.

Success Criteria

You are ready for Phase 07 when you can, from memory:

  1. State the inductive-bias-vs-data law and what DeiT changed about it.
  2. Walk ViT's patchify → CLS → position-embedding pipeline and the resolution- interpolation mechanism; explain the patch-size dial.
  3. Write CLIP's symmetric InfoNCE idea and why batch size and temperature matter.
  4. Demonstrate (on paper) zero-shot classification and name CLIP's four weak spots.
  5. Explain DINO's agreement-across-views intuition and its three emergent properties.
  6. Apply the backbone-choice table to three scenarios with cost reasoning.
  7. Draw SAM's three-part split (heavy run-once encoder / trivial prompt encoder / light two-way-attention decoder), explain both directions of two-way attention, and retell the SA-1B data-engine flywheel — and why the engine, not the architecture, is the transferable lesson.
  8. Derive detection-as-set-prediction: what the Hungarian matching cost contains, why one-to-one training makes duplicates unprofitable, and the three operational consequences of deleting NMS.
  9. Tell the register-tokens story end to end: the artifact, the hijacked-patches cause, the fix, and what it teaches about diagnosing ViTs.
  10. Sketch VLM anatomy (tower → projector → LLM; alignment then instruction tuning) and argue VLM-vs-classical-pipeline for a given workload using cost, verifiability, and hallucination-risk reasoning.

Interview Q&A

Q: You have 200 labeled images per class for a niche industrial defect task. ViT, CNN, CLIP, or DINO? 200/class is small-data regime with a domain gap. First bet: frozen DINOv2 + linear probe (label-efficient, no overfitting risk, hours of work) with k-NN as the sanity baseline; second: fine-tune a small CNN with heavy augmentation (the Phase 03 playbook) — compare honestly. CLIP zero-shot likely fails (industrial defects aren't web vocabulary) but costs nothing to measure. Training ViT from scratch is the one clearly wrong answer. The graded skill: matching supervision regime to data reality, with measurement over allegiance.

Q: Why does contrastive learning need large batches, and what breaks with small ones? The loss discriminates the true pair against in-batch negatives: batch size is the number of negatives, and gradient quality scales with how hard the discrimination is. Small batches → few, mostly-easy negatives → weak geometry (embeddings collapse toward coarse clusters; retrieval precision dies in the tail). Mitigations: memory banks/queues (MoCo-style) to decouple negatives from batch, or non-contrastive objectives (DINO/BYOL) that avoid explicit negatives entirely — connecting the two families is the senior answer.

Q: A PM asks: "Can we just use CLIP zero-shot instead of labeling data for our detector?" Decompose: zero-shot classification of crops — plausible; detection needs localization, so open-vocab detectors (OWL-ViT/Grounding-DINO-class) are the relevant tools — try them on a measured pilot. Expect: good on common categories, weak on domain-specific ones; latency far above a YOLO; and prompt sensitivity becomes the new tuning surface. Likely production answer: use open-vocab models to bootstrap labels (pre-annotation, mined hard examples), then train the small fast supervised detector — foundation models as labeling leverage, not always as the deployed artifact.

Q: You have 50k unlabeled images and budget to label 2k — design the labeling strategy. Don't spend the human budget first. (1) Auto-label all 50k with Grounded-SAM (Ch. 8): your ontology as text prompts → Grounding DINO boxes → SAM masks — offline compute, zero human hours. (2) Spend humans where they compound: QA-sample the auto-labels to measure per-class quality (Phase 02: unmeasured labels are rumors), then route the budget by active learning — images where the auto-labeler is least trustworthy (low confidence, disagreement between Grounding DINO and a first distilled model) and rare-class candidates mined by CLIP/DINOv2 retrieval over the pool. (3) Reserve a slice of gold labels (say 500) as a sacred eval set never trained on. (4) Distill: train a compact detector on cleaned auto-labels + corrected hard cases; iterate the flywheel once (retrain → re-flag → re-correct — SA-1B's loop at hobby scale). Result: 50k-image supervision at a 2k-label price; the graded skill is spending humans on measurement and hard cases, not uniform annotation.

Q: When would you deploy RT-DETR over YOLO, and what breaks? Deploy RT-DETR when: GPU serving with TensorRT at S/M/L model scale (where its accuracy/latency curve wins); crowded scenes where NMS's IoU threshold eats true positives; or when NMS-free is operationally valuable — single clean export graph (no NMS plugin), no per-dataset threshold tuning, deterministic latency for tight SLAs. What breaks or needs checking: edge/CPU targets (tiny YOLO variants dominate, and attention survives INT8 quantization worse than convolutions — measure the post-quantization accuracy drop, the decoder is the risk); the ecosystem gap (Ultralytics-class tooling depth: exports, trackers, recipes); small-object performance on your data rather than the COCO table; and training friction if the team retrains often — DETR-family training is fiddlier despite denoising-era fixes. Also concede: YOLOv10-class NMS-free heads shrink the differentiator. The senior close: it's a benchmark on your distribution and your hardware, not a brand choice.

Q: A PM wants to replace your detection+OCR pipeline with a VLM — argue both sides with numbers-shaped reasoning. For the VLM: the pipeline is N models with N failure modes plus glue code; a VLM collapses it to one prompt, absorbs layout variation the template logic can't, and ships in days, not quarters. Against, do the arithmetic aloud: at 1M docs/day the pipeline costs milliseconds and fractions of a cent per doc on amortized GPU; a VLM pushes 1–3k tokens per doc through a multi-billion-parameter decoder — orders of magnitude more compute and seconds of latency; and hallucination concentrates exactly where this product bleeds (digits, totals, counts) delivered with fluent confidence, while free-text output is harder to evaluate than field-level accuracy against GT. The senior answer decomposes instead of picking a side: benchmark VLM field-level accuracy on a few hundred gold documents against the pipeline; propose the hybrid — pipeline for the ~95% standard layouts, VLM only for rejects and weird layouts (cost stays pipeline-shaped, robustness improves); force JSON schema output and validate fields either way. Hand the PM per-1k-doc cost, p95 latency, and field-accuracy in one table — the argument settles itself.

References