Phase 06: State-of-the-Art Vision Models

Weeks 13-14 | 4 Labs

The modern CV engineer must understand the architectural innovations that power GPT-4V, SAM, CLIP, and DINO. This phase builds ViT, CLIP, DINO, and a mini-SAM from scratch — then maps the 2024–2026 layer on top: promptable segmentation (SAM/SAM2), open-vocabulary detection (OWL-ViT, Grounding DINO), NMS-free set-prediction detectors (DETR/RT-DETR), frozen DINOv2/v3 backbones, and vision-language models.

Why SOTA Models?

• Vision Transformers (ViT): replaced CNNs as backbone in most SOTA systems
• CLIP: foundation for zero-shot recognition, image search, VLMs
• Self-supervised learning (DINO/DINOv2/v3): reduce label dependency by 10-100x; frozen features + light heads is the 2026 default
• SAM/SAM2 + open-vocabulary detection: promptable masks and text-defined classes — the standard auto-labeling stack (Grounded-SAM) and the data-engine mindset
• DETR/RT-DETR: detection as set prediction — no NMS, deterministic latency, real-time DETRs competing with YOLO
• VLMs (LLaVA-class, Qwen2.5-VL-class): when perception needs reasoning — plus the cost/verifiability framework for choosing them over pipelines
• Every top-tier role (Google DeepMind, Meta AI, OpenAI) expects fluency here

Lab Structure

Key Equations

Scaled Dot-Product Attention

$$\text{Attention}(Q,K,V) = \text{softmax}!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V$$

InfoNCE Loss (CLIP)

$$\mathcal{L} = -\frac{1}{N}\sum_{i=1}^{N} \log \frac{\exp(\text{sim}(z_i^I, z_i^T)/\tau)}{\sum_{j=1}^{N}\exp(\text{sim}(z_i^I, z_j^T)/\tau)}$$

DINO EMA Update (teacher)

$$\theta_t \leftarrow \lambda,\theta_t + (1-\lambda),\theta_s$$

DETR Bipartite Matching Cost (per prediction–GT pair)

$$\mathcal{C}{ij} = -\hat{p}i(c_j) + \lambda{L1},\lVert \hat{b}i - b_j \rVert_1 + \lambda{giou},\mathcal{L}{giou}(\hat{b}_i, b_j)$$

Architectural Comparison