Warmup Guide — Vision-Language Models

Zero-to-senior primer for Phase 04. We start from "what does it even mean for a model to see an image" and end with the full pipeline that lets an LLM look at a photo and answer a question about it: the Vision Transformer that turns pixels into tokens, the CLIP contrastive objective that puts images and text in one embedding space, zero-shot classification and retrieval that fall out of that space for free, the two fusion families that hand the image to an LLM (projection à la LLaVA vs cross-attention à la Flamingo), and the systems reality — images inflate the token budget and the KV-cache, so the Phase 00 cost math comes straight back. Every term is built from first principles, and the lab makes each one runnable on nested lists.

Table of Contents

• Chapter 1: What "Seeing" Means for a Transformer
• Chapter 2: The Vision Transformer — Image → Patches → Tokens
• Chapter 3: CLIP — Forging a Shared Image/Text Space
• Chapter 4: The InfoNCE Loss, the Temperature, and In-Batch Negatives
• Chapter 5: Zero-Shot Classification & Retrieval
• Chapter 6: Fusion — Projection (LLaVA) vs Cross-Attention (Flamingo)
• Chapter 7: The Systems Reality — Tokens, KV-Cache, Resolution & Training
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: What "Seeing" Means for a Transformer

From zero. A transformer does not know what a pixel is. It only knows how to process a sequence of vectors — tokens — by mixing them with attention and transforming them with MLPs. Everything you learned in Phases 01–03 (tokenize, embed, attend, predict) assumes the input already is a sequence of vectors. So the entire problem of "making a model see" reduces to one question: how do you turn a 2D grid of pixels into a sequence of vectors that a transformer can chew on the same way it chews on words?

Why this framing matters. It collapses a scary-sounding topic ("multimodal AI") into a familiar one. There is no new kind of network. There is a front-end that converts pixels to tokens, and then it's transformers all the way down. Once you internalize that, a vision-language model stops being a black box and becomes three boxes you can name: encode the image to tokens, align those tokens with text, feed them to the LLM.

The three jobs. Pull any VLM apart and you find:

The senior's habit. When someone says "we'll add vision to the assistant," the senior does not picture a new model. They picture: which vision encoder (a ViT, probably CLIP-pretrained), how many tokens per image at our resolution, which fusion (projector, almost certainly), and what that does to our context length and KV-cache. Then they have an opinion with numbers in it.

Common misconception. "Multimodal models have some special architecture that understands images." No — the understanding is learned (Chapters 3–5), and the architecture is a vision encoder plus the same transformer you already know. The novelty is the front-end and the training objective, not a new flavor of neural network.

Chapter 2: The Vision Transformer — Image → Patches → Tokens

The problem it solves. An image of size H×W (say 224×224) has 50,176 pixels. You cannot make each pixel a token — attention is O(T²), so 50k tokens is 2.5 billion attention pairs per layer. You need far fewer, coarser tokens. The Vision Transformer's answer (Dosovitskiy et al., 2020) is gloriously blunt: cut the image into a grid of square patches and treat each patch as a token. The paper's title says it all — "An Image is Worth 16×16 Words."

Step 1 — patchify. Slice the image into non-overlapping P×P patches. An H×W image yields

$$ n_{\text{patches}} = \frac{H}{P} \cdot \frac{W}{P} $$

patches, each flattened to a vector of length P·P (for grayscale; P·P·3 for RGB). A 224×224 image with P = 16 gives 14·14 = 196 patches. This is the only place pixels enter; from here on it's vectors.

  H×W image, P×P patches            flatten each patch (row-major)
  ┌──┬──┬──┬──┐                      patch (0,0) = [p00 p01 p10 p11 ...]
  │P0│P1│P2│P3│   raster order →     patch (0,1) = [...]
  ├──┼──┼──┼──┤   left→right,        ...
  │P4│P5│P6│P7│   top→bottom         → n_patches vectors of length P·P
  └──┴──┴──┴──┘

Step 2 — patch embedding. Each flat patch vector is run through a single linear layer W of shape (P·P, d_model):

$$ \text{token}_i = \text{patch}_i \cdot W. $$

In real ViTs this is implemented as a Conv2d with kernel_size = stride = P — which is exactly a per-patch linear projection, just phrased as a strided convolution. After this step each patch is a d_model-dim token, indistinguishable in shape from a word embedding.

Step 3 — the [CLS] token and position embeddings. Two finishing touches make it a real ViT:

• [CLS] token. A single learned vector is prepended to the sequence (borrowed from BERT). After the transformer runs, the output at the [CLS] position is used as the whole-image embedding — the vector CLIP compares against text. (Modern variants sometimes mean-pool the patch tokens instead.)
• Position embeddings. Attention is permutation-invariant — it has no idea patch 5 is below patch 1. So a learned position vector is added to each token, encoding where the patch was in the grid. (Wrap the token in backticks in prose: the [CLS] token is special.)

Step 4 — it's just a transformer now. Feed the n_patches + 1 tokens into a standard transformer encoder (the attention + MLP blocks from Phase 02). Patches attend to each other; the [CLS] token aggregates the image into one vector.

Why this is the universal vision front-end. Almost every modern VLM uses a ViT (often a CLIP-pretrained one) as its eyes. The patch grid is the knob that controls everything downstream: more patches (smaller P or higher resolution) → more detail, more tokens, more cost.

Common misconception. "The ViT does some clever feature extraction like a CNN's edge detectors." Out of the box, the patch embed is a single linear layer — there's no hand-crafted vision prior. The ViT learns everything from data (which is why ViTs are data-hungry compared to CNNs, and why CLIP-pretraining on hundreds of millions of pairs is so valuable). The lab stops at the patch embed precisely to make that minimalism visible.

Chapter 3: CLIP — Forging a Shared Image/Text Space

The problem it solves. A ViT gives you an image embedding. A text encoder gives you a text embedding. But they live in different, unrelated coordinate systems — there's no reason the image-vector for a cat photo is anywhere near the text-vector for "cat." CLIP (Radford et al., 2021 — Contrastive Language-Image Pretraining) trains both encoders jointly so that a matching image/text pair lands close together and a non-matching pair lands far apart. The result is one shared embedding space where similarity means "these mean the same thing."

The setup. Take a giant dataset of (image, caption) pairs — CLIP used ~400M scraped from the web. For a batch of n pairs:

1. Encode all n images with the image encoder → n image embeddings.
2. Encode all n captions with the text encoder → n text embeddings.
3. L2-normalize every embedding to the unit sphere, so a dot product is the cosine similarity.
4. Compute the n×n similarity matrix S where S[i][j] = cos(image_i, text_j), scaled by a temperature.

Cosine similarity, precisely. For two vectors,

$$ \cos(a, b) = \frac{a \cdot b}{\lVert a\rVert , \lVert b\rVert} \in [-1, 1]. $$

It is 1 when they point the same way, -1 opposite, 0 orthogonal. The senior detail: guard the zero-norm case — if either vector is all zeros the angle is undefined, so return 0 rather than dividing by zero. (The lab tests this explicitly.)

            texts →
          t0   t1   t2   t3        the diagonal S[i][i] is the
        ┌────┬────┬────┬────┐      MATCHED pair (image i ↔ its caption).
   i0   │ ★  │    │    │    │      everything off-diagonal is a
   i1   │    │ ★  │    │    │      NON-match — a free "negative".
images  ├────┼────┼────┼────┤
   i2   │    │    │ ★  │    │      training pushes ★ up and the
   i3   │    │    │    │ ★  │      rest of each row/column down.
        └────┴────┴────┴────┘

Why "contrastive." You never tell the model the right answer in words. You only tell it which pairs go together (the diagonal) and which don't (everything else). Pulling positives together while pushing negatives apart is contrastive learning, and the magic is that the "everything else" — the in-batch negatives — comes for free: in a batch of 256, each image has 1 positive caption and 255 negatives, no extra labeling.

Common misconception. "CLIP is a classifier." It's not — it has no fixed label set. It's a similarity model. The classifier is something you build at inference time out of text (Chapter 5). That indirection is the whole reason it generalizes zero-shot.

Chapter 4: The InfoNCE Loss, the Temperature, and In-Batch Negatives

What we're optimizing. Given the n×n similarity matrix S, we want the diagonal to dominate every row and every column. CLIP frames this as two classification problems glued together:

• Image → text: treat row i as a classifier over the n texts; the correct class is i (the matched caption). Apply softmax + cross-entropy.
• Text → image: treat column j as a classifier over the n images; the correct class is j. Same loss, other direction.

The total is the symmetric InfoNCE (a.k.a. the CLIP loss), the average of the two:

$$ \mathcal{L} = \frac12\Big(\underbrace{\frac1n\sum_i \text{CE}(S_{i,:},, i)}{\text{image→text}} + \underbrace{\frac1n\sum_j \text{CE}(S{:,j},, j)}_{\text{text→image}}\Big). $$

For a single row with logits z and correct index t, cross-entropy is

$$ \text{CE}(z, t) = -\log \frac{e^{z_t}}{\sum_k e^{z_k}} = \log!\Big(\textstyle\sum_k e^{z_k}\Big) - z_t. $$

The numerical-stability rule (this is part of the lesson). e^{1000} overflows to infinity in floating point. So never exponentiate raw logits — subtract the row max first (log-sum-exp / max-subtraction):

$$ \log!\sum_k e^{z_k} = m + \log!\sum_k e^{,z_k - m}, \qquad m = \max_k z_k. $$

This is mathematically identical but every exponent is ≤ 0, so it never overflows. The lab's test_info_nce_numerically_stable_for_large_logits throws 1000.0 logits at you precisely to check you did this. (Same trick you used for softmax in Phase 08's sampler.)

The temperature τ. Before the softmax, CLIP divides the cosine similarities by a small temperature (or equivalently multiplies by a learned logit_scale):

$$ S[i][j] = \frac{\cos(\text{img}_i, \text{txt}_j)}{\tau}. $$

Cosine lives in [-1, 1] — far too flat for a sharp softmax. Dividing by τ = 0.07 rescales the gap to [-14, 14], so the softmax can actually separate the positive from the negatives. CLIP learns τ rather than fixing it (clamped to avoid collapse). Small τ → sharp, confident distribution; large τ → soft, forgiving one.

Why in-batch negatives + big batches matter. The loss's "push apart" signal comes entirely from the off-diagonal negatives in the batch. More negatives = a harder, more informative contrastive task, which is why CLIP trained with batch size 32,768. (SigLIP later swapped the softmax InfoNCE for a pairwise sigmoid loss specifically so it scales to huge batches without the softmax's global normalization — same idea, friendlier arithmetic.)

The soul of the lab. info_nce_loss is low when the diagonal dominates (matched pairs are the most similar) and high when you shuffle the text side so the positives fall off the diagonal. Those two tests — test_info_nce_low_when_pairs_matched and ...high_when_pairs_shuffled — are the lesson: a loss is just a number that's small when the model is right and big when it's wrong, and you can feel the contrastive objective by watching it move.

Common misconception. "Higher temperature is always smoother and safer." Temperature is a lever, and CLIP learns it because the right sharpness depends on the data; too small collapses training, too large gives a flat, uninformative gradient. It's tuned, not guessed.

Chapter 5: Zero-Shot Classification & Retrieval

The payoff. Once images and text share one space, you get zero-shot capabilities — solving tasks the model was never explicitly trained on — basically for free. This is the result that made CLIP famous: competitive ImageNet accuracy with zero ImageNet training images.

Zero-shot classification, mechanically. To classify an image into one of K classes:

1. Turn each class name into a text prompt — e.g. "a photo of a {class}" for cat, dog, car.
2. Encode each prompt with the text encoder → K text embeddings. These are your classifier weights, constructed on the fly from words.
3. Encode the image → one image embedding.
4. Predict argmax_k cos(image, text_k).

$$ \hat{y} = \arg\max_{k} ; \cos(\text{img}, \text{txt}_k). $$

That's clip_zero_shot_classify in the lab. No training, no fixed label set, no fine-tuning — change the prompts and you've changed the classifier. Want to add "airplane" as a class? Add a prompt. That flexibility is impossible with a softmax head trained on fixed labels.

Prompt engineering is real here. "a photo of a {class}" beats the bare class name, and prompt ensembling (averaging the embeddings of 80 templates per class) gave CLIP several extra points on ImageNet. The wording moves accuracy because it nudges where the text lands in the shared space.

Retrieval is the same operation, transposed. Instead of "which of these texts matches this image," ask "which of these images matches this text" (or vice versa) — encode everything, rank by cosine similarity, return the top-k. Image search by text query, and finding the caption for an image, are both just nearest-neighbor in the shared space (which connects forward to the vector search of Phase 11).

Common misconception. "Zero-shot means the model is guessing." No — it's doing genuine nearest-neighbor in a space that was trained to put meaning-equivalent things together. The "zero" refers to zero task-specific training examples, not zero knowledge. The knowledge is in the pretrained embedding space.

Chapter 6: Fusion — Projection (LLaVA) vs Cross-Attention (Flamingo)

The problem it solves. A CLIP-style encoder gives you image tokens (or one image embedding). An LLM expects word-embedding-shaped tokens in its embedding space. The gap between "vision encoder output space" and "LLM token space" is bridged by a connector, and there are two main families.

Family 1 — Projection (LLaVA). The simplest and now-dominant approach (Liu et al., 2023). Take the vision encoder's patch features, run each through a small MLP projector that maps vision_dim → llm_token_dim, and insert the resulting vectors directly into the LLM's input sequence as if they were ordinary tokens.

  image ──▶ ViT (frozen) ──▶ patch features (vision_dim)
                                   │
                                   ▼
                            PROJECTOR (2-layer MLP, GELU)   ← the part you train
                                   │
                                   ▼  image tokens (llm_token_dim)
  text:  "What is in <image> ?"  tokenize ──▶ [w0 w1 <IMG> w3 w4]
                                                       │
                            splice at the <image> slot ▼
                        [w0  w1  img0 img1 ... imgN  w3  w4]  ──▶  LLM

The LLM then attends over text and image tokens uniformly — it doesn't know which tokens came from pixels. That's Projector.project + assemble_multimodal_sequence in the lab: project the features, then splice them in at the <image> placeholder, giving a fused sequence of length len(text) - 1 + len(image). Cheap, simple, and shockingly effective.

Family 2 — Cross-attention (Flamingo). The earlier, heavier approach (Alayrac et al., 2022). Instead of stuffing image tokens into the text sequence, you interleave new cross-attention layers into the frozen LLM. The text tokens stay as the query stream; the image tokens become the keys and values that the text attends to. The image never enters the main sequence — the LLM reaches out to it through gated cross-attention blocks.

The senior takeaway. Projection won the popularity contest because it's simple, trains fast, and reuses the LLM unchanged — but it spends your context budget on image tokens. Cross- attention keeps the image out of the sequence (cheaper context) at the cost of more parameters and architectural surgery. Knowing which one a model uses tells you its cost profile and its multi-image story.

Common misconception. "The LLM has special image-handling logic." In the LLaVA family it absolutely does not — the projected image vectors are just tokens, and the (unmodified) LLM attends to them with the exact same mechanism it uses for words. That's why projection is so appealing: zero changes to the language model itself.

Chapter 7: The Systems Reality — Tokens, KV-Cache, Resolution & Training

Images are expensive tokens. This is where Phase 00 comes roaring back. Each image becomes (H/P)·(W/P) tokens before a word of prompt:

A single high-res image can be thousands of tokens. Recall the Phase 00 KV-cache formula — 2 · n_layers · T · d_model · bytes · batch — and note that T now includes every image token. The image, not the prompt, often dominates the context length and the KV-cache. This is why multi-image and high-resolution inputs are the real cost driver of VLMs, and why techniques like token pooling, perceiver resamplers, and tiling (split a big image, encode tiles, downsample) exist — they're all token-budget tricks.

The resolution ↔ token-count ↔ task tradeoff. Resolution is the dial that decides which tasks a VLM can even attempt:

• OCR / document reading needs high resolution — small text vanishes at 224×224. More patches = more legible glyphs = more tokens = more cost. This is the core tension in document VLMs.
• Grounding (pointing at where an object is — bounding boxes, "click the blue button") needs enough spatial tokens to localize. Too few patches and the model can't tell left from right.
• VQA / captioning (answering questions, describing) tolerate lower resolution because they're about what, not where or what-it-says.

So "what resolution?" is not a quality knob — it's a capability and cost decision. Doubling resolution roughly quadruples image tokens (it's H/P · W/P), which quadruples that part of the KV-cache and the attention cost.

How a VLM is actually trained (the senior tell). You do not train the whole thing end to end from scratch. The standard recipe (LLaVA-style):

1. Freeze the vision encoder — it's already a good CLIP-pretrained ViT; retraining it is expensive and risks wrecking the features.
2. Train the projector — the connector is the cheap, small piece that learns to translate vision features into LLM token space. Stage 1 trains only this on image-caption pairs.
3. Instruction-tune with the LLM via LoRA — Stage 2 unfreezes the LLM with LoRA (low-rank adapters) on visual-instruction data, while the encoder stays frozen and the projector keeps training.

That "encoder frozen, projector trained, LLM LoRA-tuned" answer is a strong senior signal — and the LoRA half is exactly Phase 05. The whole VLM is a small amount of new training on top of two big frozen models, which is why a lab can build a tiny working version on nested lists.

Common misconception. "Adding vision means retraining a giant model." For most teams it means training a projector (a few-million-parameter MLP) and LoRA adapters on top of frozen encoders and a frozen-ish LLM — orders of magnitude cheaper than pretraining, and entirely the point of doing it this way.

Lab Walkthrough Guidance

The lab (lab-01-clip-vit-projector) turns these chapters into code. Suggested order (matches the file top-to-bottom):

1. ViT front-end (patchify, patch_embed) — Chapter 2. patchify is careful indexing plus the divisibility check (ValueError); patch_embed is a matmul (patch · W) with an optional prepended [CLS]. Watch the raster order — left→right, top→bottom.
2. Similarity primitives (cosine_similarity, l2_normalize) — Chapter 3. The only traps are the zero-norm guard (return 0 / return zeros, never divide by zero) and clamping cosine to [-1, 1] to absorb float drift.
3. CLIP matrix + loss (clip_similarity_matrix, info_nce_loss) — Chapter 4. The matrix is scaled cosine (cos / temperature). The loss is symmetric (rows + columns) and must use max-subtraction — the large-logit test depends on it. The diagonal is the positive.
4. Zero-shot (clip_zero_shot_classify) — Chapter 5. Argmax cosine to the class text embeddings. One line of logic, but it's the whole "build the classifier from text" idea.
5. Connector (Projector, assemble_multimodal_sequence) — Chapter 6. The projector is a seeded 2-layer GELU MLP (vision_dim → hidden → llm_dim); determinism comes from random.Random(seed). The assembler splices image tokens at the placeholder index — mind the length: len(text) - 1 + len(image), and reject out-of-range indices.

Run red → green: pytest test_lab.py -v, then LAB_MODULE=solution pytest test_lab.py -v, then python solution.py for the worked numbers. The two InfoNCE tests are the soul — make those green and you've felt contrastive learning.

Success Criteria

• All tests pass against your lab.py and against LAB_MODULE=solution.
• You can compute the token count of an image from (H, W, P) and explain why doubling resolution roughly quadruples image tokens — and what that does to the KV-cache.
• You can write the symmetric InfoNCE loss from a similarity matrix from memory, say why the diagonal is the positive, and explain the max-subtraction trick that keeps it stable.
• You can explain the temperature: why cosine is too flat for softmax, why dividing by τ fixes it, and why CLIP learns τ.
• You can describe zero-shot classification as "build the classifier from text prompts, then argmax cosine," and why that generalizes where a fixed softmax head can't.
• You can contrast projection (LLaVA) vs cross-attention (Flamingo) fusion on token budget, LLM changes, and multi-image flexibility.
• You can state the training recipe — encoder frozen, projector trained, LLM LoRA-tuned — and why it's cheap.

Interview Q&A

• "How does a transformer process an image at all?" — Patchify it into a grid of P×P patches, linearly embed each patch into a d_model token, add position embeddings and a [CLS] token, then run a standard transformer. An image becomes (H/P)·(W/P) tokens; after that it's ordinary attention.
• "What is CLIP and what does it actually learn?" — Two encoders (image + text) trained jointly with a contrastive loss so matching pairs have high cosine similarity in a shared space. It learns alignment, not classification — the classifier is built later from text.
• "Write the CLIP loss." — Symmetric InfoNCE: softmax cross-entropy over the rows (image→text) plus over the columns (text→image), diagonal = positive, averaged; logits are cos / temperature; use log-sum-exp for stability.
• "Why divide by a temperature?" — Cosine sits in [-1, 1], too flat for the softmax to separate positive from negatives. Dividing by τ≈0.07 sharpens it; CLIP learns τ (clamped) so the sharpness fits the data.
• "What are in-batch negatives and why use a huge batch?" — Every non-matching pair in the batch is a free negative; bigger batches give more negatives → a harder, more informative contrastive task. CLIP used batch 32k; SigLIP's sigmoid loss scales this even further.
• "Explain zero-shot classification." — Encode "a photo of a {class}" for each class to get classifier weights from text, encode the image, and pick the argmax cosine. No task-specific training; change the prompts to change the classes.
• "How does an LLM 'see' the image — what's the fusion?" — Two families: projection (LLaVA — project image features with an MLP and splice them into the token sequence; the LLM attends to them like words) and cross-attention (Flamingo — image tokens are K/V the LLM attends to via inserted cross-attn layers). Projection is simpler and spends context; cross-attn keeps the image out of the sequence.
• "What does adding an image do to your serving cost?" — Each image is hundreds–thousands of tokens, so it inflates context length, the KV-cache (2·L·T·d·bytes·batch), and the attention cost. The image often dominates the prompt — size your KV-cache for it.
• "What resolution should a VLM use?" — A capability decision, not a quality knob. OCR and grounding need high resolution (small text/precise location need many patches); VQA/captioning tolerate less. Doubling resolution ~4×'s the image tokens and cost.
• "How is a VLM trained — what's frozen?" — Freeze the CLIP-pretrained vision encoder, train the projector on image-caption pairs, then instruction-tune with LoRA on the LLM. You train the connector, not the world (forward-ref Phase 05).
• "Projection vs cross-attention — when would you pick each?" — Projection for single/few-image, simplest training, reuse the LLM unchanged. Cross-attention for many interleaved images or when you can't afford image tokens eating the context budget.
• "What's the difference between CLIP and SigLIP?" — Same idea (contrastive image/text alignment), different loss: CLIP uses softmax InfoNCE (needs the full batch for normalization); SigLIP uses a pairwise sigmoid loss that's cheaper to scale to enormous batches.

References

• Dosovitskiy et al., An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale (ViT, 2020) — patches as tokens, the [CLS] token, position embeddings.
• Radford et al., Learning Transferable Visual Models From Natural Language Supervision (CLIP, 2021) — contrastive image/text pretraining, the symmetric InfoNCE loss, zero-shot.
• Liu et al., Visual Instruction Tuning (LLaVA, 2023) and Improved Baselines with Visual Instruction Tuning (LLaVA-1.5) — the MLP projector connector and the two-stage training recipe.
• Alayrac et al., Flamingo: a Visual Language Model for Few-Shot Learning (2022) — gated cross-attention fusion and interleaved image/text.
• Zhai et al., Sigmoid Loss for Language Image Pre-Training (SigLIP, 2023) — the sigmoid alternative to softmax InfoNCE that scales to larger batches.
• van den Oord et al., Representation Learning with Contrastive Predictive Coding (2018) — the original InfoNCE objective CLIP's loss descends from.
• HF transformers docs — ViTModel, CLIPModel, LlavaForConditionalGeneration — to verify the miniature against the real classes; OpenCLIP for the contrastive-loss and zero-shot code.