Phase 04 — Vision-Language Models

The phase where the model stops being a language model and starts being a perception model. Up to now everything entered as token IDs. Now the input is a grid of pixels — and the senior question is mechanical, not magical: how do you turn an image into tokens an LLM can attend to, how do you teach vision and text to share one embedding space, and how do you splice the picture into the prompt without blowing the context budget? Answer that on a whiteboard and you can reason about GPT-4V, Claude's vision, LLaVA, and Flamingo as systems, not products.

Why this phase exists

Every multimodal model you'll be asked about in a senior interview is the same three pieces in a trench coat: a vision encoder (almost always a Vision Transformer), a contrastive alignment step (CLIP/SigLIP) that puts images and text in one space, and a fusion mechanism that hands the image to an LLM. The reason this deserves its own phase is that the interesting failures and costs all live in the seams:

1. An image is not free tokens. A 336×336 image at 14×14 patches is 576 tokens — before a single word of prompt. Multiply by the KV-cache math from Phase 00 and you see why high-resolution and multi-image inputs are the real cost driver of VLMs.
2. CLIP's whole trick is the loss. The encoders are ordinary transformers; what makes a cat-photo embedding land near the word "cat" is the symmetric InfoNCE objective over in-batch negatives and a temperature. Understand the loss and you understand zero-shot.
3. Fusion is an architecture decision. Projection (LLaVA: vision → MLP → LLM token space) is cheap and dominant; cross-attention (Flamingo) is more flexible but heavier. Knowing which one a model uses tells you its token-budget and training profile.
4. You train the connector, not the world. Production VLMs freeze the vision encoder, train the projector, and LoRA-tune the LLM. That "what's frozen, what's trained" answer is a senior tell — and it forward-references Phase 05.

Get fluent here and "the model can see now" stops being a slogan and becomes a token count, a loss, and a splice you can draw.

Concept map

                         ┌───────────────────────────────────┐
                         │   Pixels → tokens an LLM can read   │
                         └───────────────────────────────────┘
                            │              │             │
                ┌───────────┘       ┌──────┘      ┌──────┘
                ▼                    ▼             ▼
          VISION ENCODER      ALIGNMENT (CLIP)   FUSION
          ViT: image→patches  shared embedding   project (LLaVA)
          →patch embed→tokens  InfoNCE + τ        vs cross-attn (Flamingo)
          + [CLS] + pos        in-batch negatives  splice at <image>
                │                    │             │
                └────────┬───────────┴──────┬──────┘
                         ▼                   ▼
                  zero-shot / retrieval   image tokens inflate
                  (argmax cosine to text)  the KV-cache  ── ties to Phase 00
                         │                   │
                         └────────┬──────────┘
                                  ▼
                    VQA · OCR · grounding · captioning
                  (encoder frozen, projector + LoRA trained → Phase 05)

The lab

The lab is a runnable, test-verified miniature — see the lab standard. Run it red (pytest test_lab.py), make it green, then read solution.py.

Integrated scenario ideas

• Size a VLM request: a prompt with two 768×768 images at 14×14 patches — count the image tokens, add them to the text, and use the Phase 00 KV-cache formula to show why the images, not the words, set the memory.
• Defend "freeze the encoder": explain why you train only the projector (+ LoRA on the LLM) on a small instruction set, and what would break if you fine-tuned the whole vision tower (catastrophic forgetting, cost, data hunger).
• Pick a fusion family: a model that must handle interleaved image/text documents vs one that answers a single-image VQA — argue projection vs cross-attention for each.
• Explain zero-shot to a new hire: why "no training" classification works because the classifier weights are text embeddings, and how prompt wording moves accuracy.
• OCR/grounding reality check: why a 224×224 patchified image cannot read small text, and how higher resolution (more patches) trades cost for legibility.
• Debug a contrastive run: the loss won't drop — walk through the suspects (desynced image/caption pairs so the diagonal is no longer the positive, a temperature that's collapsed, a too-small batch starving the negatives) using the similarity matrix as your microscope.
• Build a zero-shot classifier with no training data: turn a list of class names into "a photo of a {class}" prompts, embed them as the classifier weights, and argmax cosine — then show how prompt wording and template ensembling move accuracy.

Deliverables checklist

• lab.py passes pytest test_lab.py -v (and LAB_MODULE=solution does too).
• You can derive the token count of an image from (H, W, patch_size) and explain the resolution↔token-count↔cost tradeoff.
• You can write the symmetric InfoNCE loss from the similarity matrix and explain why the diagonal is the positive and off-diagonals are in-batch negatives.
• You can explain temperature, why softmax/InfoNCE use max-subtraction, and why CLIP learns its temperature.
• You can contrast projection (LLaVA) vs cross-attention (Flamingo) fusion and say what's frozen vs trained in a real VLM.
• You can take any "the model can see" prompt and turn it into ViT → CLIP → projector → LLM + a token-budget number.

Key takeaways

• A Vision Transformer is a tokenizer for pixels. Patchify → linear embed → add position and [CLS] → feed an ordinary transformer. Once an image is tokens, the LLM machinery you already built (attention, KV-cache) applies unchanged.
• CLIP's power is the loss, not the encoders. The symmetric InfoNCE objective over in-batch negatives, scaled by a learned temperature, is what forges a shared image/text space — and that shared space is what makes zero-shot classification and retrieval free.
• Fusion is a budget decision. Projection (LLaVA) is the cheap, dominant default; cross-attention (Flamingo) buys flexibility at cost. Either way, images become tokens and tokens inflate the KV-cache — the Phase 00 wall returns, now driven by image resolution.
• Zero-shot is the shared space cashing out. Because images and text live in one space, you build a classifier from text prompts and pick the argmax cosine — no task-specific training, and you add a class by writing a prompt. Retrieval is the same operation transposed.
• You train the connector, freeze the rest. The senior mental model of "how is a VLM trained" is: vision encoder frozen, projector trained, LLM LoRA-tuned — which is exactly the PEFT story of the next phase.

Next: Phase 05 — PEFT: LoRA, Adapters & Distillation.