Hitchhiker's Guide — Vision-Language Models

The compressed practitioner tour. If WARMUP.md is the professor, this is the senior who leans over and says "here's what you actually need to remember about how a model sees."

The 30-second mental model

A VLM is three machines: a ViT that cuts an image into patches and embeds each patch as a token; a CLIP-style contrastive objective that trains image and text encoders to share one embedding space (matching pairs → high cosine); and a connector that hands image tokens to an LLM — either projection (LLaVA: MLP the features into LLM token space and splice them into the sequence) or cross-attention (Flamingo: image is K/V the LLM attends to). Zero-shot classification is "build the classifier from text prompts, argmax cosine." And the catch: images are hundreds-to-thousands of tokens, so they inflate the KV-cache — the Phase 00 wall, back again.

The numbers to tattoo on your arm

Back-of-envelope one-liners

# "How many tokens is a 336×336 image at patch 14?"
(336/14) * (336/14) = 24 * 24 = 576 tokens   (before any text)

# "I send 4 images at 768×768 — what's my context cost?"
4 * (768/14)^2 ≈ 4 * 3009 ≈ 12,000 image tokens → that's your KV-cache driver

# "KV-cache of those image tokens?" (Phase 00 formula)
2 * n_layers * T_image * d_model * bytes * batch   ← T now dominated by the image

# "Why is the temperature 0.07?"
cos ∈ [-1,1] is too flat for softmax; /0.07 stretches the gap to [-14,14] → sharp

The framework one-liners (where these live in real tools)

# Encode an image / text with CLIP (embeddings already L2-normalized for cosine)
from transformers import CLIPModel, CLIPProcessor
img_emb = model.get_image_features(**proc(images=img, return_tensors="pt"))
txt_emb = model.get_text_features(**proc(text=["a photo of a cat"], return_tensors="pt"))
# Zero-shot = argmax cosine (== dot product on normalized embeddings)
logits = (img_emb @ txt_emb.T) * model.logit_scale.exp()   # the learned temperature

# The ViT patch embed IS a strided conv
nn.Conv2d(in_ch, d_model, kernel_size=patch, stride=patch)   # one token per patch

# LLaVA fusion: project vision features, splice at the <image> token
image_features = mm_projector(vision_tower(pixel_values))     # vision_dim → llm_dim
# inputs_embeds = [text_embeds[:idx], image_features, text_embeds[idx+1:]]

War stories

• The context blew up on the first screenshot. Demo worked on tiny test images; a user uploaded a 4K screenshot and the request 10×'d the token count and OOM'd. Nobody had multiplied resolution into the token budget. Image tokens = (H/P)²; size context and KV-cache for the image, not the prompt.
• The "it can't read the receipt" ticket. VQA worked great; OCR was garbage. The encoder ran at 224×224 — small text literally didn't survive patchification. Fix was higher resolution / tiling, which cost 4× the tokens. Resolution is a capability decision, not a quality slider.
• The frozen-encoder save. A team tried to fine-tune the whole vision tower on 5k in-house images and torched the CLIP features (catastrophic forgetting), tanking everything. Reverting to freeze encoder, train projector, LoRA the LLM fixed it in a day. Train the connector, not the world.
• The temperature collapse. Someone hard-coded τ too small "to make it confident"; training diverged. CLIP learns and clamps τ for a reason — it's a tuned lever, not a confidence dial.
• The shuffled-batch bug. A data loader desynced images and captions; the contrastive loss stayed high and nobody knew why. The diagonal was the positive — except the diagonal pairs no longer matched. (This is literally the lab's "shuffled ⇒ high loss" soul test, in production.)

Vocabulary (rapid-fire)

• ViT — Vision Transformer; image → P×P patches → linear embed → transformer.
• Patch embedding — the Conv2d(kernel=stride=P) that turns each patch into a token.
• [CLS] token — learned vector prepended; its output is the whole-image embedding.
• CLIP — Contrastive Language-Image Pretraining; aligns image & text in one space.
• Cosine similarity — a·b/(|a||b|) ∈ [-1,1]; the distance metric of the shared space.
• InfoNCE — the symmetric contrastive loss; diagonal = positive, off-diagonal = negatives.
• Temperature / logit_scale — sharpens the softmax over similarities; learned in CLIP.
• In-batch negatives — every non-matching pair in the batch; free training signal.
• Zero-shot — build the classifier from text prompts; argmax cosine; no task training.
• Projector / connector — the MLP that maps vision features into LLM token space (LLaVA).
• Cross-attention fusion — Flamingo-style; image is K/V the LLM attends to.
• SigLIP — CLIP with a sigmoid loss instead of softmax InfoNCE; scales to bigger batches.
• Grounding — localizing where (boxes, points); needs spatial token resolution.

Beginner mistakes

• Forgetting an image is hundreds-to-thousands of tokens, so it dominates the KV-cache.
• Treating resolution as a quality knob instead of a capability + cost decision (OCR needs it).
• Skipping the zero-norm guard in cosine similarity (a degenerate embedding → divide by zero).
• Not subtracting the max in softmax/InfoNCE → overflow on large logits.
• Computing InfoNCE one-directionally — it's symmetric (rows and columns).
• Fine-tuning the whole vision encoder instead of freezing it and training the projector.
• Hard-coding the temperature instead of letting it be a learned/tuned lever.
• Thinking the LLM has special image logic — in LLaVA the image tokens are just tokens.

The one thing to take away

A VLM is ViT (image→tokens) + CLIP (shared space) + connector (image→LLM) — and the moment you add an image you've added a pile of tokens to your context and KV-cache. If you can name those three boxes, write the symmetric InfoNCE loss, and count an image's tokens from (H, W, P), you can reason about any multimodal model as a system. If you can't, "the model can see now" is just a demo.