Selecting Multimodal Models

Phase 5 · Document 08 · Model Selection Prev: 07 — Structured-Output Models · Next: 09 — Cost-Quality-Latency Framework

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Multimodal use cases — read a screenshot, transcribe a call, answer questions about a PDF or video, generate an image — are exploding, and their selection has unique traps. The biggest: "multimodal" almost always means input, not output (a model that reads images usually can't make them — that's a separate model class). Others: image tokens cost more than you expect, OCR-style tasks have surprising failure modes, and "supports vision" doesn't mean "reliable on dense documents." This doc applies the framework to multimodal, making the in-vs-out distinction concrete and teaching what to actually measure.

2. Core Concept

Plain-English primer (modalities, in vs out)

• Modality — a type of data: text, image, audio, video. Multimodal = handles more than text.
• Input vs output modality (the critical distinction): what a model can read in is listed separately from what it can produce out.
  • Vision/image input: read images (charts, screenshots, photos, document scans). Common.
  • Audio input / speech-to-text (STT): transcribe/understand audio.
  • Text-to-speech (TTS) / audio output: generate speech. Separate capability.
  • Image generation output: create images (diffusion-style models or specific multimodal-output models). A different model class from vision-input LLMs.
  • Video input: understand video (frames + audio). Output video is rare/separate.

  "Reads images" ≠ "makes images." Always check the in column and the out column separately (Phase 1.04).

• Vision tokens — images are converted into a token-equivalent for the model; a single image can cost hundreds–thousands of tokens (often by resolution/tiles), so multimodal prompts are pricier than they look (Phase 4.04).
• OCR vs vision understanding — reading text in an image (OCR-like) vs reasoning about the image (what's happening, chart values). A model can be good at one and weak at the other.

What to optimize for (modality-specific axes)

1. The right in/out modalities — confirm the exact inputs you send and outputs you need are supported (not just "multimodal").
2. Task accuracy on your media — chart/diagram reading, document OCR, screenshot UI understanding, transcription accuracy (WER for STT), etc. — measured on your data.
3. Resolution / size limits & token cost — max image size/pages, how images are tokenized, and the resulting cost.
4. Grounding/faithfulness on visual input — does it hallucinate details not in the image? (Common on dense documents.)
5. Latency — large images/long audio/video increase prefill and cost.

Selection often means combining specialized models

Many multimodal products are pipelines of specialized models, not one omni-model:

voice assistant:  STT model → text LLM → TTS model         (three models)
document Q&A:      OCR/vision model → text/RAG generator    (often two)
image generation:  text LLM (prompt) → image-gen model      (two, different classes)

A single "omni" model can be simpler but may be weaker (and pricier) than specialized components for a given step. Selection = decide per modality step, then compose (Phase 9 for visual RAG).

Output generation is a different world

Generating images/audio/video uses model classes (diffusion image models, TTS models) with their own selection criteria (style, fidelity, speed, safety) largely outside the text-LLM framework. Don't assume your chat provider does it; check explicitly, and treat it as a separate selection.

3. Mental Model

"MULTIMODAL" → ask: which modalities IN, which OUT?   (reads images ≠ makes images)

INPUT classes:  vision (image) · audio/STT · video
OUTPUT classes: text (default) · TTS/audio · IMAGE-GEN (separate model class!)

VISION TOKENS: 1 image ≈ hundreds–thousands of tokens → multimodal is pricier than it looks (Phase 4.04)
OCR (read text in image) ≠ VISUAL REASONING (understand the image) → measure the one you need

Products often = PIPELINES of specialized models:  STT → LLM → TTS ;  OCR/vision → RAG generator
Select per modality STEP, then compose. Output generation = its own selection world.

4. Hitchhiker's Guide

What to confirm first: the exact in/out modalities you need — send your real media types, get your real output type. Don't trust "multimodal."

What to ignore at first: generic multimodal leaderboards; measure on your images/audio/docs.

What misleads beginners:

• Assuming "multimodal" includes image/audio generation (usually input only).
• Underestimating image token cost.
• Assuming vision = reliable OCR on dense docs (test it; consider a dedicated OCR/doc model).
• Trusting visual answers without checking for hallucinated details.

How experts reason: confirm in/out modalities, measure task accuracy on their media (chart reading, WER, doc OCR, grounding), price the vision tokens, and often compose specialized models (STT→LLM→TTS, OCR→RAG) rather than one omni-model.

What matters in production: modality fit, task accuracy on real media, image/audio token cost, visual grounding (low hallucination), and latency on large media.

How to verify: build a small eval with your media (images/docs/audio) and known answers; measure accuracy, grounding, cost, and latency per candidate (Phase 1.07).

Questions to ask: Exact inputs/outputs supported? Max image size/pages, how tokenized, cost per image? OCR vs visual-reasoning accuracy on my docs? Do I need image/audio generation (separate model)?

What silently gets expensive/unreliable: image-token cost blowups; hallucinated visual details; weak OCR on dense documents; assuming generation capability you don't have; high latency on large media.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• "Multimodal" usually means input, not output — reading images ≠ generating them (Phase 1.04).
• Image/audio generation are separate model classes (diffusion image models, TTS) with their own selection criteria.
• A single image can cost hundreds–thousands of tokens — multimodal prompts are pricier than they look.
• OCR ≠ visual reasoning — measure the specific skill you need on your media.
• Vision models can hallucinate image details — check grounding/faithfulness.
• Many multimodal products are pipelines of specialized models (STT→LLM→TTS, OCR→RAG).
• Confirm exact in/out modalities and limits (image size/pages, audio length) before committing.
• Evaluate on YOUR media, not generic multimodal leaderboards.

9. Observations from Real Systems

• Gemini leads on broad input multimodality (image/audio/video/PDF) and large context; GPT/Claude have strong vision input — all primarily input, with text output (Phase 3.02).
• Voice assistants compose STT → LLM → TTS — three selections, often three providers.
• Document AI pairs OCR/vision with a RAG generator for grounded answers (Phase 9).
• Image-token bill surprises are common when high-resolution images are sent at full detail.
• Hallucinated chart/figure values are a known vision failure mode — caught only by a grounded eval.

10. Common Misconceptions

11. Engineering Decision Framework

1. MAP modalities: list EXACT inputs you send and outputs you need. "reads images" ≠ "makes images."
2. OUTPUT generation needed (image/audio)? → that's a SEPARATE model class & selection (diffusion/TTS).
3. For INPUT understanding: eval task accuracy on YOUR media (OCR? chart reading? transcription WER? grounding).
4. PRICE the vision/audio tokens; check size/length limits; estimate cost & latency on real media.
5. COMPOSE if better: STT→LLM→TTS, OCR/vision→RAG generator — select per step.
6. Pick the cheapest that passes accuracy+grounding on your eval; memo (3.07).

12. Hands-On Lab

Goal

Build a small vision eval on your images: measure task accuracy, grounding (hallucinated details), token cost, and latency across 2 vision-input models — and confirm the in/out distinction.

Prerequisites

• 10–15 images with known answers (mix: a chart, a screenshot, a document scan, a photo); access to 2 vision-capable models.

Steps

1. Confirm modalities: for each candidate, verify it accepts your image type and outputs text; note if you also need image generation (separate model).
2. Build the eval: each image + a question + the correct answer; include at least one "trap" where the answer is not in the image (to test hallucination/grounding).
3. Run candidates: record accuracy, whether it hallucinated on the trap, tokens used per image (cost), and latency.
4. Compare: accuracy + grounding + cost/latency; note OCR vs reasoning strengths.
5. Decide / compose: pick a model, or design a pipeline (e.g. OCR→RAG) if no single model suffices; memo it.

def vision_eval(model, items):
    correct = halluc = 0; tok = 0
    for it in items:
        ans, usage = ask_vision(model, it["image"], it["question"])
        tok += usage.image_tokens + usage.output_tokens
        if it.get("answer_in_image", True):
            correct += int(matches(ans, it["gold"]))
        else:
            halluc += int(not refused(ans))         # should say "not in image"
    return correct/len(items), halluc, tok/len(items)

Expected output

• Accuracy + per-image token cost + a grounding (hallucination) count — showing image cost is non-trivial and some models invent details.

Debugging tips

• Token cost huge? You sent full-resolution images — check the provider's detail/resolution setting.
• One model great at OCR, weak at reasoning (or vice versa)? That's the OCR-vs-reasoning distinction — pick for your task.

Extension task

Add an audio step: transcribe a clip with an STT model and feed the text to your LLM — measure WER and end-to-end accuracy (the pipeline pattern).

Production extension

Add a grounding gate (the model must cite/locate the visual evidence or decline) and a per-image cost budget; wire into Phase 13.

What to measure

Task accuracy, grounding/hallucination rate, tokens (cost) per image, latency; (extension) STT WER + pipeline accuracy.

Deliverables

• A vision eval set (incl. a grounding trap).
• A 2-model comparison (accuracy, grounding, cost, latency).
• A selection or pipeline design + memo.

13. Verification Questions

Basic

1. Why must you check input and output modalities separately?
2. Why are multimodal prompts often pricier than text-only?
3. What's the difference between OCR and visual reasoning?

Applied 4. You need to generate images from text. Can your chat vision model do it? What do you select? 5. Design selection for a voice assistant — how many models and which roles?

Debugging 6. Your vision feature's bill is surprisingly high. Likely cause and fix? 7. The model confidently describes a chart value that isn't there. What did you fail to measure?

System design 8. Design a document-QA system over scanned PDFs: which modality models, composed how, with grounding?

Startup / product 9. Explain why composing specialized models (STT/OCR/LLM/TTS) can beat one omni-model on quality and cost.

14. Takeaways

1. "Multimodal" usually means input — reading ≠ generating; check in and out.
2. Image/audio generation are separate model classes with their own selection.
3. Image tokens are costly — price multimodal prompts on real media.
4. OCR ≠ visual reasoning, and vision models hallucinate details — measure both.
5. Compose specialized models (STT→LLM→TTS, OCR→RAG) when it beats an omni-model.
6. Evaluate on your media, then pick the cheapest that passes accuracy + grounding.

15. Artifact Checklist

• Vision eval set (incl. a grounding trap).
• 2-model comparison (accuracy, grounding, token cost, latency).
• Modality map (exact in/out needed) for your use case.
• Selection or pipeline design (compose specialized models if needed) + memo.
• (Extension) STT pipeline with WER.
• Notes: in≠out, image-token cost, OCR vs reasoning.

Next: 09 — Cost-Quality-Latency Framework