Model Capabilities — Families, Variants, and What a Model Can Do

Phase 1 · Document 04 · LLM Vocabulary and Mental Models Prev: 03 — Inference Parameters · Next: 05 — Serving Terms

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Every model page is a list of capabilities dressed up in jargon: base vs instruct, chat vs reasoning, multimodal, embedding, reranker, tool-calling, structured-output. If you can't decode these labels, you'll pick a base model for a chat product (and get garbage), an embedding model for generation (and get an error), or a model without tool calling for an agent (and rebuild later). This vocabulary is the filter you apply on models.dev or Hugging Face before you ever look at price or benchmarks — it's how you go from thousands of models to the handful that could possibly do your job.

2. Core Concept

Model families and variants

A model family is a set of models from one lab sharing architecture and training lineage (e.g. "Llama 3", "Qwen2.5", "Gemini 2.5"). A variant is a specific member, distinguished by:

• Size (8B, 70B, …) — see 02 — Parameters.
• Training stage — base vs instruct vs reasoning.
• Modality — text, vision, audio.
• Purpose — chat, code, embedding, reranking.

The model ID is the exact string you pass to an API (gpt-4o-2024-08-06, meta-llama/Llama-3.1-8B-Instruct). An alias (gpt-4o, claude-3-5-sonnet-latest) points at a moving target; a versioned ID is pinned. (Aliases vs versions are covered in depth in 08 — Business and Pricing Terms.)

Training-stage capabilities

A generative model produces tokens (all of the above). A foundation model is a large base model others build on.

Functional capabilities (the filter columns)

• Multimodal — accepts/produces more than text. Vision (image in), audio/STT (speech→text), TTS (text→speech). Inputs and outputs are listed separately; "multimodal input" ≠ "multimodal output."
• Embedding model — outputs a fixed-length vector representing meaning, not text. The engine of RAG retrieval and semantic search (Phase 9).
• Reranker — given a query and candidate documents, outputs relevance scores to reorder them. Improves retrieval precision; not a generator.
• Tool calling / function calling — the model can emit structured requests to call your functions (Phase 12).
• Structured output — the model can be constrained to a JSON schema.
• Reasoning support — exposes a thinking/effort control.

These are binary filters in catalogs: a model either supports tool calling or it doesn't. You filter first, then compare the survivors.

3. Mental Model

PICK ALONG FOUR AXES:

1. STAGE     base ──► instruct ──► chat ──► reasoning
             (raw)    (follows)   (talks)  (thinks)

2. MODALITY  text │ +vision │ +audio-in │ +audio-out      (in ≠ out)

3. PURPOSE   generate │ embed (→vector) │ rerank (→scores)

4. FEATURES  [tool calling] [structured output] [reasoning] [streaming]
             ── these are yes/no FILTERS, applied before price/benchmarks

If a product needs an agent over private docs: PURPOSE=generate + embed, STAGE≥instruct, FEATURES must include tool calling + structured output. That single sentence eliminates 95% of the catalog.

4. Hitchhiker's Guide

What to look for first: stage (instruct/chat — not base, unless you're fine-tuning), the feature flags you actually need (tool calling? structured output? vision?), and whether it's a generator vs embedding vs reranker.

What to ignore at first: benchmark leaderboard rank. Capability fit gates everything; a model that can't call tools won't power your agent no matter its MMLU score.

What misleads beginners:

• Using a base model and wondering why it won't follow instructions.
• Assuming any chat model can output a vector (it can't — you need an embedding model).
• Assuming "multimodal" means it can generate images (usually it means it can read them).
• Assuming tool calling is universal (many open-weight chat models lack reliable tool calling).

How experts reason: capability is a filter applied before comparison. They write down the required capability set for the workload, filter the catalog to matches, then rank survivors by quality/cost/latency (Phase 5).

What matters in production: verify a claimed capability actually works at the reliability you need — "supports tool calling" can mean "emits plausible tool JSON 80% of the time," which is not production-grade for an agent.

Debug/verify: for embeddings, check output dimension and that two similar texts score as similar; for tool calling, run 50 prompts and measure the valid-call rate; for vision, send an image and confirm grounded answers.

Questions to ask: Is this instruct/chat or base? What modalities are supported in vs out? Is tool calling first-class and benchmarked? Is structured output grammar-constrained? Is there a paired embedding model?

What silently gets unreliable: tool calling that "mostly works," vision that hallucinates on dense images, and embedding/generation model mismatches (always embed queries and documents with the same embedding model).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• A base model does not reliably follow instructions — products need instruct/chat variants.
• Embedding models output vectors, not text; you cannot "chat" with them, and you must use the same embedding model for queries and documents.
• "Multimodal" usually means input (e.g. vision = reading images); generating images/audio is a separate, rarer capability.
• Tool calling and structured output are not universal — many open-weight chat models support them weakly or not at all.
• Reasoning models trade latency and reasoning-token cost for quality on hard tasks; they're overkill (and pricier) for simple ones.
• A reranker is not a generator — it scores relevance to reorder retrieved candidates.
• Capability flags in catalogs are filters applied before quality/cost comparison.
• The same family often ships base + instruct + multiple sizes; read the variant, not just the family name.

9. Observations from Real Systems

• models.dev has dedicated columns for Reasoning, Tool Call, Structured output, and Input modality — designed exactly as the pre-filter described here.
• Hugging Face tags models by task (text-generation, feature-extraction/embeddings, sentence-similarity, text-ranking) and stage (-Instruct, -Base) — your fastest capability filter.
• OpenAI / Anthropic / Cohere ship separate model IDs for chat, embeddings, and rerank — proof these are distinct capabilities, not modes of one model.
• Cursor / coding tools route to a reasoning model for planning and a fast instruct/code model for autocomplete — capability-based routing in production.
• OpenRouter lets you filter models by supported features (tools, structured output, modalities) before sending traffic.

10. Common Misconceptions

11. Engineering Decision Framework

Write the required capability set FIRST, then filter the catalog:

Product = chat assistant?
  → STAGE ≥ instruct/chat; FEATURES: streaming. Tools/structured only if needed.

Product = agent / automation?
  → FEATURES MUST include tool calling + structured output; STAGE = instruct/chat;
    verify reliability (measure valid-call rate), don't trust the flag.

Product = RAG / semantic search?
  → need an EMBEDDING model (+ optional RERANKER) AND a generator.
    Embeddings and generation are different models.

Task = hard math/code/planning?
  → consider a REASONING variant; route simple sub-tasks to a cheap instruct model.

Input includes images/audio?
  → require matching MODALITY *in*; confirm whether you also need modality *out*.

Only AFTER filtering by capability → compare survivors on quality / cost / latency (Phase 5).

12. Hands-On Lab

Goal

Build a capability classifier: given a workload description, output the required capability set; then verify two capabilities (embeddings, tool calling) actually behave as claimed.

Prerequisites

• Python 3.10+, an API key with access to a chat model and an embedding model.

Setup

pip install openai numpy
export OPENAI_API_KEY=sk-...

Steps

from openai import OpenAI
import numpy as np
client = OpenAI()

# 1. Embedding capability: similar texts → high cosine similarity
def embed(t): return np.array(client.embeddings.create(
    model="text-embedding-3-small", input=t).data[0].embedding)
a, b, c = embed("a cat sat on the mat"), embed("a kitten rested on the rug"), embed("quarterly tax filing deadlines")
cos = lambda x, y: float(x @ y / (np.linalg.norm(x)*np.linalg.norm(y)))
print("similar:", round(cos(a, b), 3), "unrelated:", round(cos(a, c), 3))

# 2. Tool-calling capability: does the model emit a structured call?
tools = [{"type":"function","function":{
    "name":"get_weather",
    "parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}]
r = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role":"user","content":"What's the weather in Paris?"}],
    tools=tools, tool_choice="auto")
print("tool_calls:", r.choices[0].message.tool_calls)

Expected output

• similar cosine clearly higher than unrelated.
• A populated tool_calls with get_weather({"city":"Paris"}).

Debugging tips

• Trying to embed with a chat model → error or nonsense; use an embedding model.
• Empty tool_calls → the model/provider may not support tools, or tool_choice needs "required".

Extension task

Write required_capabilities(workload: str) -> set[str] for the five workloads in Section 11, and assert it against your expectations.

Production extension

Run 50 tool-calling prompts and compute the valid-call rate (parses + matches schema). This is the real metric for "does this model support tool calling well enough to ship."

What to measure

Cosine similarity gap (similar vs unrelated); tool-call valid rate over 50 prompts.

Deliverables

• required_capabilities() mapping for 5 workloads.
• A short reliability note: measured tool-calling valid-rate for one model.

13. Verification Questions

Basic

1. What's the difference between a base, instruct, and reasoning model?
2. What does an embedding model output, and how is it different from a chat model?
3. Does "multimodal" usually refer to input, output, or both? Why check separately?

Applied 4. List the must-have capabilities for a coding agent and for a RAG search feature. 5. Why must queries and documents be embedded with the same model?

Debugging 6. Your "tool-capable" model returns plain text instead of a tool call. Name two likely causes. 7. Retrieval quality is poor even with a good generator. Which capability/model might be missing?

System design 8. Design the model set for a voice assistant that answers questions over a private knowledge base. Name each capability and why.

Startup / product 9. A founder wants one model "that does everything." Explain, using capability axes, why a small set of specialized models is usually cheaper and more reliable.

14. Takeaways

1. Decode every model along four axes: stage, modality, purpose, features.
2. Base ≠ instruct ≠ chat ≠ reasoning — products almost always need instruct/chat.
3. Embeddings and rerankers are not generators; RAG needs all three roles.
4. Capabilities are filters applied before quality/cost/latency comparison.
5. Multimodal usually means input; confirm in vs out.
6. Verify claimed capabilities (especially tool calling) by measuring, not trusting the flag.

15. Artifact Checklist

• Code: required_capabilities(workload) classifier.
• Code: embedding-similarity and tool-call verification scripts.
• Benchmark result: tool-calling valid-rate for one model.
• Notes: the four-axis model in your own words.
• Decision record: the capability set for one real workload you care about.
• Cheat card: the workload → must-have-capabilities table.

Next: 05 — Serving Terms