How to Read Model Cards

Phase 3 · Document 00 · Model Cards and System Cards Prev: Phase 3 Index · Next: 01 — How to Read System Cards

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

A model card is the primary source of truth for a model — what it does, how it was trained, its limits, and how it fails. Reading cards poorly causes the expensive failures: wrong model for the use case, surprise behavior in production, data-retention violations, missed safety requirements, and bad cost estimates. Reading one well takes 5–10 minutes and can save weeks. This document teaches the general skill; the per-lab guides (02–05) apply it to specific publishers, and the checklist and memo template turn it into a repeatable artifact.

2. Core Concept

Plain-English primer (the capability words a card throws at you)

A model card lists capabilities in shorthand. Here's each in plain English with its purpose, so the "what to extract" list below makes sense even with no AI background:

• Tokens / context window — text is chopped into tokens (≈ ¾ word each); the context window is how many tokens the model can read at once. Purpose: bounds the document/history size you can feed it. → Phase 1.01
• Parameters; dense vs MoE (total/active) — parameters are the billions of learned numbers that make up the model ("8B" = 8 billion). A dense model uses all of them per word; a mixture-of-experts (MoE) model holds many but uses only a few at a time — so 26B-A4B means "26B in memory, ~4B used per word." Purpose: total params set memory; active params set speed. → Phase 1.02
• Tool / function calling — the model can output a structured request like get_weather(city="Paris"); your code runs the function and feeds the result back. Use case: agents and automations. The model proposes; your app executes. → Phase 1.04
• Structured output — the model can be forced to return data matching a fixed shape (e.g. valid JSON with specific fields). Use case: extraction/pipelines where you need machine-readable output.
• Reasoning / thinking — the model can do hidden step-by-step thinking before answering (better on hard tasks, but slower and billed for the extra tokens). → Phase 2.09
• Multimodal (in vs out) — handles more than text (images/audio/video). What it can read in is listed separately from what it can produce out — "reads images" ≠ "makes images."
• Versioned ID vs alias — a versioned ID (gpt-4o-2024-08-06) is a fixed snapshot; an alias (gpt-4o-latest) silently points at newer versions. Use case: pin versioned IDs in production so behavior doesn't change under you. → Phase 1.08
• Weights / license — weights = the model's learned numbers; if they're downloadable the model is "open-weight." But open weights ≠ free to use commercially — the license decides that.

Four documents, four purposes

You'll constantly conflate these — don't. They're written by different parties, for different reasons, and are not updated in sync.

The model card tells you what the model can do; the provider page tells you what you'll pay and how to access it. Read both for any production decision.

What to extract from a model card

A model card is dense; read it with a target list (the full version is the checklist):

• Identity: exact model ID, family, variant (size/stage/modality), creator, release/updated date, versioned ID vs alias (Phase 1.08).
• Architecture & training: dense vs MoE (total/active params), context window, max output, training cutoff, post-training method, tokenizer.
• Capabilities: intended tasks; tool calling; structured output; reasoning; multimodal in/out; which benchmarks and what they actually measure (Phase 1.04, 1.07).
• Limitations: stated weaknesses, out-of-scope use, hallucination patterns, language coverage, knowledge cutoff.
• Safety: evals performed, red-teaming, refusal behavior, filters/classifiers (01).
• Deployment: API-only vs weights available; license (commercial?); vLLM/GGUF/Ollama compatibility; fine-tuning support; data retention.

The reading mindset

A model card is written by the seller: benchmarks are chosen to flatter, and "supports X" can mean "emits plausible X sometimes." Treat capability claims as a shortlist signal, then verify with your own eval (Phase 1.07). The most valuable sections are often the least promotional: limitations, out-of-scope use, and data policy.

3. Mental Model

FOUR DOCS, ONE DECISION:
  model card  → CAN it do my task? (capabilities, limits, training)
  system card → is it SAFE/compliant? (evals, policy)         [Doc 01]
  provider page → what does it COST & how do I access it?      (price, limits, flags)
  HF page     → can I RUN/host it? (weights, quant, license)   [Doc 04]

READING ORDER for a decision:
  Identity → Capabilities (filter) → Limits/Safety/Data policy (gates) →
  Provider page (cost/access) → YOUR EVAL (verify) → Selection Memo [Doc 07]

Cards are written by the SELLER → trust limits/policy, VERIFY capabilities.

4. Hitchhiker's Guide

What to read first: Identity (exact ID, version vs alias), the capability flags you need, then limitations and data-retention policy. These gate everything.

What to ignore at first: glossy benchmark tables — they shortlist, they don't decide.

What misleads beginners:

• Trusting the headline benchmark as task quality (chosen by the seller; may be contaminated).
• Assuming the provider page = the model card (context limits and feature flags often differ).
• Reading "supports tool calling" as "reliable tool calling" — measure it.
• Using an alias in production and being surprised when behavior shifts.

How experts reason: card → shortlist by capability + read limits/policy as hard gates → provider page for cost/access → own eval to decide → record a memo. They cross-check the card against the provider page and HF page because the three drift.

What matters in production: exact versioned ID, data-retention/residency policy, real (not claimed) capability reliability, license for your use, and a verified cost estimate.

How to verify: diff the model card's context/limits against the provider page; run a small eval for any capability you depend on; confirm the license permits your use.

Questions to ask: Is this the lab's card or community-written? Versioned or alias ID? What's the data-retention policy? Do provider feature flags match the card? What benchmarks, and what do they measure?

What silently gets expensive/unreliable: alias drift; provider context limits below the card's number; "supported" features that are flaky; restrictive licenses discovered late; data sent under a non-compliant retention policy.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• A model card is the lab's description of capabilities, training, and limits; a system card adds safety/eval/policy depth (01).
• Provider pages ≠ model cards and aren't synced — context limits, pricing, and feature flags can differ.
• Pricing is never on the model card — it lives on the provider page and changes often.
• Cards are written by the seller — benchmarks flatter; verify capabilities with your own eval.
• "Open weights" ≠ "open source" — read the license on the card/HF page (Phase 1.02).
• The most decision-relevant sections are often limitations, out-of-scope use, and data retention.
• Use versioned IDs, not aliases, in production.
• A good card read takes 5–10 minutes and prevents weeks of debugging.

9. Observations from Real Systems

• Google DeepMind publishes structured model cards (overview, training data, capabilities, limitations, evaluation) — Doc 02.
• Anthropic/OpenAI publish system cards emphasizing safety evals and deployment policy — Doc 01/03.
• Hugging Face pages mix lab cards, community cards, files/quantizations, and a community tab — Doc 04.
• Unsloth pages add quantized GGUF variants and hardware/benchmark claims that are environment-specific — Doc 05.
• OpenRouter / models.dev normalize cross-model comparison (price, context, flags) — the provider-page view (Phase 4).

10. Common Misconceptions

11. Engineering Decision Framework

Reading a card for a decision:
  1. IDENTITY: exact versioned ID, family, variant, release/updated.
  2. CAPABILITY FILTER: required features present? (tools/structured/reasoning/modality)
  3. GATES: limitations acceptable? data-retention/residency ok? license permits use?
  4. PROVIDER PAGE: real cost (in/out/cached), context limit, feature flags — does it match the card?
  5. VERIFY: run your eval on the capability you depend on (Phase 1.07).
  6. RECORD: write a model-selection memo (Doc 07).

If any GATE fails → reject regardless of benchmarks.
If card and provider page DISAGREE → trust the provider page for cost/limits, re-verify capability.

12. Hands-On Lab

Goal

Build model-card fluency by reading three cards from different publishers and producing a completed checklist + summary for each.

Prerequisites

• Browser access; the checklist and memo template.

Steps

1. A lab model card (e.g. a current Gemini or Llama card). Complete the checklist. Note knowledge cutoff and out-of-scope use.
2. A Hugging Face page for an open model (e.g. a Llama/Qwen Instruct). Complete the checklist; note whether the HF page agrees with the official lab card, and which GGUF/quant variants exist.
3. A provider page (OpenRouter/Bedrock) for one of those models. Record price (in/out/cached), context limit, and feature flags — and any disagreement with the model card.
4. For each model write: a 3-sentence "good for" summary and one use case where you would NOT use it.

Expected output

Three completed checklists, three summaries, and a list of card-vs-provider discrepancies.

Debugging tips

• Card vague on a field? Mark it "unspecified" — that gap is itself a finding (and a vendor question).
• Two context numbers? The provider often limits below the card; use the provider's.

Extension task

Turn the strongest candidate into a full model-selection memo with a cost estimate (reuse the Phase 1.08 calculator).

Production extension

Script a "card vs provider" diff: pull context/feature fields from a provider/catalog API and flag mismatches against the documented card values.

What to measure

Fields you could fill vs "unspecified"; number of card-vs-provider discrepancies; capability claims you'd want to verify by eval.

Deliverables

• 3 completed checklists + 3 summaries.
• A discrepancy list (card vs provider).
• One full selection memo for the top candidate.

13. Verification Questions

Basic

1. What's the difference between a model card, a system card, and a provider page?
2. Why isn't pricing on the model card?
3. Why prefer a versioned model ID over an alias?

Applied 4. The provider page lists a 32K context but the model card says 128K. Which do you trust for capacity planning, and why? 5. A card says "supports structured output." What do you do before relying on it?

Debugging 6. Production behavior changed with no deploy on your side. Which card-reading habit would have prevented the surprise? 7. You discover post-launch that the license forbids your commercial use. Where should you have caught it?

System design 8. Design a repeatable "read → gate → verify → memo" process for model adoption across a team.

Startup / product 9. For a regulated customer, which card sections become hard gates, and how do they shape your provider choice?

14. Takeaways

1. Four documents, four purposes: model card (capabilities), system card (safety), provider page (cost/access), HF page (run/host).
2. The card says what it can do; the provider page says what you'll pay — read both; they drift.
3. Cards are written by the seller — trust limits/policy, verify capabilities with your eval.
4. The gates (limitations, data retention, license) can reject a model regardless of benchmarks.
5. Pin versioned IDs, not aliases.
6. Produce a selection memo for every non-trivial model decision.

15. Artifact Checklist

• 3 completed model-card checklists (lab, HF, provider).
• 3 "good for / not for" summaries.
• Card-vs-provider discrepancy list.
• One full model-selection memo with cost estimate.
• (Stretch) card-vs-provider diff script.
• Notes: the four-document model and the read→gate→verify flow.

Next: 01 — How to Read System Cards