Cheatsheet 02 — Reading Model Cards & System Cards

What to extract, in what order, and the red flags. Full: Phase 3.

What a model card answers (read in this order)

1. What is it? Base vs instruct/chat vs reasoning; size (params, active vs total for MoE); modality.
2. Context window — max input+output tokens. (And max output separately — often smaller.)
3. Training data & cutoff — knowledge cutoff date; domains; languages.
4. Intended use & limits — what it's for, what it's not for.
5. Capabilities — tool calling? structured output? reasoning? vision?
6. Evaluations — which benchmarks, which numbers (and whether contaminated/self-reported).
7. License & weights — open weights? commercial use? restrictions.
8. Safety — refusals, red-team results, known failure modes (system cards go deep here).

Model card vs system card

The numbers to write down (for selection)

• Context window (and max output).
• Price (input $/Mtok, output $/Mtok — output is usually pricier).
• Knowledge cutoff.
• Capabilities flags: tool calling, structured output, reasoning, vision.
• Open vs closed weights + license terms.
• Latency/throughput class (if stated).

Red flags 🚩

• Self-reported benchmarks only, no methodology → discount them; run your own eval.
• No knowledge cutoff stated → unknown staleness.
• Vague license ("research only", custom terms) → check commercial use (cheatsheet 05).
• Benchmark contamination not addressed → numbers may not transfer.
• Max output ≪ context window → can't generate long outputs even with big context.
• "State of the art" with no comparison baseline → marketing.

Interview line

"I never select on benchmark numbers alone — model cards shortlist; my own task eval + cost/latency decide. From a card I extract context, price, capabilities, cutoff, and license, then verify claims on my golden set." (Phase 5)

Next: 03 — models.dev Columns · Full: Phase 3