Column-by-Column Catalog Guide

Phase 4 · Document 01 · Model Catalogs and Trend Reading Prev: 00 — How to Use models.dev · Next: 02 — How to Track Model Releases

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

A catalog row packs a model's whole identity into ~13 columns of shorthand. If you can read every column and know which decision it drives, you can screen a model in 30 seconds and avoid the classic mistakes: filtering on the wrong context number, confusing creator with host, or treating a feature checkmark as a guarantee. This doc turns each column into a concrete engineering question and the action it implies — making 00's filtering precise and feeding the registry schema in Phase 8.

2. Core Concept

Plain-English primer (every column, from zero)

Each column answers one question. Here's the full set with plain meaning, the decision it drives, and the gotcha.

The two distinctions people get wrong

1. Context vs Output are different limits. A "1M context" model might only generate 8K tokens. If you need a long report, the Output column gates you, not Context.
2. Lab vs Providers. "Llama" (lab: Meta) is served by many providers; "GPT" (lab: OpenAI) may be served by OpenAI and Azure. Pick the provider for cost/latency/region, having chosen the model for capability.

Reading flags as filters, not guarantees

Tool Call / Structured / Reasoning / modality are boolean filters: use them to exclude models you can't use, then verify the survivors ("supported" ≠ "reliable enough to ship"). This two-step — filter then verify — is the whole skill.

3. Mental Model

A CATALOG ROW = 13 answers to 13 questions. Group them:

  IDENTITY   Model-ID · Lab · Providers     → "what exactly, by whom, served where?"
  CAPACITY   Context · Output                → "does my input fit / can it write enough?"  (DIFFERENT limits!)
  FEATURES   Input · Reasoning · ToolCall · Structured · Temperature → boolean FILTERS (then verify)
  ECONOMICS  Weights · Price                 → "self-host vs API; what does it cost?"
  FRESHNESS  Release · Updated               → "current? data stale?"

Use IDENTITY+CAPACITY+FEATURES to FILTER → ECONOMICS to compare → FRESHNESS to sanity-check.

4. Hitchhiker's Guide

What to read first: Context and Output (separately), the feature flags you require, and Weights (open vs API). These decide candidacy.

What to ignore at first: Release date as a quality proxy — verify quality by eval, not recency.

What misleads beginners:

• Filtering on Context when the binding limit is Output.
• Treating a checkmark as a reliability guarantee.
• Comparing models by input price alone (ignore output price and token efficiency at your peril).
• Conflating Lab and Providers.

How experts reason: they map each column to a question, filter on the hard ones, then verify the soft ones (flags, price) on the provider page before trusting them.

What matters in production: the versioned Model ID, verified Context/Output for your shape of work, confirmed feature reliability, and ≥2 Providers for fallback.

How to verify: pick a candidate and confirm Context, Output, one feature flag, and price against the provider's own docs; note any mismatch.

Questions to ask: Is this the versioned ID? Output limit (not just context)? Which providers + prices? Is the flag grammar-enforced? License for the weights?

What silently gets expensive/unreliable: assuming Output = Context; ignoring cached-input pricing; trusting a flag; using an alias that drifts.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Context and Output are different limits — long-report use cases are gated by Output.
• Lab ≠ Providers — choose the model for capability, the provider for cost/latency/region.
• Feature flags are filters, not guarantees — verify reliability before shipping.
• Use the versioned Model ID, never the friendly name or a -latest alias, in production.
• Output usually costs 2–5× input; check the cached price too.
• "Weights = open" doesn't mean commercial-use-free — check the license.
• The Updated date is a staleness flag — recheck old rows on the provider page.
• A single model often has multiple providers with different prices/latency — gold for fallback.

9. Observations from Real Systems

• models.dev columns map almost 1:1 to a gateway model registry schema (Phase 8, Lab 2) — Context, Output, supports_tools, supports_structured, prices, weights.
• OpenRouter makes the Providers column tangible: one model, a list of providers with live price/latency to route across.
• Output-limit surprises are common: teams pick a huge-context model for long generation, then hit a small Output cap.
• Catalog vs provider drift shows up most in the Price column — always reconcile.
• Capability-flag overconfidence (shipping an agent on a "tool-calling" model that's flaky) is a recurring incident the verify-step prevents.

10. Common Misconceptions

11. Engineering Decision Framework

Screen a row in 30 seconds:
  IDENTITY  → versioned ID? which providers (≥2 for fallback)?
  CAPACITY  → Context ≥ my input AND Output ≥ my answer length?
  FEATURES  → all required flags present? (then mark "verify")
  ECONOMICS → open-weight if I need self-host; price (in/out/cached) in budget?
  FRESHNESS → Updated recent? Release not deprecated?
  → survivors go to the provider page for verification, then eval.

Pick the column that's your binding constraint and sort by it:
  long reports → sort by Output ; agents → filter Tool Call ; private → filter Weights ; cost → cost/task.

12. Hands-On Lab

Goal

Build a "row reader": given a catalog entry, output the 13 answers, flag the binding constraint, and list what to verify on the provider page.

Prerequisites

• The SQLite catalog from 00's lab (or the raw api.json).

Steps

def read_row(m, need_context, need_output, required_flags):
    report = {
        "identity": {"id": m["id"], "lab": m.get("lab"), "providers": m.get("providers")},
        "capacity": {"context": m.get("context"), "output": m.get("output"),
                     "context_ok": (m.get("context") or 0) >= need_context,
                     "output_ok": (m.get("output") or 0) >= need_output},
        "features": {f: bool(m.get(f)) for f in required_flags},
        "economics": {"weights": bool(m.get("weights")), "in": m.get("input_price"), "out": m.get("output_price")},
        "freshness": {"release": m.get("release"), "updated": m.get("updated")},
    }
    report["binding_constraint"] = (
        "output" if not report["capacity"]["output_ok"]
        else "context" if not report["capacity"]["context_ok"]
        else "features" if not all(report["features"].values())
        else "none — verify price & flags on provider page")
    report["verify_on_provider"] = ["price(in/out/cached)", "context", "output"] + list(required_flags)
    return report

# Example
m = {"id":"example-1","lab":"ExampleAI","providers":["ExampleAI","OpenRouter"],
     "context":200000,"output":8192,"tool_call":True,"structured":True,
     "input_price":0.2,"output_price":0.6,"weights":False,"release":"2026-04","updated":"2026-06"}
import json; print(json.dumps(read_row(m, need_context=100000, need_output=16000,
                                       required_flags=["tool_call","structured"]), indent=2))
# → binding_constraint: "output" (needs 16k, model caps at 8192)

Expected output

• A structured read of the row that correctly flags Output as the binding constraint in the example (16K needed > 8192 cap) — exactly the trap this doc warns about.

Debugging tips

• All *_ok True but you still fail in practice? The constraint was a feature reliability issue — that's why verify_on_provider lists flags.
• Field missing? Map to the real catalog keys (inspect a record first).

Extension task

Run read_row across your whole SQLite catalog for one requirement set and print only rows where binding_constraint == "none..." — your verified-candidate shortlist.

Production extension

Emit the report as the input to a model registry insert (Phase 8), tagging each candidate with "verified: false" until a provider-page check flips it true.

What to measure

How many models pass capacity vs features; how often Output (not Context) is the binding constraint in your filters.

Deliverables

• A read_row tool + a candidate shortlist for one requirement set.
• A note: in your filter, what was the most common binding constraint?

13. Verification Questions

Basic

1. What's the difference between the Context and Output columns?
2. What does the Lab column tell you that the Providers column doesn't?
3. What does a checkmark in the Tool Call column actually promise?

Applied 4. You need to generate 20-page reports. Which column gates you, and how do you filter? 5. Two providers serve the same model at different prices — what do you do, and why is that good for reliability?

Debugging 6. A model "passed" all catalog filters but your agent is unreliable. Which column did you over-trust? 7. Your cost is double the estimate from the Price column. Name two likely reasons.

System design 8. Map the catalog columns to a model-registry table schema for a gateway. Which become filters, which become routing inputs?

Startup / product 9. Explain how reading Context vs Output correctly avoids a feature that "demos fine but truncates real reports."

14. Takeaways

1. Each catalog column maps to one decision — read them as questions.
2. Context ≠ Output — pick the binding limit for your workload.
3. Lab ≠ Providers — capability vs cost/latency/region.
4. Feature flags are filters, then verify — "offered" isn't "reliable."
5. Use the versioned Model ID; weigh output + cached price, not just input.
6. The Updated date is your staleness alarm.

15. Artifact Checklist

• Code: read_row catalog reader flagging the binding constraint.
• Candidate shortlist for one requirement set.
• Column→registry schema mapping (for Phase 8).
• Notes: the IDENTITY/CAPACITY/FEATURES/ECONOMICS/FRESHNESS grouping.
• Verify list: which fields you'd confirm on the provider page.
• Finding: most common binding constraint in your filters.

Next: 02 — How to Track Model Releases