How to Use models.dev (Model Catalogs)

Phase 4 · Document 00 · Model Catalogs and Trend Reading Prev: Phase 4 Index · Next: 01 — Column-by-Column Guide

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

There are thousands of models across dozens of providers, and the list changes weekly. A model catalog like models.dev is the fastest way to go from "thousands of models" to "the three that could do my job" — comparing context, price, features, and providers in one normalized table instead of crawling dozens of vendor pages. This is the practical entry point to model selection (Phase 5): you filter in the catalog, then verify on the provider page and decide with your own eval. The deep dives (columns, releases, benchmarks, pricing, watchlists) get their own docs; this one teaches the catalog as a tool.

2. Core Concept

Plain-English primer (what a "catalog" is and the words on it)

• Model catalog — a comparison table of models, like a price-comparison site for laptops, but the rows are LLMs and the columns are their specs. models.dev is the open one this phase uses; it also exposes a JSON API so you can query it in code.
• Lab vs provider — the lab made the model (Meta, OpenAI); a provider serves it over an API (OpenAI, Bedrock, OpenRouter). The same model can have many providers at different prices (Phase 1.08).
• Context / output — max input tokens it can read vs max output tokens it can generate (often different numbers). → 1.01
• Feature flags — yes/no columns: tool calling, structured output, reasoning, input modalities. These are filters you apply before comparing quality. → 1.04
• Weights — whether you can download and self-host the model (open-weight) vs API-only. → 1.02
• Price — input / output / cached cost per 1 million tokens. → 1.08
• Release / Updated — when the model launched, and when the catalog row was last refreshed (a staleness signal).

What a catalog is for (and what it isn't)

A catalog normalizes scattered metadata into one comparable shape. Use it to:

• Filter to candidates by hard requirements (context ≥ X, tool calling = yes, open-weight, price ≤ Y).
• Discover which providers host a model and at what price/latency.
• Spot new releases and changed pricing (the Updated date).

It is not ground truth for production. Catalogs lag provider docs — pricing and limits especially. The rule: catalog filters → provider page verifies → your eval decides (Phase 1.07). And capability flags say "supported," not "reliable" — confirm by testing.

The JSON API (why this is an engineering tool, not just a website)

models.dev publishes machine-readable data (e.g. https://models.dev/api.json). That means you can pull it into SQLite/Postgres and build your own filters, watchlists, and routing rules — which is exactly the lab below and a building block for the Phase 8 gateway's model registry.

3. Mental Model

THOUSANDS of models ──[catalog filters]──► a handful of CANDIDATES
                          (context, price, features, weights, provider)
                                   │
                                   ▼
                        provider page  →  VERIFY cost/limits/flags
                                   │
                                   ▼
                          your EVAL    →  DECIDE  →  selection memo (Phase 3.07)

Catalog = FAST SHORTLIST, not the final word.
Rule:  catalog filters · provider page verifies · your eval decides.
The "Updated" date is a freshness warning — recheck before production.

4. Hitchhiker's Guide

What to do first: apply your hard filters (context, required features, open vs API, price ceiling) to get a short candidate list — don't browse by vibes.

What to ignore at first: an overall "ranking." Catalogs aren't leaderboards; fit beats rank.

What misleads beginners:

• Trusting catalog pricing/limits for production (they lag — verify on the provider page).
• Reading a feature flag as "works reliably" (it means "offered" — test it).
• Sorting by cheapest input price alone (a cheap model that needs more tokens or retries can cost more — compare cost per resolved task).
• Confusing the Lab column with the Providers column.

How experts reason: filter to candidates in the catalog → open each candidate's provider page to confirm real price/limits/flags → run a small eval → record a memo. They also pull the JSON API to automate this for routing.

What matters in production: verified (not catalog-listed) pricing/limits, confirmed capability reliability, multiple providers for fallback, and the Updated/Release dates to catch drift and deprecation.

How to verify: cross-check 2–3 catalog fields (price, context, a feature flag) against the provider's own docs; if they disagree, trust the provider.

Questions to ask: When was this row updated? Which providers host it and at what price? Is the feature flag grammar-enforced or best-effort? Open-weight under what license?

What silently gets expensive/unreliable: stale catalog prices; picking on headline price not cost-per-task; assuming a flag = reliability; missing that a model is deprecated despite being listed.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• A catalog is a fast shortlist tool, not production ground truth — verify on the provider page.
• Catalog data lags provider docs, especially pricing and rate limits.
• A feature flag means "offered," not "reliable" — confirm by testing.
• The same model has multiple providers at different prices/latency — catalogs surface this.
• models.dev exposes a JSON API, so you can build your own filters/watchlists/routing.
• Lab ≠ provider — don't conflate the creator and the host.
• Compare cost per resolved task, not just headline input price.
• The Updated/Release dates catch drift and possible deprecation.

9. Observations from Real Systems

• models.dev is the de-facto open catalog: columns for lab, providers, context, output, modality, reasoning, tools, structured, weights, price, release/updated (Phase 4.01).
• OpenRouter is effectively a live catalog + gateway: it shows per-provider price/latency and lets you route across them (Phase 8).
• Artificial Analysis / LMArena add independent quality/speed/price indices to cross-check catalog claims (Phase 4.03).
• Gateways' model registries (Phase 8) are essentially a private catalog table — the lab below is its seed.
• Catalog vs provider drift is real: a price or context number on a catalog can trail a provider change by days.

10. Common Misconceptions

11. Engineering Decision Framework

Use the catalog in this order:
  1. HARD FILTERS: context ≥ need · required feature flags = yes · open/API · price ≤ ceiling.
  2. CANDIDATES: keep the few survivors (note their providers).
  3. VERIFY: open each provider page — confirm price/limits/flags (catalog can be stale).
  4. EVAL: run your task-specific eval on survivors (Phase 1.07).
  5. DECIDE + MEMO: pick + fallback; record a selection memo (Phase 3.07).

Automate it (lab): pull the JSON API → SQLite → your own filter/watchlist queries.

12. Hands-On Lab

Goal

Import the models.dev catalog into SQLite and build a filterable explorer — the seed of a model registry.

Prerequisites

• Python 3.10+; internet to fetch the catalog.

Setup

curl https://models.dev/api.json -o models.json

Steps

import json, sqlite3
data = json.load(open("models.json"))

conn = sqlite3.connect(":memory:")
conn.execute("""CREATE TABLE models(
  id TEXT, name TEXT, lab TEXT, context INT, output INT,
  tools INT, structured INT, reasoning INT,
  in_price REAL, out_price REAL, weights INT, release TEXT, updated TEXT)""")

# NOTE: adapt these .get() keys to the ACTUAL JSON shape you downloaded —
# inspect data structure first (print a sample row) since schemas evolve.
def rows(d):
    items = d.values() if isinstance(d, dict) else d
    for m in items:
        yield (m.get("id"), m.get("name"), m.get("lab") or m.get("provider"),
               m.get("context"), m.get("output"),
               int(bool(m.get("tool_call"))), int(bool(m.get("structured"))),
               int(bool(m.get("reasoning"))), m.get("input_price"), m.get("output_price"),
               int(bool(m.get("weights") or m.get("open_weights"))),
               m.get("release"), m.get("updated"))

conn.executemany("INSERT INTO models VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?)", rows(data))

# Filter: cheap, long-context, tool-capable
for r in conn.execute("""SELECT id, lab, context, in_price, out_price
                         FROM models
                         WHERE context >= 100000 AND tools = 1 AND in_price <= 1.0
                         ORDER BY in_price ASC LIMIT 10"""):
    print(r)

Expected output

• A short list of catalog models meeting your filters — your candidate set.

Debugging tips

• Empty/odd results? The JSON schema differs from the placeholder keys — print one record first and map fields accordingly (catalog schemas change; this is the Phase 0.02 "verify primary source" lesson).
• Prices missing for some rows? Catalogs don't have every field for every model — handle None.

Extension task

Add a CLI with flags (--min-context, --max-price, --tools, --open-weights, --lab) and a "cost per request" column given assumed in/out token counts.

Production extension

Turn this table into a model registry seed for the Phase 8 gateway (add cached_price, providers, supports_streaming), and a nightly refresh that flags new/changed rows (→ Phase 4.05 watchlist).

What to measure

Candidate count per filter set; catalog-vs-provider discrepancies for 2 fields on 2 models.

Deliverables

• A working SQLite catalog + filter queries (or CLI).
• A candidate shortlist for one real requirement set.
• A 2-field catalog-vs-provider discrepancy note.

13. Verification Questions

Basic

1. What is a model catalog, and what's it best used for?
2. What's the difference between the Lab and Providers columns?
3. Why isn't catalog pricing safe to trust for production?

Applied 4. You need ≥100K context + tool calling at ≤ $1/1M input. How do you find candidates and then confirm them? 5. Why might the "cheapest" model by input price not be the cheapest in practice?

Debugging 6. Your import returns nonsense rows. What's the first thing to check, and why? 7. A model is in the catalog but your API calls 404. What might explain it?

System design 8. Design a nightly job that ingests the catalog and alerts on new models or price changes relevant to your stack.

Startup / product 9. How does owning a catalog-backed model registry help your unit economics and reliability (routing, fallback, cost tracking)?

14. Takeaways

1. A catalog turns thousands of models into a few candidates via hard filters.
2. Catalog filters → provider page verifies → your eval decides.
3. Catalogs lag provider docs (especially pricing) — always verify before production.
4. Feature flags mean "offered," not "reliable" — test them.
5. The JSON API makes the catalog programmable (registry, watchlist, routing).
6. Compare cost per resolved task, not headline price.

15. Artifact Checklist

• Code: models.dev → SQLite importer + filter queries/CLI.
• Candidate shortlist for one real requirement set.
• Discrepancy note: catalog vs provider for ≥2 fields.
• (Stretch) registry seed with cost-per-request column.
• Notes: the filter→verify→decide flow.
• Selection memo for the chosen candidate (Phase 3.07).

Next: 01 — Column-by-Column Guide