Phase 4 — Model Catalogs and Trend Reading

How to navigate thousands of fast-changing models without drowning: use catalogs to shortlist, read benchmarks and pricing skeptically, track releases/deprecations, and maintain one automated watchlist that feeds your decisions.

Why this phase matters

Phase 3 taught you to read a single model's documents. Phase 4 zooms out to the market: comparing many models efficiently and keeping current as it churns weekly. The output is operational — a catalog explorer, a release detector, a benchmark-skeptic's checklist, a cost/margin calculator, and a maintained watchlist — all feeding model selection (Phase 5) and the gateway (Phase 8).

Documents

#DocumentWhat you'll be able to do
00How to Use models.devFilter thousands of models to candidates; import the catalog
01Column-by-Column GuideRead every catalog column; flag the binding constraint
02How to Track Model ReleasesRun a tiered system; triage releases; catch deprecations
03How to Read BenchmarksRead benchmarks skeptically; decide with your eval
04How to Read Pricing PagesTurn a pricing page into cost-per-request and margin
05How to Build a Model WatchlistMaintain one automated, dated, eval-backed watchlist

How to work through it

Read in order — 00/01 are the catalog mechanics, 02–04 the skeptical reading skills, and 05 the capstone that ties them into a maintained artifact. The labs compound: the catalog importer (00) → column reader (01) → release diff (02) → benchmark + cost tools (03/04) → all feed the watchlist (05), which feeds the Phase 8 gateway.

Phase 4 artifacts

  • A models.dev → SQLite explorer + candidate shortlist
  • A read_row catalog reader that flags the binding constraint
  • A catalog-diff release/deprecation detector + 4-question triage
  • A trust-weighted benchmark reading + golden-set comparison
  • A pricing → cost/margin calculator with caching/reasoning/batch scenarios
  • A maintained watchlist.yaml + updater (feeding gateway routing/fallback)

Next

Phase 5 — Model Selection