How to Track Model Releases

Phase 4 · Document 02 · Model Catalogs and Trend Reading Prev: 01 — Column-by-Column Guide · Next: 03 — How to Read Benchmarks

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

New models drop weekly, and the hype-to-signal ratio is brutal. Two failure modes hurt teams: missing a release that would've cut cost or improved quality, and chasing every release into churn and instability. The skill is a lightweight, tiered tracking system that surfaces what matters and filters the noise — without you doom-scrolling. This operationalizes Phase 0.02 ("invariants vs specifics; primary sources; dated notes") into a concrete release-tracking workflow, and feeds the watchlist tooling in 05.

2. Core Concept

Plain-English primer

• Release vs deprecation — a release is a new (or updated) model; a deprecation is a model being retired on a schedule. You must track both — a deprecation can break production silently if you used an alias.
• Primary vs secondary sources — primary = the lab/provider's own changelog/blog/model card; secondary = aggregators, newsletters, social posts. Primary is authoritative; secondary is for discovery/triage (Phase 0.02).
• Changelog / release notes — the official feed where a provider announces new models, price changes, and deprecations. The single highest-signal source.
• Snapshot vs alias — a snapshot/versioned ID (...-2026-04) is fixed; an alias (...-latest) silently moves to new releases. Tracking releases is partly about deciding when to move pinned IDs.

A tiered tracking system (the whole method)

Don't follow everything equally. Triage by importance to you:

• Tier 1 — follow closely (push): the labs/providers you run in production. Subscribe to their changelogs/deprecation notices and blogs. A deprecation here is an incident risk.
• Tier 2 — check periodically (pull, ~weekly/monthly): catalogs and aggregators that surface new models fast — models.dev sorted by Release, OpenRouter's new-models list, Hugging Face trending, an independent index (Artificial Analysis), 1–2 high-signal newsletters.
• Tier 3 — as-needed: specific labs you might adopt (Mistral, Qwen, Cohere, DeepSeek), papers, social threads (pointers only — verify before believing, see Phase 0.02).

What to actually evaluate in a new release (the triage checklist)

A release matters to you only if it changes a decision. Ask:

1. Does it beat what I run today on my axis (quality on my eval, price, context, a feature I needed)?
2. Is it real or marketing? Benchmarks are gameable (03) — discount headline claims; verify on your eval.
3. Is it production-ready? Day-1 models often have shifting prices, rate limits, and bugs; wait for stability unless you have a reason to be first.
4. What would adopting it cost (migration, re-eval, regression risk) vs the gain?

Most releases fail step 1 for your use case — and that's fine. The system's job is to make that judgment in minutes.

3. Mental Model

TIER 1 (push)  ─ providers you RUN: changelogs + DEPRECATION notices   → incident risk
TIER 2 (pull)  ─ catalogs/aggregators (models.dev, OpenRouter, HF, an index, 1-2 newsletters) → weekly/monthly skim
TIER 3 (ad hoc)─ specific labs, papers, social (pointers only)         → when considering adoption

For any release, run the 4-question triage:
  beats what I run on MY axis?  →  real or marketing?  →  production-ready/stable?  →  adoption cost vs gain?
  most releases stop at Q1 — log it, move on.

Track DEPRECATIONS as carefully as releases. Pin snapshot IDs; plan migrations.

4. Hitchhiker's Guide

What to set up first: subscriptions to the changelogs of the providers you run in production (Tier 1) — including deprecation notices. That's the part that prevents outages.

What to ignore: the daily social-media model hype. Let Tier 2 surface what's worth a look on your schedule.

What misleads beginners:

• Reacting to launch-day benchmark claims (gameable; not yet stable).
• Tracking everything equally (burnout, no signal).
• Forgetting deprecations — the silent production breaker, especially with aliases.
• Adopting on day 1 and inheriting price/limit/quality instability.

How experts stay current: push alerts for production providers, a brief weekly pull-skim of catalogs/indices, and the 4-question triage. They keep dated notes so "is this still true?" is answerable, and they pin snapshot IDs so they migrate deliberately.

What matters in production: never miss a deprecation; re-eval before swapping models; pin versioned IDs; have a migration checklist.

How to verify a release claim: go to the primary changelog/model card, then run it on your eval before believing "beats X."

Questions to ask: Is there a changelog/RSS and a deprecation policy + notice period? Snapshot IDs available? Has this model's price/limits stabilized since launch?

What silently gets expensive/unreliable: surprise deprecations; chasing releases into constant migrations; trusting day-1 benchmarks; alias drift moving you onto a new model unannounced.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Track deprecations as carefully as releases — a retirement can break production, especially via aliases.
• Primary sources (changelogs/cards) are authoritative; social/aggregators are for discovery and triage.
• Day-1 models are often unstable (shifting price/limits/bugs) — wait for stability unless you must be first.
• A release only matters if it changes a decision — most don't, for your use case.
• Benchmark claims at launch are marketing-adjacent — verify on your eval (03).
• Pin snapshot/versioned IDs so you migrate on purpose, not by surprise.
• A tiered system (push/pull/ad-hoc) beats following everything equally.
• Dated notes make "is this still current?" answerable later.

9. Observations from Real Systems

• OpenAI/Anthropic/Google publish changelogs and deprecation schedules — the Tier-1 feeds that prevent surprise breakage.
• models.dev / OpenRouter / Hugging Face surface new models within days — ideal Tier-2 pull sources.
• Artificial Analysis / LMArena provide independent comparisons to sanity-check launch hype (03).
• Alias-driven incidents happen when a -latest alias rolls to a new model and behavior shifts under a team that didn't pin a snapshot.
• Gateways (Phase 8) centralize model versions so a deprecation is a one-place migration, not a hunt across services.

10. Common Misconceptions

11. Engineering Decision Framework

Set up the system (once):
  Tier 1: subscribe to changelogs + deprecation notices for providers you RUN.
  Tier 2: bookmark models.dev (by Release), OpenRouter new, HF trending, 1 index, 1-2 newsletters.
  Tier 3: note labs/papers to watch.

On a new release (4-question triage):
  1. Beats what I run on MY axis (eval/price/context/feature)?  No → log + stop.
  2. Real or marketing?  → check primary source; plan an eval.
  3. Stable/production-ready?  → if day-1, usually WAIT.
  4. Adoption cost (migration + re-eval + regression risk) < gain?  → if yes, pilot.

On a deprecation notice:
  identify affected pinned IDs → pick replacement → re-eval → migrate before the deadline.

12. Hands-On Lab

Goal

Build a release-detector: diff today's catalog against yesterday's to surface new and disappeared (possibly deprecated) models, and run the 4-question triage on one new entry.

Prerequisites

• The models.dev importer from 00; two snapshots of the catalog (or fetch twice over time).

Steps

import json

def load_ids(path):
    d = json.load(open(path))
    items = d.values() if isinstance(d, dict) else d
    return {m["id"]: m for m in items if m.get("id")}

old = load_ids("models_prev.json")   # yesterday's snapshot
new = load_ids("models.json")        # today's

added   = [new[i] for i in new.keys() - old.keys()]
removed = [old[i] for i in old.keys() - new.keys()]   # candidates for deprecation
changed = [(i, old[i].get("input_price"), new[i].get("input_price"))
           for i in new.keys() & old.keys()
           if old[i].get("input_price") != new[i].get("input_price")]

print("NEW models:", [m["id"] for m in added])
print("GONE (check deprecation):", [m["id"] for m in removed])
print("PRICE CHANGES:", changed)

Expected output

• Lists of newly added models, vanished models (deprecation candidates), and price changes — your tracking signal, automatically.

Debugging tips

• No prior snapshot? Save today's as models_prev.json, fetch again tomorrow, then diff. (Or simulate by deleting a few rows.)
• "GONE" can also mean a catalog hiccup — confirm against the provider's deprecation page before acting.

Extension task

For one added model, fill the 4-question triage in a markdown note (beats-my-current? real? stable? worth adopting?), citing the primary changelog.

Production extension

Schedule the diff (cron/CI) to post new/removed/price-changed models to Slack/email — a real release-and-deprecation alert feeding your watchlist.

What to measure

Count of new/removed/changed models between snapshots; how many added survive Q1 of the triage.

Deliverables

• A catalog-diff release detector.
• A completed 4-question triage for one new model.
• A 3-source Tier-1 subscription list (with deprecation feeds).

13. Verification Questions

Basic

1. Why must you track deprecations, not just releases?
2. What's the difference between a primary and a secondary source here?
3. Why is a snapshot ID safer in production than an alias?

Applied 4. A new model claims "beats GPT-X." Walk through your 4-question triage. 5. Design Tier-1 vs Tier-2 sources for a team running Claude + a self-hosted Llama.

Debugging 6. Production output changed with no deploy. Which tracking gap likely caused it, and how do you prevent recurrence? 7. You adopted a day-1 model and hit shifting rate limits. What rule did you skip?

System design 8. Design an automated job that alerts on new models and deprecations relevant to your stack and feeds a watchlist.

Startup / product 9. How do you balance "use the best new model" against product stability and unit-economics predictability?

14. Takeaways

1. Use a tiered system: push alerts for providers you run, periodic pulls for discovery, ad-hoc for the rest.
2. Track deprecations as carefully as releases — and pin snapshot IDs.
3. Run the 4-question triage: beats-my-current? real? stable? worth the adoption cost?
4. Primary sources are authoritative; verify launch claims on your eval.
5. Don't chase day-1 releases unless you have a reason — wait for stability.
6. Automate detection (catalog diff) and keep dated notes.

15. Artifact Checklist

• Tier-1/2/3 source list (with deprecation feeds).
• Code: catalog-diff release/deprecation detector.
• 4-question triage completed for one new model.
• (Stretch) scheduled alert to Slack/email.
• Migration checklist for handling a deprecation.
• Dated notes entry for one tracked model.

Next: 03 — How to Read Benchmarks