How to Build a Model Watchlist

Phase 4 · Document 05 · Model Catalogs and Trend Reading Prev: 04 — How to Read Pricing Pages · Up: Phase 4 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

This doc is the capstone of Phase 4: it ties together the catalog (00/01), release tracking (02), benchmark skepticism (03), and pricing (04) into one automated, dated artifact you actually maintain. A watchlist answers, on demand: which models do I run, which are my fallbacks, which am I evaluating, and what changed since last week? Without it you either fly blind (surprise deprecations, missed cheaper models) or drown in noise. With it, staying current becomes a 20-minute monthly ritual instead of constant anxiety — the practical embodiment of Phase 0.02.

2. Core Concept

Plain-English primer

• Watchlist — a small, maintained table of the models you care about (run, back up, or are considering) with their key specs and a dated note. Like a stock watchlist, but for LLMs.
• Production / fallback / candidate / archive — the four states a watched model can be in: actively serving, a backup to route to on failure, under evaluation, or retired (kept for history).
• Diff / change detection — automatically comparing today's catalog/prices against yesterday's to surface what changed (02 lab).
• Dated note — every fact stamped with "as of <date>" so staleness is visible (Phase 0.02).

What a good watchlist contains

For each model you track:

• Identity: versioned model ID, lab, provider(s).
• Specs (dated): context, output, key feature flags, input/output/cached price.
• State: production | fallback | candidate | archived.
• Your eval result: score on your golden set (Phase 1.07) — the number that actually decides.
• Watch reasons: what would make you act (cheaper alternative, deprecation, a needed feature).
• Dates: released, last-verified, deprecation (if announced).

What a watchlist is for

1. Don't get surprised — deprecations and price hikes on production models trigger alerts.
2. Don't miss upgrades — new models that beat your current on your axis surface for evaluation.
3. Make decisions fast — when you need to choose/route, the comparison is already there, eval-backed.
4. Feed the gateway — the watchlist is the human-curated front-end of your gateway's model registry and routing/fallback config (Phase 8).

The maintenance loop (why automation matters)

A watchlist only works if it stays current. The pattern: automate detection (catalog/price diff from 02) → human triage (the 4-question test from 02) → re-eval candidates on your golden set → update states + dated notes. Run detection daily (cheap, automated); do triage/eval on a monthly cadence (or when an alert fires).

3. Mental Model

SOURCES                       WATCHLIST (your maintained table)                ACTIONS
catalog/api (00,01) ─┐        ┌─────────────────────────────────────┐
release/deprec (02) ─┼─diff──►│ id · lab · provider · specs(dated)   │──┬─► production: monitor cost/deprecation
benchmarks (03)      │        │ price · STATE · MY-eval · watch-why  │  ├─► fallback: keep ready, re-verify
pricing (04)        ─┘        │ dates: released/verified/deprecated  │  ├─► candidate: run my eval → promote/drop
                              └─────────────────────────────────────┘  └─► archived: retired, kept for history
        AUTOMATE detection (daily) → HUMAN triage (4 questions) → RE-EVAL → UPDATE states + dated notes (monthly)
                              │
                              ▼
                   feeds the Phase 8 gateway: model registry + routing/fallback config

4. Hitchhiker's Guide

What to build first: a 5–15 row table of the models you run + their fallbacks, with versioned IDs, dated specs, and your eval scores. Small and maintained beats big and stale.

What to ignore: tracking hundreds of models. Track what you run, back up, or seriously consider — the rest lives in the catalog when you need it.

What misleads beginners:

• Building a huge static list that's never updated (stale = worse than nothing).
• Tracking benchmark scores instead of your eval scores.
• Omitting deprecation dates (the surprise that breaks production).
• Manual-only updates (you'll stop; automate detection).

How experts run it: automated daily diff for new/changed/removed models + price changes → alerts → monthly human triage and re-eval → states and dated notes updated → the result drives gateway routing/fallback.

What matters in production: the watchlist is the source of truth for what you run and fall back to, eval-backed, with deprecation alerts wired up.

How to verify it's working: a price hike or deprecation on a production model should reach you before it bites; a clearly-better-and-cheaper model for your task should surface for eval within a cycle.

Questions to ask: Is every production model's deprecation tracked? Are states current? Is each candidate scored on my eval? Does the watchlist feed routing/fallback?

What silently gets expensive/unreliable: a stale watchlist (false confidence); tracking the wrong signal (benchmarks not your eval); no automation (it rots); no link to the gateway (decisions don't propagate).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• A watchlist tracks the models you run, back up, or are considering — not the whole market.
• It records your eval scores, not just benchmarks — that's the decisive number.
• Deprecation dates belong in the watchlist — the top cause of surprise outages.
• Automate detection, human-triage decisions — a manual-only watchlist rots.
• Date every fact so staleness is visible (Phase 0.02).
• The watchlist is the human front-end of the gateway's model registry and routing/fallback (Phase 8).
• Small and current beats large and stale.
• A monthly 20-minute ritual keeps it alive; daily automated diffs do the watching.

9. Observations from Real Systems

• Gateways (LiteLLM/OpenRouter-style) carry a model list + fallback config that is an executable watchlist — the doc here is its curated source (Phase 8).
• Teams that pin versioned IDs + track deprecations migrate calmly; those on aliases get surprised (Phase 4.02).
• models.dev/OpenRouter diffs are the practical detection feed for new/changed models and prices.
• Eval-backed watchlists (each candidate scored on a golden set) make "should we switch?" a data question, not a debate (Phase 1.07).
• The trending-LLMs reference (references/trending-llms-landscape) is a broad market snapshot; your watchlist is the narrow, personal operational version.

10. Common Misconceptions

11. Engineering Decision Framework

Build it:
  1. SEED with models you run + fallbacks + serious candidates (versioned IDs, dated specs, MY eval score).
  2. STATE each: production / fallback / candidate / archived.
  3. WATCH-WHY per row: what change would make me act (deprecation, cheaper, feature).

Maintain it (loop):
  DAILY (automated): diff catalog/prices (Phase 4.02) → alert on new/changed/removed + price moves + deprecations.
  ON ALERT / MONTHLY (human): 4-question triage (Phase 4.02) → re-eval candidates on golden set (Phase 1.07)
                              → update states + dated notes.

Use it:
  decisions & routing/fallback (Phase 8) read from the watchlist; production rows monitored for cost/deprecation.

12. Hands-On Lab

Goal

Create a maintained watchlist file + an updater that ingests the catalog diff, flags changes on your tracked models, and prints what needs human attention.

Prerequisites

• The 00 importer and 02 diff; a golden set from Phase 1.07.

Steps

1. Create watchlist.yaml:

# watchlist.yaml — dated; small and maintained
updated: 2026-06-13
models:
  - id: example-instruct-8b-2026-04
    lab: ExampleAI
    providers: [ExampleAI, OpenRouter]
    state: production            # production | fallback | candidate | archived
    context: 128000
    output: 8192
    price: {in: 0.20, out: 0.60, cached: 0.05}   # as of 2026-06-13
    my_eval: 0.91                # score on MY golden set
    watch_why: "cheaper alt; watch for deprecation"
    deprecation: null
  - id: claude-haiku-...           # fallback example
    state: fallback
    my_eval: 0.93
    watch_why: "primary fallback on 429/timeout"

2. Write the updater:

import yaml, json
wl = yaml.safe_load(open("watchlist.yaml"))
catalog = {m["id"]: m for m in (json.load(open("models.json")) or [])}  # adapt to real shape

alerts = []
for m in wl["models"]:
    cat = catalog.get(m["id"])
    if cat is None:
        alerts.append(f"⚠ {m['id']}: NOT in catalog — check deprecation")
        continue
    if cat.get("input_price") not in (None, m["price"]["in"]):
        alerts.append(f"💲 {m['id']}: input price {m['price']['in']} → {cat.get('input_price')}")
    if m.get("deprecation"):
        alerts.append(f"⏳ {m['id']}: deprecation {m['deprecation']} — plan migration")
print("\n".join(alerts) or "No changes — watchlist current.")
# Monthly: for each `candidate`, re-run your golden-set eval and update my_eval, then promote/drop.

Expected output

• A short alert list: price changes, vanished (deprecation-candidate) models, and upcoming deprecations on the models you track.

Debugging tips

• Catalog shape differs? Map fields as in 00's lab.
• Too many alerts? You're tracking too many models — prune to what you run/back up/consider.

Extension task

Emit the production + fallback rows as a LiteLLM-style routing config (primary + fallbacks) — connecting the watchlist to executable infra (Phase 8).

Production extension

Run the updater on a daily schedule (cron/CI) posting alerts to Slack/email, and add a monthly reminder to re-eval candidates on your golden set and re-date specs.

What to measure

Number of tracked models per state; alerts per cycle; lead time between a price/deprecation change and your awareness.

Deliverables

• A maintained watchlist.yaml (5–15 models, dated, with your eval scores).
• An updater that flags changes/deprecations.
• (Stretch) a generated routing/fallback config from the watchlist.

13. Verification Questions

Basic

1. What does a watchlist track, and how is it different from the trending-LLMs landscape reference?
2. Why record your eval score rather than benchmark scores?
3. Why are deprecation dates essential in the watchlist?

Applied 4. Design the columns/fields for a watchlist that drives a gateway's routing + fallback. 5. Describe the daily-automated vs monthly-human split of the maintenance loop.

Debugging 6. A production model was deprecated and you found out from a 404. What watchlist gap caused it? 7. Your watchlist hasn't changed in 3 months. What's wrong, and how do you fix it?

System design 8. Design the end-to-end pipeline: catalog diff → alert → triage → re-eval → watchlist update → gateway config.

Startup / product 9. How does an eval-backed, automated watchlist protect both reliability and unit economics as the market churns?

14. Takeaways

1. A watchlist is the maintained, dated artifact that operationalizes catalog + releases + benchmarks + pricing.
2. Track what you run, back up, and consider — small and current.
3. Record your eval scores and deprecation dates, not just benchmarks.
4. Automate detection, human-triage decisions; date every fact.
5. It's the human front-end of the gateway's registry + routing/fallback.
6. A daily diff + monthly 20-minute ritual keeps you current without churn.

15. Artifact Checklist

• watchlist.yaml (5–15 models, states, dated specs, your eval scores, deprecation dates).
• Updater that flags price/deprecation/availability changes.
• (Stretch) generated routing/fallback config from the watchlist.
• Maintenance ritual documented (daily auto + monthly human).
• Alert channel wired (Slack/email).
• Notes: how the watchlist feeds the Phase 8 gateway.

Up: Phase 4 Index · Next: Phase 5 — Model Selection