Model Registry

Phase 8 · Document 04 · LLM Gateways Prev: 03 — Provider Adapters · Up: Phase 8 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

The model registry is the gateway's source of truth: the database of every model it can serve, with each model's capabilities, limits, pricing, availability, and routing entries. Nearly every other component reads from it — the routing engine filters candidates by capability and price, usage metering computes cost from registry prices, adapters check capability facts, and the admin dashboard renders it. If the registry is wrong or stale, everything downstream is wrong: you route a tool call to a non-tool model, bill at last quarter's prices, or recommend a deprecated model. It's the Phase 4 catalog (models.dev) turned into a live, queryable, gateway-owned table — and keeping it fresh is an ongoing operational job, not a one-time seed.

2. Core Concept

Plain-English primer: the gateway's catalog

The registry answers, for any model the gateway exposes: what is it, what can it do, what does it cost, who serves it, is it still active, and which routes use it? It's the same metadata you learned to read off model cards and catalogs (Phase 3, Phase 4) — now stored so code can query and act on it.

Three kinds of facts live here:

1. Identity & availability — exact model ID, provider/host, lab, version/alias, release/updated dates, is_active/deprecated (Phase 3.00).
2. Capabilities & limits — context window, max output, supports tools / structured output / reasoning / vision / streaming (Phase 1.04).
3. Pricing — input / output / cached-input (and reasoning, image) prices, for cost computation (Phase 4.04).

Plus a routes table mapping aliases (auto/coding) → ordered model candidates with constraints (05).

The schema

CREATE TABLE models (
  id              TEXT PRIMARY KEY,        -- "anthropic/claude-..."  (provider/model namespace [01])
  name            TEXT NOT NULL,
  provider        TEXT NOT NULL,           -- who SERVES it (may differ from lab) [01,5.10]
  lab             TEXT NOT NULL,           -- who BUILT it
  version         TEXT,                    -- pin versions, not aliases [5.10]
  context_window  INTEGER,
  max_output_tokens INTEGER,
  supports_tools  BOOLEAN DEFAULT FALSE,
  supports_structured_output BOOLEAN DEFAULT FALSE,
  supports_reasoning BOOLEAN DEFAULT FALSE,
  supports_vision BOOLEAN DEFAULT FALSE,
  supports_streaming BOOLEAN DEFAULT TRUE,
  input_price_per_1m  DECIMAL(10,6),
  output_price_per_1m DECIMAL(10,6),
  cached_input_price_per_1m DECIMAL(10,6),
  is_active       BOOLEAN DEFAULT TRUE,    -- flip off on deprecation
  release_date    DATE,
  updated_at      TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE model_routes (              -- aliases → candidates (the routing source) [05]
  route_name  TEXT NOT NULL,             -- "auto/coding"
  priority    INTEGER NOT NULL,          -- try order
  model_id    TEXT REFERENCES models(id),
  constraints JSONB,                     -- {"max_input_price":5.0,"require_tools":true}
  PRIMARY KEY (route_name, priority)
);

Why provider vs lab is a first-class distinction

A subtle but critical column pair: lab (who built the weights) vs provider (who serves this row). The same model (one lab) may appear as multiple rows with different provider, version, price, and even quantization — the serving-fidelity reality from Phase 5.10 and the aggregator's multi-upstream model (01). The registry must represent (model × provider), not just model, so routing can choose the right served instance.

Keeping it fresh (the operational job)

The hardest part isn't the schema — it's freshness. Prices change, models deprecate, new versions ship, capabilities get added. A stale registry mis-bills and mis-routes. Strategies:

• Seed + sync from catalogs (e.g., models.dev / provider pricing pages) on a schedule (Phase 4.05).
• Track updated_at and alert on staleness; reconcile metered cost against actual provider invoices (06).
• Deprecate, don't delete — flip is_active=false so historical usage rows keep a valid model reference, and routing stops selecting it (Phase 1.08 deprecation).
• Pin versions, not aliases so a provider can't silently change behavior under you (Phase 5.10).

Who reads it

ROUTING [05]   → filter candidates by capability + price + active
METERING [06]  → price × tokens = cost
ADAPTERS [03]  → capability facts (don't send tools to a non-tool model)
ADMIN [08]     → render/edit models, routes, pricing

LiteLLM/OpenRouter ship a built-in model database; building your own registry matters when you need custom capability flags, internal models, or your own pricing/routing logic (02).

3. Mental Model

   MODEL REGISTRY = the gateway's SOURCE OF TRUTH (Phase 4 catalog, made live + queryable)
   ┌───────────────────────────────────────────────────────────────────────┐
   │ (model × PROVIDER) rows:  identity/version · caps/limits · PRICING · active │
   │   lab = who BUILT it   ≠   provider = who SERVES this row [5.10]            │
   │ routes: alias (auto/coding) → ordered candidates + constraints  [05]       │
   └───────────────────────────────────────────────────────────────────────┘
        ▲ read by → ROUTING [05] · METERING(price) [06] · ADAPTERS(caps) [03] · ADMIN [08]
   FRESHNESS is the hard part: sync from catalogs, track updated_at, DEPRECATE don't delete,
                               PIN versions, reconcile cost vs invoices

Mnemonic: the registry is the live catalog of (model × provider) facts — caps, limits, price, active — that every component queries; its enemy is staleness, so sync, pin versions, and deprecate (don't delete).

4. Hitchhiker's Guide

What to look for first: the capability + price + active fields and the (model × provider) modeling. Those drive correct routing and billing. Then the routes table.

What to ignore at first: modeling every exotic field. Start with identity, context/output, the capability booleans, prices, and is_active; add nuance as needed.

What misleads beginners:

• Modeling "model" not "(model × provider)". You can't represent serving variance (price/quant per host) without the provider dimension (Phase 5.10, 01).
• Conflating lab and provider. They're different facts; routing needs the served instance.
• Letting it go stale. Wrong prices → wrong bills; dead models still selected. Freshness is the job.
• Deleting deprecated models. Breaks historical usage references — flip is_active instead.
• Using aliases as the version. A floating alias lets behavior change silently — pin versions.

How experts reason: they treat the registry as (model × provider) source-of-truth, keep it fresh via scheduled sync + staleness alerts, pin versions, deprecate not delete, and reconcile metered cost against invoices to catch price drift. They drive routing, metering, adapters, and admin entirely off it so there's one place to change a fact.

What matters in production: price accuracy (billing integrity), capability accuracy (correct routing), is_active hygiene (don't route to dead models), version pinning (no silent drift), and freshness SLAs.

How to debug/verify: when a route or bill looks wrong, check the registry row first (price/caps/active/version); diff against the provider's current pricing/card; verify routes reference active models.

Questions to ask: does the registry model (model × provider)? how/when is it synced? are versions pinned? how are deprecations handled? does metering read prices from here?

What silently gets expensive/unreliable: stale prices (billing drift), wrong capability flags (failed/over-priced requests), routing to deprecated models, and alias drift changing behavior unnoticed.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• The registry is the gateway's source of truth — routing, metering, adapters, and admin all read it.
• Model it as (model × provider), not just model — same model can have many served rows differing in price/quant (Phase 5.10, 01).
• lab (built) ≠ provider (serves) — keep both.
• It stores identity/version, capabilities/limits, and pricing — the Phase 3/4 metadata, made queryable.
• Metering computes cost from registry prices — stale prices ⇒ wrong bills (06).
• Routing filters candidates by capability + price + is_active (05).
• Deprecate (flip is_active), don't delete — preserve historical usage references.
• Pin versions, not aliases, and keep it fresh via scheduled sync + staleness alerts (Phase 4.05).

9. Observations from Real Systems

• LiteLLM ships a public model-cost/context JSON that is a registry — many gateways seed from or reuse it (02).
• OpenRouter exposes a models API with per-provider pricing/capabilities — a live (model × provider) catalog (01).
• models.dev is the human-facing version of the same data — the seed for a custom registry (Phase 4.00).
• Billing incidents often trace to a stale price row; routing incidents to a wrong capability flag or a deprecated-but-still-active model.
• Enterprises add internal models (self-hosted vLLM) as registry rows alongside cloud models, with their own pricing (amortized GPU cost) (Phase 7.01).

10. Common Misconceptions

11. Engineering Decision Framework

BUILD/OPERATE the registry:
 1. SCHEMA: (model × provider) rows — identity/version · caps/limits · pricing · is_active; routes table. 
 2. SEED: import from models.dev / LiteLLM cost map / OpenRouter models API.                 [4.00,4.05]
 3. CAPABILITIES: set tools/structured/reasoning/vision/streaming + context/output accurately. [1.04]
 4. PRICING: input/output/cached (+reasoning/image); metering reads these.                     [06,4.04]
 5. FRESHNESS: scheduled sync; track updated_at; alert on staleness; reconcile vs invoices.
 6. LIFECYCLE: pin versions; deprecate via is_active (never delete); record release/updated.   [5.10]
 7. SERVE downstream: routing [05], metering [06], adapters [03], admin [08] all read here.

12. Hands-On Lab

Goal

Build a model registry (seed it from a real catalog), then drive a capability+price query that routing and metering would use.

Prerequisites

• SQLite/Postgres; pip install requests (to fetch a catalog JSON).

Steps

1. Create the schema (§2: models + model_routes).
2. Seed it: import a real catalog — e.g., LiteLLM's model_prices_and_context_window.json or the OpenRouter models API — mapping each (model × provider) into a row with caps + prices.
3. Capability query (routing input): SELECT * FROM models WHERE supports_tools AND is_active AND input_price_per_1m <= 5 ORDER BY input_price_per_1m; — the candidate set 05 would use.
4. Cost query (metering input): given a usage row (in/out/cached tokens), join to models and compute cost from registry prices — exactly what 06 does.
5. Define routes: insert auto/coding → ordered candidates with constraints (require_tools, max_input_price).
6. Freshness check: add an updated_at staleness query and a is_active=false deprecation (verify routing would skip it).

Expected output

A seeded registry that answers capability/price queries, computes a request's cost from stored prices, and exposes routes — the data backbone the rest of the gateway reads.

Debugging tips

• Cost off by a lot → seeded prices are per-1M vs per-1K mismatch; normalize units.
• A deprecated model still selected → routing query didn't filter is_active.

Extension task

Add a self-hosted vLLM model row with an amortized price (GPU $/hr ÷ tokens/hr) and confirm it appears in cost-sorted candidate queries (Phase 7.01, Phase 5.02).

Production extension

Schedule a daily sync from the catalog with updated_at tracking + a staleness alert, and a monthly reconciliation of metered cost vs provider invoices (06).

What to measure

Query correctness (candidates/cost); freshness (max updated_at age); reconciliation delta vs invoices.

Deliverables

• A registry schema + a seeded dataset (model × provider).
• A capability query and a cost-from-registry computation.
• A routes definition + a deprecation demonstration.

13. Verification Questions

Basic

1. What is the model registry and which components read it?
2. Why model (model × provider) instead of just model?
3. What's the difference between lab and provider?

Applied 4. Write the registry query the routing engine uses to find eligible candidates. 5. How does the registry enable correct cost metering?

Debugging 6. Bills are wrong for one model. Where do you look first? 7. A deprecated model keeps getting selected. Two likely causes.

System design 8. Design a registry + freshness pipeline that stays accurate as prices and models change.

Startup / product 9. Why is an accurate registry foundational to gateway unit economics and trust?

14. Takeaways

1. The registry is the gateway's source of truth — identity, capabilities, limits, pricing, routes — read by every component.
2. Model (model × provider) to capture serving variance; keep lab ≠ provider.
3. Metering computes cost from registry prices; routing filters on capability + price + is_active.
4. Freshness is the hard part — sync from catalogs, pin versions, alert on staleness, reconcile vs invoices.
5. Deprecate (flip is_active), don't delete; reuse LiteLLM's/OpenRouter's catalog to seed.

15. Artifact Checklist

• A registry schema (models + model_routes) modeling (model × provider).
• A seeded dataset from a real catalog.
• A capability query (routing) + a cost-from-registry computation (metering).
• Routes with constraints + a deprecation (is_active) demo.
• A freshness/sync plan + invoice-reconciliation note.

Up: Phase 8 Index · Next: 05 — Routing Engine