How to Study Fast-Moving AI

Phase 0 · Document 02 · Orientation Prev: 01 — Roles and Career Paths · Next: 03 — How to Use This Guide

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

AI moves faster than any field you've studied: models are superseded in months, "best practices" rot in quarters, and most blog posts are out of date or marketing. If you study the way you'd study a stable subject — memorizing current model names and prices — your knowledge decays before you can use it. The only durable strategy is to learn the invariants (the things that don't change), build a system for tracking change (primary sources, a watchlist), and develop a filter for hype. This document teaches that meta-skill. It's what keeps the rest of this curriculum useful a year from now.

2. Core Concept

Separate invariants from specifics

Spend ~80% of study time on the left column; it stays true. Track the right column with a system, not memorization.

Go to primary sources

Reliability ladder, best to worst:

1. Official docs, model cards, system cards, API references (the lab/provider's own words).
2. Source code (vLLM, llama.cpp, OpenAI SDK) — the ground truth when docs are ambiguous.
3. Papers (for invariants and mechanisms).
4. Reputable practitioner blogs (for synthesis and gotchas).
5. Social media / influencer threads (signal of what to investigate, never the conclusion).

When a detail affects a cost, capacity, or safety decision, always confirm at level 1–2.

Build a change-tracking system

• A model watchlist (which families you follow, where their releases post).
• A small set of primary feeds (provider changelogs, a few high-signal newsletters).
• A habit of dated notes ("as of 2026-06, model X costs Y") so stale facts are obvious later.
• A re-check ritual before any production decision: prices, limits, deprecations change silently.

Learn by building, verify by measuring

Reading about LLMs produces fragile knowledge; building small labs produces durable knowledge. Every claim ("4-bit is fine," "this model is faster") should be verified by a measurement on your task, not accepted from a post.

Filter hype

Most "X changes everything" claims are environment-specific or cherry-picked. Default skepticism: under what hardware/quant/prompt/batch/task does this hold? (This is the explicit lesson of the Phase 2 screenshot deep-dive: a "2× faster" claim rarely applies to every prompt, device, or batch size.)

3. Mental Model

                    KNOWLEDGE HALF-LIFE
  invariants (years)  ████████████████████  ← study DEEPLY, build labs
  practices  (months) ████████              ← track, re-verify
  specifics  (weeks)  ██                     ← look up on demand, date your notes

  PIPELINE FOR ANY NEW CLAIM:
    social/blog (what to look at)
        → primary source / code (is it true?)
        → small lab on MY task (does it help ME?)
        → dated note + watchlist update

  Before any prod decision: RE-CHECK prices · limits · deprecations.

4. Hitchhiker's Guide

What to do first: internalize the invariants (Phases 1–2). They make every new release legible on sight.

What to ignore: memorizing current model names, prices, and leaderboard order — you'll look them up, dated, when you need them.

What misleads beginners:

• Treating influencer threads as conclusions instead of pointers.
• Believing benchmark/marketing claims without re-testing on their own task.
• Re-learning the field with every release instead of mapping it onto invariants.
• Hoarding tutorials without building anything.

How experts stay current: they read primary sources, follow a small set of high-signal feeds, build a quick lab to test anything that matters, and keep dated notes so they know what's stale. They spend most energy on mechanisms, not model-of-the-week.

What matters in production: a re-verification habit (prices/limits/deprecations drift), version pinning, and evals that tell you whether a "better" new model is actually better for you.

How to verify a claim: find the primary source; if it affects a decision, reproduce it in a 20-minute lab on your data; record the result with a date.

Questions to ask about any hot take: What's the primary source? Under what conditions does it hold? Does it replicate on my task? Is it already out of date?

What silently wastes time/money: chasing every release; trusting stale prices; over-investing in a framework that may be deprecated; learning specifics that expire before use.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Invariants outlive specifics — mechanisms and economics persist; model names and prices don't.
• Primary sources beat summaries; confirm decision-relevant details in docs or code.
• Benchmark/marketing claims are environment-specific — replicate on your task before believing them.
• Prices, rate limits, and model availability change silently — re-verify before every production decision.
• Dated notes are essential — an undated fact is a future trap.
• Building + measuring produces durable knowledge; passive reading produces fragile knowledge.
• A small, high-signal feed set beats firehose consumption.
• Version-pin in production so the ground doesn't shift under you (see Phase 1.08).

9. Observations from Real Systems

• models.dev publishes an "Updated" date precisely because catalog facts go stale — use it as a freshness signal.
• Provider changelogs (OpenAI/Anthropic/Google) are the authoritative deprecation channel; teams that ignore them get surprise breakages.
• vLLM/llama.cpp source and release notes are the real spec when docs lag the fast-moving code.
• Model/system cards are primary disclosures of capabilities, evals, and limitations — the antidote to leaderboard hype (Phase 3).
• The Phase 2 "2× faster" screenshot is the canonical lesson: a headline benchmark that doesn't generalize across prompt/device/batch — always re-test.

10. Common Misconceptions

11. Engineering Decision Framework

Encounter a new claim/release:
  1. Is it an INVARIANT or a SPECIFIC? Invariant → study; specific → note (dated) + watchlist.
  2. Decision-relevant? → confirm in PRIMARY source/code.
  3. Affects cost/quality/latency for me? → REPRODUCE in a small lab on my task.
  4. Record a DATED note; update the watchlist.

Before any production decision:
  re-check prices · rate limits · deprecations · pin versioned IDs · run the eval (Phase 13).

Allocating study time:
  ~80% invariants (Phases 1–2 + mechanisms) · ~20% tracking current specifics.

12. Hands-On Lab

Goal

Stand up a personal AI-tracking system: a model watchlist, a primary-source feed list, a dated-notes file, and one reproduced claim.

Prerequisites

• A notes repo; optional: a feed reader.

Steps

1. Create watchlist.md: list 3–5 model families you'll follow and where each posts releases (changelog URLs).
2. Create feeds.md: 5 primary/high-signal sources (provider changelogs, models.dev, 1–2 newsletters, arXiv listing).
3. Create notes-dated.md: record 5 current facts (a model price, a context window, a rate limit) each with an "as of <date>".
4. Pick one popular claim ("4-bit barely hurts quality" or a "Nx faster" claim) and reproduce it minimally on a task you care about (reuse Phase 1 labs).
5. Write a 3-line verdict: did it replicate for you, and under what conditions?

Expected output

Three tracking files plus one reproduced-claim verdict.

Debugging tips

• Can't find a primary feed? Search "<provider> changelog/release notes/deprecations."
• Claim won't replicate? Note the conditions — that is the finding.

Extension task

Add a monthly "re-verify" checklist item that revisits notes-dated.md and flags anything stale.

Production extension

Wire the watchlist to the Phase 4 models.dev explorer so new/changed models surface automatically.

What to measure

Number of decision-relevant facts you've dated; whether your reproduced claim held on your task.

Deliverables

• watchlist.md, feeds.md, notes-dated.md, and a one-claim reproduction verdict.

13. Verification Questions

Basic

1. Give three invariants and three specifics in LLM engineering.
2. Rank the reliability of: a viral thread, a provider doc, source code, a paper.
3. Why date your notes?

Applied 4. A new model claims "2× faster." What's your verification process before relying on it? 5. How do you allocate study time between invariants and specifics, and why?

Debugging 6. Your app broke overnight with no deploy. What change-tracking habit would have caught it? 7. You "knew" a model's price but the bill is higher. What went wrong?

System design 8. Design a lightweight system for a team to track model releases and avoid deprecation surprises.

Startup / product 9. As a founder, how do you keep model choices current without churning your product every month? (Hint: invariants + evals + version pinning + routing.)

14. Takeaways

1. Learn invariants deeply; track specifics with a system — don't memorize the model-of-the-week.
2. Go to primary sources (docs, cards, code); treat social media as pointers, not conclusions.
3. Reproduce claims on your task before believing benchmarks or hype.
4. Date your notes and re-verify prices/limits/deprecations before production decisions.
5. Build and measure for durable knowledge; passive reading decays.
6. A small high-signal feed set + a watchlist beats firehose consumption.

15. Artifact Checklist

• watchlist.md of model families + release sources.
• feeds.md of primary/high-signal sources.
• notes-dated.md with 5+ dated facts.
• One reproduced-claim verdict on your own task.
• Re-verify ritual added to your workflow.
• Notes: the knowledge half-life model.

Next: 03 — How to Use This Guide