Start Here — LLM Inside-Out Mastery
Read this before anything else.
What You Are Building Toward
You are building the mental model and hands-on skills of a seasoned LLM engineer.
That means you can:
- Talk to the model — Understand tokens, context, sampling, generation
- Choose the right model — Read model cards, use catalogs, compare providers
- Run models anywhere — Local, cloud, self-hosted, API, gateway
- Build with models — RAG, agents, tools, structured output, streaming
- Serve models — vLLM, continuous batching, PagedAttention, KV cache
- Route and meter — LLM gateway, fallbacks, cost tracking
- Evaluate rigorously — Correctness, latency, cost, safety, regression
- Operate safely — Security, privacy, prompt injection, governance
- Build products — Startup playbook, unit economics, MVP
The Right Mental Model From Day 1
An LLM is a token probability machine.
It takes a sequence of tokens as input and predicts the probability distribution over the next token. It does this one token at a time. Everything else — tool calling, structured output, agents, RAG, reasoning — is application logic wrapped around this core.
You control quality through context.
The model only knows what is in its context window. Everything the model "knows" beyond that came from training. Your job as an LLM engineer is to ensure the right context is in the window at the right time.
Cost is tokens times price.
Every call costs (input tokens × input price) + (output tokens × output price). Model routing, caching, context compression, and chunked retrieval are cost controls.
Latency has two phases.
TTFT (time to first token) depends on prefill speed. TPOT (time per output token) depends on decode speed. Streaming makes products feel faster. Batching increases throughput at the cost of individual TTFT.
Safety is your application's responsibility.
The model's safety training helps but cannot be relied on. You must add output validation, prompt shields, tool approval, rate limits, and audit logs.
Recommended Entry Path
Week 1-2: Vocabulary and Mental Models
Start with Phase 1 (vocabulary) and Phase 0 (orientation).
Goal: be able to read a model card and understand every term.
→ 01-llm-vocabulary/00-core-mental-model.md → 01-llm-vocabulary/09-complete-glossary.md → 00-orientation/00-what-is-an-llm-engineer.md
Lab: Read three model cards from scratch. Extract every field using the model card checklist.
Week 3-4: How LLMs Work
Phase 2 (transformer foundations).
Goal: explain tokens → embeddings → attention → generation → KV cache to a non-engineer.
→ 02-transformer-foundations/00-transformer-big-picture.md → 02-transformer-foundations/06-kv-cache.md → 02-transformer-foundations/07-prefill-vs-decode.md
Lab: Run a small model locally and observe tokens/sec, TTFT, and memory usage.
Week 5-6: Model Cards, Catalogs, and Model Selection
Phases 3 and 4.
Goal: build a model selection matrix for five real use cases.
→ 03-model-cards-and-system-cards/00-how-to-read-model-cards.md → 04-model-catalogs-and-trend-reading/00-how-to-use-models-dev.md → 05-model-selection/00-model-selection-framework.md
Lab: Build a model selection memo for a coding agent, a RAG assistant, and a document summarizer.
Week 7-8: Local Inference
Phase 6.
Goal: run three different models locally, benchmark them, compare quantization.
→ 06-local-inference/00-local-inference-overview.md → 06-local-inference/03-gguf-and-llama-cpp.md → 06-local-inference/06-quantization-guide.md
Labs: labs/lab-02-local-llama-cpp/ and labs/lab-05-quantization-benchmark/
Week 9-10: Production Serving
Phase 7.
Goal: run vLLM, benchmark throughput, understand PagedAttention and continuous batching.
→ 07-production-serving/00-serving-architecture.md → 07-production-serving/01-vllm.md
Lab: labs/lab-04-vllm-serving/
Week 11-12: LLM Gateway
Phase 8 + Lab 7.
Goal: build a functional LLM gateway with routing, fallback, cost tracking, and streaming.
→ 08-llm-gateways/00-what-is-an-llm-gateway.md
Lab: labs/lab-07-model-gateway/
Month 2: RAG + Agents + Eval
→ Phases 9, 10, 12
Labs: labs/lab-08-rag-system/ and labs/lab-09-agent-with-tools/
Month 3: Fine-tuning, Security, Startup
→ Phases 13, 14, 15
Lab: labs/lab-13-capstone-platform/
What to Produce As You Go
| Phase | Artifact |
|---|---|
| Vocabulary | Personal glossary with notes |
| Transformer foundations | Architecture diagram with annotations |
| Model cards | Model card reading checklist + 3 sample memos |
| Model selection | Model selection matrix for 5 use cases |
| Local inference | Benchmark report for 3 models |
| Production serving | vLLM deployment and benchmark report |
| LLM gateway | Working gateway with routing config |
| RAG | Working RAG system with eval results |
| Agents | Working agent with tool calls and traces |
| Eval | Eval harness with model comparison report |
| Startup | MVP spec, cost model, and demo narrative |
What This Is Not
- Not a machine learning research curriculum.
- Not a deep learning from scratch course.
- Not a data science course.
- Not a Python tutorial.
You do not need to implement backpropagation. You do not need to train models from scratch. You need to understand how to engineer, serve, evaluate, and build on top of LLMs.
How to Study Fast-Moving AI
- Primary sources first. Model cards, official docs, arxiv abstracts, changelogs.
- Ignore hot takes. Most "new technique" posts describe ideas that are not yet production-ready.
- Benchmark everything yourself. External benchmarks are environment-specific.
- Build fast, evaluate quickly. A working lab beats a finished reading list.
- Track the spec, not the demo. Demos are curated. Read the methodology.
- Watch release notes, not announcements. Announced capabilities vs shipped capabilities differ.
How to Use the External References
Every document in this curriculum lists external references with:
- Title and URL
- Why it matters
- What sections to read first
- How it connects to the lab
Do not try to read every reference. Use them as on-demand lookups when you hit a concept you need to understand more deeply.
→ Ready? Start with 01-llm-vocabulary/00-core-mental-model.md