Senior AI Engineer — LLMs, VLMs & Agentic Systems — Curriculum

Target Role: Senior AI Engineer at a Robotics Research Center — design, fine-tune, align, and deploy Large Language Models (LLMs), Vision-Language Models (VLMs), and autonomous agentic systems as scalable, low-latency production software (the reference JD — jd.md).

Also prepares you for:

Frontier-lab & applied-research roles — Member of Technical Staff / Research Engineer on training, inference, alignment, or multimodal teams.
Any company productionizing LLMs/VLMs — staff/senior AI engineer owning the model stack end-to-end: tokenizer → transformer → fine-tune → align → quantize → serve → agent → evaluate → observe.
Embodied-AI / robotics-foundation-model teams — the differentiator of this JD: agents that perceive, plan, and act, with the sim-to-real and VLA-action layer built explicitly.
The "I can use the API" floor is not the ceiling here: this track makes you build the mechanisms the frameworks only expose — PagedAttention, the LoRA delta, the DPO loss, the ReAct loop, the all-reduce ring — so you can debug them at 2 a.m. and defend them in a principal interview.

Duration: ~36 weeks core (≈9 months) — extendable with the capstone and the C++/CUDA track Seniority Target: Senior → Staff IC — the person who owns the model stack, makes the quality/latency/cost/safety tradeoff, and is the escalation point when generation is wrong, slow, or unsafe. Goal: Make you the engineer who can architect a multimodal agentic AI system at the system level and debug it at the token-logit, attention-mask, KV-cache-block, LoRA-rank, quantization-scale, gradient-shard, reward-margin, and tool-call level — across training, alignment, inference serving, retrieval, agents, evaluation, and embodied control.

Why This Curriculum Exists

Most "learn LLMs" material teaches you to call an API: which SDK method, which prompt, which framework abstraction. That gets you to "AI-adjacent engineer." It does not get you to Senior AI Engineer at a research center, because that job is to make correct tradeoffs under conflicting requirements (quality vs latency vs cost vs memory vs safety) and to debug the layer everyone else treats as magic — and you cannot do either if the transformer, the sampler, the scheduler, the optimizer, and the agent loop are black boxes.

So this track is built on two non-negotiables:

Every mechanism in the JD gets implemented, not just described. Scaling-law and FLOPs/KV-cache arithmetic, a BPE tokenizer, scaled-dot-product + multi-head attention with RoPE and a KV-cache, a reverse-mode autograd engine that trains a tiny LM, a CLIP-style contrastive VLM with a LLaVA-style projector, the LoRA/QLoRA delta and a distillation loss, affine + GPTQ/AWQ-style quantization, a Bradley-Terry reward model and the DPO/PPO losses, temperature/top-k/top-p/min-p sampling and a grammar-constrained decoder, a PagedAttention block manager with continuous batching and speculative decoding, data/tensor/pipeline parallelism and ZeRO memory accounting, an HNSW/IVF ANN index with reranking, a ReAct agent with typed tools and memory, a multi-agent orchestrator, a neurosymbolic verifier, an embodied RL planner with a VLA action decoder, an eval harness with LLM-as-judge and calibration, and a GPU roofline/occupancy model — you build a working, tested miniature of each.
Everything is taught to the depth interviewers and incidents actually demand — the 6ND training-FLOPs rule, why softmax overflows without the max-subtraction trick, the merge-priority off-by-one in BPE, why a Durable… (sorry, wrong track) — why a Durable orchestrator is replay-safe but an agent loop is not, the R+W of the KV-cache block table, the LoRA α/r scaling, the symmetric-vs-affine zero-point, the DPO β temperature, the ring-all-reduce 2(N−1) step count, the speculative-decoding acceptance criterion. We assume you are building the system that serves millions of multimodal requests with quality, latency, and safety guarantees — not a notebook demo.

How Each Phase Teaches (the "many ways of explaining")

Every phase deliberately explains the same material through several lenses, because senior-level understanding is the intersection of all of them:

Document	Voice	What it gives you
`README.md`	the syllabus	why the phase exists, concept map, lab specs, integrated-scenario ideas, deliverables checklist, key takeaways
`WARMUP.md`	the professor	zero-to-senior primer — every term from first principles, the mechanism under the hood, the math, lab guidance, success criteria, interview Q&A, references
`HITCHHIKERS-GUIDE.md`	the senior who's been there	the compressed practitioner tour — 30-second mental model, the numbers to tattoo on your arm, the framework one-liners, war stories, beginner mistakes
`BROTHER-TALK.md`	your brother, off the record	candid real-talk — what nobody tells you, the 2 a.m. pager truth, the career framing
`lab-*/`	the lab bench	a runnable, test-verified implementation: `lab.py` (TODOs) → your code → `solution.py` (reference) → `test_lab.py` (the proof). Validation is `pytest`.

The lab engineering standard is documented in LAB-STANDARD.md. Read it before you start the first lab.

The Phases

#	Phase	You build (flagship lab)	JD competency
00	Foundations: Scaling Laws, FLOPs/Memory Math & the Tradeoff Calculus	transformer FLOPs/KV-cache/latency + Chinchilla-optimal + cost modeler + quality/latency/cost tradeoff resolver	productionizing at scale; the design-review arithmetic
01	Tokenization & the Multimodal Input Pipeline	a BPE tokenizer (train merges, encode/decode, byte fallback, special tokens) + chat-template renderer	LLM/VLM input; transformers
02	The Transformer From First Principles	scaled-dot-product + multi-head attention, causal mask, RoPE, RMSNorm, SwiGLU, KV-cache — a tiny GPT forward pass	transformers; LLM internals
03	Autograd & Training Dynamics	a reverse-mode autograd engine + Adam/LR-schedule/grad-clip that trains a tiny LM	PyTorch/TF internals; training
04	Vision-Language Models & Multimodal Fusion	ViT patch embedding + CLIP contrastive dual-encoder (InfoNCE) + LLaVA-style projector into token space	VLMs; multimodal architectures
05	Parameter-Efficient Fine-Tuning: LoRA/QLoRA & Distillation	the LoRA delta (`BA·α/r`, merge), QLoRA NF4 quant of the base, and a KD soft-target loss	fine-tuning, LoRA/QLoRA, distillation
06	Quantization & Model Compression	affine/symmetric quant (scale/zero-point), per-channel INT8 GEMM, GPTQ + AWQ-style scaling	quantization; edge deployment
07	Alignment: Reward Models, RLHF, RLAIF & DPO	a Bradley-Terry reward model, the DPO closed form, and a PPO-with-KL policy step	RLHF/RLAIF, alignment
08	Sampling, Decoding & Constrained Generation	greedy/temperature/top-k/top-p/min-p, repetition penalty, and a grammar/JSON-constrained logit-mask decoder	tool-use I/O; structured output
09	Inference Serving Internals (vLLM/TensorRT-class)	a PagedAttention KV-block manager + continuous-batching scheduler + speculative-decoding acceptance sampler	DeepSpeed/vLLM/TensorRT; low-latency serving
10	Distributed Training & Model Parallelism	ring all-reduce + data/tensor/pipeline parallelism (1F1B bubble) + ZeRO memory accounting	distributed training, model parallelism
11	Retrieval-Augmented Generation & Vector Search Internals	an HNSW/IVF ANN index, chunking, MMR + cross-encoder reranking, and recall@k eval	FAISS/Pinecone/Milvus/Weaviate; RAG
12	Agentic AI: Reasoning Loops, Tool Use & Memory	a ReAct loop, a typed JSON-schema tool registry/dispatcher, and a tiered memory store (buffer + summary + semantic recall)	LangChain/LlamaIndex; reasoning loops, memory
13	Multi-Agent Orchestration & Coordination	a message-bus/blackboard, planner/executor/critic roles, a turn scheduler, and a critique-revise consensus loop	CrewAI/AutoGen; multi-agent coordination
14	Neurosymbolic Reasoning	a forward-chaining rule engine + constraint solver coupled as LLM-proposer / symbolic-verifier	neurosymbolic methodologies
15	Embodied AI & Robotics: RL, Planning & VLA	an MDP + Q-learning/policy-gradient, a hierarchical task planner, and a VLA action-token decoder	embodied AI; RL; robotics/simulation
16	Evaluation, Observability & Guardrails	an eval harness (EM/F1/pass@k/LLM-judge), calibration (ECE), and guardrails	W&B/MLflow/Evals; safety
17	MLOps: Experiment Tracking, Model Registry & Production Platform	3 labs — an MLflow-style tracking server + model registry (stages, lineage, run comparison); drift detection + production monitoring; and a CI/CD deployment pipeline (eval gates, canary, rollback)	MLflow/W&B/MLOps; productionizing at scale
18	C++/CUDA Performance & Systems Integration	a roofline + occupancy + memory-coalescing + tiled-GEMM model (CUDA mental model, offline)	C++, CUDA, GPU optimization, HPC
19	Capstone: Production Multimodal Agentic Serving Platform	an end-to-end model that composes the scheduler, KV-cache, RAG, agent loop, eval, MLOps gates, and cost into one scored design	the whole stack; system design

Use the sidebar for the full clickable table of contents, and see GLOSSARY.md / CHEATSHEET.md for the running references.

How to Use This Track

Start here, then read Phase 00 end to end — it builds the mental model (scaling laws, the FLOPs/memory/latency dials, the tradeoff calculus) that every later phase plugs into.
For each phase: read WARMUP.md (professor), skim HITCHHIKERS-GUIDE.md (the numbers), do the lab (lab.py red → green against test_lab.py), then read BROTHER-TALK.md.
Work phases roughly in order — the transformer (P02), autograd (P03), and the FLOPs/memory math (P00) underpin everything after them; agents (P12–P13) assume the serving and tool layers below them.
Use interview-prep/ and system-design/ as running references; the capstone (P19) ties the whole stack together.

Cross-Cutting Themes (the senior AI engineer's lenses)

Training compute vs inference compute — 6ND to train once; 2N per token forever. The bill, the latency, and the carbon all live on the inference side at scale, which is why quantization (P06), serving (P09), and the KV-cache (P02/P09) get their own phases.
Quality is a distribution, not a number — every generation is a sample. Sampling (P08), evaluation (P16), and alignment (P07) all exist because the model is a probability distribution over tokens, and "it worked once" is not a guarantee.
The KV-cache is the memory wall — autoregressive decoding is memory-bandwidth-bound, not compute-bound; PagedAttention, continuous batching, and quantized KV exist to fight that one fact. Internalize it in P00 and you understand half of inference engineering.
Agents are loops over an unreliable oracle — the LLM is a stochastic function that can hallucinate a tool call. Every agent design (P12–P14) is about constraining and verifying that oracle: typed tools, memory, multi-agent critique, symbolic verification.
Quantify the tradeoff — FLOPs, KV-cache bytes, tokens/sec, $/1M-tokens, recall@k, reward margin, ECE — these are arithmetic you do before the design review, not after the incident. Every phase ends with a number you can defend.
Embodiment changes the contract — when the agent's action moves a motor, latency is a safety property and "regenerate" is not an option. P15 makes the perception→plan→act loop and the sim-to-real gap explicit.