Glossary — Senior AI Engineer

The running vocabulary for the whole track. Terms are grouped by area; each entry is the one-sentence definition plus the why it matters a senior is expected to know. Where a phase builds the mechanism, the phase number is noted.

Foundations, math & scaling (P00)

  • Parameter (N) — a learned weight. Model "size" is the parameter count; memory for weights ≈ N × bytes_per_param (2 for fp16, 1 for int8, ~0.5 for int4).
  • Token (D in training, T in a sequence) — the atomic unit the model reads/writes (a sub-word, byte, or image patch). Training cost scales with total tokens seen.
  • Training FLOPs ≈ 6 × N × D — the rule of thumb: forward+backward over D tokens for an N-param dense model. Forward alone ≈ 2N per token.
  • Inference FLOPs ≈ 2 × N per generated token — one forward pass per token; prefill processes the whole prompt at once, decode emits one token at a time.
  • Chinchilla-optimal — for a fixed compute budget, the loss-minimizing split is roughly D ≈ 20 × N tokens per parameter; most "big" models were under-trained on data.
  • Scaling law — loss falls as a power law in compute/params/data: ( L(N) = L_\infty + (N_c/N)^{\alpha} ). The reason "make it bigger" worked.
  • FLOPs vs memory-bandwidth bound — prefill and training are compute-bound; autoregressive decode is memory-bandwidth bound (you re-read the whole weight matrix per token).
  • Arithmetic intensity — FLOPs per byte moved; where it sits relative to the hardware roofline ridge tells you whether you're compute- or bandwidth-limited (P17).
  • KV-cache — the stored keys/values of every past token so decode is O(1) per step instead of O(T). Size ≈ 2 × layers × T × d_model × bytes; the real inference memory wall.
  • MFU (Model FLOPs Utilization) — achieved FLOPs ÷ peak hardware FLOPs; 40–55% is good for training, far lower for memory-bound decode.

Tokenization & input (P01)

  • BPE (Byte-Pair Encoding) — learn a merge table by greedily merging the most frequent adjacent pair; encode by applying merges in learned priority order.
  • Byte fallback — represent any unknown character as its raw UTF-8 bytes so the vocab is closed (no [UNK] data loss).
  • Special tokens — <bos>, <eos>, <pad>, <|im_start|>, <image> — reserved IDs the model is trained to treat structurally; mishandling them is the #1 chat bug.
  • Chat template — the exact string format (roles, turn markers) the model was trained on; the prompt you send must match it byte-for-byte or quality collapses.
  • Context window — max tokens the model attends over; the KV-cache and attention cost scale with it.

Transformer internals (P02)

  • Embedding — lookup table mapping token ID → vector of size d_model.
  • Self-attention — ( \text{softmax}(QK^\top/\sqrt{d_k})V ); each token mixes information from others, weighted by query-key similarity.
  • Causal mask — set future positions to -∞ before softmax so a token can't see the future (required for autoregressive LMs).
  • Multi-head attention (MHA) — run attention in h parallel subspaces and concatenate; lets the model attend to several relations at once. MQA/GQA share K/V heads to shrink the KV-cache.
  • RoPE (Rotary Position Embedding) — encode position by rotating Q/K by an angle proportional to position; relative, extrapolates better than learned absolute embeddings.
  • RMSNorm / LayerNorm — normalize activations for stable training; RMSNorm drops the mean subtraction (cheaper, ~same quality).
  • SwiGLU / GeLU FFN — the per-token MLP (~2/3 of params); SwiGLU is a gated activation used in Llama-class models.
  • Residual stream — the additive x + sublayer(x) highway every block reads/writes; the conceptual "bus" of the transformer.
  • Softmax stability — subtract max(logits) before exp to avoid overflow; mathematically identical, numerically safe.

Training (P03, P10)

  • Autograd / reverse-mode AD — build a computation graph on the forward pass, then apply the chain rule backward to get every gradient in one pass.
  • Cross-entropy loss — ( -\sum y \log \hat{p} ); the LM objective (next-token prediction).
  • Adam / AdamW — adaptive optimizer tracking 1st/2nd moment estimates; AdamW decouples weight decay. Optimizer state is 2× the params in memory (the ZeRO motivation).
  • LR schedule — warmup then cosine/linear decay; the single biggest non-architecture knob.
  • Gradient clipping — cap the gradient norm to stop loss spikes / divergence.
  • Mixed precision (AMP) — compute in fp16/bf16, keep an fp32 master copy + loss scaling.
  • Gradient accumulation — sum grads over micro-batches to simulate a big batch on small memory.
  • Data parallel (DP) — replicate the model, split the batch, all-reduce gradients.
  • Tensor parallel (TP) — shard a single matmul across GPUs (Megatron); needs all-reduce inside the layer.
  • Pipeline parallel (PP) — split layers across GPUs, stream micro-batches; the idle gap is the bubble (1F1B scheduling shrinks it).
  • ZeRO — partition optimizer state (stage 1), gradients (2), and params (3) across data- parallel ranks to cut per-GPU memory; the core of DeepSpeed.
  • All-reduce — every rank ends with the sum of all ranks' tensors; ring all-reduce does it in 2(N−1) steps moving ~2× the data, bandwidth-optimal.
  • FSDP — PyTorch's native ZeRO-3-style fully-sharded data parallel.

Multimodal / VLMs (P04)

  • ViT (Vision Transformer) — split an image into patches, linearly embed each as a "token," run a transformer; images become sequences.
  • CLIP — train an image encoder and text encoder so matching pairs have high cosine similarity (contrastive InfoNCE loss over a batch); enables zero-shot classification and retrieval.
  • Projector / connector — the MLP that maps vision-encoder features into the LLM's token embedding space (LLaVA pattern), so images become "tokens" the LLM reads.
  • InfoNCE / contrastive loss — cross-entropy over a similarity matrix where the diagonal (matched pairs) is the target; temperature τ sharpens it.
  • Cross-attention fusion — alternative to projection: the LLM attends to vision features via added cross-attention layers (Flamingo).
  • VLM — Vision-Language Model: takes interleaved image+text, outputs text (captioning, VQA, grounding).

Fine-tuning & compression (P05, P06)

  • PEFT — Parameter-Efficient Fine-Tuning: adapt a model by training a tiny fraction of weights.
  • LoRA — freeze W, learn a low-rank update ( \Delta W = \frac{\alpha}{r} BA ) with B∈ℝ^{d×r}, A∈ℝ^{r×k}; trains <1% of params, merges back at inference for zero overhead.
  • QLoRA — LoRA on top of a 4-bit (NF4) quantized frozen base with double quantization and paged optimizers; fine-tune a 65B model on one GPU.
  • Knowledge distillation (KD) — train a small student to match a large teacher's soft probabilities; loss = KL of softened logits at temperature T (scaled by T²).
  • Quantization — store/compute weights (and sometimes activations) in fewer bits. Affine: q = round(x/scale) + zero_point; symmetric: zero_point = 0.
  • Per-tensor vs per-channel — one scale for the whole tensor vs one per output channel; per-channel is far more accurate for weights.
  • PTQ vs QAT — Post-Training Quantization (calibrate, no retrain) vs Quantization-Aware Training (simulate quant during training).
  • GPTQ — second-order, layer-wise weight quantization minimizing output error.
  • AWQ — Activation-aware Weight Quantization: scale up salient weight channels before quantizing so important weights keep precision.
  • Outlier / salient channels — a few activation channels with huge magnitude that dominate quant error; the thing AWQ/SmoothQuant protect.

Alignment & RL (P07, P15)

  • SFT — Supervised Fine-Tuning on instruction/response pairs; the first alignment step.
  • RLHF — Reinforcement Learning from Human Feedback: train a reward model on human preferences, then optimize the policy against it with PPO + a KL penalty to the SFT model.
  • RLAIF — same loop, with an AI model (a "constitution") generating the preferences instead of humans.
  • Reward model — predicts a scalar preference; trained with the Bradley-Terry loss ( -\log \sigma(r_{\text{chosen}} - r_{\text{rejected}}) ).
  • DPO (Direct Preference Optimization) — skip the RL loop; optimize the policy directly on preference pairs with a closed-form loss using the SFT model as reference and temperature β. Simpler and more stable than PPO.
  • KL penalty / reference model — keep the aligned policy from drifting too far from the base, preventing reward hacking and gibberish.
  • PPO — Proximal Policy Optimization: clipped policy-gradient update; the classic RLHF optimizer.
  • Reward hacking — the policy exploits flaws in the reward model to score high while getting worse; the KL term and good eval fight it.
  • MDP / Q-learning / policy gradient — the RL substrate (states, actions, rewards); Q-learning learns action values, policy gradient learns the policy directly (P15).

Decoding & generation (P08)

  • Logits — the model's raw pre-softmax scores over the vocab for the next token.
  • Greedy / argmax — always take the highest-probability token; deterministic, repetitive.
  • Temperature — divide logits by T before softmax; T<1 sharpens, T>1 flattens, T→0 ⇒ greedy.
  • Top-k — sample only from the k highest-probability tokens.
  • Top-p (nucleus) — sample from the smallest set whose cumulative probability ≥ p.
  • Min-p — keep tokens with prob ≥ min_p × max_prob; adapts the cutoff to confidence.
  • Repetition / presence penalty — down-weight already-emitted tokens to reduce loops.
  • Constrained / structured decoding — mask logits to only-valid tokens via a grammar/FSM so output is guaranteed-valid JSON or matches a schema (the backbone of reliable tool calls).
  • Beam search — keep the b best partial sequences; better for translation, worse for open-ended/creative generation.

Inference serving (P09, P17)

  • Prefill vs decode — prefill processes the prompt in one parallel pass (compute-bound); decode emits tokens one at a time (memory-bound). Different bottlenecks, different tuning.
  • PagedAttention — vLLM's trick: store the KV-cache in fixed-size blocks (like OS paging) so memory isn't fragmented and sequences can share prefixes; enables high batch occupancy.
  • Continuous (in-flight) batching — add/remove sequences from the running batch every step instead of waiting for the slowest; 2–4× throughput over static batching.
  • Speculative decoding — a small draft model proposes k tokens, the big model verifies them in one pass; accept the longest correct prefix. 2–3× speedup with identical output distribution.
  • TTFT / TPOT / ITL — Time-To-First-Token (prefill latency), Time-Per-Output-Token, Inter-Token Latency; the two SLOs that matter for serving.
  • Throughput vs latency — bigger batches raise tokens/sec (throughput) but raise per- request latency; the core serving tradeoff.
  • vLLM / TensorRT-LLM / DeepSpeed-Inference / ONNX Runtime — the production serving stacks the JD names; all implement variants of the above.

Retrieval & RAG (P11)

  • Embedding — a dense vector encoding semantic meaning; similarity = cosine / dot product.
  • ANN (Approximate Nearest Neighbor) — sub-linear similarity search trading a little recall for huge speed. HNSW (navigable small-world graph) and IVF (inverted-file clusters) are the two dominant families.
  • Recall@k / nDCG — fraction of true neighbors found in the top-k / rank-weighted gain; how you measure a retriever.
  • Chunking — splitting documents into passages; size and overlap trade recall vs precision and cost.
  • Reranking — re-score retrieved candidates with a stronger (cross-encoder) model; MMR re-ranks for relevance and diversity.
  • Vector DB — FAISS (library), Pinecone/Weaviate/Milvus (managed/serving); store embeddings
    • metadata, serve ANN queries with filters.
  • RAG — Retrieval-Augmented Generation: retrieve relevant context, put it in the prompt, generate grounded answers; the dominant pattern for factuality.

Agents (P12, P13, P14)

  • Agent — an LLM in a loop that can take actions (tool calls) and observe results until a goal is met.
  • ReAct — Reason+Act: interleave a "thought," an "action" (tool call), and an "observation" each step; the canonical agent loop.
  • Tool / function calling — the model emits a structured call (name + JSON args) that the runtime executes and feeds back; typed schemas + validation keep it safe.
  • Memory — short-term (the running transcript), summarized (compressed history), and long-term/semantic (vector-recalled facts); how an agent persists beyond the context window.
  • Planning — decomposing a goal into steps (Plan-and-Execute, Tree-of-Thoughts, hierarchical task networks).
  • Multi-agent — multiple specialized agents (planner/executor/critic) coordinating via a shared blackboard or message bus; frameworks: CrewAI, AutoGen, LangGraph.
  • Reflection / critique-revise — an agent (or critic agent) reviews and improves a draft; the cheapest reliable quality boost.
  • Neurosymbolic — couple a neural proposer (LLM) with a symbolic verifier (rule engine, solver, type checker) so answers are checked, not just generated (P14).

Evaluation, observability & safety (P16)

  • Exact match / F1 — string-overlap metrics for closed-form answers.
  • pass@k — probability that at least one of k samples passes (code/agent eval).
  • LLM-as-judge — use a strong model with a rubric to score open-ended outputs; cheap, scalable, biased — calibrate against humans.
  • Calibration / ECE — does a model's confidence match its accuracy? Expected Calibration Error bins predictions and measures the gap.
  • Guardrails — input/output filters (PII, jailbreak, toxicity, schema validation) wrapping the model.
  • Run tracking (W&B / MLflow) — log params, metrics, artifacts per run for reproducibility and comparison; the experiment system of record.
  • Hallucination — confident, fluent, wrong; the failure mode RAG, grounding, and verification fight.
  • Drift — input/output distribution shifts over time; monitored in production to catch silent quality decay.

Systems & hardware (P17)

  • CUDA / kernel / thread / warp / block / SM — the GPU execution hierarchy; a kernel runs on a grid of blocks of threads scheduled in 32-thread warps on Streaming Multiprocessors.
  • Occupancy — active warps per SM ÷ max; too-low occupancy starves the latency-hiding that makes GPUs fast.
  • Memory coalescing — adjacent threads reading adjacent memory = one transaction; the #1 GPU performance lever.
  • Roofline model — plot of attainable FLOPs vs arithmetic intensity; the ridge point is where you flip from bandwidth- to compute-bound.
  • Operator fusion — combine ops (e.g. matmul+bias+GeLU) into one kernel to cut memory round-trips; what torch.compile/FlashAttention do.
  • FlashAttention — fused, IO-aware attention that never materializes the full T×T score matrix; the reason long context is feasible.