Cheat Sheet — Senior AI Engineer

The numbers, formulas, and one-liners to tattoo on your arm. Pair with GLOSSARY.md.

The arithmetic that opens every design review (P00)

Decode is memory-bandwidth bound. tokens/sec ≈ mem_bandwidth / bytes_per_token_read. A 7B fp16 model on 1 TB/s HBM: 1e12 / 14e9 ≈ 70 tok/s single-stream ceiling — batching amortizes the weight read across requests, which is the whole point of P09.

Softmax & attention (P02) — the numerically-safe form

softmax(x)_i = exp(x_i - max(x)) / Σ_j exp(x_j - max(x))      # subtract max → no overflow
Attention(Q,K,V) = softmax( Q Kᵀ / √d_k  +  causal_mask ) V    # mask future = -inf

• √d_k keeps the dot products from saturating softmax as d_k grows.
• MHA: split d_model into h heads of d_k = d_model/h, attend, concat, project.
• GQA/MQA: fewer K/V heads than Q heads ⇒ smaller KV-cache (the modern default).
• RoPE: rotate (q,k) pairs by θ_pos; relative position falls out of the dot product.

Autograd & training (P03, P10)

• Reverse-mode: forward builds the graph; backward does grad_input = grad_output · ∂out/∂in in reverse topological order. One backward pass ⇒ all gradients.
• Cross-entropy + softmax gradient simplifies to (p − y) — memorize this.
• AdamW step: m=β₁m+(1−β₁)g; v=β₂v+(1−β₂)g²; θ −= lr·(m̂/(√v̂+ε)) + lr·wd·θ.
• Ring all-reduce: 2(N−1) steps, each moving tensor/N; total ≈ 2·tensor bytes/GPU regardless of N — bandwidth-optimal.
• ZeRO memory: stage 1 shards optimizer (÷N), stage 2 +gradients, stage 3 +params.
• Pipeline bubble fraction ≈ (P−1)/(M + P−1) for P stages, M micro-batches — more micro-batches ⇒ smaller bubble.

Fine-tuning & quantization (P05, P06)

• LoRA: W' = W + (α/r)·B·A; trainable params = r·(d+k) vs d·k — orders of magnitude fewer. r=8, α=16 is a common default; merge B·A into W for zero inference cost.
• QLoRA: NF4 4-bit base (frozen) + double-quant scales + paged optimizer + LoRA adapters.
• KD loss: T² · KL(softmax(z_t/T) ‖ softmax(z_s/T)) + α·CE(y, z_s) — the T² keeps gradient magnitude scale-stable.
• Affine quant: q = clamp(round(x/s) + z, qmin, qmax), x̂ = s·(q − z). Symmetric ⇒ z=0.
• Per-channel (weights) beats per-tensor; keep activation outliers in higher precision (SmoothQuant/AWQ) or accuracy tanks.

Alignment (P07)

• Bradley-Terry reward loss: −log σ(r(chosen) − r(rejected)).
• DPO loss: −log σ( β·[ (logπ_θ(y_w) − logπ_ref(y_w)) − (logπ_θ(y_l) − logπ_ref(y_l)) ] ). No reward model, no RL loop, uses the frozen reference policy. β≈0.1 typical.
• PPO objective: clipped min(ratio·Â, clip(ratio,1±ε)·Â) − λ·KL(π_θ ‖ π_ref).
• Always keep the reference/KL term or the policy reward-hacks into gibberish.

Decoding (P08)

top_k=1 ≡ greedy. temperature → 0 ≡ greedy. Constrained decode > "please return JSON."

Serving (P09)

• PagedAttention: KV in fixed blocks (e.g. 16 tokens) ⇒ ~0 fragmentation, prefix sharing, high batch occupancy.
• Continuous batching: 2–4× throughput vs static — sequences join/leave the batch per step.
• Speculative decoding: draft k tokens with a small model, verify in one big-model pass, accept longest correct prefix ⇒ 2–3× faster, identical output distribution.
• SLOs: TTFT (prefill) for responsiveness, TPOT (decode) for streaming feel.
• Throughput↑ ⇒ batch↑ ⇒ latency↑. Pick the SLO first, then the max batch that meets it.

Retrieval / RAG (P11)

• Similarity: cosine = (a·b)/(‖a‖‖b‖); normalize once, then it's a dot product.
• HNSW: greedy descent through a multi-layer small-world graph; M (degree) and ef (search width) trade recall vs latency. IVF: cluster, search nprobe nearest clusters.
• Eval the retriever (recall@k, nDCG) separately from the generator (faithfulness) — most "RAG is bad" bugs are retrieval bugs.
• MMR rerank: argmax λ·rel(d) − (1−λ)·max_sim(d, selected) — relevance and diversity.

Agents (P12–P14)

• ReAct step: Thought → Action(tool, args) → Observation → …→ Final. Cap the loop (max steps) or it runs forever / burns tokens.
• Tools: typed JSON-schema args, validate before executing, return errors as observations so the agent can recover.
• Memory tiers: buffer (recent turns) → summary (compressed) → semantic (vector recall).
• Multi-agent: planner decomposes, executors act, critic verifies; a blackboard holds shared state. Termination detection or it deadlocks.
• Neurosymbolic: LLM proposes, a deterministic verifier (rules/solver/types) accepts or rejects → trustworthy answers.

Evaluation & safety (P16)

• Eval the retriever, the generator, and the system separately; one number hides regressions.
• LLM-as-judge: cheap and scalable but biased (position, verbosity, self-preference) — randomize order, use a rubric, spot-check against humans.
• ECE: bin by confidence, measure |accuracy − confidence| per bin, weight by bin size.
• Guardrails are layered: input filter → constrained decode → output validator → human review for high-stakes.

GPU performance (P17)

• Roofline: attainable = min(peak_FLOPs, bandwidth × arithmetic_intensity); left of the ridge = bandwidth-bound, right = compute-bound.
• Coalesced access (adjacent threads → adjacent addresses) can be 10× uncoalesced.
• Occupancy = active warps / max warps per SM; raise it by cutting registers/shared-mem per thread — but past "enough to hide latency" it stops helping.
• Fuse to cut HBM round-trips (FlashAttention, torch.compile); the bottleneck is usually memory traffic, not math.

The senior tradeoff one-liners

• "Make it bigger" trades inference cost forever for one-time training gains — quantize and distill before you scale.
• "Add an agent step" trades latency and a failure mode for autonomy — verify every tool call.
• "Multi-region / multi-GPU" trades complexity and $ for availability/throughput — show the SLA and MFU math first.
• "Lower the temperature" trades diversity for reliability — and constrained decoding beats both for structured output.
• Quality is a distribution: report a metric with a confidence interval, never a single cherry-picked generation.