Glossary — Model Accuracy & AI Performance

Every term the track (and the Qualcomm Senior Staff JD) assumes, defined from first principles. Grouped by phase area. Cross-references in [[brackets]] point to related entries.

Framework Internals (P01)

  • Dispatcher (c10::Dispatcher) — PyTorch's C++ router that, for every operator call, selects the kernel based on dispatch keys. The layer between the Python API and the backend kernels.
  • Dispatch key — a tag (CPU, CUDA, Autograd, Functionalize, Meta, …) that determines which registered kernel handles an op. A custom op must behave correctly under all relevant keys.
  • Autograd engine — builds a DAG during the forward pass (each op records its backward) and walks it in reverse-topological order on .backward(), accumulating gradients at the leaves.
  • AccumulateGrad — the autograd node that sums gradients into a leaf tensor's .grad.
  • retain_graph / create_graph — keep the graph after backward (for multiple backwards) / build a differentiable backward graph (for higher-order gradients / double backward).
  • torch.autograd.Function — a custom op with hand-written forward/backward; save_for_backward stashes tensors for the backward pass.
  • TORCH_LIBRARY / TORCH_LIBRARY_IMPL — C++ macros to declare an op schema and register a kernel for a given dispatch key.
  • Meta kernel — a kernel that computes only shapes/dtypes (no data), used for tracing, torch.compile, and shape inference. Missing it is a common custom-op bug.
  • Functionalization — a pass that rewrites in-place/mutating ops into functional equivalents so the graph is pure (needed by compile/export).
  • tf.function — TensorFlow's tracing decorator: runs the Python once to build a graph (FuncGraph), then executes the graph. Python side-effects happen at trace time only.
  • Retracing — TF re-runs tracing when it sees a new input signature; excessive retracing is a performance bug.
  • BF16 / FP16 / FP32 — brain-float16 (8-bit exponent, 7-bit mantissa — FP32's range, less precision), half (5-bit exp, 10-bit mantissa), single. [[accumulation-dtype]].
  • Catastrophic cancellation — large precision loss when subtracting nearly-equal floats; a driver of low-precision instability.

Model Architecture (P02)

  • Self-attention — each token attends to all others; cost is O(N²) in sequence length N (the motivation for [[FlashAttention]] and sparse/linear variants).
  • FFN / MLP block — the per-token feed-forward in a transformer; the bulk of the params.
  • Mixture-of-Experts (MoE) — replace the dense FFN with N experts; a router sends each token to the top-k experts. Cuts FLOPs/token but adds a dynamic gather/scatter the NPU dislikes.
  • Load-balancing loss — auxiliary loss that prevents MoE router collapse (a few experts hogging all tokens). Without it, "8 experts" becomes effectively 2.
  • State-Space Model (SSM) / Mamba — sequence model using a selective scan recurrence with near-linear cost instead of quadratic attention; the scan is sequential (hardware tradeoff).
  • ViT (Vision Transformer) — images as patch sequences fed to a transformer; trades CNN inductive bias for scale.
  • DINO — self-supervised distillation: an EMA teacher trains a student with no labels; centering + sharpening prevent representation collapse.
  • CLIP bridge — projecting vision encoder features into an LLM's token embedding space to make a multimodal model.

Quantization & Compression (P03, P10)

  • Quantization — mapping FP values to a small set of integers via a scale and zero-point.
  • Scale (s) — the step size: s = (x_max − x_min) / (q_max − q_min). Smaller scale → smaller error.
  • Zero-point (z) — the integer that maps to real 0 (asymmetric quant). Symmetric quant has z = 0.
  • Symmetric vs asymmetric — range centered at 0 (z=0, simpler hardware) vs arbitrary [min,max] (uses z). Watch the dtype: 8-bit unsigned is [0,255], signed is [−128,127].
  • Per-tensor vs per-channel — one scale for the whole tensor vs one per output channel (per-channel is more accurate; the hardware must support the layout).
  • PTQ (Post-Training Quantization) — quantize a trained model using a little calibration data. Fast; first resort.
  • QAT (Quantization-Aware Training) — simulate quantization during training (fake-quant forward, [[straight-through-estimator]] backward) so the model learns robustness. For when PTQ drops too much.
  • Straight-through estimator (STE) — treat the non-differentiable round() as identity in the backward pass so gradients flow through fake-quant.
  • Calibration — choosing the clipping range from observed activations (min/max, percentile, MSE-optimal, entropy/KL).
  • GPTQ — second-order weight quantization: quantize column-by-column, compensating remaining weights for the error using the (approx) Hessian. Holds accuracy at 4-bit.
  • AWQ (Activation-aware Weight Quantization) — protect the small set of salient weight channels (found via activation magnitudes) and crush the rest to 4-bit.
  • NF4 — a 4-bit "normal float" datatype tuned to the normal distribution of NN weights (used by [[QLoRA]]).
  • QLoRA — fine-tune a frozen 4-bit base model via small high-precision low-rank adapters.
  • Mixed precision — keep sensitive layers high-precision, crush the rest; the real size/accuracy win.
  • Sensitivity analysis — quantize one layer at a time, measure the accuracy hit, rank layers.

Compilers & IR (P04, P05)

  • torch.compile — PyTorch's JIT: [[Dynamo]] captures the graph, a backend lowers it.
  • TorchDynamo — captures Python bytecode into an [[FX graph]]; graph-breaks on code it can't trace (data-dependent control flow, .item(), unsupported ops).
  • Graph break — Dynamo splitting a function into multiple graphs with eager glue; the #1 reason torch.compile underdelivers.
  • TorchInductor — the default backend; lowers FX to Triton (GPU) / C++ (CPU) kernels.
  • FX graph — PyTorch's inspectable IR of nodes (placeholder, call_function, call_module, call_method, output). Dialect matters: symbolic_trace of modules gives call_module; aot/make_fx gives aten ops.
  • Operator fusion — combining ops (e.g. Linear→GELU) into one to cut memory traffic / kernel launches; only safe when the producer's output has a single consumer.
  • Custom backend — a function registered with Dynamo that receives an FX GraphModule, transforms it (classify NPU-eligible nodes, offload, fall back to CPU), and returns a callable.
  • ONNX — Open Neural Network Exchange; a cross-framework model interchange format. Export can drop/decompose ops — verify numerically after export.
  • opset — the ONNX operator-set version; mismatches cause export/runtime failures.
  • TVM — an ML compiler with auto-tuning (searches loop schedules for the fastest kernel on a given target). IRs: Relay (high-level), TIR (low-level).
  • MLIR — Multi-Level IR: a compiler infrastructure of progressively-lowered dialects; the backbone of many modern hardware compilers.
  • Lowering — translating a higher-level IR to a lower-level one (coarse ops → fine loops); each step optimizes and risks bugs.

Hardware / NPU (P06)

  • NPU (Neural Processing Unit) — a fixed-function accelerator for tensor math; on Qualcomm, the Hexagon Tensor Processor (HTP) in the AI Engine. Loves static, dense, fixed-point work.
  • Coverage / offload rate — fraction of model compute that runs on the NPU vs falls back to CPU. The key health metric; low coverage silently wrecks latency/power.
  • CPU fallback — running an unsupported op on the CPU mid-graph; the round-trip is expensive.
  • Fixed-point accumulator — the NPU's integer accumulator; finite width → can overflow if scales are mis-set.
  • AI Hub — Qualcomm's service for compiling/profiling/deploying models to target devices.
  • Operator coverage / workload synthesis — ensuring (or making) every op in a model runnable on the hardware; co-design feedback to the HW team.
  • Energy per inference (joules) — the real edge budget; power and latency couple via thermal throttling.

Profiling & Inference (P07, P08)

  • Roofline model — plots achievable performance vs arithmetic intensity (FLOPs/byte); classifies a kernel as compute-bound or memory-bound, which dictates the fix.
  • Arithmetic intensity — FLOPs per byte of memory traffic; low intensity → memory-bound.
  • Compute-bound — limited by FLOPs/s; helped by faster math, not by moving fewer bytes.
  • Memory-bound — limited by bytes/s; helped by fusion, quantization, layout — not by more FLOPs. Most LLM inference is here.
  • CPU↔device sync — a .item()/.cpu() in a hot loop that serializes execution; the most common "accelerator at 30% utilization" cause.
  • Warmup — the first iterations (compile/alloc/cache) you must discard before timing steady-state; report the median of many runs.
  • FlashAttention — computes attention in tiles using online softmax, never materializing the N×N matrix; cuts memory traffic O(N²)→O(N). Exact, not approximate.
  • Online softmax — running-max + running-sum recurrence that lets softmax be computed incrementally over tiles.
  • KV cache — cached keys/values for past tokens so autoregressive decode is O(1)/token in attention; grows with sequence length (a memory killer; can be quantized/paged).
  • Speculative decoding — a small draft model proposes several tokens; the big model verifies them in one parallel pass. Exact (identical output), pure latency win.
  • Continuous batching — evicting finished sequences and admitting new ones every step to keep the accelerator saturated (server-side throughput).
  • Paged KV cache (vLLM) — managing KV memory in fixed pages to avoid fragmentation.

Accuracy & Evaluation (P09)

  • Pareto frontier — the set of models where you can't gain accuracy without adding latency; deployment picks a point on it.
  • Knee point — the frontier's point of diminishing returns; usually the deployment sweet spot.
  • Per-example delta / churn — which specific inputs changed verdicts between two models; hides regressions a matching aggregate score conceals.
  • Regression CI — automated accuracy gates that block a merge if an "optimization" drops accuracy below threshold.
  • Bootstrap CI / McNemar's test — statistical tools to decide whether an accuracy difference is real vs noise (paired model comparison).
  • Reproducibility — same seed/data/config → same number; the precondition for any comparison.

The numbers behind these terms (bit-widths, quant formulas, roofline math, memory sizing) live in CHEATSHEET.md.