Concepts Cheatsheet — Top 40 Q&A

Use this for rapid review. Answers are at the Senior Staff level — go deeper than the surface definition.

Section 1: Quantization (12 questions)

Q1. What is the quantization formula for INT8?

For affine (asymmetric) quantization: $$x_q = \text{clip}\left(\text{round}\left(\frac{x}{s}\right) + z,\ 0,\ 255\right)$$

where scale $s = (x_{max} - x_{min}) / 255$ and zero-point $z = -\text{round}(x_{min}/s)$.

Dequantization: $x \approx s \cdot (x_q - z)$

For symmetric quantization: $z = 0$, scale = $\max(|x|) / 127$.

Q2. What is the difference between per-tensor and per-channel quantization?

Per-tensor: one (scale, zero_point) for the entire tensor. Simple, low overhead.

Per-channel: one (scale, zero_point) per output channel. 4–8× better accuracy for weights because each output filter has its own dynamic range. Standard for weight quantization in deployment. Activations typically use per-tensor.

Q3. What is SQNR and what values are acceptable?

Signal-to-Quantization-Noise Ratio: $\text{SQNR (dB)} = 10 \log_{10}\frac{\mathbb{E}[x^2]}{\mathbb{E}[(x - \hat{x})^2]}$

For INT8: SQNR ≈ 48–52 dB (acceptable for most tasks). For INT4: SQNR ≈ 24–28 dB (may cause accuracy degradation on sensitive tasks). Below 20 dB: typically unacceptable for production.

Q4. What is the Straight-Through Estimator and why is it used in QAT?

In QAT, the quantize-dequantize operation is inserted in the forward pass: $\hat{x} = \text{round}(x/s) \cdot s$

The round() function has zero gradient almost everywhere (gradient is 0 except at discontinuities). Without STE, gradients don't flow backward through the quantizer.

STE approximation: $\frac{\partial \hat{x}}{\partial x} \approx 1$ (treat the quantizer as identity in the backward pass). This allows gradients to flow to weights during QAT training.

Q5. Explain GPTQ in 3 sentences.

GPTQ quantizes each row of each weight matrix sequentially using second-order information. For each weight column $j$, it quantizes $w_j$, computes the quantization error, and propagates that error to the remaining columns using the inverse Hessian of the layer's activation covariance. This is based on the OBQ algorithm but uses lazy batch updates for efficiency.

Q6. What is AWQ's key insight?

Not all weights matter equally. Weights associated with large-magnitude activations (salient channels) cause disproportionately large quantization error. AWQ scales up salient weight channels before quantization (making them "easier" to quantize), then scales down the corresponding activations by the same factor. The product $W \cdot X$ is preserved exactly.

Q7. Explain SmoothQuant.

SmoothQuant migrates quantization difficulty from activations to weights. In transformers, outliers in activations (up to 600× larger than mean) make per-tensor INT8 quantization of activations lossy. SmoothQuant: multiply activations by $\text{diag}(s)^{-1}$ and weights by $\text{diag}(s)$ where $s_j = \max|X_j|^\alpha / \max|W_j|^{1-\alpha}$. Activations become easier to quantize; weights are slightly harder but manageable.

Q8. When would you choose QAT over PTQ?

QAT is warranted when: PTQ accuracy is unacceptable (typically when target bit-width ≤ INT4, or task is sensitive), enough labeled training data is available (typically 10K–100K examples), compute budget allows fine-tuning (hours to days on GPU). PTQ is preferred for large models (7B+) where fine-tuning is expensive. AWQ/GPTQ are the middle ground: PTQ with smarter algorithms that close much of the QAT accuracy gap.

Q9. What is the "quantization cliff" in LLMs?

LLMs have a phase transition in accuracy degradation: from FP16 → INT8 is nearly lossless (< 0.5% MMLU), from INT8 → INT4 causes a moderate drop (1–3%), but from INT4 → INT3 can cause catastrophic degradation (10–30%). This is because at INT3 and below, the quantization noise exceeds the information content of the activations in some layers. Outlier channels in LLMs (large-magnitude activations) are particularly vulnerable.

Q10. What is double quantization in QLoRA?

Double quantization quantizes the quantization constants (scale factors). Standard INT4 quantization stores one FP32 scale per group of 64 weights = 0.5 bits/weight overhead. Double quantization further quantizes those scales from FP32 to INT8: reduces the scale overhead from 0.5 to ~0.125 bits/weight — about 0.37 bits/weight total savings for 7B parameter models.

Q11. How do you choose the calibration dataset for PTQ?

The calibration set should match the deployment distribution as closely as possible. For language models: WikiText-2 or C4 for general language PTQ; task-specific text for domain PTQ. Size: 128–512 samples is typically sufficient (diminishing returns beyond that). Calibration data affects activation statistics → scale computation → quantization accuracy. Mismatch between calibration and deployment distributions is a common source of unexpected accuracy degradation.

Q12. What is the quantization error accumulation problem in deep networks?

In a deep network, quantization error in early layers gets amplified by subsequent layers. A 1-bit quantization error in the embedding layer affects all subsequent attention computations. Mitigation strategies: (1) use higher precision for first/last layers, (2) apply SmoothQuant for outlier-prone layers, (3) use mixed-precision quantization (sensitive layers → INT8, bulk → INT4), (4) apply error correction via GPTQ's Hessian-based column-by-column quantization.

Section 2: PyTorch Framework Internals (8 questions)

Q13. Describe the PyTorch dispatch stack.

A tensor operation (e.g., x + y) goes through:

1. Python: __add__ calls torch.add
2. C++ dispatcher: resolves via dispatch key priority list (Functionalize → Autograd → CUDA/CPU)
3. Autograd: creates AddBackward0 node in the autograd graph (if requires_grad)
4. CompositeImplicitAutograd or backend-specific: calls into cuBLAS/CUDA kernels

Key dispatch keys in priority order: Functionalize, FuncTorchDynamicLayerFrontMode, InplaceOrView, Autograd (per-backend), CUDA, CPU.

Q14. What is TorchDynamo and how does it capture graphs?

TorchDynamo intercepts CPython bytecode evaluation using torch._dynamo.eval_frame.set_eval_frame(). It installs a frame evaluation callback that observes each operation, builds an FX graph, and breaks the graph when it encounters Python control flow that depends on tensor values. It generates "guards" (Python predicates) that must be true for the compiled graph to be valid — if guards fail, Dynamo recompiles.

Q15. What is a graph break in torch.compile and how do you debug it?

A graph break is a point where TorchDynamo cannot continue tracing — it stops, returns to Python, and starts a new compiled subgraph after the break point. Common causes: tensor.item(), tensor.numpy(), if tensor > 0 (data-dependent control flow), non-PyTorch library calls, unsupported Python constructs.

Debug: torch._dynamo.explain(model)(x) prints each graph break reason. torch._dynamo.config.verbose = True prints breaks during compilation.

Q16. What is AOTAutograd?

Ahead-of-Time Autograd: lifts the backward pass computation (autograd graph) into a static graph before execution. Instead of building the backward graph at runtime, AOTAutograd traces through both the forward and backward pass during compilation, producing a "joint" forward+backward graph that can be optimized end-to-end by the backend. This enables the backend to fuse forward ops with their backward kernels.

Q17. What is the difference between torch.fx.symbolic_trace and torch.export?

symbolic_trace: Python-level tracer; follows Python code; may not handle dynamic control flow or data-dependent shapes; suitable for static model analysis.

torch.export: TorchDynamo-based; handles dynamic shapes via symbolic shape expressions; produces a strict aten-level graph with no Python dependencies; intended for mobile/server deployment; verified correct by construction.

Q18. Describe the FX node types.

• placeholder: function input (graph edge from outside)
• get_attr: access module parameter/buffer
• call_function: free function call (torch.relu(x))
• call_method: tensor method call (x.reshape(...))
• call_module: nn.Module forward call (self.linear(x))
• output: function return value

Q19. What is torch.compile's performance overhead?

First call: compilation overhead (Dynamo tracing + backend compilation) — 5–30 seconds depending on model size. Subsequent calls (cache hit): near-zero overhead (guard check is O(1)). Cold start is the main cost. In production: precompile before serving, or use torch.compile(mode='reduce-overhead') which uses CUDA graphs to eliminate kernel launch overhead.

Q20. How do custom ops interact with torch.compile?

Custom ops registered with TORCH_LIBRARY / torch.library participate in the dispatch system and are visible to the FX tracer as opaque nodes. To make them compilable: (1) register a fake/meta kernel that computes output shapes without CUDA, (2) register an autograd kernel for gradient support. Without a meta kernel, torch.compile cannot trace through the op (graph break or error).

Section 3: Hardware and Profiling (8 questions)

Q21. Explain the roofline model.

Performance (FLOPS/s) = min(Peak FLOPS, Peak BW × Arithmetic Intensity).

Arithmetic Intensity = FLOPS / bytes transferred from memory.

Crossover point = Peak FLOPS / Peak BW. On A100: 312 TFLOPS / 2 TB/s = 156 FLOPS/byte.

Ops with AI < 156 on A100 are memory-bound — benefit from reducing memory traffic (fusion, tiling). Ops with AI > 156 are compute-bound — benefit from better compute utilization (tensor cores, mixed precision).

Q22. What is the arithmetic intensity of decode-time attention?

At batch_size=1, seq_len=n: attention reads KV cache (O(nd) bytes), computes O(nd) FLOPS. AI ≈ 4nd / 8nd = 0.5 FLOPS/byte — deeply memory-bound regardless of hardware. This is why LLM decode performance scales with DRAM bandwidth, not TOPS.

Q23. When do you use nsys vs ncu?

nsys (Nsight Systems): system timeline — CPU/GPU timeline, kernel launches, data transfers, synchronization gaps. Use when you need to understand where time is spent at a high level — is GPU busy, are there transfers, is there CPU overhead?

ncu (Nsight Compute): per-kernel deep analysis — FLOP utilization, memory bandwidth, warp stalls, occupancy. Use when you've identified which kernel is slow and want to know why.

Q24. What is CUDA occupancy and when does it matter?

Occupancy = active warps / max warps per SM. Limited by: register pressure, shared memory usage, block size.

High occupancy matters when the kernel is latency-bound (waiting for memory): more warps → scheduler hides latency by switching. Low occupancy is fine if the kernel is compute-bound (100% FLOP utilization).

Low occupancy from excessive register use is a common issue with complex kernels — use launch_bounds to limit registers and increase occupancy.

Q25. What is the HTP on Qualcomm Snapdragon?

Hexagon Tensor Processor: dedicated ML accelerator. Components: HMX (Hexagon Matrix Extension) for matrix multiplications, HVX (Hexagon Vector Extension) for elementwise/activation ops, 8 MB VTCM on-chip SRAM (2 TB/s bandwidth). Peak: ~73 TOPS INT8 on 8 Gen 3. Ops that fall back to CPU cause expensive data transfers.

Q26. Why do some ops fall back to CPU on the HTP?

1. Dynamic output shapes (e.g., nonzero(), unique())
2. Op not in QNN's HTP library for the given data type (e.g., FP32 ops are CPU-only; use INT8 or FP16)
3. Data type mismatch between compile-time specs and runtime values
4. Graph topology that breaks HTP's requirements (e.g., non-supported data flows)

CPU fallback causes a data copy between CPU and HTP memory — often 100× more expensive than the op itself.

Q27. How do you benchmark a PyTorch function reliably?

Use torch.utils.benchmark.Timer:

1. It calls torch.cuda.synchronize() before and after to account for async GPU execution
2. It runs warm-up iterations first (eliminates JIT compilation and caching artifacts)
3. It reports mean, median, and IQR over multiple runs
4. It disables garbage collection during timing

Never use time.time() directly for CUDA timing — it does not synchronize the GPU.

Q28. What is memory coalescing?

When 32 threads in a CUDA warp access 32 consecutive memory addresses (aligned to 128 bytes), the GPU combines them into one DRAM transaction. Non-coalesced access (e.g., strided access where each thread's address is not adjacent) requires one transaction per thread — 32× more DRAM bandwidth.

Fix: ensure matrix access patterns are row-major for row-parallel kernels, or use shared memory tiling to convert non-coalesced → coalesced access patterns.

Section 4: Model Accuracy and Evaluation (6 questions)

Q29. How do you compute perplexity correctly for long documents?

Use the stride trick: process the document in overlapping windows so each token has the maximum context from preceding tokens. The crucial detail: only count the loss for tokens that were NOT in the previous window (they had full context). See Phase 09 HITCHHIKERS-GUIDE for the stride implementation.

Q30. Why is McNemar's test preferred over a t-test for accuracy comparison?

McNemar is for paired binary outcomes — each test example is either correct (1) or incorrect (0). The t-test assumes continuous, independently sampled values. Two models tested on the same examples are not independent — their errors are correlated. McNemar's test exploits this correlation (only examples where models disagree are informative).

Q31. You observe MMLU dropped 0.8% after INT4 quantization. Significant?

Compute McNemar's test: $\chi^2 = (|n_{01} - n_{10}| - 1)^2 / (n_{01} + n_{10})$. With 14k examples and 0.8% drop ≈ 112 net changes, if the standard error is 0.37% (typical), then 0.8% is ~2.2 SE. Apply McNemar to get exact p-value. With evenly split changes (56+56 disagreements): $\chi^2 = (0-1)^2/112 ≈ 0.01$, p ≈ 0.92 — not significant. Need to see the actual per-example results.

Q32. What is calibration in the context of model accuracy?

A calibrated model's stated confidence = empirical accuracy in that confidence bucket. Measured by ECE (Expected Calibration Error). Quantization often worsens calibration — the model becomes more overconfident or underconfident even if overall accuracy is maintained. Check calibration before shipping quantized models, especially in high-stakes domains.

Q33. What benchmarks would you run to characterize a new quantization method?

Perplexity on WikiText-2 (sensitivity to language modeling), MMLU 5-shot (knowledge/reasoning), HellaSwag (common sense), ARC-Challenge (hard reasoning), GSM8K (math). Also: calibration (ECE), throughput and latency benchmark, memory footprint. Run at multiple bit-widths to map the accuracy-efficiency frontier.

Q34. How do you detect accuracy regression caused by a new optimization?

Set up: (1) eval harness that produces per-example results (not just aggregate), (2) baseline stored for the current main branch, (3) McNemar's test comparing per-example results, (4) policy: fail if p < 0.05 AND absolute drop > threshold (0.5%). The threshold prevents noise-level changes from failing CI while still catching real regressions.

Section 5: GenAI Systems and Edge (6 questions)

Q35. How would you run a 7B LLM on a device with 4 GB DRAM?

1. INT4 quantization: 7B × 0.5 bytes = 3.5 GB (just fits)
2. Use NF4 or AWQ for best accuracy at INT4
3. Sliding window attention to limit KV cache growth
4. Chunked prefill to bound peak memory
5. If 3.5 GB still too much: use INT2 (AQLM) for ~1.75 GB, with LoRA accuracy recovery

Q36. What is speculative decoding and prove it is lossless.

Speculative decoding: draft model proposes k tokens; target model verifies all k+1 positions in one forward pass; accept tokens where draft distribution ≤ target distribution; reject and resample from corrected distribution otherwise.

Losslessness proof: acceptance probability = min(1, p(x)/q(x)), rejection probability = 1 - ΣP(accept). Corrected distribution = norm(max(0, p(x)-q(x))). The marginal output distribution = min(p,q) + correction = p(x). ∎

Q37. What is the difference between SNPE and QNN?

SNPE (Snapdragon Neural Processing Engine) is the legacy SDK (2nd generation). QNN (Qualcomm Neural Network SDK) is the current SDK (3rd generation) with per-channel INT8/INT16/FP16, better op coverage, wider device support (Snapdragon 8 Gen 1+), and a cleaner API. Use QNN for all new projects; SNPE only for legacy embedded targets.

Q38. Explain LoRA. Why does it work?

LoRA adds trainable low-rank matrices $W = W_0 + BA$ where $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times k}$, $r \ll \min(d,k)$. It works because pre-trained model weight updates have low intrinsic rank — the fine-tuning signal lives in a much lower-dimensional subspace than the full weight matrix. A rank-64 LoRA captures most of the useful adaptation with only $64(d+k)$ parameters vs $dk$ for full fine-tuning.

Q39. What is paged attention and why does it matter?

Paged attention stores the KV cache in non-contiguous memory pages (e.g., 16 tokens/page). A page table maps logical sequence positions to physical pages. Benefits: (1) no memory waste from pre-allocating max-sequence-length buffers, (2) multiple requests can share common prefix pages (prefix caching), (3) easy eviction/swap for memory-pressure situations. Enables 2–3× higher GPU utilization in serving systems vs naive contiguous KV cache.

Q40. Design the accuracy evaluation cadence for a new chip tapeout.

1. Pre-tapeout (simulation): run full accuracy suite on ISA simulator; compare to GPU golden reference; any regression blocks tapeout
2. Post-tapeout bringup: run on 3–5 representative models; verify INT8 accuracy within 0.5% of FP16 baseline; identify HTP fallback ops
3. Production release: run full benchmark suite (20+ models, 3 bit-widths each); publish accuracy comparison table; run McNemar tests for all significant deltas
4. Ongoing CI: every SDK release triggers automated accuracy regression check; any model that regresses >1% accuracy flags a review