Interview Prep 01 — Concepts Rapid-Fire

60+ questions a JD2 interviewer actually asks, with whiteboard-grade answers — concise enough to say out loud, deep enough to prove you understand the system. Grouped by topic. Each links to the full treatment. Drill until each answer is reflexive.

Table of Contents

A. Hardware & roofline

Q1. Why are GPUs better than CPUs for inference? CPUs are latency-optimized (few powerful cores, branch prediction) for serial, branchy work. GPUs are throughput-optimized (thousands of simple lanes) for massive regular parallel arithmetic. Inference is ~90% matrix multiply — massively parallel — so the GPU's "many weak lanes" matches the problem's "many independent multiply-adds." (K00 §1)

Q2. What's the memory hierarchy and why does it matter? Registers → shared/L1 SRAM → L2 → HBM → host RAM → disk. On-chip SRAM is ~10–30× faster than HBM but ~100× smaller; HBM is ~50× faster than PCIe. The whole game of optimization is touching HBM as little as possible — that's why FlashAttention and operator fusion exist. (K00 §3)

Q3. Compute-bound vs memory-bound — how do you tell? Compute arithmetic intensity (FLOPs ÷ bytes moved) and compare to the chip's ridge point (peak FLOPs ÷ peak bandwidth, ~295 FLOP/byte on H100). Below it → memory-bound (you're on the bandwidth slope); above → compute-bound (on the FLOPs roof). (K00 §4–5)

Q4. Draw the roofline. (Do it.) Log-log: a sloped memory roof (slope = bandwidth) meeting a flat compute roof at the ridge point. A kernel plots at its arithmetic intensity; its attainable performance is the lower of the two roofs. (K00 §5)

Q5. FLOPs of a matmul? 2·M·N·K (factor 2 = multiply + add). A P-parameter model costs ~2P FLOPs per token at inference. (K00 §4)

Q6. FP16 vs BF16? Same 16 bits; FP16 has 5 exponent / 10 mantissa (more precision, narrow range, overflows), BF16 has 8 exponent / 7 mantissa (FP32-range, less precision, more stable). BF16 is preferred when range/stability matters. (K00 §6)

Q7. What's a tensor core? A hardware unit that does a small matrix multiply-accumulate (e.g., 16×16 tile) in one op, only at peak in reduced precision (FP16/BF16/FP8/INT8). ~95% of an LLM's FLOPs run on them. (K00 §2,6)

Q8. Why do CUDA Graphs help decode? Decode launches dozens of tiny kernels per token; per-launch CPU overhead (~µs each) can dominate the tiny kernels. CUDA Graphs record the launch sequence once and replay it as one submission, killing the overhead. (K00 §8)

Q9. NVLink vs PCIe — why care? NVLink ~900 GB/s intra-node vs PCIe ~64 GB/s. Tensor parallelism all-reduces every layer, so it must run over NVLink; over PCIe the comms erase the speedup. (K00 §9)

Q10. How does a Groq LPU differ from a GPU? SRAM-only, no HBM — all weights in on-chip SRAM for deterministic, lowest single-stream latency, paying with tiny per-chip capacity (needs many chips). It's the "bandwidth/capacity" roofline trade made physical. (K00 §10)

B. LLM inference internals

Q11. Prefill vs decode? Prefill processes the whole prompt in parallel (one pass, compute-bound, sets TTFT). Decode generates one token at a time, sequentially, attending to cached KV (memory-bound, sets TPOT). Different bottlenecks → different fixes. (K01 §2,4)

Q12. Why is decode memory-bound? Batch-1 decode does ~2P FLOPs but reads ~2P bytes of weights → arithmetic intensity ~1, far below the ~295 ridge point. Tensor cores starve waiting on HBM. (K01 §4)

Q13. What is the KV cache and why does it exist? Cached Key/Value vectors of all prior tokens, so each decode step computes K/V only for the new token instead of recomputing all (O(t) per step instead of O(t²)). It's what makes autoregressive decoding affordable. (K01 §3)

Q14. KV cache size formula? 2 · n_layers · n_kv_heads · head_dim · seq_len · batch · bytes. The "2" is K and V. It frequently caps concurrency before compute does. (K01 §3)

Q15. MHA vs MQA vs GQA vs MLA? MHA: each query head has its own K/V (max KV). MQA: all heads share one K/V (KV ÷ n_heads, small quality cost). GQA: groups share K/V (the modern compromise, Llama-3/Mistral). MLA: low-rank latent KV (DeepSeek, smallest KV). KV size ∝ n_kv_heads. (K01 §5)

Q16. What does FlashAttention actually do? Computes attention without materializing the N×N score matrix in HBM — tiles Q/K/V into SRAM and uses online softmax. Identical math, far less memory traffic → big speedup and long-context feasibility. An IO optimization, not an approximation. (K01 §6)

Q17. Why does batching help so much? Decode reads the weights once per step regardless of batch size, but produces B tokens. So weight-read cost amortizes across the batch → throughput grows ~linearly with batch until the compute roof or KV memory caps it. (K01 §7)

Q18. TTFT vs TPOT vs goodput? TTFT = time to first token (prefill-bound). TPOT/ITL = time per output token (decode-bound). Goodput = throughput counting only SLO-meeting requests — the honest metric. (K01 §8)

Q19. Explain speculative decoding. A cheap draft model proposes k tokens; the big target verifies all k in one parallel pass and accepts the longest correct prefix. 2–3× decode speedup, exact output distribution preserved. Helps most at low batch where the GPU has spare compute. (K01 §9)

Q20. What's prefix caching and when is it huge? Reuse the KV of a shared prefix (system prompt, few-shot, RAG context) across requests instead of re-prefilling. Massive for agent/RAG with big fixed prompts. (K01 §9, K04 §5)

C. Quantization

Q21. Why quantize? Memory ÷, bandwidth ÷ (→ faster memory-bound decode), tensor-core throughput × — for a small, recoverable accuracy cost. The highest-leverage inference optimization. (K02 §1)

Q22. Walk the quantization equation. q = round(r/s) + z; r ≈ s(q−z). s = scale (step size), z = zero-point. INT8 symmetric: s = max(|r|)/127, z=0. Matmul accumulates integer products in INT32, applies scales at the margins. (K02 §2)

Q23. Per-tensor vs per-channel vs per-group? One scale per tensor (cheap, outlier-sensitive) / per channel (isolates outliers, near-free, default for weights) / per group of N (best, used for INT4). Put the scale boundary where outliers live. (K02 §3)

Q24. Why does naive INT8 break LLMs? Activations develop systematic outlier channels 10–100× larger than the rest; per-tensor scaling lets those outliers crush everyone else's resolution. Fixes: keep outliers in FP16 (LLM.int8), migrate them into weights (SmoothQuant), or protect salient channels (AWQ). (K02 §8)

Q25. GPTQ vs AWQ? Both W4 weight-only. GPTQ: layer-wise, uses second-order/Hessian info to compensate quantization error. AWQ: activation-aware, scales the salient ~1% of weight channels to protect them, no backprop, faster to produce. Both great; AWQ is often the easier high-quality default. (K02 §9)

Q26. Weight-only vs weight+activation quant? Weight-only (W4A16: GPTQ/AWQ) attacks the memory/bandwidth bottleneck → great for decode, easy, robust. Weight+activation (W8A8: SmoothQuant/FP8) attacks compute too → great for prefill, mandatory for integer-only accelerators, harder (activation outliers). (K02 §9)

Q27. PTQ vs QAT? PTQ: quantize a trained model with a small calibration set, no retraining — 90% of cases. QAT: simulate quantization during training, recovers more accuracy at low bits, needs the pipeline. Start PTQ, escalate to QAT only if PTQ misses the floor. (K02 §6)

Q28. Why FP8 over INT8? FP8 is a float, so it keeps dynamic range → tolerates activation outliers far better than INT8, near-FP16 quality at ~2× speed with less fuss. Needs Hopper+. (K02 §9)

Q29. How do you prove quantization didn't break the model? Perplexity (weak signal) → task benchmarks on the customer's hardest task (MMLU/GSM8K/HumanEval/domain) → side-by-side on real traffic. Report baseline, quantized, and delta with conditions. Watch for reasoning/math/code cliffs. (K02 §11)

Q30. What's mixed precision really? Per-layer bit-width assignment via sensitivity analysis — keep sensitive layers (embeddings, LM head, first/last blocks) higher precision, push robust middle layers to INT4. Produces an accuracy/size Pareto frontier. (K02 §10)

D. Model conversion & compilers

Q31. Why convert at all? Training frameworks are eager/flexible/slow; inference compilers freeze the graph and specialize it to one chip — fusion, kernel selection, quantization, memory planning. (K03 §1)

Q32. What is ONNX? A vendor-neutral serialized graph format with a standardized opset — the on-ramp from PyTorch/TF to most accelerators (TensorRT, OpenVINO, Qualcomm AIC, ORT). (K03 §3)

Q33. Tracing vs scripting export? Tracing runs the model and records ops (simple, but loses data-dependent control flow and bakes in shapes). Scripting/graph-capture analyzes the code (handles control flow, pickier). A trace that captured one branch is the first suspect when a port misbehaves on some inputs. (K03 §2)

Q34. What is operator fusion and why does it help? Merge a chain of ops into one kernel so intermediates stay in registers/SRAM instead of round-tripping HBM. Same math, far less data movement — 2–5× on memory-bound chains, plus less launch overhead. (K03 §5)

Q35. Top 3 things that break in a port? (a) Unsupported/custom ops → rewrite, decompose, fallback, or plugin. (b) Dynamic shapes → optimization profiles / dynamic axes / bucketing. (c) Numerical divergence → bisect precision, compare layer-by-layer against the FP16 reference. (K03 §6)

Q36. Why does TensorRT build on the deployment hardware? It autotunes kernels for the specific GPU SKU and precision; an engine built on an H100 may not load/perform on an A100. (K03 §4)

Q37. torch.compile vs TensorRT? torch.compile (Dynamo+Inductor+Triton) stays in PyTorch — fast iteration, watch for graph breaks. TensorRT/vendor compiler — max performance, INT8/FP8 autotuning, no Python runtime, non-NVIDIA targets. Often both. (K03 §8)

Q38. How do you validate a port? End-to-end output comparison (allclose, cosine of logits), layer-by-layer to localize divergence, task-level eval, and confirm it's actually faster. Ship a one-page report with max error and accuracy/throughput deltas. (K03 §10)

E. Serving

Q39. Why is static batching bad for LLMs? Ragged completion (short sequences wait idle for the longest) and head-of-line blocking (new requests wait for the whole batch). Wastes GPU on padding. (K04 §2)

Q40. Explain continuous batching. Operate per decode step: finished sequences leave the batch immediately and waiting requests take their slots mid-stream. GPU never idles on padding → 5–20× throughput. (K04 §3)

Q41. What is PagedAttention? Virtual memory for the KV cache: split KV into fixed blocks allocated on demand and tracked by a block table, eliminating fragmentation and over-reservation → fit far more concurrent sequences. Enables prefix block sharing. (K04 §4)

Q42. Chunked prefill — what problem? A long prefill stalls everyone's decode (TPOT spikes). Chunk it and interleave with decode steps → smoother TPOT under mixed load, small TTFT cost. (K04 §5)

Q43. vLLM vs TensorRT-LLM vs Triton? vLLM: open-source, multi-vendor, PagedAttention, default. TensorRT-LLM: max NVIDIA perf, FP8, compiled engines. Triton: general multi-framework serving host (not LLM-specific) for CV/multi-model/ensembles; can host TRT-LLM. (K04 §7,9)

Q44. What do you tune to hit an SLO? max_num_seqs/batch, gpu_memory_utilization, max_num_batched_tokens (prefill chunk), KV dtype, quantization, prefix caching, parallelism, CUDA graphs. Tune for goodput (throughput subject to the latency SLO), not raw throughput. (K04 §8,10)

F. Distributed inference

Q45. When do you distribute? Two reasons only: capacity (doesn't fit on one chip) or throughput/latency (fits but too slow). Capacity → split the model (TP/PP); too-slow-but-fits → replicate (DP). (K05 §1)

Q46. Tensor vs pipeline parallelism? TP splits matrices within each layer → all-reduce every layer → chatty → needs NVLink → lives in a node. PP splits by layers into stages → only activations cross stages → tolerates slow inter-node links → spans nodes, but has a pipeline bubble needing many in-flight requests. (K05 §3–4)

Q47. Canonical topology for a big model? TP within each node over NVLink, PP across nodes over InfiniBand, DP to replicate for throughput. Map chatty parallelism to fast links. (K05 §8,9)

Q48. Why does TP fail over PCIe? TP all-reduces ~2× per layer, cost ∝ hidden·batch / interconnect bandwidth. On PCIe (64 GB/s) the per-layer, per-token all-reduces dominate → TP becomes a slowdown. (K05 §3,8)

Q49. Name the collectives. All-reduce (TP), all-gather, reduce-scatter, all-to-all (MoE routing), broadcast/scatter/gather, point-to-point (PP handoff). NCCL on NVIDIA. (K05 §7)

Q50. What's expert parallelism? For MoE: place experts on different GPUs; route each token to its experts via all-to-all. Memory holds all experts, but only a few compute per token — capacity-heavy, FLOP-light sizing. (K05 §5)

G. Profiling & benchmarking

Q51. First profiling question? "Is the GPU busy or idle?" Idle + CPU busy → overhead/launch-bound (CUDA graphs, fusion, batch). Busy → roofline question (compute vs memory). Read it off an nsys timeline. (K06 §4)

Q52. The four bottleneck classes? Compute-bound, memory-bandwidth-bound, memory-capacity-bound, overhead/launch-bound. Each has a distinct symptom and fix. (K06 §2)

Q53. nvidia-smi shows 100% — are we good? No — that's residency, not efficiency. Check MFU (tensor-core use, for prefill) or MBU (bandwidth use, for decode). Decode SHOULD have low MFU; check its MBU. (K06 §5)

Q54. nsys vs ncu? nsys = timeline, where wall-clock goes (start here). ncu = per-kernel, why one kernel is slow (occupancy, roofline). Don't open ncu until nsys names the dominant kernel. (K06 §3)

Q55. Rules for an honest benchmark? Warm up, synchronize, report percentiles (p95/p99) not just mean, fix all non-variables, use realistic input/output lengths, report single-stream AND loaded throughput, state all conditions. (K06 §7)

Q56. How do vendor benchmarks mislead? Cherry-picked I/O lengths, throughput without latency, mean without tail, peak (theoretical) FLOPs, mismatched precision, best-case lab conditions. You reproduce on the customer's workload and show the goodput curve. (K06 §10)

H. CV & video

Q57. FFmpeg vs GStreamer? FFmpeg = a codec tool (transcoding, batch, file processing). GStreamer = a live pipeline framework (multi-stream real-time decode→infer→track→sink, dynamic graphs; DeepStream is built on it). Live multi-camera → GStreamer/DeepStream; offline transcode → FFmpeg. (K07 §4)

Q58. What's the hidden bottleneck in video pipelines? Decode (and color-convert/copies), not the model. Use hardware decode (NVDEC) directly into GPU memory; decode capacity (NVDEC streams), not FLOPs, often caps the camera count. (K07 §3)

Q59. Detection→tracking→recognition — why a cascade? Each stage is specialized and cheap on its narrow job; run the expensive recognizer only on detected crops, not every pixel. Coarse-to-fine, like retrieval-then-rerank. (K07 §5)

Q60. The #1 fix in real video deployments? Keep pixels on the GPU — decode→preprocess→infer→postprocess all in GPU memory, copy only tiny metadata back. nsys full of big HtoD/DtoH memcpy = CPU round-trips killing you. (K07 §8)

I. Sizing & customer

Q61. How do you size a deployment? Extract model + I/O lengths + throughput + concurrency + SLO + accuracy floor + budget. Compute memory/instance → parallelism → per-instance goodput at the SLO → #instances × headroom → total accelerators. State every assumption. (K08 §2)

Q62. What's $/1M tokens and how do you halve it? (accelerator $/hr × #accelerators) ÷ (tokens/hr at SLO) × 1e6. Halve via quantization, better batching, spec decode, higher utilization, or a more efficient chip. The whole commercial game. (K08 §3)

Q63. Customer says 'make it faster and cheaper.' Response? Convert to numbers: which latency (TTFT/TPOT), at what concurrency, against what percentile SLO; what $/token or budget; what accuracy floor on which eval. Then present the three-point trade-off curve. (K08 §1,5)

Q64. When do you recommend a SMALLER model? When the smallest model that clears the accuracy floor on their eval is much cheaper — a tuned 8B+RAG often beats a 70B at ¼ cost. Recommending down builds trust and is literally a JD bullet. (K08 §5)

Q65. Why is perf/watt a sales weapon? At scale, power/cooling rival hardware cost and many sites are power-capped — "same throughput at half the watts" can be the only way a customer can grow. Frame efficiency as tokens-per-dollar and tokens-per-watt. (K08 §3)

J. The curveballs

Q66. "We benchmarked on an A100; will chip X be faster?" Depends on your workload's arithmetic intensity vs chip X's roofline. Decode-heavy chat wants bandwidth/SRAM; prefill/vision/embedding wants FLOPs. Let me see your I/O length distribution and I'll tell you which roof binds — the spec sheet alone can't.

Q67. "Our LLM is slow." Which phase — TTFT (prefill) or TPOT (decode)? Is the GPU busy or idle (overhead vs roofline)? KV-memory-bound (tiny batch/preemption)? One word — "slow" — has four different root causes and fixes.

Q68. "Quantization is lossless, right?" Often near-lossless on chat/summarization, but it can quietly cost points on reasoning/math/code where error chains amplify. I never claim lossless without an eval on your hardest task and a reported delta.

Q69. "Let's just buy more GPUs." Maybe — but if you're memory-bandwidth-bound at batch 1, more chips help throughput via replication, not per-request latency; and if you're under-utilizing (30% MFU/MBU) you're paying for idle silicon. Let's profile and quantize first — often cheaper than buying.

Q70. "Can it run air-gapped / on-prem?" Yes — freeze weights and deps, ship the engine, use llama.cpp/Ollama or a self-hosted vLLM/Triton with no external calls. (This is where this track meets the original Digital Twin folder.) The constraint changes the toolchain, not the physics.

Meta-skill for curveballs: never answer the literal question with a number. Restate it as a measurable spec, name the bottleneck, then give the trade-off. That reflex is the principal.

K. Frontier (2025–2026)

Q71. An MoE says "671B total / 37B active" — which number goes where in your sizing? Memory sizes on total (all experts must be resident: 671 GB at FP8, multi-node before you've served a token). Per-token compute and batch-1 bandwidth follow active (~37B). Quoting one number for both is the tell of someone who's never sized an MoE. (K09 §2)

Q72. Why does batching amortize worse on MoE than dense? Dense decode reads all weights once per step, so bytes/token = weights/B. MoE must read every expert any token in the batch selected: expected hot experts = E·(1−(1−k/E)^B) — at E=256, k=8, batch 32 you're already reading ~64% of all experts. Weight reads grow with B, so per-token bandwidth can be several× worse than a dense model of equal active size at moderate batch; amortization only resumes once coverage saturates. (K09 §2)

Q73. EP vs TP? TP slices every matrix across GPUs and all-reduces every layer — same work, split fine. EP places whole experts on different GPUs and moves tokens to them via two all-to-alls per MoE layer (dispatch + combine), with data-dependent, imbalance-prone volumes. TP's cost scales with hidden size; EP's with k·d_model per token and the router's whims — and wide-EP is what lets each device stream only its resident hot experts. (K09 §2, K05 §5)

Q74. Disaggregate prefill/decode, or chunked prefill? Size the KV transfer first: 1k-token prompt on a 70B GQA model ≈ 0.32 GB → ~0.4 ms over NVLink, ~6.5 ms over 400G RDMA (fine, overlappable), ~260 ms over 10 GbE (worse than the prefill itself — disqualifying). Disaggregate only with long prompts AND a strict TPOT SLO AND fleet scale on a fast fabric; otherwise colocation + chunked prefill captures most of the win for free. (K09 §3)

Q75. What's the tension in prefix-cache-aware routing? Cache affinity vs load-spreading. Routing to the replica holding the prefix makes the request ~10× cheaper (skip prefill) but concentrates a tenant's traffic into hotspots; spreading load evenly keeps queues flat but makes every cache cold. Production routers score both (expected hit length minus queue penalty). Cache hit rate is now a first-class load-balancing objective, not a bonus. (K09 §4)

Q76. RadixAttention vs plain paged prefix caching? Plain prefix caching shares KV blocks for exact, hash-matched prefixes. RadixAttention keeps all cached sequences in one radix tree (edges = token substrings) and matches any new request against the longest shared path — automatic, nested sharing (system prompt → few-shot → user history → turn) with LRU eviction on the leaves. Structural sharing for agent/tree workloads instead of operator-predicted sharing. (K09 §4)

Q77. MTP-as-draft vs EAGLE? EAGLE(-3) bolts a trained draft head onto an existing checkpoint — works on any open model, needs its own training run, drafts several tokens via a dynamic tree. MTP is trained into the base model at pretraining (DeepSeek-V3 predicts t+2 natively) — at inference the MTP head is the draft: no second model, ~85–90% acceptance because it shares the trunk, but only if the model shipped with it. Both die at high batch: speculation spends spare FLOPs, and a saturated batch has none. (K09 §5)

Q78. Why do rotations (QuaRot/SpinQuant) fix the outlier problem? Insert an orthogonal R around the matmul: (xR)(RᵀW) = xW — mathematically free, Rᵀ folds into the weights. Statistically, a Hadamard-type rotation rewrites each channel as a ±1/√d mix of all channels, smearing a 100× outlier channel into a small bump across thousands of channels. Kurtosis collapses → uniform INT4/FP4 quantizers stop clipping. The outlier problem from K02 §8, dissolved by a change of basis. (K09 §6)

Q79. Why does FP4 need block scaling? E2M1 has only eight magnitudes — no per-tensor or per-channel scale can stretch that across a real dynamic range. MXFP4: blocks of 32 share a power-of-two E8M0 scale; NVFP4: blocks of 16 with an FP8 scale + per-tensor second level. The block is the outlier blast radius — one outlier now coarsens 16–32 neighbors, not a whole channel. Blackwell runs FP4 natively at ~2× FP8 rate, which is why this is a serving topic, not a research one. (K09 §6)

Q80. How does grammar-constrained decoding actually guarantee valid JSON? Compile the schema to an FSM/pushdown automaton; each decode step, mask to −∞ every vocab token that can't continue a valid string from the current state. The subtlety: token boundaries don't align with grammar terminals (one token can close a string, an object, and add a comma), so masks are computed per automaton state over the whole vocab — XGrammar precomputes masks for context-independent states and overlaps the rest with the forward pass → near-zero overhead. And constrained ≠ correct: schema-valid hallucinations pass. (K09 §7)

Q81. What changes in capacity math when the customer moves to a reasoning model? Output tokens dominate: out 500 → ~3,000 (high variance) means ~6× fewer requests/s per instance at the same tok/s, KV grows during generation (unshareable thinking tokens), slots are held ~6× longer, and TTFT stops being the UX metric (time-to-first-answer-token ≈ thinking×TPOT can be minutes). Levers in order: thinking budgets, difficulty routing, prefix caching across turns, quantized KV, length-aware scheduling — then hardware. (K09 §8)