Knowledge 06 — Performance Profiling & Benchmarking

The JD bullet: "Identify bottlenecks across compute, memory, and runtime, and guide optimization strategies," "Excellent … debugging, and performance analysis." This module turns the roofline theory into a practical skill: read a profiler trace, name the bottleneck, fix it, and prove the fix with an honest benchmark. This is the difference between guessing at optimizations and knowing. A principal never optimizes without measuring first.

Table of Contents

• 1. The cardinal rule: measure, don't guess
• 2. The four bottleneck classes
• 3. The tools and what each one tells you
• 4. Reading a Nsight Systems timeline
• 5. Computing the metrics that matter (MFU, MBU)
• 6. The roofline in practice
• 7. Honest benchmarking methodology
• 8. Load testing a serving system
• 9. The bottleneck-hunting playbook
• 10. Common traps and how vendors lie (and how not to)
• 11. References

1. The cardinal rule: measure, don't guess

Performance intuition is wrong more often than right. The weight-read you assumed dominated is actually a tiny launch-overhead gap; the "slow model" is actually a single un-fused op falling back to the CPU; the "GPU is maxed" reading is nvidia-smi showing the GPU busy but at 5% MFU. Every optimization starts with a measurement that localizes the cost, and ends with a measurement that proves the win. Engineers who skip this burn weeks optimizing the wrong thing.

Amdahl's law, restated for the job: speeding up something that's 5% of runtime by 10× saves 4.5%. Find the dominant cost first. The profiler tells you which 5% is actually 80%.

2. The four bottleneck classes

Every kernel/workload is limited by exactly one of four things at any moment. Naming which is 80% of the diagnosis.

The four map cleanly to the stack layers: compute/bandwidth/capacity are hardware roofline problems (Knowledge 00); overhead is a runtime problem (Knowledge 00 §8, Knowledge 03). The diagnostic question: is the GPU busy doing math (compute), busy moving data (bandwidth), out of room (capacity), or idle waiting (overhead)? The profiler answers it directly.

3. The tools and what each one tells you

The workflow: nsys first to find where time goes (which is usually "the GPU is idle 40% of the time" or "this kernel is 70%"), then ncu to understand why a specific dominant kernel is slow. Don't open ncu until nsys told you which kernel matters — per-kernel profiling without a timeline is a needle in a haystack.

4. Reading a Nsight Systems timeline

nsys shows parallel timelines: CPU threads, CUDA API calls, GPU kernel execution, memory copies, NCCL. What you're looking for:

• GPU rows with gaps = the GPU is idle. If the CPU row is busy during those gaps → launch/overhead-bound (CPU can't feed the GPU fast enough). Fix: CUDA Graphs, fusion, async, bigger batches. This is extremely common in LLM decode — dozens of tiny kernels per token with gaps between them.
• GPU rows packed solid with kernels = GPU is busy → then ask ncu whether those kernels are compute- or memory-bound.
• Big memcpy (HtoD/DtoH) bars = you're shuffling data across PCIe → are you needlessly moving data host↔device? Pin memory, keep tensors on-device, overlap copy with compute.
• Long NCCL bars = communication-bound → too much TP over a slow link (Knowledge 05 §8).
• Serialized CPU→GPU dependency chains = no overlap; you're waiting synchronously. Use async/streams.

The single most valuable read: "Is the GPU busy or idle?" Idle GPU = overhead problem (fix the host/launch side). Busy GPU at low efficiency = roofline problem (fix the kernel/precision/batch side). That fork sends you down completely different optimization paths, and reading it correctly off an nsys trace is a top JD2 skill.

5. Computing the metrics that matter (MFU, MBU)

nvidia-smi "100% utilization" only means a kernel was resident, not that it was efficient. The real efficiency metrics:

• MFU (Model FLOPs Utilization) = (FLOPs the model actually needed) / (peak FLOPs × time). How well you use the tensor cores. Well-tuned prefill hits 40–70%; decode is often 1–10% (it's memory-bound — low MFU is expected, not a bug).
• MBU (Model Bandwidth Utilization) = (bytes the model must read) / (peak bandwidth × time). How well you use HBM bandwidth. For decode, high MBU is the real target — if MBU is ~80%+ you're near the bandwidth roofline and doing well for a memory-bound workload.
• The pairing rule: judge prefill by MFU, judge decode by MBU. Reporting low decode MFU as "bad utilization" is a classic misread — decode should have low MFU; check its MBU instead.

Principal nuance: "Our decode MFU is only 4%!" is not a problem statement — decode is memory-bound, so MFU is supposed to be low. The question is MBU: are we near the bandwidth roof? If MBU is 75%, the GPU is working as hard as physics allows; the only levers are fewer bytes (quantize) or more bandwidth (different chip) or more arithmetic intensity (batch). Knowing which utilization metric applies to which phase is a senior tell.

6. The roofline in practice

Turning Knowledge 00 §5 into a measurement:

1. From ncu (or by hand), get a kernel's FLOPs and bytes moved → its arithmetic intensity.
2. Plot it against the chip's roofline (Nsight Compute has a built-in roofline chart).
3. Below the ridge / on the memory slope → memory-bound → the kernel is at the bandwidth limit; only fusion / fewer bytes / higher AI helps.
4. On the compute roof → compute-bound → only lower precision / more FLOPs helps.
5. Far below both rooflines → you're overhead-bound or low-occupancy → the kernel isn't even hitting either limit → fix launch overhead, occupancy, or tiling.

That third case (below both lines) is the most actionable: the hardware isn't the limit, your execution is — usually overhead or a badly-shaped kernel.

7. Honest benchmarking methodology

Benchmarks are where credibility is won or lost (you'll publish them to customers — a JD duty). The rules:

1. Warm up. Discard the first iterations (CUDA context init, autotuning, caches cold). Measure steady state.
2. Synchronize. GPU work is async; torch.cuda.synchronize() before stopping the timer or you measure launch, not execution. Use CUDA events for kernel timing.
3. Report percentiles, not just mean. p50/p95/p99 — the tail is what users feel. A great mean with a terrible p99 is a bad system.
4. Fix everything that isn't the variable: same hardware, driver, precision, batch, input/output lengths, sampling, power/clock settings. Lock GPU clocks (nvidia-smi -lgc) if you need run-to-run stability.
5. Use realistic inputs. Benchmarking 128-in/128-out when the customer runs 2000-in/500-out is meaningless — prefill/decode ratio changes everything (Knowledge 01 §2).
6. Measure the right thing: single-stream latency and loaded throughput are different questions. Report both, and the curve between them.
7. State the conditions with every number: model, precision, hardware, batch, in/out lengths, framework version. A number without conditions is noise.

The honesty standard: a benchmark you'd defend under cross-examination from the customer's own engineer. "2.3× faster" must come with "at FP8 vs FP16, 70B, H100, 2k-in/500-out, p95, vLLM 0.6.x." Vague benchmarks destroy the "trusted advisor" relationship the instant the customer reproduces them and gets a different number.

8. Load testing a serving system

Single-request timing ≠ production behavior. To benchmark a server (Knowledge 04):

• Generate realistic load: a concurrency or request-rate sweep (e.g., 1, 8, 32, 128, 256 concurrent) with a realistic input/output length distribution and arrival pattern (Poisson/bursty, not uniform). Tools: vLLM benchmark_serving.py, genai-perf/Triton Perf Analyzer, locust, custom async client.
• Measure at each load: throughput (tok/s, req/s), p50/p95/p99 TTFT and TPOT, queue time, and goodput (requests meeting the SLO).
• Plot throughput vs p95 latency — the goodput Pareto curve. The "right" operating point is the highest throughput still inside the SLO box.
• Find the knee: where latency starts climbing steeply as load rises — that's your capacity limit. Beyond it, queueing dominates and the system falls over.
• Soak test: run for hours — catch memory leaks, KV fragmentation, throughput decay, OOM under sustained load.

The output is the same shape as the serving playbook deliverable: a curve, a chosen point, and the headroom — which feeds directly into capacity sizing.

9. The bottleneck-hunting playbook

A repeatable procedure (this is the JD's "guide optimization strategies"):

1. Define the metric you're optimizing (p95 TPOT? throughput? $/token?). Don't optimize blind.
2. nsys: is the GPU busy or idle?
     IDLE + CPU busy        → overhead-bound  → CUDA Graphs, fusion, async, bigger batch
     IDLE + waiting on memcpy→ data-movement   → keep tensors on device, pin, overlap
     IDLE + waiting on NCCL  → comms-bound      → fix parallelism/interconnect (Knowledge 05)
     BUSY                    → go to step 3
3. ncu the dominant kernel(s): roofline position?
     memory slope  → memory-bound  → fuse, quantize, raise arithmetic intensity (batch)
     compute roof  → compute-bound → lower precision (FP8/INT8), better tiling
     below both    → low occupancy/overhead → tune launch config, registers, tiling
4. Apply ONE change. Re-measure with the same harness. Did the target metric move?
5. Repeat until you hit the roofline or the SLO. Then stop — you're at physics; further work is wasted.
6. Document: the bottleneck, the fix, the before/after numbers, the conditions.

The discipline of one change at a time, re-measured is what separates real performance work from cargo-culting flags. And knowing when to stop — "we're at 78% MBU, decode is at the bandwidth roofline, there's no more to get without different hardware" — is itself principal-level judgment that saves the customer money.

10. Common traps and how vendors lie (and how not to)

The misleading benchmark patterns to recognize (in others' claims) and never commit (in yours):

• nvidia-smi 100% util ≠ efficient. It means a kernel was resident, not that the tensor cores or bandwidth were used. Always check MFU/MBU.
• Cherry-picked input/output lengths. A vendor quoting throughput at 128-in/8-out is hiding decode cost. Demand the customer's real distribution.
• Throughput without latency. "Tokens/s" at huge batch with 10-second TTFT is useless for interactive use. Always pair with latency percentiles → goodput.
• Mean without tail. Great p50, awful p99. Users live in the tail.
• Best-case batch, single model, warm cache, locked clocks in the vendor lab vs. the customer's noisy multi-tenant reality.
• Peak (theoretical) TFLOPs vs achievable. Spec sheets quote peak (often with sparsity); real workloads see a fraction. The roofline is built on achievable numbers.
• Different precision in the comparison (their FP8 vs your FP16) presented as apples-to-apples.

The advisor's edge: when a customer waves a competitor's benchmark, you don't argue — you reproduce it on their workload and show the honest goodput curve. Being the person whose numbers survive scrutiny is the entire basis of "trusted technical advisor." Your benchmarks are your reputation.

11. References

• Williams et al., "Roofline" (CACM 2009); NVIDIA Nsight Compute roofline analysis docs.
• NVIDIA Nsight Systems and Nsight Compute user guides; PyTorch Profiler docs; Perfetto trace viewer.
• Horace He, "Making Deep Learning Go Brrrr From First Principles" (2022) — compute/memory/overhead bound, the practical view.
• MLPerf Inference benchmark methodology (the industry-standard rules for honest inference benchmarking).
• Databricks/MosaicML, "LLM Inference Performance Engineering: Best Practices" — MFU/MBU and TTFT/TPOT in practice.
• vLLM benchmark_serving, NVIDIA genai-perf, Triton Performance Analyzer / Model Analyzer docs.

➡ Next: Knowledge 07 — CV & Video Pipelines — the multi-model, real-time side of JD2.