Lab 04 — Profile & Find the Bottleneck

Goal: take a deliberately-suboptimal workload, profile it with Nsight Systems, correctly classify the bottleneck (compute / memory-bandwidth / memory-capacity / overhead), fix it, and prove the fix with a before/after measurement. This is the JD's "identify bottlenecks across compute, memory, and runtime."

Knowledge prereq: K06 — Profiling & Benchmarking, K00 — Hardware Foundations.

Stack: torch, nsys (Nsight Systems), ncu (Nsight Compute), torch.profiler, a CUDA GPU.

Steps

  1. Create (or find) a bottlenecked workload. Three deliberate ones to diagnose — do at least two:
    • (A) Overhead-bound: a loop of many tiny GPU ops (e.g., element-wise ops on small tensors) in eager mode. You should find the GPU idle with the CPU busy launching.
    • (B) Memory-bandwidth-bound: a large element-wise/normalization op or batch-1 decode. Low arithmetic intensity; HBM saturated.
    • (C) Compute-bound: a big square matmul. Tensor cores saturated, high MFU.
  2. Profile with nsys: nsys profile -o trace python workload.py. Open the timeline (K06 §4). First question: is the GPU busy or idle?
    • Idle + CPU busy → overhead/launch-bound (case A).
    • Busy → go to ncu for the roofline question.
  3. Profile a hot kernel with ncu: ncu --set full -k <kernel> python workload.py. Read the roofline chart and the memory-vs-compute throughput → memory-bound (B) or compute-bound (C)?
  4. Compute MFU / MBU (K06 §5) for the workload. Confirm your classification: low MFU + high MBU = memory-bound; high MFU = compute-bound; low on both = overhead/occupancy.
  5. Apply the matching fix (K06 §2,9):
    • Overhead → torch.compile (fusion) and/or CUDA graphs → re-profile, watch the gaps close.
    • Memory-bound → fuse ops / raise batch (arithmetic intensity) → re-profile.
    • Compute-bound → lower precision (FP16/FP8) → re-profile.
  6. Measure before/after with the same harness (warmup + sync + percentiles — K06 §7). Did the target metric move?
  7. Write the root-cause report: the symptom, the trace evidence ("GPU idle 45%, CPU launching"), the classification, the fix, and the measured delta. One change at a time, re-measured — the discipline is the point.

What you must be able to explain

  • The "GPU busy vs idle" fork and why it sends you down different paths (K06 §4).
  • Why low decode MFU is expected, not a bug — and why you check MBU instead (K06 §5).
  • When to stop optimizing (you've hit the roofline) (K06 §9).

Result / résumé bullet

"Diagnosed and fixed three GPU bottleneck classes using Nsight Systems/Compute: eliminated launch overhead with CUDA graphs (idle GPU 45%→8%), raised arithmetic intensity on a memory-bound kernel via fusion (1.8×), and confirmed a matmul at the compute roofline — each classified by MFU/MBU and proven with before/after traces."