The Hitchhiker's Guide — Phase 07: Profiling & Performance Engineering

"Measure don't guess. Guess don't ship." — Performance engineering maxim

Section 1: The Roofline Model

The Core Idea

Every op on a GPU (or NPU) is limited by one of two things:

1. Compute throughput: how fast the ALUs can multiply (FLOPS/s)
2. Memory bandwidth: how fast data moves from DRAM to the ALUs (bytes/s)

The roofline model formalizes this:

$$\text{Performance (FLOPS/s)} = \min\left(\text{Peak FLOPS/s},\ \text{Peak BW (bytes/s)} \times I\right)$$

where $I$ = arithmetic intensity = FLOPS / bytes transferred.

The "roofline" is the shape formed by this function: a sloped line (memory-bound region) that flattens at the peak FLOPS ceiling (compute-bound region).

Performance
(TFLOPS/s)
   │
   │           __________ Peak FLOPS (A100: 312 TFLOPS FP16)
   │          /
   │         /  ← compute-bound ops (slope = Peak BW)
   │        /
   │       / ← memory-bound ops (below the roofline)
   │      /
   └──────────────────────────────────── Arithmetic Intensity (FLOPS/byte)
       L1   L2   L3   DRAM              (crossover point: Peak FLOPS / Peak BW)

Crossover point (arithmetic intensity where the boundary is): $$I_{\text{crossover}} = \frac{\text{Peak FLOPS}}{\text{Peak BW}}$$

For A100: $312 \times 10^{12} / 2 \times 10^{12} = 156$ FLOPS/byte

Any op with $I < 156$ FLOPS/byte is memory-bound on A100.

Arithmetic Intensity of Common Ops

Key insight: a matmul with $K=4096$ has AI ≈ 1024 FLOPS/byte — firmly compute-bound. A matmul in decode-time attention (batch=1, $K$ is small) may be only AI = 2 FLOPS/byte — firmly memory-bound.

The Decode-Time Attention Problem

During LLM generation (decode step), batch_size=1, seq_len grows with each token. The attention operation:

• Reads K, V caches: $8 n d$ bytes (grows with sequence length)
• Compute: $4nd$ FLOPS

AI = $4nd / 8nd = 0.5$ FLOPS/byte — deeply memory-bound regardless of hardware.

This is why decode-time LLM performance scales with memory bandwidth, not FLOPS. A100 has 2 TB/s HBM bandwidth; Snapdragon 8 Gen 3 has 77 GB/s DRAM → A100 is 26× better on LLM decode, even though the TOPS ratio is much larger.

Section 2: Profiling Tools — When to Use Which

Tool Selection Guide

Question: "Why is my model slow?"
                │
        ┌───────┴───────┐
   "Which ops?"     "Why is op X slow?"
        │                    │
  torch.profiler          Nsight Compute (ncu)
  (Python-level)          (kernel-level)
        │
   "Is the GPU busy?"
        │
  Nsight Systems (nsys)
  (system-level timeline)

torch.profiler — Python-Level Profiling

import torch
from torch.profiler import profile, record_function, ProfilerActivity

model = MyModel().cuda()
x = torch.randn(32, 512).cuda()

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    record_shapes=True,
    with_flops=True,
    with_stack=True,
) as prof:
    with record_function("model_inference"):
        model(x)

# Print top ops by CUDA time
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=20))

# Export Chrome trace (open in chrome://tracing)
prof.export_chrome_trace("trace.json")

# With scheduler for training loops
with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    schedule=torch.profiler.schedule(wait=2, warmup=2, active=3, repeat=1),
    on_trace_ready=torch.profiler.tensorboard_trace_handler('./log/resnet18'),
) as prof:
    for step, (x, y) in enumerate(dataloader):
        loss = model(x).mean()
        loss.backward()
        optimizer.step()
        prof.step()  # must call step() at end of each iteration

Nsight Systems (nsys) — System Timeline

# Profile a Python training script
nsys profile \
    --trace=cuda,nvtx,osrt,cudnn,cublas \
    --output=profile \
    python train.py

# Open the output
nsys-ui profile.nsys-rep  # opens GUI

In the nsys timeline, look for:

• CPU-GPU sync points (CPU waiting for GPU to finish — gaps in GPU timeline)
• Memory transfer (H2D/D2H) — often the hidden bottleneck
• CUDA launch gaps — if GPU timeline has gaps, it's CPU-limited (not submitting kernels fast enough)
• SM occupancy — visible in NVTX rows; low occupancy = underutilized GPU

Nsight Compute (ncu) — Per-Kernel Analysis

# Profile specific kernel
ncu \
    --metrics l1tex__t_bytes_pipe_lsu_mem_global_op_ld.sum,\
              l1tex__t_bytes_pipe_lsu_mem_global_op_st.sum,\
              sm__sass_thread_inst_executed_op_ffma_pred_on.sum \
    --target-processes all \
    python -c "import torch; torch.nn.functional.softmax(torch.randn(1024, 1024).cuda(), dim=-1)"

# Full report (slow but comprehensive)
ncu --set full python train.py 2>&1 | head -100

Key ncu metrics:

• sm__throughput.avg.pct_of_peak_sustained_elapsed: % of peak SM utilization
• l1tex__t_bytes_pipe_lsu_mem_global_op_ld.sum: bytes loaded from L1 cache
• dram__bytes_read.sum: bytes read from DRAM (global memory)
• smsp__warp_issue_stalled_long_scoreboard_per_warp_active.pct: % time stalled waiting for memory — high = memory-bound kernel

torch.utils.benchmark — Reliable Timing

import torch.utils.benchmark as benchmark

# Timer: handles warmup automatically
t = benchmark.Timer(
    stmt='model(x)',
    setup='import torch; model = MyModel().cuda(); x = torch.randn(32, 512).cuda()',
    num_threads=1,
)

# Measure
result = t.timeit(100)  # 100 repetitions
print(result)
# <torch.utils.benchmark.utils.common.Measurement object at 0x...>
# model(x)
# 2.47 ms
# 1 measurement, 100 runs , 1 thread

# Compare two implementations
results = benchmark.Compare([
    benchmark.Timer(stmt='fast_model(x)', setup=fast_setup).timeit(100),
    benchmark.Timer(stmt='slow_model(x)', setup=slow_setup).timeit(100),
])
results.print()

Why torch.utils.benchmark over time.time():

1. Handles warmup automatically (GPU JIT compilation, caching)
2. Reports statistics (mean, median, IQR)
3. Handles CUDA synchronization correctly (calls torch.cuda.synchronize() before timing)
4. Uses perf_counter_ns for high-resolution timing

Section 3: CUDA Occupancy

Occupancy = (active warps on SM) / (max theoretical warps on SM)

Occupancy is limited by the most constraining resource per SM:

1. Register file: each SM has 65,536 registers; if a kernel uses 64 registers per thread and block has 256 threads: 64 × 256 = 16,384 registers per block; SM can run 4 blocks = 1024 threads = 32 warps (100% occupancy on A100 which has 64 max warps)
2. Shared memory: SM has 192 KB; if kernel uses 48 KB per block: SM can run 4 blocks
3. Block size: launch config must match hardware; e.g., A100 has max 1024 threads/block and max 32 blocks/SM

Calculate occupancy:

# Using PyTorch's occupancy calculator
from torch.cuda.amp import autocast

@torch.jit.script
def my_kernel(x: torch.Tensor) -> torch.Tensor:
    return x.relu()

# Get theoretical occupancy
import pycuda.compiler as cuda
# Or use ncu --metrics sm__occupancy_avg...

Low occupancy is not always bad: a kernel with 25% occupancy but 100% FLOP utilization is fine. Occupancy matters when the kernel is latency-bound (lots of memory waits) — higher occupancy allows the scheduler to hide latency by switching to other warps.

Section 4: Memory Coalescing

A CUDA warp (32 threads) executes the same instruction simultaneously. When all 32 threads access a 128-byte-aligned, contiguous segment of memory, this is a coalesced access — one DRAM transaction for 32 threads.

Coalesced (efficient):

# Each thread accesses array[thread_id] — contiguous, aligned
# 32 threads → 1 memory transaction (128 bytes)
output[idx] = input[idx]  # good: sequential access pattern

Non-coalesced (inefficient):

# Each thread accesses array[thread_id * stride] — strided
# 32 threads → 32 memory transactions (one per thread)
output[idx] = input[idx * 64]  # bad: stride-64 access, each in different cache line

Real example — transposed access:

# Matrix stored row-major: A[row][col]
# Kernel: thread i reads column i of row j — sequential → coalesced ✅
# Kernel: thread i reads row i of column j — non-sequential → non-coalesced ❌

# Fix: use shared memory tiling
# Load a TILE×TILE block into shared memory (coalesced)
# Operate on shared memory (any pattern, fast)

Section 5: The Five Common GPU Performance Bugs

These are the bugs you will find and fix in Lab 03:

Bug 1: Missing KV Cache (2–4× slowdown in generation)

• Symptom: generation time grows O(n²) with sequence length
• Fix: cache K, V in past_key_values; compute attention only for new tokens

Bug 2: Non-Contiguous Tensors (10–30% overhead)

• Symptom: tensor.is_contiguous() returns False; copies inserted automatically
• Fix: insert .contiguous() before ops that require contiguous memory, or refactor the transpose/slice that created the non-contiguous tensor

Bug 3: CPU-GPU Synchronization in Loop (100–1000× slowdown)

• Symptom: tensor.item(), .numpy(), print(tensor) inside generate loop
• Fix: remove synchronizations from the hot path; accumulate and print after generation completes

Bug 4: FP32 Ops in FP16 Model (2–4× slowdown)

• Symptom: in the profile, LayerNorm/Softmax runs in FP32 despite model in FP16
• Fix: use torch.autocast(device_type='cuda', dtype=torch.float16) or explicitly cast ops to FP16; or use torch.nn.LayerNorm(..., dtype=torch.float16)

Bug 5: Sequential Generation (Nx slowdown for batch=N)

• Symptom: calling model.generate() N times in a loop instead of batching
• Fix: pad to equal length, pass batch to model.generate(input_ids=batch); if variable length, use continuous batching

Section 6: Interview Pitfalls

Section 7: Resources

1. Roofline Model paper — Samuel Williams et al., "Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multiprocessor Architectures" (CACM 2009)
2. NVIDIA Nsight Compute documentation — https://docs.nvidia.com/nsight-compute/
3. NVIDIA Nsight Systems documentation — https://docs.nvidia.com/nsight-systems/
4. torch.profiler tutorial — https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html
5. torch.utils.benchmark documentation — https://pytorch.org/docs/stable/benchmark_utils.html
6. "CUDA C++ Best Practices Guide" — NVIDIA (memory coalescing, occupancy, shared memory)
7. "GPU Performance Background" — Tim Dettmers blog: https://timdettmers.com/2020/09/07/which-gpu-for-deep-learning/
8. MLPerf inference benchmarks — https://mlcommons.org/benchmarks/inference/ — real performance numbers on real hardware