Brother Talk — C++/CUDA & GPU Performance Engineering

Off the record. The stuff I'd tell you over a beer, not in a design review.

This is the phase that makes you the person they can't outsource. Tons of engineers can call torch.compile and hope. Almost nobody can look at a slow kernel and say "it's memory-bound, you're register-limited at 25% occupancy, your inner loop is uncoalesced — fix the layout, then tile." The day you can do that out loud, you stop being a model user and become the person the infra team pulls into the room when the GPU bill or the p99 latency is on fire. That's a rare, durable, well-paid identity, and this is where you earn it.

You do not need a GPU to learn the most valuable 80%. This trips people up — they think "I can't do CUDA, I don't have an H100." Wrong. The thinking — roofline placement, occupancy and the limiter, coalescing, tiling, fusion — is arithmetic over a mental model, and that's exactly what the lab makes you do, on a laptop, offline. The hardware part (writing the actual .cu, reading Nsight) comes fast once the model is in your head. Build the intuition first; the syntax is the easy part.

"Memory-bound, so move fewer bytes" is the cheat code, again. You met it in Phase 00 at the model level; here it is at the kernel level, and it's still the answer 80% of the time. Coalesce, tile, fuse, drop precision. When a teammate says "let's get a faster GPU," and the kernel is sitting far left of the ridge, you'll save the team a quarter and a pile of money with one sentence. Most GPU "optimization" is people relearning this one fact the hard way.

Coalescing is the single highest ratio of (impact ÷ effort) you'll ever see. A data-layout flip — array-of-structs to struct-of-arrays — can be a 10× win on a memory-bound kernel. No new algorithm, no new hardware, no PhD. It feels almost too cheap. Train yourself to check it first on any slow memory kernel; it's the thing juniors never look at and seniors check reflexively.

Don't fetishize CUDA C++. There's a macho pull toward "real engineers write CUDA." Resist it. The senior move is restraint: torch.compile first (free), Triton when you need a custom kernel without the pain, and hand-written CUDA only when you've measured that nothing else clears the bar. Being able to drop to CUDA is the moat; needing to every time is a smell. The best kernel engineers I know reach for Triton far more than C++.

Profile, or you're just decorating. I've watched smart people spend a week "optimizing" a kernel that wasn't the bottleneck because they guessed instead of running ncu. The discipline is boring and it always wins: profile, find the one binding cause, fix it, re-profile. Nsight Compute will literally tell you the roofline placement and the occupancy limiter — the exact things this lab taught you to compute. Let the tool confirm your mental model; don't let your ego skip it.

nvidia-smi at 100% is a lie you'll see senior people believe. It means a kernel ran, not that the math units were busy. A memory-bound decode shows 100% util at single-digit MFU. The number of times this exact misunderstanding derails a perf conversation is genuinely funny once you know better. Know better, and be gentle about it.

What's actually worth caring about: the roofline placement, the occupancy limiter, coalescing, and the move-fewer-bytes reflex (tile/fuse). What's not worth losing sleep over: memorizing every register-allocation quantum, every SM's exact smem split, or hand-tuning a kernel that's 3% off cuBLAS — cuBLAS/CUTLASS/Triton autotune that, and your time is better spent finding the next bottleneck. Be right to within 2× on the whiteboard; let the autotuner sweat the last few percent.

Bank conflicts and launch overhead are the two "wait, that's a thing?" gotchas. Nobody warns you that a shared-memory column access can be 32× slower because all 32 lanes hit one bank, or that your single-stream decode is slow because the CPU can't launch tiny kernels fast enough. You'll hit both, you'll be baffled, and then you'll remember this paragraph and fix it in ten minutes (pad the tile; capture a CUDA Graph). Knowing the failure mode exists is 90% of the battle.

The career framing. This is the last technical phase before the capstone, and it's deliberately the deepest into the metal — because "I understand the model as a physical system, all the way down to the warp" is the complete version of the identity this whole track has been building. Every phase before this gave you a mechanism; this one gives you the ground they all run on. Get it cold and you can credibly say you own the stack top to bottom. Now go make pytest green, then go write a real kernel when you get your hands on a GPU.