🛸 Hitchhiker's Guide — Phase 03: GPU Compilers & Codegen

Read this if: you can write CUDA but nvcc is a magic box, "PTX" is a word you nod at, and you couldn't yet audit an accelerator vendor's "we support PyTorch" claim. Compressed field notes; derivations live in the WARMUP.

0. The 30-second mental model

CUDA C++ ──(cicc/LLVM)──► PTX (virtual ISA, stable forever)
                            └─(ptxas, AOT)──► SASS for sm_80, sm_90...   } fatbin
                            └─(driver JIT, load time)──► SASS for whatever ships next

PTX = the contract. SASS = the truth. Fatbin = the deployment policy. JIT = the cold-start you forgot to plan for. Everything else is detail.

1. The toolbox

2. PTX reading card

• %tid.x %ctaid.x %ntid.x = threadIdx / blockIdx / blockDim
• mad.lo.s32 %r4, %ctaid, %ntid, %tid = THE index formula
• .param → ld.param → cvta.to.global = kernel ABI prologue (your Lab 02 compiler emits this trio)
• setp.ge.s32 %p1, … + @%p1 bra = tail guard; @%p on a non-branch = if-conversion (no divergence machinery at all)
• fma.rn.f32 = contracted multiply-add; one rounding, not two — the reason for tolerance-based testing, and for -fmad=false in numerical disputes
• Infinite %f/%r/%rd registers — pressure is invisible here; spills only exist in SASS

3. Production policies you now own

• Build matrix: AOT SASS for every arch in the fleet + newest compute_XX PTX for the future. Cost: build minutes, container GB. Skipping an arch = JIT cold-start storm (or hard load failure if you also skipped PTX).
• JIT hygiene: CUDA_MODULE_LOADING=LAZY (default since 12.x), pre-warm compute cache in images, watch CUDA_CACHE_MAXSIZE on multi-model fleets.
• Numerical policy: FMA contraction + reduction order = CPU/GPU diffs in the last ulp; tests compare with tolerance, never == (Phase 02), and -fmad changes are reviewed like API changes.
• The vendor audit (the leadership move of this phase): "Top-20 ops for our workload: % of roofline at p50/p99, on our shapes and precisions. Which came from a kernel library, which from your compiler, which are unimplemented?" Watch them sweat on attention variants and odd head dims.

4. The portability map, one line each

• NVPTX (LLVM): how every non-NVIDIA frontend reaches NVIDIA hardware.
• ROCm/HIP: CUDA-shaped API, mechanical porting, public ISA — but no PTX equivalent: per-gfx code objects, no forward-compat promise. Plan rebuilds.
• SPIR-V: truly neutral, historically below peak for ML; the "standards" position.
• Triton: block-level Python DSL → MLIR → PTX/ROCm; ~90% of hand-tuned on LLM kernels; the pragmatic kernel layer for a hardware-agnostic platform.
• MLIR: the infrastructure bet; what you adopt so you don't invent an IR.

Three platform postures: wrap vendor libraries behind a HAL (fast, Phase 09), adopt Triton/MLIR for kernels (velocity), build a full compiler (moonshot). Blend of first two = the defensible default. Be ready to defend it to a board.

5. War stories to recognize

• "New A-series GPUs took 8 minutes to first token" — no sm_XX cubin, JIT storm. Fix: build matrix + cache pre-warm. (You will meet this one.)
• "Same model, different last digit on GPU vs CPU" — FMA contraction + parallel reduction order. Not a bug; a numerical policy.
• "We set maxrregcount=32 and got slower" — spills to local memory; the SASS shows STL in the inner loop. Occupancy isn't a score (Phase 01/02).
• "Vendor demo hit 80% of peak; our model hit 19%" — demo ran their three tuned kernels; your model hit the unimplemented long tail. The audit in §3 exists because of this meeting.

6. Exit bar

You can: narrate saxpy's PTX cold; find spills/FMA/predication in SASS; draw the fatbin/JIT decision tree and its incident; defend a build-matrix policy; argue the three portability postures; and your own little compiler emits PTX that a real driver loads. That's "technical credibility with engineering teams" at the compiler layer — on to Phase 04, where we descend below the runtime into the driver and the kernel (the OS one).