Phase 03 — GPU Compilers: NVCC, PTX, SASS, and Codegen

Difficulty: ⭐⭐⭐⭐☆ | Estimated Time: 2 weeks Roles supported: Head of Engineering [GPU], GPU Compiler Engineer, Runtime Engineer Hardware needed: none for Lab 02 (the compiler you build emits and interprets PTX); any NVIDIA GPU (or Godbolt) for Lab 01

Why This Phase Exists

A platform that "operates independently of the underlying hardware" (the JD's first bullet) is, at its core, a compiler and runtime story: something must translate a hardware-neutral program representation into each vendor's ISA, and do it well enough that customers don't pay a performance tax for portability. Every player in this market — CUDA itself (PTX as the stable virtual ISA), Triton, MLIR, ROCm, OpenAI/vLLM kernels, every startup accelerator's "we support PyTorch" claim — is a position in the compiler argument.

You cannot lead that platform if the compiler layer is a black box. This phase opens it: you will trace nvcc's real pipeline, read PTX and SASS with comprehension, understand JIT vs AOT and the fatbin compatibility model (a production operations concern, not trivia), and then write a working compiler that lexes, parses, and emits legal PTX — plus an interpreter that executes your PTX per-thread so you can verify semantics without a GPU.

Concepts

  • The nvcc pipeline end-to-end: host/device split → C++ frontend → NVVM IR (LLVM) → ptxas → SASS; what cudafe++, cicc, ptxas, fatbinary each do
  • PTX: a virtual ISA — typed registers (infinite), explicit state spaces (.global, .shared, .param, .reg), predication, the %tid/%ctaid special registers
  • SASS: the real ISA — register allocation, dual-issue, stall counts in control bits; why SASS changes every architecture but PTX doesn't
  • JIT vs AOT: -arch/-code, compute_XX vs sm_XX, fatbins, the driver JIT cache (~/.nv/ComputeCache), forward compatibility — and the production incident pattern ("new GPU, 5-minute cold start")
  • What the compiler does for you: loop unrolling, if-conversion to predication, FMA contraction, register allocation vs occupancy (-maxrregcount, the Phase 02 connection)
  • The portability landscape a hardware-agnostic platform must navigate: LLVM NVPTX vs NVIDIA's NVVM, AMD's ROCm/LLVM path (GCN/RDNA ISA is public!), SPIR-V, Metal
  • Triton in one lab-sized bite: block-level programming model, why it can fuse and auto-tune, where it fits between CUDA C++ and TVM/MLIR
  • Compiler-as-moat: why kernel libraries + compiler maturity, not TFLOPs, decide which accelerators win (ties to Phase 01 Ch. 10, Phase 09)

Labs

Lab 01 — PTX/SASS Dissection (CUDA toolkit or Godbolt)

FieldValue
GoalCompile small kernels and read what comes out: PTX first, then SASS; observe predication, unrolling, FMA contraction, and register pressure.
ConceptsThe nvcc pipeline, PTX syntax, SASS reading basics, -Xptxas=-v output, fatbins.
Steps1) nvcc -ptx the four provided kernels; annotate every PTX line of saxpy (template provided). 2) cuobjdump --dump-sass (or Godbolt) the same kernels; find the predicated branch and the FMA. 3) Rebuild with -maxrregcount 16; find the spills (local loads/stores). 4) Build a fatbin for two archs and dissect it with cuobjdump --list-elf.
StackCUDA toolkit (nvcc, cuobjdump, nvdisasm) — or godbolt.org with CUDA selected, zero install
OutputAn annotated saxpy.ptx (every line explained) + a findings sheet for the other three kernels.
How to TestYour annotations are checked against the provided answer key (saxpy-annotated.md).
Talking PointsWhy PTX is the stability boundary (CUDA's actual moat); what ptxas does that the PTX level can't express; how if (x>0) y=a becomes a predicated instruction with no branch.
Resume Bullet"Dissected NVCC compilation pipeline from CUDA C++ through PTX to SASS; documented predication, FMA contraction, and register-spill behavior across optimization flags for team training."
ExtensionsCompare clang --cuda-device-only -S (LLVM NVPTX) output against nvcc's for the same kernel; compile the same kernel for AMD with hipcc --genco and read the GCN ISA.

Lab 02 — Mini Compiler: Expression Language → PTX (Python)

FieldValue
GoalBuild a real (small) compiler: lex → parse → typed AST → PTX codegen for elementwise GPU kernels — plus a PTX interpreter that runs your output per-thread and verifies it against a Python reference.
ConceptsLexing, recursive-descent parsing, AST, register allocation (virtual → PTX .reg), the PTX kernel ABI (.param space, cvta, %tid math, bounds guard), codegen correctness.
Steps1) Run python solution.py — compiles 5 test kernels to PTX and executes them in the bundled interpreter against references. 2) Read the compiler in pipeline order: Lexer → Parser → TypeChecker → CodeGen → PTXInterp. 3) Extensions: add min/max intrinsics, constant folding, common-subexpression elimination, then a fused relu(a*b+c) kernel.
StackPure Python (stdlib). Optional final step: load your PTX on a real GPU via cuda-python/PyCUDA and confirm the interpreter told the truth.
OutputA working expr → PTX compiler whose output is legal PTX (loads on a real GPU) and passes the built-in differential tests.
How to Testpython solution.py — every kernel's interpreter output must match the Python reference elementwise; PTX text is also syntax-checked against the grammar subset.
Talking PointsWhat Triton does beyond this (tiling, smem staging, auto-tuning at the block level); why PTX's infinite virtual registers push register allocation to ptxas; what changes to target AMD (different ABI + ISA, same compiler shape).
Resume Bullet"Built an expression-to-PTX compiler (lexer, parser, type checker, codegen) with differential testing via a PTX interpreter; output verified as legal PTX loadable by the CUDA driver."
ExtensionsConstant folding + CSE passes (measure instruction-count reduction); emit the same AST to a second backend (C source) and reflect on what a multi-backend HAL compiler needs (Phase 09 foreshadowing).

Deliverables Checklist

  • Fully annotated saxpy.ptx matching the answer key
  • SASS findings: one predicated instruction, one FMA, one spill identified with flags that caused them
  • Mini compiler passes all differential tests
  • At least one optimization pass (folding or CSE) implemented and measured
  • You can draw the nvcc pipeline and the JIT/AOT decision tree from memory

Interview Relevance

  • "What does nvcc actually do? What's PTX vs SASS?"
  • "Your service takes 5 minutes to start on new GPUs. Why, and what's the fix?" (JIT + fatbin story)
  • "Why did Triton succeed where so many kernel DSLs failed?"
  • "An accelerator startup says 'we run PyTorch.' What's actually behind that claim, and what do you audit?"
  • "Walk me through compiling c[i] = a[i] * 2 + b[i] down to PTX by hand."