Phase 01 — GPU Architecture: From Silicon to SM
Difficulty: ⭐⭐⭐☆☆ | Estimated Time: 2 weeks Roles supported: Head of Engineering [GPU], GPU Platform Engineer, Performance Engineer Hardware needed: none — both labs run on any CPU
Why This Phase Exists
Every decision you will make as a GPU platform leader — scheduler design, KV-cache budgets, which inference engine to standardize on, what to tell an OEM partner their chip needs — reduces to a handful of physical facts about how GPUs are built. Leaders who can't derive "decode is memory-bandwidth-bound" from silicon get out-negotiated by vendors and out-designed by competitors.
This phase builds that base. Not trivia — mechanism. You will be able to explain, with numbers, why a GPU dedicates its transistor budget to ALUs instead of caches and branch predictors, why 32 threads execute in lockstep, why divergence halves throughput, and why "how many GB/s" matters more than "how many TFLOPs" for the workloads this JD cares about.
Concepts
- Why GPUs exist: latency-oriented vs throughput-oriented design; the transistor-budget argument
- The compute hierarchy: GPU → GPC → SM → warp scheduler → CUDA core / Tensor Core
- SIMT execution: warps, lockstep, active masks, divergence and reconvergence
- Latency hiding: zero-cost warp switching vs CPU context switches; occupancy as a latency-hiding budget
- The memory hierarchy: registers → shared memory/L1 → L2 → HBM; sizes, latencies, bandwidths for A100/H100
- HBM: why stacked DRAM, why bandwidth is the headline number
- Arithmetic intensity and the roofline model — the single most useful performance abstraction
- Tensor Cores: what an MMA unit actually computes; why FP16/BF16/FP8 exist
- NVLink, PCIe, and why interconnect shows up in Phase 08
- Reading a spec sheet critically: A100 vs H100 vs MI300X vs a startup accelerator
Labs
Lab 01 — SIMT Simulator (Python)
| Field | Value |
|---|---|
| Goal | Build a warp-level SIMT interpreter that executes a tiny ISA in lockstep with active masks, then measure divergence cost. |
| Concepts | Warps, lockstep execution, active masks, divergence/reconvergence, why branchy code kills GPU throughput. |
| Steps | 1) Read solution.py top-down. 2) Run it; study the divergence report. 3) Do the extension exercises: add a reconvergence stack, implement vote.any, write a maximally-divergent kernel and predict its slowdown before running. |
| Stack | Pure Python, no dependencies |
| Output | Working simulator + a divergence-cost table you produced and can explain. |
| How to Test | python solution.py — built-in asserts verify lockstep semantics and the 32× worst-case divergence measurement. |
| Talking Points | Why SIMT ≠ SIMD; what the active mask is; why if (threadIdx.x % 2) halves throughput but if (blockIdx.x % 2) doesn't. |
| Resume Bullet | "Built a SIMT execution simulator demonstrating warp divergence semantics; quantified 2×–32× throughput loss across divergence patterns and used it to teach GPU execution to a platform team." |
| Extensions | Add predication (execute-both-sides) and compare against branch-based masks; simulate a warp scheduler choosing among 4 resident warps to hide a 400-cycle memory stall. |
Lab 02 — Memory Hierarchy & Roofline Microbenchmarks (C)
| Field | Value |
|---|---|
| Goal | Measure your own machine's memory hierarchy with C microbenchmarks, then build a roofline model and place kernels on it. |
| Concepts | Cache lines, latency vs bandwidth, pointer-chasing, streaming, arithmetic intensity, the roofline. |
| Steps | 1) make && ./membench. 2) Identify L1/L2/L3/DRAM plateaus in the latency curve. 3) Compute your machine's roofline ridge point. 4) Place vector_add (AI≈0.08) and matmul (AI grows with tile size) on the roofline; predict then verify which is memory-bound. |
| Stack | C (C11), make, optionally perf |
| Output | A filled-in ROOFLINE.md worksheet with your measured numbers. |
| How to Test | Latency plateaus match your CPU's published cache sizes within 2×; streaming bandwidth within 30% of spec. |
| Talking Points | Why pointer-chasing measures latency but streaming measures bandwidth; how the same methodology applies to HBM on a GPU; the A100/H100 ridge-point calculation. |
| Resume Bullet | "Wrote C microbenchmarks measuring cache-hierarchy latency/bandwidth; derived roofline models for CPU and GPU targets and used arithmetic-intensity analysis to predict kernel-bound regimes before profiling." |
| Extensions | Add a strided-access benchmark and explain the cache-line cliff at stride 64; rerun pinned to a different core with taskset and explain NUMA effects. |
Deliverables Checklist
- SIMT simulator runs; you can explain every line of the execution loop
- Divergence-cost table produced and explained
-
membenchresults with identified cache plateaus - Roofline worksheet: ridge points computed for your CPU and for A100/H100 from spec sheets
- You can do the "512 KB per token" style bandwidth math cold (see WARMUP Ch. 7)
Interview Relevance
- "Walk me through what happens when a warp hits a branch."
- "Why do GPUs have thousands of cores but no branch predictors?"
- "A100 has 1,555 GB/s HBM and 312 TFLOPS BF16 — what's the ridge point, and what does it mean for LLM decode?"
- "Your vendor claims their chip beats H100. What three numbers do you ask for?"