Head of Software Engineering [GPU] — Complete Apprenticeship Curriculum
Target Roles:
- Head of Engineering / VP Engineering — AI Infrastructure (deep-tech startups)
- Principal / Staff Engineer — GPU Platforms, AI Compute Clouds (CoreWeave, Lambda, Together, Modal, NVIDIA, AMD)
- Platform Lead — LLM Serving & GPU Orchestration (vLLM-style platforms, inference clouds)
- Founding Engineer — Sovereign AI / On-Premise LLM Infrastructure
- Engineering Lead — Hardware-Agnostic AI Runtime / GPU Compiler Stack
Duration: 26–36 weeks core — extendable to 18 months for full head-of-engineering readiness Goal: Build the rare combination this JD demands: silicon-level GPU understanding, compiler and driver literacy, production LLM-serving depth (GPU scheduling, KV-cache management), hardware-decoupled platform architecture, security/licensing primitives for commercial software, and the leadership judgment to ship it all through a real release cycle.
Why This Curriculum Exists
The "Head of Software Engineering [GPU]" role is one of the hardest hiring profiles in the industry, because it sits at the intersection of five disciplines that are usually separate careers:
- GPU systems engineering — chips, SMs, warps, memory hierarchies, CUDA, drivers, compilers down to PTX/SASS.
- AI infrastructure — LLM serving, KV-cache management, continuous batching, multi-engine deployment (vLLM, TensorRT-LLM, SGLang).
- Distributed systems & HPC — collectives, NCCL, multi-node scheduling, failure domains.
- Platform productization — decoupling software from hardware (the JD's core mandate: "a platform that operates independently of the underlying hardware"), developer ecosystems, OEM/cloud integrations.
- Commercial software protection — hardware fingerprinting, license enforcement, code obfuscation, air-gapped/sovereign deployment for regulated industries.
Most engineers know one or two of these. This curriculum is built backward from the JD so that every "required" and "nice to have" line maps to phases and hands-on labs:
| JD Requirement | Where It's Trained |
|---|---|
| GPU-accelerated workloads, CUDA, GPU drivers | Phases 01, 02, 03, 04 |
| Runtime systems programming | Phases 03, 05, 09 |
| GPU scheduling | Phase 06 |
| KV-cache management, LLM serving optimisation | Phase 07 |
| Distributed systems, HPC | Phase 08 |
| Decouple software from hardware environments | Phase 09 (the keystone phase) |
| Linux internals, containers, observability tooling | Phases 04, 10 |
| Hardware fingerprinting, license enforcement, software protection, code obfuscation | Phase 11 |
| Sovereign AI, on-premise, air-gapped, regulated industries | Phases 07, 11 |
| Ship through a full release cycle; lead teams; OEM/partner strategy | Phases 10, 12 |
| Python, C++, systems language (Rust/Go/C), JS/TS tooling | Labs span C, C++, CUDA, Rust, Go, Python, TypeScript |
What You Will Build
By the end of this curriculum you will have produced, with your own hands:
- A SIMT execution simulator that demonstrates warp divergence and lockstep execution (Phase 01)
- Memory-hierarchy microbenchmarks in C that measure cache lines, bandwidth ceilings, and the roofline of your own machine (Phase 01)
- CUDA kernels from vector-add to tiled shared-memory matmul, with occupancy/coalescing experiments (Phase 02)
- A mini compiler that emits real PTX from an expression language — plus annotated dissections of NVCC's pipeline, PTX, and SASS (Phase 03)
- A Linux character-device driver in C with an ioctl interface that mimics how GPU drivers expose command submission (Phase 04)
- A container-from-scratch lab (namespaces + cgroups) showing exactly how GPUs pass into containers (Phase 04)
- A caching GPU-style memory allocator in Rust (the PyTorch-allocator design) and a stream scheduler in C++ (Phase 05)
- A GPU cluster scheduler simulator (bin-packing, gang scheduling, fragmentation metrics) and a Kubernetes device-plugin walkthrough (Phase 06)
- A paged KV-cache allocator with prefix sharing and a continuous-batching simulator with TTFT/TPOT/goodput metrics (Phase 07)
- Ring all-reduce implemented over real sockets, benchmarked against the theoretical 2(N-1)/N bound (Phase 08)
- A hardware abstraction layer in C with dlopen-loaded vendor backends — the literal architecture of "decouple software from hardware" (Phase 09)
- A GPU metrics exporter (Prometheus format), failure-drill runbooks, and an SLO definition pack (Phase 10)
- A hardware fingerprinting tool, an Ed25519-signed license-enforcement system with grace periods and air-gap activation, and an anti-tamper/obfuscation lab in C (Phase 11)
- A capstone: mini GPU orchestration platform — scheduler + HAL + serving + licensing + observability composed into one system, with a design doc and release-cycle artifacts (Phase 12)
Folder Structure
Head of Software Engineering [GPU]/
├── README.md ← You are here
├── jd.md ← Target role profile
├── phase-01-gpu-architecture-silicon/ ← From transistor to SM: how the chip actually works
├── phase-02-cuda-programming/ ← CUDA C++: kernels, memory, streams, occupancy
├── phase-03-gpu-compilers-codegen/ ← NVCC, PTX, SASS, LLVM NVPTX, Triton; build a mini PTX compiler
├── phase-04-linux-drivers-containers/ ← Kernel modules, ioctl, the GPU driver stack, cgroups, container GPU passthrough
├── phase-05-runtime-systems-memory/ ← Caching allocators, streams/events, async runtimes (Rust + C++)
├── phase-06-gpu-scheduling-virtualization/ ← MPS, MIG, time-slicing, k8s device plugins, cluster scheduling
├── phase-07-llm-serving-kv-cache/ ← KV-cache management, PagedAttention, continuous batching, multi-engine ops
├── phase-08-distributed-hpc-collectives/ ← NCCL, ring all-reduce, topology, multi-node failure domains
├── phase-09-hardware-abstraction-portability/ ← HAL design, plugin ABIs, dlopen backends — the decoupling mandate
├── phase-10-observability-production/ ← DCGM, exporters, eBPF, SLOs, release engineering
├── phase-11-security-licensing-protection/ ← Fingerprinting, license enforcement, obfuscation, attestation, air-gap
├── phase-12-leadership-capstone/ ← Head-of-engineering practice + mini orchestration platform capstone
├── interview-prep/ ← Concepts, coding, systems, leadership interviews
└── system-design/ ← GPU cloud, serving platform, HAL, licensing-at-scale walkthroughs
26-Week Core Schedule
| Week | Phase | Focus |
|---|---|---|
| 1 | 01 | GPU vs CPU design philosophy; SMs, warps, SIMT; build the SIMT simulator |
| 2 | 01 | Memory hierarchy: HBM, L2, shared memory, registers; roofline microbenchmarks in C |
| 3 | 02 | First CUDA kernels: grid/block/thread, vector add, error handling |
| 4 | 02 | Tiled matmul in shared memory; coalescing and occupancy experiments |
| 5 | 03 | The NVCC pipeline; reading PTX and SASS; Godbolt dissections |
| 6 | 03 | Build a mini expression→PTX compiler; Triton and LLVM NVPTX overview |
| 7 | 04 | Linux kernel modules; write a char device with ioctl command submission |
| 8 | 04 | The real GPU driver stack (KMD/UMD); containers from scratch; GPU passthrough |
| 9 | 05 | Memory allocators: buddy, slab, caching; build the Rust caching allocator |
| 10 | 05 | Streams, events, dependency graphs; build the C++ stream scheduler |
| 11 | 06 | GPU sharing: MPS vs MIG vs time-slicing; fractional GPUs |
| 12 | 06 | Cluster scheduling: bin-packing, gang scheduling; build the scheduler simulator |
| 13 | 07 | KV-cache memory math; build the paged KV allocator with prefix sharing |
| 14 | 07 | Continuous batching simulator; TTFT/TPOT/goodput; vLLM/TensorRT-LLM/SGLang ops |
| 15 | 08 | Collectives from first principles; implement ring all-reduce over sockets |
| 16 | 08 | NCCL, topology (NVLink/IB), multi-node training/serving failure modes |
| 17 | 09 | ABI vs API; plugin architectures; build the C HAL with dlopen backends |
| 18 | 09 | Portability strategy: CUDA/ROCm/Level Zero/Metal; capability negotiation |
| 19 | 10 | Observability: DCGM metrics, build the Prometheus exporter; eBPF tracing |
| 20 | 10 | SLOs, failure drills, release engineering: versioning, rollback, canary |
| 21 | 11 | Hardware fingerprinting: build the tool; threat-model the spoofing adversary |
| 22 | 11 | License enforcement: Ed25519-signed licenses, grace, air-gap activation |
| 23 | 11 | Software protection: obfuscation, anti-tamper, attestation; sovereign AI deployment |
| 24 | 12 | Leadership: org design, technical strategy memos, OEM/partner engineering |
| 25 | 12 | Capstone build: mini orchestration platform integration |
| 26 | 12 | Capstone release cycle: design doc, runbook, demo, retrospective |
Phase Structure
Every phase follows a consistent format:
| File | Purpose |
|---|---|
README.md | Why the phase exists, concepts list, lab tables (goal/steps/output/testing/talking points/resume bullet) |
WARMUP.md | Zero-to-expert primer: every concept from first principles, with lab walkthrough guidance, success criteria, interview Q&A, references |
HITCHHIKERS-GUIDE.md | The practitioner's deep-dive: mental models, production numbers, the things people actually get asked |
lab-XX-*/ | Hands-on labs: README walkthrough + working reference code you run, break, and extend |
Labs deliberately span the JD's language matrix: C (drivers, HAL, anti-tamper), C++/CUDA (kernels, stream scheduler), Rust (allocator, fingerprinting), Go (device plugin), Python (simulators, serving, licensing), TypeScript (platform CLI tooling in the capstone).
No GPU? You can still do ~85% of this track
Labs are designed in three tiers:
- CPU-only (most labs): simulators, allocators, schedulers, compilers-to-PTX (text output), licensing, HAL — need only a Linux/macOS box.
- Any-NVIDIA-GPU (Phase 02, parts of 03/10): a free Google Colab T4 or any consumer card works.
- Cluster-scale (parts of 06/08): simulated locally; the WARMUPs explain what changes at real scale.
Prerequisites
- Solid Python; working C or C++ (you can read pointers, structs, makefiles)
- Comfort in a Linux shell (you'll compile kernels modules and use
perf,strace) - Basic ML literacy: what a transformer forward pass is (or run the LLM Inference Engineer track Phases 01–04 first)
- Git, Docker basics
- No prior CUDA, driver, or compiler experience required — Phase 01 starts at the silicon
Related Tracks in This Repo
- LLM Inference Engineer — deeper on models/training; its Phase 09 pairs with our Phase 07
- Security Engineer — deeper on offensive/defensive security; pairs with our Phase 11
- AI Specialist — air-gapped serving labs that pair with our Phase 11 sovereign-AI work