Lab 01 — fakegpu: A GPU-Shaped Linux Character Device Driver

Phase: 04 — Linux & Drivers | Difficulty: ⭐⭐⭐⭐☆ | Time: 5–8 hours Language: C (kernel module + userspace client) | Hardware: Linux VM (snapshot it first!)

Concept primer: ../WARMUP.md Ch. 1–4. ⚠️ Kernel code runs with no safety net — use a VM (Lima/Multipass/UTM/WSL2/cloud), snapshot before insmod. A panic here is a lesson, not a disaster.

Run

sudo apt install build-essential linux-headers-$(uname -r)   # Debian/Ubuntu
make                       # builds fakegpu.ko (kernel) and client (user)
sudo insmod fakegpu.ko
dmesg | tail -3            # "fakegpu: loaded, /dev/fakegpu ready"
sudo ./client              # exercises the full ioctl ABI; prints PASS/FAIL
strace -e ioctl,openat ./client 2>&1 | head -30   # compare to a real CUDA trace
sudo rmmod fakegpu

0. The mission

Implement the architectural skeleton every GPU driver shares, in ~300 honest lines:

fakegpu conceptReal GPU equivalent
per-open() context structCUDA context (per-process GPU state)
ALLOC/FREE ioctls → handlescudaMalloc → device pointers (via KMD page tables)
SUBMIT ioctl → job queuecommand-buffer submission (real HW: mapped ring + doorbell)
WAIT ioctl → completed seqnofence wait (cudaEventSynchronize)
per-context byte quotaper-context VRAM accounting
INFO ioctlcudaGetDeviceProperties

The client is the test suite: it asserts context isolation (two fds can't see each other's handles), quota enforcement (over-alloc returns ENOSPC), seqno ordering, and input validation (bad handle → EINVAL, oversized request → rejected — the adversarial-userspace discipline from WARMUP Ch. 3).

1. Reading order (fakegpu.c)

  1. The ABI header (fakegpu_ioctl.h) — shared by kernel and client; note _IOWR encodings and fixed-size structs. ABI files are forever; this one is written like it.
  2. fakegpu_open/release — per-fd context allocation. This is the pattern: everything else hangs off file->private_data.
  3. fakegpu_ioctl — the dispatcher: copy_from_user → validate → act → copy_to_user. Count the validation lines vs the "work" lines (the ratio is the lesson).
  4. The job model — SUBMIT enqueues and immediately completes jobs (incrementing completed_seq). Real hardware completes asynchronously; the extension adds a timer_list to simulate that.

2. The strace comparison (step 3 of the lab)

Your client:

openat(AT_FDCWD, "/dev/fakegpu", O_RDWR)       = 3
ioctl(3, FAKEGPU_INFO, ...)                    = 0
ioctl(3, FAKEGPU_ALLOC, {size=1048576, ...})   = 0
ioctl(3, FAKEGPU_SUBMIT, ...)                  = 0
ioctl(3, FAKEGPU_WAIT, ...)                    = 0

A real CUDA init (from WARMUP Ch. 3): same shape — open device, info/version ioctls, allocation ioctls, then mmap calls you don't have (the ring/doorbell — because real drivers move submission out of ioctl; your extension 3 closes that gap). Write the side-by-side into TRACE-NOTES.md — that artifact is your "technical credibility" exhibit for this phase.

3. Extensions

  1. Quota tightening: per-context and global device quota; correct errno for each (ENOSPC vs ENOMEM). Add a /sys/module/fakegpu/ or sysfs stats attribute (your Phase 10 metrics on-ramp).
  2. Async completion: a timer_list completes jobs 10 ms after submit; WAIT blocks on a wait_queue_head_t until completed_seq >= target. You've just implemented a fence.
  3. The mmap ring (hard, the real pattern): fakegpu_mmap exposes a page containing completed_seq; client polls it with zero syscalls in steady state. Measure waits/sec: ioctl-WAIT vs mmap-poll (~100× — the entire reason GPU submission works the way it does).
  4. Adversarial client: write evil.c — bad handles, huge sizes, ioctls on a closed-and-reopened fd, two threads racing SUBMIT. Every case must fail cleanly. (Phase 11 will call this "attack surface review"; here it's just good driver hygiene.)

4. Common pitfalls

  1. Dereferencing the user pointer instead of copy_from_user — works in your VM (same address space mapping luck), then oopses elsewhere. The compiler __user annotation + sparse exists for this.
  2. Forgetting mutex_lock around context state — two threads on one fd corrupt the buffer list; the racing-SUBMIT test in extension 4 catches it.
  3. Leaking contexts on release — rmmod then dmesg shows the leak warning the module prints; the kernel's own kmemleak would find it in real review.
  4. Building against the wrong headers (uname -r mismatch after a kernel update) — insmod: invalid module format. This is the DKMS lesson from WARMUP Ch. 6, self-inflicted once for learning.

5. What this lab proves about you

You can read (and write) the layer most GPU engineers only gesture at: when a fleet incident says "Xid 31, context poisoned, ioctl returned EIO," every word of that sentence now maps to code you've written. And nvidia's open-gpu-kernel-modules repo is now readable to you — start with kernel-open/nvidia/nv.c and recognize the shapes.