Warmup Guide — Linux Internals, GPU Drivers & Containers

Zero-to-expert primer for Phase 04. Assumes you can use a Linux shell and read C. No kernel programming background assumed — that's what we build. By the end the stack between cudaLaunchKernel and silicon is glass, not fog.

Table of Contents

Chapter 1: Kernel Space, User Space, and What a Driver Is

Zero background: the CPU runs in (at least) two privilege modes. Kernel mode can touch hardware, page tables, interrupts; user mode cannot — any attempt traps. Your process lives in user mode; the only doorway into kernel mode is a syscall (or an interrupt/fault). That hard boundary is why "the process crashed" doesn't take the machine down, and why everything hardware-related — including every GPU operation — ultimately funnels through the kernel at least once.

What a driver is, demystified: a driver is not a process or a daemon. It's a bundle of callbacks registered with the kernel ("when someone open()s this device, call my open; when they ioctl(), call my ioctl") plus interrupt handlers ("when the hardware raises IRQ N, call this"). A loadable kernel module (.ko) is just a relocatable object the kernel links into itself at runtime — insmod is closer to dlopen than to exec. Lab 01 makes this concrete: your fakegpu.ko is ~300 lines of callbacks and one struct file_operations.

The cost model that explains GPU driver design: a syscall costs ~100–300 ns plus cache pollution; an interrupt costs more. Drivers for high-rate devices (GPUs, NICs, NVMe) are therefore architected to get the kernel out of the hot path: set up shared memory once, then submit work from user space without syscalls (Ch. 4). Keep this in mind — it's the single design force behind everything in this phase.

Misconception: "the GPU driver runs on the GPU." The driver runs on the CPU, in the kernel; the GPU runs firmware (increasingly — GSP, Ch. 4) and your kernels. The driver's job is bookkeeping, memory management, scheduling, and shoveling command buffers.

Chapter 2: Character Devices — /dev, major/minor, file_operations

Unix's grand unification: devices are files. open("/dev/nvidia0") returns an fd; read/write/ioctl/mmap on that fd dispatch into driver callbacks.

The plumbing, end to end:

  • A character device ("char dev") is the streams-of-bytes/commands flavor (vs block devices, which are the page-cached storage flavor). GPUs are char devices.
  • Each device has a major number (selects the driver) and minor number (selects the instance — /dev/nvidia0 vs /dev/nvidia1). ls -l /dev/nvidia* shows them (major 195 for NVIDIA, historically).
  • /dev nodes are created by udev at hotplug (or mknod by hand, or devtmpfs). A node is just a (type, major, minor) triple in a filesystem — this fact is load-bearing for containers (Ch. 9): injecting a GPU into a container is largely creating/allowing this node inside it.
  • In the driver, you register a struct file_operations:
static const struct file_operations fakegpu_fops = {
    .owner          = THIS_MODULE,
    .open           = fakegpu_open,      // allocate per-fd state — "a context"
    .release        = fakegpu_release,   // free it
    .unlocked_ioctl = fakegpu_ioctl,     // the command path
    .mmap           = fakegpu_mmap,      // shared-memory path (extension)
};

The idea to carry away: per-open-file state. Each open() gets its own context structure (Lab 01 hangs a buffer registry and job queue off it). This is precisely what a CUDA context is at the driver level — and why two processes on one GPU don't see each other's allocations (and why a poisoned context dies alone, Phase 02 Ch. 4).

Chapter 3: ioctl — The GPU's Actual Control Path

read/write suit byte streams. Devices need commands: "allocate 2 GB," "map this buffer," "submit this command stream." The escape hatch is ioctl(fd, CMD, arg) — a generic "do device-specific thing," where CMD encodes direction/size/type and arg points to a command struct copied across the boundary.

The ABI discipline (Lab 01's core lesson):

struct fakegpu_alloc { __u64 size; __u32 handle; /* out */ };
#define FAKEGPU_IOC_ALLOC  _IOWR('F', 1, struct fakegpu_alloc)

// kernel side — NEVER dereference user pointers:
if (copy_from_user(&req, (void __user *)arg, sizeof(req))) return -EFAULT;
...validate everything...
if (copy_to_user((void __user *)arg, &req, sizeof(req))) return -EFAULT;

copy_from_user/copy_to_user are the checked crossings; validation is mandatory because user space is adversarial (a security boundary you'll revisit in Phase 11). Handles — small integers the driver maps to kernel-side objects — instead of raw pointers, for the same reason fds exist.

Proof that this is real: strace any CUDA program:

openat(AT_FDCWD, "/dev/nvidiactl", O_RDWR) = 3
ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x46, 0x2a, 0x20), 0x7ffd...) = 0
openat(AT_FDCWD, "/dev/nvidia0", O_RDWR)  = 4
ioctl(4, ...)                             = 0     // dozens at init
mmap(NULL, 0x200000, ..., fd=4, ...)      = 0x7f...   // ring buffers!

Initialization is a flurry of ioctls; then it goes quiet — because the steady state uses the mmap'd path (next chapter). Lab 01 step 3 has you place these two traces side by side.

Chapter 4: The Real NVIDIA Stack — KMD, UMD, Ring Buffers, Doorbells

The division of labor:

  • KMD (kernel-mode driver) — nvidia.ko (+ nvidia-uvm.ko for unified memory, nvidia-modeset.ko for display): privileged setup. Memory management (GPU page tables!), context creation, channel/ring allocation, interrupt handling, recovery (Xid errors — Phase 10), power.
  • UMD (user-mode driver) — libcuda.so (the driver API from Phase 02 Ch. 1 lives here): everything per-launch. Builds command buffers in memory shared with the GPU, manages your virtual address space, JIT-compiles PTX (Phase 03 Ch. 7!), implements the CUDA API surface.

The hot path — why launches take ~5 µs, not ~5 ms:

  1. At context setup, the KMD maps a ring buffer (command queue) and a doorbell register into the process's address space (those mmaps in the strace).
  2. To launch a kernel, libcuda.so writes the command into the ring (a memory write!) and writes the doorbell (another memory write, which the hardware sees) — zero syscalls.
  3. The GPU's front end fetches commands, executes, and writes a fence (completion marker) back to shared memory; the UMD polls or sleeps on an interrupt for it. (cudaEventSynchronize = wait-on-fence.)

This user-space-submission pattern is universal in high-performance I/O — NVMe queues, io_uring, DPDK NICs, every modern GPU (AMD's is in the open amdgpu driver — same shapes, readable source). Your Lab 01 extension (mmap'd completion ring) implements the fence half.

Two modern wrinkles worth knowing: GSP firmware — recent NVIDIA drivers offload much KMD logic to a RISC-V management processor on the GPU, shrinking kernel-side code (and enabling the open-gpu-kernel-modules release: the kernel module is now MIT/GPL dual-licensed and on GitHub — readable!); the proprietary crown jewels moved to firmware + UMD. And nouveau — the reverse-engineered FOSS driver — historically lacked the firmware interfaces for reclocking, hence its performance gap; GSP is slowly changing that equation.

Chapter 5: DMA, IOMMU, and Pinned Memory From the Kernel's Side

Phase 02 Ch. 5 told you pinned memory makes transfers ~2× faster and truly async. Now the mechanism:

  • DMA: the GPU's copy engines read/write host RAM directly, without the CPU touching the data. To do that safely, the physical pages must (a) stay resident — not be swapped or migrated — and (b) be known to the device.
  • Pinning (cudaMallocHost → get_user_pages/page-locking in the kernel) guarantees (a). For pageable memory the driver must instead copy through an internal pre-pinned bounce buffer — there's your 2× and your fake-async.
  • IOMMU provides (b)-with-safety: it's a page table for devices, translating device-visible addresses and blocking DMA outside allowed ranges. Cloud passthrough (vfio), confidential computing, and "why does iommu=off make it faster but scarier" all live here. (Security relevance returns in Phase 11: DMA from a malicious device is a classic attack class.)

Chapter 6: Version Skew — The Driver/CUDA Compatibility Matrix

The unglamorous chapter that pages people at 2 a.m.:

  • One kernel module serves all processes; each container/app brings its own CUDA toolkit/runtime. The contract: driver ≥ what the toolkit needs. nvidia-smi's "CUDA Version" = the max the driver supports, not what's installed.
  • Failure signatures to recognize on sight: CUDA driver version is insufficient for CUDA runtime version (driver too old); forward compatibility was attempted on non supported HW (compat package misuse); a container that worked on node A failing on node B (driver skew across the fleet); kernel upgrade → module won't load → DKMS didn't rebuild (or secure-boot signing failed — Phase 11 connection).
  • Fleet upgrade choreography (the head-of-engineering deliverable): inventory driver/toolkit pairs → canary nodes → drain GPU workloads (Phase 06 scheduler integration) → upgrade KMD (reboot or careful unload) → health-check (Phase 10's checks) → progressive rollout. The forward-compatibility packages (newer libcuda on older driver branch, datacenter GPUs only) buy flexibility; pinning exact driver versions in node images buys reproducibility. Pick a policy and write it down — Lab 02's explainer and Phase 10's release engineering both cite this chapter.

Chapter 7: Namespaces — What a Container Actually Is

There is no container object in the kernel. "Container" = a process started with unshared namespaces + resource limits (Ch. 8) + (usually) reduced privileges. Namespaces virtualize visibility:

NamespaceIsolatesDemo
mountfilesystem treedifferent / via pivot_root
PIDprocess IDsyour shell is PID 1; can't see host
netinterfaces, portsempty ip a until you add a veth
UTShost/domain namehostname container1
IPCSysV/POSIX queues
userUID mappingsroot inside = UID 100000 outside (rootless)
cgroupcgroup tree view

One syscall family creates them: clone(CLONE_NEWPID | CLONE_NEWNS | ...) / unshare(2). Lab 02's runtime is ~200 lines because that's genuinely all it takes: clone with flags, set hostname, pivot_root into a rootfs, mount fresh /proc, drop into a shell. Docker/Podman/containerd add image management, networking, and an API — the isolation itself is these kernel primitives.

Why this matters for a GPU platform: every scheduling, isolation, and security promise you make to customers (Phase 06, 11) bottoms out in which namespaces/cgroups/devices a workload got. You can't reason about "can tenant A see tenant B's model weights" without this chapter.

Chapter 8: cgroups v2 — Resources, and the GPU-Shaped Hole

Namespaces control what a process sees; cgroups control what it uses. v2 is a unified tree mounted at /sys/fs/cgroup; directories are groups; writes to control files set limits:

mkdir /sys/fs/cgroup/demo
echo 100M  > /sys/fs/cgroup/demo/memory.max
echo "50000 100000" > /sys/fs/cgroup/demo/cpu.max   # 0.5 CPU
echo $$    > /sys/fs/cgroup/demo/cgroup.procs       # join

Exceed memory.max → the OOM killer fires inside the group (Lab 02 step 2 demonstrates). The devices controller (an eBPF program in v2) allow-lists device-node access — the enforcement half of GPU injection (Ch. 9).

The GPU-shaped hole — say this sentence in interviews: there is no GPU cgroup controller. No gpu.max, no kernel-enforced fraction of SMs or HBM. Consequences:

  • "GPU limits" in k8s are allocation (whole devices via the device plugin), not enforcement.
  • Actual sharing requires GPU-side mechanisms — MPS (cooperative, weak isolation), MIG (hardware partition, strong), time-slicing (none) — Phase 06's entire subject matter, motivated by this one missing kernel feature.
  • Vendors building "fractional GPU" products (and this JD's company likely among them) are filling exactly this hole with userspace interception or scheduler tricks — knowing why the hole exists (GPU contexts are opaque to the kernel; the KMD, not the scheduler, owns GPU time) is what lets you evaluate those products honestly.

Chapter 9: How a GPU Enters a Container

Now compose Chapters 2, 7, 8 — the full anatomy, which Lab 02 step 3 replicates manually:

A container needs three things to use a GPU:

  1. The device nodes: /dev/nvidia0, /dev/nvidiactl, /dev/nvidia-uvm visible in its mount namespace (bind-mount or mknod), and allowed by the devices cgroup/eBPF program.
  2. The user-mode driver libraries: libcuda.so matching the host kernel module (Ch. 6!). The container image cannot ship these — it would skew against the host KMD. So they're injected from the host at start.
  3. No special kernel work — the KMD is shared; namespaces don't apply to it.

nvidia-container-toolkit is precisely this: an OCI hook (now CDI spec) that, at container creation, mounts the device nodes + host driver libs + nvidia-smi et al. into the container and writes the cgroup allow rule. CUDA_VISIBLE_DEVICES (or NVIDIA_VISIBLE_DEVICES at the hook level) then filters which minors get this treatment — note carefully: visibility, not security — a process that can mknod or escape the allow-list can reach other GPUs; the cgroup rule is the actual enforcement, and per-tenant isolation worthy of the name means dedicated devices or MIG (Phase 06) plus the Phase 11 story.

Debugging ladder for "container can't see GPU" (memorize; it's a weekly occurrence somewhere in any fleet): node has driver? (nvidia-smi on host) → toolkit hook configured? (runtime config) → nodes present in container? (ls /dev/nvidia*) → libs injected & matching? (ldconfig -p | grep cuda, compare versions) → cgroup allows? → only then blame the app.

Chapter 10: The Debugging Toolkit — strace, dmesg, /proc, /sys

The five tools that solve most layer-4 mysteries:

  • strace -f -e trace=openat,ioctl,mmap — what is the process actually asking the kernel? (CUDA init failures become obvious: which open/ioctl returned what errno.)
  • dmesg -T — the kernel's side of the story: module load errors, Xid errors (the GPU's error codes: Xid 79 = fell off the bus, Xid 48 = ECC DBE… Phase 10 builds alerting on these), OOM kills, IOMMU faults.
  • /proc/PID/ — maps (see the mmap'd rings!), fd/ (which devices are open), status, cgroup.
  • /sys — device topology (/sys/bus/pci/devices/...), cgroup tree, module params (/sys/module/nvidia/).
  • lsof /dev/nvidia* — who's holding the GPU (the "why won't the driver unload" question).

Lab 01 step 3 and Lab 02's debugging exercises are drills on exactly these.

Lab Walkthrough Guidance

Order: Lab 01 → Lab 02 (Lab 02 injects Lab 01's device into a container — the punchline depends on both).

Lab 01 (fakegpu driver):

  1. Read fakegpu.c top-down against Ch. 2–3 before building; predict what dmesg will print at insmod.
  2. Build/load (any Linux VM; headers via apt install linux-headers-$(uname -r)). Run ./client; all asserts PASS.
  3. strace ./client → annotate against the chapter-3 CUDA trace. Two columns, same shape — that's the lab's thesis made visible.
  4. Extensions in order: quotas (validation discipline) → timer-based async completion (interrupt/fence flavor) → mmap'd completion ring (the real pattern; hardest).
  5. Safety: it's a VM, snapshot first; a buggy module can panic the kernel — which is itself the lesson about why KMDs are small and UMDs are big.

Lab 02 (mini container runtime):

  1. Run first, read after: sudo ./mininer run /bin/sh, poke around (ps, hostname, ls /), exit, then read mininer.c against Ch. 7.
  2. cgroup demo: apply memory.max, run the included hog, watch the group OOM while the host stays healthy.
  3. The synthesis step: inject /dev/fakegpu into the container (bind-mount + devices allow) and run Lab 01's client inside. When that passes, write the explainer — you have personally performed what nvidia-container-toolkit automates.

Success Criteria

  • You can define "driver" as callbacks + interrupts, and narrate open→ioctl→copy_from_user→validate→copy_to_user (Ch. 1–3)
  • You can explain why steady-state GPU submission uses zero syscalls (rings + doorbells + fences) and what the KMD vs UMD each own (Ch. 4)
  • You can diagnose the four version-skew signatures and sketch the fleet upgrade choreography (Ch. 6)
  • You can build a container from syscalls and state precisely what namespaces, cgroups, and capabilities each contribute (Ch. 7–8)
  • You can recite the GPU-into-container anatomy and its debugging ladder, and you've performed it by hand (Ch. 9)
  • "There is no GPU cgroup controller" — and you can derive Phase 06 from it

Interview Q&A

Q1: What happens at the OS level when a CUDA app starts and launches a kernel? A: Init: libcuda.so opens /dev/nvidiactl and /dev/nvidia0, issues a series of ioctls to create a context — the KMD allocates GPU page tables, a command ring, and maps the ring plus a doorbell into the process (visible as mmaps in strace). It also JIT-compiles any PTX lacking a matching SASS (Phase 03). Launch: the UMD writes the launch command into the mapped ring and taps the doorbell — two memory writes, no syscall — the GPU front end fetches, executes, and writes a fence; synchronization waits on that fence (poll or interrupt). That's why launches cost microseconds and why strace goes quiet after init.

Q2: Why doesn't CUDA_VISIBLE_DEVICES provide isolation? A: It's an environment variable read by the CUDA library in the untrusted process — it filters enumeration, nothing more; the process can unset it, and the device nodes are what they are. Enforcement lives lower: the devices cgroup/eBPF allow-list controls which nodes can be opened (what nvidia-container-toolkit configures), and even that only gates access to a whole device — within a shared GPU there's no kernel-enforced resource split because no GPU cgroup controller exists. Real isolation: dedicated GPU per tenant, or MIG hardware partitions, plus process-level (not context-level) multi-tenancy. The env var is UX; the cgroup is policy; MIG is enforcement.

Q3: A container sees /dev/nvidia0 but cudaInit fails. Debug. A: Ladder: (1) host sanity — nvidia-smi on the node (driver loaded? GPU healthy? check dmesg for Xids). (2) Library injection — ldconfig -p | grep libcuda inside the container; missing → toolkit hook not configured for this runtime; present → compare its version to the host KMD (nvidia-smi driver version) for skew, the classic "image ships its own libcuda" bug. (3) Permissions — devices cgroup allow rule, node major/minor match, non-root user in the right group. (4) The other nodes — nvidia-uvm missing breaks managed memory only, nvidiactl missing breaks everything. (5) strace the init and read which exact open/ioctl fails with which errno — it almost always names the culprit directly.

Q4: Why do GPU drivers split into a kernel module and libcuda.so, and why is the kernel part getting smaller? A: The split follows the privilege/performance line: things requiring kernel mode (page tables, interrupts, PCI config, cross-process arbitration) sit in the KMD; everything per-launch sits in user space for speed (ring submission) and shipability (a library updates with the toolkit; a kernel module update risks the host). The KMD is shrinking because GSP firmware on the GPU's own RISC-V core absorbed initialization and management logic — less GPL-adjacent kernel code (enabling NVIDIA's open kernel modules), faster context ops, and the proprietary logic now lives in firmware+UMD where it's both protected and updatable. Same pattern as NVMe/io_uring/DPDK: kernel for setup and safety, user space for the data path, firmware for vendor secrets.

Q5: Design the ioctl ABI for a simple accelerator. What discipline applies? A: Fixed-size, explicitly-padded structs (the ABI is forever — version field or reserved fields for growth); _IOWR macros encoding direction and size; copy_from_user, then validate every field against per-context state (handles not pointers, bounds on sizes, quota checks) because user space is adversarial; return handles, never kernel addresses; per-fd context so isolation falls out of the file model; errno discipline (-EINVAL vs -ENOMEM vs -EPERM mean different things to the caller's retry logic). And keep the hot path out of ioctl entirely — submission belongs in a mapped ring once setup is done (Lab 01's extension, and every real GPU's design).

Q6 (leadership): Plan a driver upgrade across a 10,000-GPU inference fleet. A: Inventory first: current driver/toolkit pairs per node pool, and every container image's CUDA runtime requirement — build the compatibility matrix before touching anything (driver must satisfy the max toolkit; forward-compat packages where datacenter GPUs allow). Canary: one node pool, full Phase 10 health suite (Xid monitoring, perf regression benchmarks — driver upgrades have shipped perf regressions). Choreography per node: cordon → drain via the scheduler (Phase 06; respect long-running jobs with checkpointing policy) → upgrade KMD + container toolkit together → reboot (cleanest; hot-unload is fragile with GPUs) → health-check → uncordon. Progressive rollout with automatic halt on Xid-rate or perf anomaly. Rollback path tested before rollout, image-pinned node pools so a rollback is a re-image, not an archaeology dig. And the boring decisive control: driver versions pinned in infrastructure-as-code, never apt upgrade'd by hand.

References

  • Linux Device Drivers, 3rd ed. (Corbet, Rubini, Kroah-Hartman) — free online; Ch. 1–3, 6 map directly to Lab 01 — https://lwn.net/Kernel/LDD3/
  • The kernel's own docs: The Linux Kernel documentation — driver-api, cgroup-v2 — https://docs.kernel.org
  • NVIDIA open GPU kernel modules (read a real KMD!) — https://github.com/NVIDIA/open-gpu-kernel-modules
  • NVIDIA Container Toolkit architecture + CDI spec — https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/
  • NVIDIA CUDA Compatibility (the skew matrix) — https://docs.nvidia.com/deploy/cuda-compatibility/
  • NVIDIA Xid errors reference — https://docs.nvidia.com/deploy/xid-errors/
  • man 2 clone, man 7 namespaces, man 7 cgroups — genuinely excellent
  • Liz Rice, "Containers from Scratch" (the famous live demo) — https://www.youtube.com/watch?v=8fi7uSYlOdc
  • freedesktop.org DRM/amdgpu docs (the open-source GPU driver world) — https://docs.kernel.org/gpu/
  • Cross-track: Phase 06 WARMUP (what fills the cgroup hole), security track (kernel/container hardening at depth)