🛸 Hitchhiker's Guide — Phase 04: Linux, Drivers & Containers

Read this if: CUDA works for you but /dev/nvidia0, nvidia.ko, and "the toolkit hook" are folklore. Field notes; derivations in the WARMUP.

0. The 30-second mental model

your app ─ libcudart ─ libcuda.so (UMD: rings, JIT, contexts)   [user space]
──────────────────────── ioctl/mmap on /dev/nvidia* ────────────[boundary]
nvidia.ko (KMD: page tables, channels, interrupts, recovery)    [kernel]
GSP firmware (management offload)  +  silicon                    [device]

Init = ioctl flurry + ring/doorbell mmaps. Steady state = zero syscalls: write command to mapped ring, tap doorbell, wait on fence. A container is a process with unshared namespaces + cgroup limits; a GPU enters it as device nodes + injected host libs. There is no GPU cgroup controller — that sentence generates Phase 06.

1. Commands that teach

strace -f -e openat,ioctl,mmap python -c "import torch; torch.zeros(1).cuda()"
ls -l /dev/nvidia*                      # major 195, minors = GPU index
dmesg -T | grep -i -e nvidia -e xid     # the kernel's side of the story
lsof /dev/nvidia0                       # who's holding the GPU
cat /proc/$(pgrep -f myapp)/maps | grep nvidia   # the mapped rings, visible
nvidia-smi -q | head -30                # driver version = max CUDA supported

2. The stack's division of labor

LayerOwnsUpdates withBreaks as
KMD nvidia.koGPU page tables, channels, IRQs, Xid recoverynode image (reboot)module won't load (DKMS/secure-boot), Xids
UMD libcuda.sorings, contexts, PTX JIT, APIdriver package (must match KMD)"driver insufficient for runtime"
toolkit/runtimeinjecting nodes+libs into containershost packagecontainer sees no GPU
app's CUDA runtimeconvenience APIcontainer imageversion skew vs driver

Rule that prevents half of all incidents: the container never ships libcuda.so; the host injects it. Images pinning their own = guaranteed eventual skew.

3. Container reality check

  • Namespaces = visibility (mount/PID/net/UTS/user). cgroups = resources (cpu/memory/io + devices allow-list). Capabilities/seccomp = privilege. A "container" is just a process wearing all three.
  • GPU "limits" in k8s are whole-device allocation, not enforcement. Fractions need MPS (cooperative), MIG (hardware, real), or time-slicing (none) — Phase 06.
  • CUDA_VISIBLE_DEVICES is UX, the devices cgroup is policy, MIG is enforcement. Repeat before promising tenant isolation to a customer.

4. Failure signatures → causes (the 2 a.m. table)

SymptomLikely cause
driver insufficient for runtime versiontoolkit newer than driver — upgrade driver or use fwd-compat package
works on node A, not node Bfleet driver skew — inventory + pin in IaC
module won't load after kernel updateDKMS rebuild failed or unsigned module + secure boot
Xid 79 in dmesgGPU fell off the bus — hardware/power/PCIe; drain the node
container: nodes present, init failsinjected lib missing/mismatched — check ldconfig -p vs host
can't rmmod nvidiasomething holds the fd — lsof /dev/nvidia*
pinned-memory alloc fails at scaleunpageable RAM exhausted — audit cudaMallocHost users

5. War stories

  • The image that shipped libcuda: passed CI (lucky match), died in prod region with older drivers. Policy since: images are scanned for driver libs; injection only.
  • The quiet strace: engineer "proved" CUDA was hung because strace showed nothing — it was decoding correctly in the mmap'd ring the whole time. Steady state is silent by design.
  • The OOM that wasn't: container kept dying at 100 MB despite "no limit" — inherited memory.max from a parent cgroup nobody remembered. cat /proc/self/cgroup first, always.
  • The 40-minute driver upgrade that took a quarter: nobody had the image-pinned rollback path; one bad canary turned into fleet archaeology. WARMUP Q6 is the plan that avoids this.

6. Exit bar

You've written a driver (small, but real), built a container from syscalls, injected your own device into your own container, and can run the debugging ladder cold. The stack is glass. Phase 05 climbs back up one level: the runtime systems (allocators, stream schedulers) that live on top of this driver and under every inference engine.