Lab 02 — Container From Scratch + GPU Passthrough Anatomy

Phase: 04 — Linux & Containers | Difficulty: ⭐⭐⭐⭐☆ | Time: 4–6 hours Language: C + shell | Hardware: Linux VM with root (same VM as Lab 01)

Concept primer: ../WARMUP.md Ch. 7–9.

Run

make                         # builds mininer + hog
sudo ./mkrootfs.sh           # builds ./rootfs from busybox (one download)
sudo ./mininer run /bin/sh   # you are now PID 1 in an isolated container
# inside: ps  /  hostname  /  ls /   — observe isolation
exit

sudo ./mininer run-limited 100 /bin/sh    # same, with memory.max=100M
# inside: /hog 200      — allocates 200 MB -> OOM-killed at ~100 MB

0. The mission

Build the smallest honest container runtime — then put a "GPU" inside it by hand, so that nvidia-container-toolkit becomes a thing you could have written.

mininer.c (~200 lines) does, in order:

1. clone(CLONE_NEWUTS | CLONE_NEWPID | CLONE_NEWNS | SIGCHLD) — new hostname, PID, and mount namespaces for the child.
2. In the child: sethostname("minibox"); make mounts private (MS_REC|MS_PRIVATE — so our mounts don't leak to the host); chroot("./rootfs") + chdir("/") (honest note: real runtimes use pivot_root, which unmounts the old root rather than hiding it — the extension asks you to upgrade); mount a fresh /proc (so ps sees only the namespace).
3. (run-limited) create /sys/fs/cgroup/mininer, write memory.max, add the child PID to cgroup.procs.
4. execv the requested command.

That's a container: namespaces (visibility) + cgroups (resources) + a rootfs (filesystem). Docker adds images, networking, and an API — not different physics.

1. The observation checklist (do these inside the container)

On the host, in another terminal: ps aux | grep sh still shows the container shell (as a normal PID) — namespaces virtualize the container's view, not the host's. Also cat /sys/fs/cgroup/mininer/memory.current while the hog runs.

2. The GPU passthrough synthesis (the point of the whole phase)

With Lab 01's module loaded on the host:

# 1. Device node into the container's rootfs (what the toolkit's hook does):
sudo mknod ./rootfs/dev/fakegpu c $(grep fakegpu /proc/misc | awk '{print $1}') \
     $(grep fakegpu /proc/misc | awk '{print $1}' > /dev/null; \
       ls -l /dev/fakegpu | awk '{print $6}' | tr -d ',')
#    (or simply: sudo cp -a /dev/fakegpu ./rootfs/dev/  — preserves maj:min)
sudo chmod 666 ./rootfs/dev/fakegpu

# 2. The "driver library injection" analog: copy the client in
sudo cp ../lab-01-fakegpu-driver/client ./rootfs/bin/gpuclient

# 3. Run it inside:
sudo ./mininer run /bin/gpuclient        # ALL CHECKS PASSED — inside a container

Now write GPU-IN-CONTAINER.md (one page): map what you just did to the three requirements of WARMUP Ch. 9 (device node + cgroup allowance + injected userspace libs; note that our mininer doesn't set up a device cgroup — add it as the extension and show /dev/fakegpu works while a non-allow-listed node fails). Close with the debugging ladder. This document is the deliverable — it's the team-training artifact the phase README promises.

3. Extensions

1. pivot_root upgrade: replace chroot with pivot_root + umount2(old, MNT_DETACH); explain in a comment why chroot alone is escapable (classic chroot jailbreak: keep an fd to the outside, fchdir).
2. Devices cgroup: attach a cgroup-v2 eBPF device filter (or use v1 devices.allow if available) permitting only /dev/fakegpu and /dev/null; demonstrate mknod + open of anything else failing.
3. Network namespace: CLONE_NEWNET + a veth pair (ip link add ... netns) — ping from container to host.
4. User namespace: CLONE_NEWUSER with uid_map so the container's root is unprivileged outside — then re-run the GPU test and document what breaks (device access vs UID mapping is exactly the rootless-container GPU pain point in real deployments).
5. Compare with runc: strace -f -e unshare,clone,mount,pivot_root runc run on a real container; diff against strace -f ./mininer run.

4. Common pitfalls

1. Forgetting MS_PRIVATE — your container's mounts propagate to the host (systemd makes / shared by default); the host's /proc gets shadowed. Symptom: weird host ps after exit. The MS_REC|MS_PRIVATE remount is non-negotiable.
2. Mounting /proc before the PID namespace exists — ps shows host processes; the fresh /proc must be mounted by the child, after clone.
3. cgroup v1 vs v2 confusion — this lab assumes v2 (/sys/fs/cgroup with memory.max). Check mount | grep cgroup2; hybrid hosts need the v2 path.
4. Running on your laptop instead of the VM — nothing here should damage a host, but you're root, juggling mounts; the VM habit from Lab 01 stands.

5. What this lab proves about you

You can state — and demonstrate, with code you wrote — exactly what a container is, what GPU injection consists of, and where the isolation boundaries actually sit. When a customer asks "how is tenant A isolated from tenant B on shared nodes," your answer comes from mechanism, not marketing. Phase 06 builds the scheduling layer on top of this; Phase 11 hardens it.