🛸 Hitchhiker's Guide — Phase 05: Runtime Systems
Read this if: you call
torch.cuda.empty_cache()superstitiously, you've never asked whynvidia-smiand your framework disagree, or "stream-ordered deallocation" sounds like jargon. Field notes; derivations in the WARMUP.
0. The 30-second mental model
The runtime layer = pools + DAG. Pools because the driver's alloc/free are slow, synchronizing, and fragmentation is permanent (no paging, no compaction in HBM). DAG because the GPU is async: streams are FIFOs, events are edges, and everything — including freeing memory — must be positioned in that DAG, not in host wall-clock time. Every framework (PyTorch, vLLM, TensorRT) is a specific set of answers to those two problems.
1. The allocator card
- Design: size-binned free lists; round sizes (512B+); split big cached blocks; never return to driver in steady state; OOM → flush cache → retry.
- reserved (nvidia-smi, driver's view) vs allocated (framework's view): the gap is cache, not a leak. Watch largest-free-block for fragmentation.
- Fragmentation defenses (no compaction possible): size bucketing, pool segregation (weights/activations/KV), fixed-block paging for the dominant consumer (= PagedAttention), and — the confession — scheduled restarts.
cudaMallocAsync= this design, inside the driver, stream-ordered.expandable_segments= VMM tricks to grow reservations.
2. The stream card
- Same stream = ordered. Different streams = concurrent until an event edge says otherwise. Legacy default stream = a global serializer; ban it.
- Review method: draw the intended DAG; extra edges = lost overlap, missing edges = race. Most stream bugs are visible in 5 minutes this way.
- Free is an event: host-time free + async GPU = use-after-free that reproduces once a week. Stream-ordered free (or stream-tagged pools) makes the class impossible, not unlikely.
- CUDA Graphs: capture the decode step, replay in one call; 10–30% at small batch; frozen shapes/pointers; no allocs/syncs inside capture.
3. Numbers
| Thing | Number |
|---|---|
| pool hit | ~100 ns |
| cudaMalloc | ~10–50 µs (driver round trip) |
| cudaFree | device sync — unbounded |
| kernel launch (CPU cost) | ~5–10 µs — why Graphs exist |
| decode step kernel count | hundreds — × 8 µs = CPU-bound issuance |
| pinned alloc | ms-scale — pool these too |
| "OOM at X% utilization" | fragmentation if X < ~70 |
4. Incident patterns
- Day-3 OOM at 60%: checkerboard fragmentation. Look at largest-free-block trend; bucket sizes; segregate pools. Restarts are the tourniquet, not the fix.
- Throughput sag after a "harmless" logging change: someone synced (D2H of a metric) inside the decode loop → killed overlap. nsys timeline shows the gap.
- Garbage tokens, unreproducible: cross-stream use-after-free. Audit every free for stream order; poison freed blocks in a debug build.
- "empty_cache() fixed it": it didn't — it papered over fragmentation by returning blocks to the driver (with syncs). Treat as a smell, not a tool.
- Graph capture failing after a refactor: someone added an alloc/sync in the inner loop. The capture failure is useful — it's a cleanliness linter.
5. What changes at head-of-engineering level
You stop reviewing allocator PRs and start writing the runtime contract: no driver alloc/free in request paths (CI-enforced), stream-ordered lifetime only, segregated pools with reserved/allocated/largest-block metrics exported, no device-wide syncs in engine code, inner loops capture-clean. Each rule maps to an incident class; that mapping is how you defend the contract to a team that finds it pedantic — until day 3.
Next: Phase 06 — with memory and execution under control inside one process, who gets the GPU between processes, pods, and tenants?