« Phase 09 README · Warmup · Hitchhiker's · Deep Dive · Core Contributor · Staff Notes

Phase 09 — Principal Deep Dive: Secure Execution: Sandboxing & Capabilities

The Sandbox in the lab is one box holding one capability grant. A production agent-execution platform — Docker's agentic runtime, Vercel Sandbox, E2B — is a fleet of those boxes, born and killed thousands of times a minute, each one a Firecracker microVM or a gVisor pod, and the interesting engineering is entirely in the layer around the box: how fast you can mint one, how you keep ten thousand tenants' boxes from bleeding into each other, what happens when one of them tries to take the host down, and how much all of it costs. This document is that layer.

Where the box sits in the platform

The lab's Sandbox.run(op) is the inner control loop. In production the outer loop is: an agent proposes an action → a scheduler picks or provisions a sandbox → the action executes inside it → the observation returns → the sandbox is either reused or destroyed. The capability grant that the lab freezes into a Capabilities object is, in production, an OCI security config materialized per task: --cap-drop=ALL --cap-add=…, --read-only with a writable tmpfs scratch mount, --network=none plus an egress proxy, --pids-limit, --memory, a seccomp profile. The fs_read/fs_write split is a mount namespace; net_hosts is a network namespace plus a Kubernetes NetworkPolicy; the three budgets are cgroup limits plus an execution deadline. The architectural insight the lab makes literal is that the grant is per-task data, not per-service config — every agent task gets its own minimal grant, computed from what that task needs, and thrown away with the sandbox.

The isolation spectrum is a capacity/latency/cost decision

The spectrum — in-process → OS process → container → gVisor → Firecracker microVM → full VM — is usually taught as a security ranking. At the principal level it is a cost curve, and you are choosing a point on it under a latency and density budget:

  • Container (namespaces + cgroups + seccomp, shared host kernel): cold start ~10–100 ms, near-zero memory overhead, thousands per host. Escape surface: the entire ~350-syscall Linux kernel. One kernel CVE is a container escape.
  • gVisor (runsc, userspace Sentry kernel): cold start ~100–150 ms, per-sandbox memory for the Sentry process, syscall-heavy workloads pay a userspace round-trip per syscall. Escape surface: the much smaller, memory-safe Sentry reimplementation.
  • Firecracker microVM (per-workload guest kernel behind a minimal Rust VMM): cold start ~125 ms, a few MB overhead, hundreds to low-thousands per host. Escape surface: the tiny VMM. A guest-kernel bug does not reach the host kernel.

The counterintuitive fact is that Firecracker's ~125 ms cold start is comparable to a container's, not the seconds a full VM takes — that is the entire reason microVMs are viable for per-request agent code. The design rationale that "looks wrong but is intentional": you accept a slightly heavier boot and a few MB per box to buy a per-tenant kernel, and for hostile multi-tenant code that trade is always worth it. The number to hold: at ~125 ms cold start, a naive "fresh microVM per op" design adds 125 ms of tail latency to every single tool call, which is why nobody does that — see pooling.

Pooling, warm pools, and the cold-start math

If every agent action paid a full cold start, an agent that makes 20 tool calls in a task would eat 2.5 seconds of pure boot latency. So the platform layer runs warm pools: pre-booted sandboxes kept idle, checked out per task, and returned or destroyed. Firecracker's snapshotting makes this cheaper still — boot one microVM to a known-good state, snapshot the memory and device state, and restore from the snapshot in single-digit milliseconds instead of re-booting a guest kernel. The capacity math becomes a queueing problem: pool size N, checkout rate λ, mean hold time T; by Little's Law the pool needs at least λ·T warm boxes to avoid falling back to cold boots under load, plus headroom for burst. Undersize the pool and tail latency spikes to the cold-start number; oversize it and you pay for idle microVMs. This is the capacity decision the lab's max_ops budget gestures at — "how many operations before we recycle" is directly "how long does a box stay checked out," which sets T.

The tension the lab's ephemerality principle exposes: warm pools want to reuse boxes (amortize boot cost), but security wants each hostile task in a fresh box (no persistence for a foothold). The production resolution is to pool the boot, not the tenant: restore a clean snapshot per task so each task gets a pristine environment cheaply, and never reuse a box across tenants. Reuse the template, never the instance.

Failure modes and blast radius

The lab teaches four controls because each caps a different blast radius, and a principal reasons in blast radii:

  • Kernel-escape blast radius. The container's shared kernel means one CVE crosses all tenants on that host — the worst blast radius in the taxonomy. This is why "container or microVM?" is a tenancy question, not a preference. Same-tenant helper scripts: a container's blast radius is your own workload, acceptable. Arbitrary code from arbitrary users on a shared host: the blast radius is cross-tenant data, and you pay for the per-tenant kernel.
  • Denial-of-service blast radius. A filesystem/network jail with no cgroups still lets a workload exhaust CPU, RAM, or PIDs and take the node down, dragging its neighbors with it — a noisy-neighbor escape that is still an escape. The lab's max_ops/max_output_bytes/max_wall model the cgroup pids/memory/cpu caps that bound this. The pids cap specifically is the fork-bomb defense; without it while True: os.fork() is a node-killer.
  • Exfiltration blast radius. The headline incident is never "the agent read a file" — it is "the agent read a file and posted it to the internet." Egress control is the load-bearing wall here, and its blast radius when missing is your customer data on someone else's server. The SSRF variant — an injected instruction steering the agent to fetch 169.254.169.254/latest/meta-data/ and lift the host's IAM credentials — turns a sandbox escape into a cloud-account compromise, the largest blast radius of all. Default-deny egress with a host allow-list, checked before the packet leaves, is the only reliable containment.

The ordering the lab enforces — check before effect — is what keeps a blocked op from ever appearing in net.calls. At platform scale the same guarantee is a network namespace with no default route: the kernel drops the connection, the workload's socket never reaches the wire.

Cross-cutting concerns

Security as layers. The sandbox is one layer of defense-in-depth, and its job is to hold when the others fail. Argument validation (Phase 02) rejects a malformed tool call; injection detection (Phase 10) flags the poisoned document; per-tenant authorization (Phase 13) gates who may call what. All three are probabilistic or upstream; the sandbox is the deterministic layer that contains what they let through. A principal designs the sandbox assuming the classifier was wrong — because it will be — and never as the only control.

Cost. Isolation strength maps to unit economics. Containers pack thousands per host; microVMs pack hundreds-to-thousands but carry per-box overhead. For a code-execution product, the cost model is dominated by idle warm-pool boxes and boot amortization, not by the running work. Choosing gVisor over Firecracker can double density for medium-trust workloads; choosing Firecracker for hostile multi-tenant code is a cost you eat because the alternative is a breach.

Observability. The lab's Sandbox.log list — every SandboxResult in order — is the audit trail. In production this is the decision log a security team reads after an incident: every op, its verdict, its violation code. The machine-readable violation field (egress_denied, path_traversal, max_ops) is what you alert on — a spike in egress_denied from one tenant is an active exfiltration attempt, not noise. Design the denial reasons to be aggregatable, because the signal is in the distribution of denials, not the individual one.

Multi-tenancy. The whole spectrum exists because the shared kernel is a shared fate. The architectural rule: the isolation boundary must be at least as strong as the trust boundary between tenants. Two tenants sharing a host kernel are one kernel bug from sharing data; that is why hostile multi-tenant execution lives at the microVM end of the spectrum and nowhere else.

Design decisions that look wrong but are intentional

  • Output bytes are counted flowing back to the agent, not on writes. Counterintuitive until you ask what the budget protects: the context window and the caller's memory. Data the agent reads fills its context; data it writes to scratch does not. The lab meters the direction that causes the blowup, and meters writes separately.
  • Every op is charged against max_ops even when denied. A denied op is not free — probing ten thousand paths to brute-force one that slips through is work, and it must burn the budget or the budget is bypassable. Charging on attempt, not on success, is what makes the ceiling real.
  • Hard halt is absorbing; soft denial is not. A resource breaker that resets would let a loop retry past it; a capability denial that halted would let one typo kill a legitimate task. The two denial classes exist because the correct response to "over budget" is "stop the workload" and the correct response to "not allowed" is "refuse this op, continue." Conflating them is a real bug.
  • The clock is injected. Reading wall-time directly makes the deadline non-deterministic and, in a replayable system, incorrect — the same seam durable workflows (Phase 08) depend on. Time is data the platform controls, not ambient state the workload reads.

The synthesis: a code-execution platform is the four controls — least privilege, filesystem isolation, egress control, resource limits — implemented at a chosen point on the isolation/cost curve, wrapped in a provisioning layer that hides cold start behind warm pools and enforces ephemerality by recycling from a clean template per task. Build the box once and the platform is the fleet management around it.