« Phase 09 README · Warmup · Hitchhiker's · Deep Dive · Principal Deep Dive · Core Contributor

Phase 09 — Staff Engineer Notes: Secure Execution: Sandboxing & Capabilities

The person who uses a sandbox reaches for --network=none and calls it done. The person trusted to own agent execution is the one who can say, out loud and under pressure, whose code is this, what is the blast radius if it is hostile, and which point on the isolation curve does that buy — and then defend the answer against both "just use containers, ship it" and "Firecracker everything." This phase sits in a rare seam: the AI people build agents that "work" and wave at the execution; the security people know containers cold but do not model prompt injection as an authority problem. The engineer who holds both — "the model is an untrusted input source, the tool call is an attacker-controllable syscall, so the sandbox is where least privilege lives" — is who gets handed the make-it-production problem. That is the signal these notes are about.

The decisions a staff engineer actually owns here

Nobody senior owns "write the traversal check." They own the choices that are expensive to reverse:

  • Isolation strength per workload. Not a global default — a per-trust-tier decision. This is the call reviewers listen for, below.
  • The egress policy as the primary control. Deciding, before anything else, that egress is default-deny and the allow-list is the shortest it can be — and that the cloud metadata endpoint is on the permanent threat list. Most teams treat egress as an afterthought; the incidents say it is the load-bearing wall.
  • The trust-boundary architecture. Where the sandbox sits relative to argument validation (Phase 02), injection detection (Phase 10), and per-tenant authorization (Phase 13) — and the explicit stance that the sandbox is the deterministic layer that holds when the probabilistic ones fail. Owning this means designing the sandbox as if the classifier is already wrong.
  • Ephemerality and recycling policy. Fresh box per hostile task, recycled from a clean template, never reused across tenants. This is a lifecycle decision with real cost implications, and it is the difference between "an escape is bounded to one task" and "an escape leaves a foothold."

The decision framework: container vs gVisor vs Firecracker

The junior move is dogma in either direction. The staff move is a two-question framework, answered out loud:

Question 1 — whose code is this? Your own team's helper scripts, same tenant, semi-trusted: a hardened container is defensible (--cap-drop=ALL, seccomp, read-only rootfs, --network=none or an egress allow-list, cgroup limits). Arbitrary code from arbitrary users: you are past what a shared kernel should hold.

Question 2 — what is the blast radius if it is hostile? Single-tenant blast radius (your own workload): container is fine. Cross-tenant blast radius (other customers' data on the same host): microVM, because you are one kernel CVE from a cross-tenant breach and ~125 ms of cold start is nothing against that. Need OCI ergonomics but stronger-than-container isolation for medium-trust code: gVisor is the middle, at the cost of a per-syscall userspace tax.

The framework in one line: match isolation strength to the trust boundary between whoever's code runs on the shared substrate. Then add ephemerality on top regardless — fresh, one per task, thrown away — so an escape has nothing to persist into. Being able to name the cost of each choice — boot latency, syscall overhead, kernel attack surface, per-box memory — is what turns a recited ranking into a judgment. Interviewers are buying the tradeoff reasoning, not the ranking.

Code-review red flags (auto-comment on sight)

These are the tells that the author has not internalized containment:

  1. A raw path.startswith(root) confinement check. Two bugs in one line: no canonicalization (/work/../etc/passwd walks out) and no boundary (/workshop passes a /work prefix). Canonicalize with normpath, then boundary-match against root + "/".
  2. A deny-list anywhere a security decision is made — "block /etc, block evil.com." It fails open: the host you forgot is allowed. Every grant is an allow-list, so the unknown case fails closed. This is the single most important line-level judgment in the phase.
  3. Act-then-check / act-then-rollback. self.vfs.write(...) before the validation, or a try/rollback around a real effect. The window is an observable partial state; the invariant is no side effect on denial, and a rollback is two side effects, not zero.
  4. eval/exec with a stripped __builtins__, called a sandbox. The escape is a three-line object-graph walk (().__class__.__bases__[0].__subclasses__()) back to os. Isolation must come from a layer the code cannot reason about — kernel or hypervisor — never your own interpreter.
  5. Reading time.time() for a deadline. Non-deterministic, untestable, and in a replayable system incorrect. Inject the clock; time is data you control.
  6. Namespaces without cgroups. A perfect filesystem/network jail with no pids/memory/cpu cap is still a node-killer via fork-bomb or memory exhaustion. The wall and the meter are both required.
  7. A mutable capability object. If authority can widen mid-run, a hostile op sequence will find the line that widens it. The grant must be frozen; authority only ever shrinks.
  8. "The system prompt tells it not to do that." A prompt is a suggestion to an untrusted stochastic generator that follows instructions in its input. It is not a boundary and never appears in the containment argument.

War stories, framed as judgment

  • SSRF to the metadata endpoint. An agent with an unrestricted fetch read a poisoned page telling it to fetch http://169.254.169.254/latest/meta-data/…, lifted the host's IAM credentials, and (would have) exfiltrated them — a sandbox escape that became a cloud-account compromise. The judgment: egress is the primary control, default-deny, and link-local plus internal ranges are denied even when a hostname slips through.
  • The fork-bomb with no pids limit. A coding agent ran generated tests in a container with namespaces but no cgroup pids cap; one while True: os.fork() took the node down and its neighbors with it. The judgment: namespaces are a wall, cgroups are a meter, and you always need both.
  • The prefix path check. A path.startswith("/work") confinement let /work/../../etc/passwd and /workshop/secrets both through. The judgment: never trust a raw path string; canonicalize then boundary-check.
  • The eval "sandbox." Shipped as "safe," escaped in three lines. The judgment: in-process is not isolation, full stop.

Each story is the same reflex applied to a different surface: assume the honored action is hostile, and bound what hostility can accomplish.

The interview signal

An interviewer asking "an agent runs model-generated Python — how do you run it safely?" is listening for a specific arc, not a keyword. The weak answer is "put it in a Docker container." The staff answer:

  1. Names the four controls without prompting — least privilege, filesystem isolation, egress, resource limits — and that all four are allow-lists that fail closed.
  2. Defines a container correctly and unhesitatingly: a normal Linux process the kernel isolates (namespaces), meters (cgroups), and muzzles (seccomp + dropped caps + read-only rootfs), sharing the host kernel — which is why a kernel CVE is a container escape but not a microVM escape.
  3. Makes the isolation call as a tradeoff against blast radius and tenancy, and prices it (~125 ms cold start, syscall overhead, kernel surface).
  4. Names egress as the anti-exfiltration control and reaches for the metadata endpoint unprompted.
  5. Frames the sandbox as defense-in-depth — the deterministic layer that holds when injection detection (probabilistic) misses — rather than as the only control.
  6. Disambiguates the two "capability" meanings cleanly: capability-based security (unforgeable authority tokens, POLA, the design behind the grant) versus Linux capabilities (the ~40 CAP_* bits you --cap-drop to harden a container). Same word, different layer — a small but reliable seniority tell.

The meta-signal underneath all six: the candidate treats containment as architecture, not configuration. They can whiteboard the boundary from a Python if up to Firecracker and explain why the shape is identical at every level.

Closing takeaways

  1. A sandbox bounds consequences; it does not make the act safe. It is insurance, not permission — you still minimize what you run, still validate, still authorize.
  2. Allow-lists beat deny-lists everywhere — paths, hosts, tools, syscalls — because allow-lists fail closed and the thing you forgot is the thing that breaches you.
  3. Match isolation to blast radius, and prove you can price the tradeoff. "Firecracker good" is not an answer; "here is when, here is why, here is what it costs" is.
  4. Egress is the load-bearing wall. The incident is not "read a file," it is "read a file and posted it to the internet." Design default-deny egress first.
  5. A prompt is never a boundary, and eval in your own process is never a sandbox. Isolation must come from a layer the untrusted code cannot reason about.
  6. The rarest, most valuable skill in this track is being the person who can contain an agent — the one holding both the kernel and the injection model at once. The market has noticed there are only a few hundred of them.