Design 02 - Untrusted Code Execution Platform

Prompt

Design a service that runs customer-submitted code for up to 10 minutes and returns results. Assume hostile code, multi-tenancy, bursty demand, and no need for arbitrary internet access.

Architecture

API authenticates user, validates job policy, stores immutable input, and creates signed job intent.
Scheduler issues a short-lived capability to a worker pool.
Each job receives a fresh microVM or strong sandbox, read-only image, bounded scratch storage, no host mounts/devices/runtime socket, and default-deny egress.
CPU, memory, PIDs, disk, runtime, output, and request budgets are enforced outside the guest.
Secrets are scoped, short-lived, explicitly requested, never injected by default, and redacted.
Results are scanned/size-limited, stored separately, and linked to tenant/job identity.
Workers are recycled; control plane and image pipeline are isolated and signed.

Guest escape, side channel, scheduler confused deputy, metadata/control-plane access, fork or disk exhaustion, malicious output, poisoned base image, secret leakage, cross-job storage reuse, and orphaned execution after cancellation.

Validation

Negative fixtures for syscalls, mounts, devices, network, metadata, and cross-job access.
Resource exhaustion and cancellation/cleanup tests.
Image-signature and rollback tests.
Host and control-plane audit events joined by job ID.
Compromised-worker tabletop and fleet reimage exercise.
Performance benchmark by workload class.

Key Decisions

Choose process/container isolation only for trusted workloads. For mutually untrusted tenants, prefer stronger isolation such as microVMs, potentially combined with an inner sandbox. No design can honestly claim perfect side-channel elimination.

Operations

Stage rollouts, maintain warm capacity without reusing tenant state, patch hosts rapidly, rotate worker identity, monitor denied actions and cleanup failures, and keep emergency global/tenant kill switches under controlled authorization.

Security Engineer