Lab 03 — MicroVM Lifecycle State Machine

Difficulty: 4/5 | Runs locally: yes

Pairs with the Phase 06 WARMUP Chapter 11 (KVM, Firecracker, microVMs) and the HITCHHIKERS-GUIDE ("Firecracker Operator Path").

Why This Lab Exists (Purpose & Goal)

Starting an isolated workload is easy; tearing it down completely is where multi-tenant platforms leak. A microVM (Firecracker-style) allocates a process, a cgroup, a network namespace, a TAP device, an API socket, a jail, scratch storage, and a result channel — and if any of those survives the job, you have a resource leak, a path back into the host, or cross-tenant residue. The goal of this lab is to model the microVM lifecycle as a state machine that guarantees complete cleanup before a job is considered destroyed.

The Concept, In the Weeds

The invariant that's deceptively hard to satisfy: cancellation must destroy the full process tree and release every resource. "Kill the process I started" is not "clean up everything it did" — a workload can fork descendants, create mounts, and leave network devices behind. The state machine enforces two properties:

  • Idempotent transitions — a cancellation can be retried safely; a crash mid-teardown can be reconciled. Real platforms have a reconciler that periodically scans for leaked resources and removes or quarantines them, which only works if transitions are idempotent.
  • Complete cleanup before DESTROYED — the job cannot reach the terminal state until the process tree, cgroup, network namespace, TAP, API socket, jail, scratch, and result channel are all released. Each of those is a specific leak: a surviving TAP device is a network path; a surviving scratch volume is cross-tenant data residue; a surviving API socket is a control channel.

This is the lifecycle dimension of the composed-isolation proof (the sixth row of the Phase 06 matrix): isolation is not just about what a running job can do, but about guaranteeing nothing persists after it.

Why This Matters for Protecting the Company

In a multi-tenant platform, a cleanup leak is a cross-tenant vulnerability that needs no exploit — a new job lands on residue (memory, scratch, a network device) from a previous tenant's job and inherits its data or reach. Snapshots compound this: they contain memory and disk state (tokens, plaintext, RNG state) that must be bound to tenant/image/config identity and never reused across tenants. A rigorous lifecycle state machine with guaranteed teardown is how serverless and CI platforms keep tenants isolated over time, not just at a single instant. This is the unglamorous reliability work that prevents the quiet, exploit-free breaches.

Build It

Implement the state machine: model Firecracker-style allocation, configuration, start, cancellation, collection, and cleanup. Enforce idempotent transitions and require cleanup of the process tree, cgroup, network namespace, TAP, API socket, jail, scratch, and result channel before reaching DESTROYED.

LAB_MODULE=solution pytest -q
# On a KVM host, bind each transition to the operator commands in HITCHHIKERS-GUIDE.md.

Validation — What You Should Be Able to Do Now

  • Enumerate every resource a microVM allocates and explain the specific leak each represents if it survives.
  • Explain why cancellation must destroy the full process tree, and why idempotent transitions enable crash-safe reconciliation.
  • Reason about snapshot risk (memory/disk residue) and why snapshots must be bound to tenant/image/ config identity.

The Broader Perspective

This lab teaches that isolation has a time dimension: a system can be perfectly isolated at any instant and still leak tenant data across jobs if teardown is incomplete. That insight — clean up completely, make teardown idempotent and reconcilable, treat residue as a cross-tenant risk — is the same discipline behind cloud resource teardown and secret rotation (Phase 07), behind incident recovery that proves old access fails (Phase 09), and behind the engagement teardown of Phase 00. Building things up is half the job; tearing them down provably is the other half, and it is where the subtle breaches hide.

Interview Angle

  • "A compromised build job — how do you contain it, and how do you ensure it leaves nothing behind?" — Ephemeral isolated runner (microVM/gVisor), no host credentials/socket, egress allowlist, per-job short-lived creds; and a lifecycle controller that, on completion or cancellation, destroys the full process tree and removes the cgroup, netns, TAP, socket, jail, and scratch — idempotently, with a reconciler scanning for leaks.

Extension (Stretch)

On a KVM host, bind each transition to real jailer/Firecracker operations and inject failures after each step; prove a restarted controller reconciles leaked resources (HITCHHIKERS "Firecracker Build and Failure Lab").

References

  • Phase 06 WARMUP Chapter 11; Firecracker design + jailer documentation; KVM API docs.