Hitchhiker's Guide — Hardening Containers, gVisor, and Firecracker

The operational companion to the Phase 06 WARMUP. The WARMUP derives isolation from first principles (namespaces, capabilities, seccomp, cgroups, OCI/runc, gVisor, KVM/Firecracker, control plane); this guide is the operator's manual — the decision model, the hardened manifests, the gVisor and Firecracker operator paths, the under-the-hood experiment sequence, the incident playbook, and the acceptance packet. The runtime you build is educational and must carry a prominent not-production-ready warning. Run benign adversarial fixtures only.

Table of Contents

• 30-Second Decision Model
• Hardened Container Checklist
• gVisor Operator Path
• Firecracker Operator Path
• Comparing the Boundaries
• What Not to Claim
• Under-the-Hood Experiment Sequence
• gVisor Comparison Lab
• Firecracker Build and Failure Lab
• Control-Plane Threats
• Incident Playbook
• Isolation Acceptance Packet
• References

30-Second Decision Model

Isolation strength is only one axis. Also weigh compatibility, startup latency, throughput, image operations, observability, patching, fleet recovery, and control-plane complexity.

Hardened Container Checklist

securityContext:
  runAsNonRoot: true
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop: ["ALL"]
  seccompProfile:
    type: RuntimeDefault

Also require:

• user namespace/rootless where supported;
• no host PID/IPC/network namespaces;
• no privileged mode or added broad capabilities;
• no hostPath, runtime socket, SSH agent, kubeconfig, or cloud credential mounts;
• explicit requests/limits, PID and ephemeral-storage controls;
• default-deny ingress/egress;
• signed image and immutable digest;
• admission policy that tests all of the above (Phase 07).

Attack fixtures

• process attempts to read /host/etc/shadow: file absent/denied;
• process attempts to connect to the metadata IP: denied and logged;
• process forks beyond limit: cgroup denial, host remains healthy;
• process writes the root filesystem: denied; scratch write succeeds within quota;
• process calls a prohibited syscall: visible seccomp denial;
• process exits/cancels: no descendants or mounts remain.

gVisor Operator Path

Install runsc only from official signed/reproducible distribution channels. Register it as an alternate runtime, then select it explicitly for the sandboxed workload or RuntimeClass.

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc
---
spec:
  runtimeClassName: gvisor      # on the workload

Hardening and validation:

1. Keep the same pod security, capability, filesystem, identity, resource, and network controls — gVisor is additional, not a replacement (Phase 06 WARMUP, Chapter 10).
2. Pin and inventory runsc; patch separately from containerd/Kubernetes.
3. Test application compatibility: /proc, networking, signals, file locks, performance counters, tracing, unusual syscalls.
4. Benchmark representative workloads, not synthetic syscall loops alone.
5. Verify runtime selection from node/runtime telemetry — a requested RuntimeClass is not proof it ran under runsc.
6. Restrict runtime debug endpoints and logs; they may contain workload metadata.
7. Test runtime failure, node drain, checkpoint assumptions, and forensic collection.

Firecracker Operator Path

Do this only on a dedicated Linux host with KVM support and an isolated test network.

Artifacts

• Firecracker binary and matching jailer; a minimal guest kernel; a read-only base rootfs; a per-job overlay/scratch; machine configuration; a TAP device or no-network mode; an API socket path inside a protected launcher directory.

Lifecycle

validate signed job intent
  → allocate job UID, cgroup, namespace, scratch, TAP
  → launch jailer/firecracker
  → configure kernel/rootfs/network through the protected API socket
  → start instance
  → enforce external timeout/output budget
  → collect result through a narrow channel
  → kill VMM/process tree
  → remove TAP, namespace, socket, jail, cgroup, scratch
  → emit cleanup-complete event

Host hardening

• dedicated worker pool; no developer credentials;
• minimal patched kernel and packages;
• KVM device permissions for the launcher identity only;
• IOMMU/device posture reviewed if passthrough is used;
• the launcher cannot be invoked directly by a tenant;
• API socket directory mode/ownership checked before and after launch;
• seccomp/jailer enabled;
• host firewall + namespace rules deny metadata, management, storage control planes, and other tenants' networks;
• immutable host with a rapid reimage path.

Guest hardening

• minimal kernel config and modules; no SSH daemon or package-manager credentials;
• read-only root, bounded scratch, no unnecessary services; a dedicated workload UID;
• init reaps children and handles shutdown;
• the application protocol authenticates the host-side controller if a channel exists;
• logs/results size-limited and treated as untrusted.

Snapshot risk

Snapshots may contain tokens, plaintext data, RNG state, and workload memory. Bind snapshot reuse to tenant, image digest, kernel, configuration, secret epoch, and policy version. Prefer no secrets at snapshot creation. Encrypt, expire, authorize, and audit snapshot access.

Comparing the Boundaries

What Not to Claim

• "Firecracker prevents all VM escapes."
• "gVisor means pod security no longer matters."
• "Rootless means harmless."
• "Seccomp prevents file access." (it filters syscall numbers, not paths)
• "A microVM solves control-plane compromise."
• "No network" — if DNS, proxy, metadata, shared storage, logging, or result channels still exist.

Under-the-Hood Experiment Sequence

1. Print UID/GID, capabilities, namespaces, mounts, cgroup, descriptors, routes, and seccomp mode.
2. Add one primitive at a time and record what changed.
3. Show that a mount namespace hides a path while an inherited descriptor can still read it (Phase 06 WARMUP, Chapter 3 — the leak principle).
4. Apply no_new_privs and test a purpose-built privilege-gain fixture.
5. Trace syscalls across startup, normal, error, DNS, threading, and shutdown paths.
6. Enforce seccomp and test required plus prohibited calls.
7. Stress memory, CPU, PIDs, output, disk, and wall time.
8. Cancel midway and scan for descendants, mounts, namespaces, TAPs, sockets, cgroups, and scratch.

Explain which kernel check denied each action and which state/log proves it.

gVisor Comparison Lab

Run the same deterministic workload under runc and runsc. Compare compatibility, output, host syscall surface, startup, throughput, memory, signals, /proc, filesystem behavior, and telemetry. Confirm runtime selection from runtime/node evidence. The lesson is attack-surface movement, not elimination (Phase 06 WARMUP, Chapter 10).

Firecracker Build and Failure Lab

On a dedicated KVM host, verify binaries, build a minimal kernel/rootfs, and create per-job UID, jail, cgroup, network namespace, TAP, scratch, and protected API socket. Prove no-network first.

Inject failure after allocation, VMM launch, configuration, start, result collection, timeout, and teardown. Restart the controller and prove reconciliation removes or quarantines leaked resources (invariant 4 — complete cleanup).

Control-Plane Threats

Test: malicious image metadata, path traversal in names, duplicate job IDs, tenant confusion, stale capabilities, replay, cancellation/launch races, worker impersonation, forged completion, oversized logs, and cleanup failure. Construct host paths and arguments from typed values, never tenant-built command strings (Phase 06 WARMUP, Chapter 12).

Isolation Acceptance Packet

Include: threat model, authority inventory, compiled specs, runtime/image provenance, negative fixtures, a compatibility/performance report, lifecycle-failure and cleanup proof, a reimage/rotation playbook, the trusted computing base, and an honest residual-escape-risk statement.

References

• man pages: namespaces(7), user_namespaces(7), capabilities(7), seccomp(2), cgroups(7).
• OCI Image/Runtime specs; runc docs; NCC Group "Understanding and Hardening Linux Containers".
• gVisor architecture docs; Firecracker design + NSDI 2020 paper; the jailer documentation.
• NIST SP 800-190 (container security); CIS Docker/Kubernetes Benchmarks; KVM API docs.
• Container-escape CVE writeups (e.g. CVE-2019-5736); Kata Containers architecture (comparison).