System Design 04 — Multi-Tenant Isolation & Secure Boot

"Design isolation and trust for a multi-tenant, sovereign rack platform." Phases: 09 (network/boot/ tenancy), 06 (k8s tenancy), 04 (accelerator partitioning), 01 (planes/domains).

1. Drive the requirements

  • Tenancy: trusted internal tenants or untrusted external? Paid vs free tiers? → soft vs hard isolation.
  • Sovereignty: regulated/air-gapped (a Kingdom data center)? → provable trust (attestation), RKE2, on-prem keys.
  • Surface: network, compute, accelerators, storage, and the management plane.
  • Threat model: a malicious tenant, a compromised node, an exposed BMC.

2. Architecture (defense in depth)

 NETWORK:   VLAN-per-tenant (802.1Q) | mgmt VLAN = HARD boundary (control-plane only)
            host: netns/veth/bridge + ACLs ; QoS (HTB) reserved mgmt class (P09)
 COMPUTE:   namespaces + RBAC + ResourceQuota + NetworkPolicy + PodSecurity (P06)
 ACCEL:     no co-placement of competing tenants on one device; MIG-style partitioning (P04)
 TRUST:     secure boot (enforce) + measured boot (TPM PCRs) + REMOTE ATTESTATION (prove) (P09)
            -> a node earns tenant work/keys ONLY by attesting a known-good state
 OPS:       least-privilege control plane, mTLS, secrets mgmt, audit (P07/P12), RKE2 (P06)

3. The hard decisions (and the tradeoffs)

  • Soft vs hard isolation: namespaces+quotas+RBAC+NetworkPolicy (soft) for trusted tenants; separate nodes/clusters or strong sandboxing (hard) for untrusted. Tradeoff: utilization vs. isolation strength — never co-place competing/untrusted tenants on one physical accelerator.
  • Management plane as a hard boundary (P09): its own VLAN, control-plane-only, never tenant/ internet-facing. Non-negotiable — even a misconfigured ACL must not expose it (an exposed BMC = full compromise). Tradeoff: none worth taking; this is a hard invariant.
  • QoS with a reserved management class (P09): HTB guarantees per tenant + a reserved mgmt slice so a tenant DDoS can't make the rack unmanageable. Invariant: sum of guarantees ≤ link capacity.
  • Attestation-gated trust (P09): a node joins/gets keys/runs tenant work only after a valid PCR quote against known-good values. Tradeoff: operational complexity (golden measurements, key management) vs. provable trust — required for sovereign deploys.
  • Secure boot vs measured boot: enforce and record — secure boot stops untrusted code; measured boot + attestation prove the state even where enforcement can't reach (and enable sealing secrets to PCR state).

4. Failure modes & operations

  • Malicious tenant tries to reach another tenant/management → blocked by VLAN/netns + NetworkPolicy
    • the hard mgmt boundary; QoS prevents bandwidth starvation.
  • Compromised node (bad firmware/OS) → fails attestation → quarantined, never trusted with tenant work/keys; management isolation limits its reach.
  • Exposed BMC → prevented by the hard mgmt-VLAN boundary; audited; the classic catastrophe avoided.
  • Key compromise / revocation → dbx revokes a vulnerable bootloader (P09); rotate keys; re-attest.
  • Operations: per-tenant audit (P07/P12), incident response (P10), and a written threat model that shows each layer fails safe — what a sovereign customer's security review demands.

5. What changes at scale / honest limits

Isolation must hold across thousands of nodes and tenants: attestation infra (golden measurements per firmware/OS combo — and they change with firmware, P08), VLAN/IP management, and key lifecycle become real operational systems. The limit I'd call out: hard isolation costs utilization, and attestation adds operational burden — I'd tier it (hard isolation + attestation for untrusted/sovereign workloads, soft for trusted internal ones) and document the threat model and residual risks explicitly (P12).