Phase 09 — Networking, Secure Boot & Multi-Tenancy
Difficulty: ⭐⭐⭐⭐☆ | Estimated Time: 2 weeks
Roles supported: Rack Management SWE (Senior/Staff), Infra Security SWE, Platform SWE
Hardware needed: none — simulators + real ip/tc commands (Linux/VM for the live extensions)
Why This Phase Exists
The JD: "Ensure secure boot, VLAN separation, and QoS enforcement for multi-tenant environments." When a rack hosts multiple tenants (or separates the management plane from tenant data — Phase 01 Ch. 6), isolation isn't a feature, it's a contract: tenant A must not see tenant B's traffic, the management network must be unreachable from tenants, no tenant can starve another of bandwidth, and the software/firmware that boots must be trusted. These are the security and fairness primitives of a shared rack.
Three pillars:
- Network isolation — VLANs, network namespaces, the management-vs-data plane separation, and segmentation that keeps tenants and the control plane apart.
- QoS — guaranteeing/limiting bandwidth so one tenant can't degrade another (and the management plane stays responsive).
- Secure/measured boot & attestation — a chain of trust from firmware to OS so you know a node booted trusted software, with remote attestation (TPM) to prove it — the foundation for sovereign/regulated deployments (Phase 06 RKE2).
You'll build a tenant-isolation + QoS model (with the real ip/tc commands) and a
secure/measured-boot + attestation simulator (TPM PCRs, signature chain).
Concepts
Network isolation
- The three planes again (Phase 01 Ch. 6): management (OOB) vs data (tenant) vs scale-up; why management is physically/logically isolated
- VLANs (802.1Q): tags, trunk vs access ports, the broadcast-domain boundary; VLAN-per-tenant
- Linux network namespaces, veth, bridges, VLAN sub-interfaces — how isolation is implemented on a host
- Segmentation & microsegmentation; firewalls/ACLs; the BMC/management VLAN as a hard boundary
- VXLAN/overlays (briefly) for cross-rack tenant networks; SR-IOV/VFs for accelerator NICs in tenant pods
- mTLS, network policy (Phase 06), secrets management, least privilege on the control plane
QoS (Quality of Service)
- Why QoS: the noisy-neighbor problem; protecting the management plane and paying tenants
- Linux
tc(traffic control): qdiscs (HTB, fq_codel), classes, filters; rate limiting, shaping, prioritization - DSCP/marking; bandwidth guarantees vs limits; per-tenant fairness
- QoS for storage/PCIe (briefly) and for the management network
Secure & measured boot, attestation
- UEFI Secure Boot: signed shim → bootloader → kernel; the key hierarchy (PK/KEK/db/dbx); what it does and doesn't protect
- Measured boot: each stage measures (hashes) the next into a TPM PCR before executing it; the difference from Secure Boot (measure vs enforce)
- TPM: PCRs (extend-only hash registers), the root of trust, sealing, quotes
- Remote attestation: a node proves its boot state (a signed PCR quote) to a verifier, which checks it against known-good values before trusting it (e.g., before handing it tenant work/keys)
- The chain of trust from hardware root → firmware → OS; anti-rollback (Phase 08) as part of it
- Sovereign/air-gapped/regulated deployment (Phase 06 RKE2): why attestation matters in the Kingdom's data centers
Multi-tenancy synthesis
- Soft vs hard isolation; defense in depth (VLAN + namespace + RBAC + NetworkPolicy + attestation)
- Mapping tenant boundaries across network, compute (Phase 06), and accelerators (Phase 04 MIG/partitioning)
- The threat model: a malicious tenant, a compromised node, an exposed BMC
Labs
Lab 01 — VLAN Separation & QoS Tenant Isolation
| Field | Value |
|---|---|
| Goal | Model (and, optionally, build with real ip/tc) tenant network isolation via VLANs/namespaces and QoS bandwidth guarantees, and verify that tenants can't cross boundaries or starve each other — and that the management plane is unreachable from tenants. |
| Concepts | VLANs, namespaces/veth/bridges, the management-vs-data boundary, tc/HTB QoS, noisy-neighbor, reachability policy. |
| Steps | 1) python3 solution.py — a reachability + QoS policy engine: who can talk to whom, and bandwidth shares. 2) Read the isolation invariants. 3) Try to make a tenant reach the BMC VLAN (it's blocked). 4) Extensions: build it for real with ip netns/tc. |
| Stack | Python 3 stdlib (policy model) + real ip/tc command examples |
| Output | A policy engine enforcing tenant isolation + QoS, with invariants (no cross-tenant, no tenant→management). |
| How to Test | Asserts verify cross-tenant traffic is denied, management VLAN is unreachable from tenants, and QoS guarantees sum within link capacity. |
| Talking Points | Why management is a hard boundary; VLAN as a broadcast-domain boundary; noisy-neighbor and HTB; defense in depth. |
| Resume Bullet | "Built a tenant network-isolation and QoS policy engine (VLAN/namespace segmentation + HTB bandwidth guarantees) enforcing no-cross-tenant and management-plane-isolation invariants." |
| Extensions | Build real isolation with ip netns add/veth/bridge/VLAN + tc HTB and test with iperf; add SR-IOV VF assignment; add a NetworkPolicy (Phase 06) equivalent. |
Lab 02 — Secure/Measured Boot + Attestation
| Field | Value |
|---|---|
| Goal | Build the chain of trust: a secure-boot signature chain (shim→bootloader→kernel) and a measured-boot model that extends component hashes into TPM PCRs, then a remote-attestation verifier that accepts/rejects a node based on a signed PCR quote. |
| Concepts | UEFI Secure Boot key hierarchy, signature verification, measured boot, TPM PCR extend, attestation quotes, known-good policy. |
| Steps | 1) python3 solution.py — boot a trusted node (verifies + attests OK) and a tampered node (fails). 2) Read the PCR-extend and quote-verify logic. 3) See attestation reject a node with an unexpected boot measurement. 4) Extensions: real tpm2-tools/swtpm. |
| Stack | Python 3 stdlib (hashlib/hmac) |
| Output | A secure+measured boot chain + an attestation verifier that admits only known-good boot states. |
| How to Test | Asserts verify: signed chain boots, tampered component is rejected (secure boot) or detected (measured boot/PCR mismatch), and attestation rejects an unknown PCR quote. |
| Talking Points | Secure boot (enforce) vs measured boot (measure); what a PCR is; why attestation gates trust before handing a node keys/work; the sovereign deployment angle. |
| Resume Bullet | "Built a secure+measured boot chain (signed shim→bootloader→kernel, TPM PCR extend) with a remote-attestation verifier admitting only known-good boot states; the trust foundation for multi-tenant/sovereign deployment." |
| Extensions | Use real swtpm + tpm2-tools (tpm2_pcrextend, tpm2_quote); seal a secret to a PCR state; integrate attestation into the Phase 05 provisioning gate. |
Deliverables Checklist
- You can explain the three planes and why management is a hard isolation boundary
- You can explain VLANs (802.1Q) and how Linux namespaces/bridges implement isolation
- Policy engine enforces no-cross-tenant and no-tenant→management invariants
-
You can explain the noisy-neighbor problem and how
tc/HTB QoS solves it - You can explain Secure Boot (enforce) vs measured boot (measure) and the key hierarchy
- You can explain what a TPM PCR is and how measured boot extends it
- Attestation verifier admits only known-good boot states; you can explain why that gates trust
- You can describe defense-in-depth tenant isolation across network/compute/accelerator
Interview Relevance
- "How do you isolate tenants on a shared rack at the network level?"
- "Why must the management/BMC network be isolated, and how do you enforce it?"
- "What's the noisy-neighbor problem and how does QoS solve it?" (
tc/HTB) - "Explain Secure Boot. What does it protect against and what doesn't it?"
- "Secure boot vs measured boot — what's the difference?"
- "What is a TPM PCR and what is remote attestation for?"
- "Design multi-tenant isolation for a rack — defense in depth."