Warmup Guide — Networking, Secure Boot & Multi-Tenancy

Zero-to-expert primer for Phase 09. You need no security/networking depth: this guide starts at "what is a VLAN" and "what does Secure Boot do" and ends with you designing defense-in-depth tenant isolation and explaining remote attestation. These are the isolation and trust primitives that make a shared rack safe.

Table of Contents

Chapter 1: The Isolation Mandate

A shared rack hosts multiple tenants and a management plane on the same physical hardware. Isolation is the contract that makes that safe:

  • Confidentiality/integrity: tenant A cannot see or alter tenant B's traffic or data.
  • Management protection: tenants cannot reach the BMC/PDU/CDU management network (an exposed BMC = total compromise, Phase 03 Ch. 7).
  • Fairness: no tenant can starve another of bandwidth (QoS).
  • Trust: a node runs only trusted firmware/OS, and you can prove it (attestation) before giving it tenant work or secrets.

The senior framing: isolation is defense in depth — no single mechanism is trusted alone. Network (VLAN/namespace), compute (Phase 06 RBAC/NetworkPolicy/quota), accelerator (Phase 04 MIG/partitioning), and trust (secure boot/attestation) layers combine so that a breach of one doesn't breach the tenant boundary. This is especially acute for sovereign/regulated deployments (a government data center in the Kingdom — Phase 06 RKE2), where isolation and provable trust are compliance requirements, not nice-to-haves.

Chapter 2: VLANs and Network Segmentation

A VLAN (Virtual LAN, IEEE 802.1Q) partitions one physical network into isolated broadcast domains. A 12-bit VLAN tag added to the Ethernet frame marks which VLAN it belongs to; switches keep traffic within its VLAN. Two port types:

  • Access port: belongs to one VLAN; untagged frames (the host doesn't know about VLANs).
  • Trunk port: carries many tagged VLANs (switch-to-switch, or to a host that handles tags).

For a rack, the canonical use is VLAN-per-plane and VLAN-per-tenant: a management VLAN (BMCs/ PDUs/CDU — reachable only by the control plane), and a VLAN (or set) per tenant for their data traffic. Crossing VLANs requires a router/firewall with an explicit ACL — so "tenant traffic can reach the management VLAN" is a misconfiguration, not a default, and your software enforces that it never happens (Lab 01's invariant).

Beyond VLANs (vocabulary): VXLAN/overlays stretch a tenant network across racks/sites (encapsulating L2 in UDP); microsegmentation applies fine-grained per-workload policy; SR-IOV carves a physical NIC into virtual functions (VFs) assignable directly to tenant pods (fast accelerator networking) — each VF needs its own isolation.

Chapter 3: Linux Isolation — Namespaces, veth, Bridges

How isolation is actually implemented on a host (and the basis of containers, Phase 06):

  • Network namespace: an isolated copy of the network stack (its own interfaces, routes, firewall). A container/tenant gets its own namespace; it can't see the host's or another namespace's interfaces.
  • veth pair: a virtual cable — two connected interfaces, one in the namespace, one outside — to plug a namespace into the host network.
  • bridge: a virtual switch in the kernel; you attach veths/VLAN sub-interfaces to it to connect namespaces, and use VLAN sub-interfaces (eth0.100) to put a namespace on a VLAN.
  • firewall/ACL (nftables/iptables): deny cross-tenant and tenant→management flows.

The real commands (Lab 01 extension):

ip netns add tenantA
ip link add veth-a type veth peer name veth-a-br
ip link set veth-a netns tenantA
ip link add link eth0 name eth0.100 type vlan id 100   # VLAN 100 sub-interface

This is the same machinery Kubernetes CNIs use; understanding it demystifies pod networking (Phase 06) and lets you reason about where isolation could leak.

Chapter 4: QoS and the Noisy Neighbor

The noisy-neighbor problem: on a shared link, one tenant saturating bandwidth degrades everyone — including the management plane (so you can't even manage the rack during the incident). QoS prevents it by guaranteeing and/or limiting bandwidth per tenant/class.

On Linux, tc (traffic control) shapes traffic with qdiscs (queuing disciplines), classes, and filters:

  • HTB (Hierarchical Token Bucket): the workhorse — a tree of classes, each with a guaranteed rate (rate) and a ceiling (ceil). You give each tenant a guaranteed share and a cap, and reserve a slice for the management plane. Idle tenants' bandwidth can be borrowed up to the ceiling — fairness without waste.
  • fq_codel: fights bufferbloat (latency under load) — good as the leaf qdisc.
  • DSCP marking: tag packets by priority so the network honors classes end to end.

Example shape (Lab 01 extension):

tc qdisc add dev eth0 root handle 1: htb default 30
tc class add dev eth0 parent 1: classid 1:1 htb rate 100gbit
tc class add dev eth0 parent 1:1 classid 1:10 htb rate 10gbit ceil 40gbit  # mgmt: guaranteed
tc class add dev eth0 parent 1:1 classid 1:20 htb rate 40gbit ceil 90gbit  # tenant A

The invariant your software enforces (Lab 01): the sum of guarantees must not exceed link capacity (you can oversubscribe ceilings, never guarantees), and the management class always has a reserved guarantee so the rack stays manageable under tenant load.

Chapter 5: Secure Boot — Enforcing Trust

UEFI Secure Boot ensures a machine only runs signed, trusted boot software, defending against bootkits/rootkits that would otherwise load below the OS and be nearly invisible.

The chain: firmware verifies the shim (signed by Microsoft's key, in practice), shim verifies the bootloader (GRUB), the bootloader verifies the kernel, and the kernel verifies modules — each stage checks the next's signature before executing it. If a signature doesn't match a key in the allowed database, the boot stops (enforcement).

The key hierarchy (know the names):

  • PK (Platform Key) — the root; owns the platform, authorizes KEK changes.
  • KEK (Key Exchange Keys) — authorize updates to db/dbx.
  • db — the allowed signatures/keys (what may run).
  • dbx — the forbidden list (revoked/known-bad — e.g., a vulnerable bootloader).

What Secure Boot does: stops unsigned/tampered boot code from running. What it doesn't: it doesn't encrypt anything, doesn't protect a running system from runtime attacks, and doesn't prove to a third party what booted (that's attestation, Chapter 7). It's enforcement at boot, and it ties to firmware signing/anti-rollback (Phase 08) and to PXE (Phase 05 — your network boot chain must be signed or Secure Boot blocks it).

Chapter 6: Measured Boot and the TPM

Measured boot is the complement to Secure Boot: instead of (or in addition to) enforcing, each boot stage measures (cryptographically hashes) the next component before running it and records the measurement in the TPM — building an unforgeable record of exactly what booted.

The TPM (Trusted Platform Module) is a hardware (or firmware, fTPM) security chip — the hardware root of trust. Its key feature for this is PCRs (Platform Configuration Registers): special registers you can't write arbitrarily, only extend:

PCR_new = Hash(PCR_old || measurement)

Because extend is a one-way hash chain, the final PCR value is a fingerprint of the entire boot sequence in order — change any component (or its order) and the PCR differs. Different PCRs hold different parts (firmware, bootloader, kernel, config). Lab 02 implements PCR-extend.

Secure boot vs measured boot (the interview distinction): Secure Boot enforces (won't run unsigned code — stops a bad boot); measured boot measures (records what booted — lets you detect and prove it, even if it booted). They're complementary: enforce and record. Measured boot also enables sealing — encrypting a secret so the TPM only releases it if the PCRs match a known-good state (so a tampered boot can't unseal the disk-encryption key).

Chapter 7: Remote Attestation

Measured boot records what booted locally; remote attestation lets a node prove its boot state to a remote verifier so the verifier can decide whether to trust it — before handing it tenant workloads, secrets, or cluster membership.

The flow:

  1. The verifier sends a nonce (freshness, anti-replay).
  2. The node asks its TPM for a quote: the current PCR values, signed by a TPM-resident key (an AK/Attestation Key, rooted in the TPM's endorsement key), including the nonce.
  3. The verifier checks the signature (is this a genuine TPM?) and compares the PCRs against known-good reference values (golden measurements for the approved firmware/OS).
  4. If they match → the node booted approved software → grant trust (join the cluster, release keys, schedule tenant work). If not → quarantine it (Phase 08/10).

Why it matters for this role: in a multi-tenant or sovereign/air-gapped deployment, you cannot just assume a node is trustworthy — a compromised firmware/OS could exfiltrate tenant data. Attestation makes trust provable and gateable: a node earns tenant work only by attesting to a known-good state. It's a natural addition to the Phase 05 provisioning validation gate and the Phase 06 node-join flow. Lab 02 builds a verifier that admits only known-good quotes.

Chapter 8: Defense in Depth & the Threat Model

Tie it together with the adversary in mind:

  • A malicious tenant: tries to reach another tenant or the management plane, or to hog bandwidth. Defenses: VLAN/namespace isolation (Ch. 2–3), NetworkPolicy/RBAC/quota (Phase 06), QoS (Ch. 4), accelerator partitioning isolation (Phase 04), no co-placement of competing tenants on one physical device.
  • A compromised node (bad firmware/OS): tries to lie about its state or attack the fabric. Defenses: secure boot (won't run untrusted code), measured boot + attestation (won't be trusted unless it proves a good state), management-plane isolation (limits reach).
  • An exposed BMC (the classic catastrophe): full power/firmware control from the network. Defense: the management VLAN is a hard boundary reachable only by the control plane; never on a tenant/internet-facing network (Phase 03 Ch. 7).

Defense in depth means no single control is load-bearing: even if a tenant breaks out of a namespace, the VLAN still contains them; even if they reach a node, attestation/RBAC limits the damage; even if firmware is tampered, attestation refuses to trust it. As a Staff engineer you articulate the threat model and show that each layer fails safe — which is exactly what a security review (and a sovereign-deployment customer) will probe.

Lab Walkthrough Guidance

Order: Lab 01 (network/QoS) → Lab 02 (boot trust). Isolation of traffic first, then trust of what's running.

Lab 01 (VLAN/QoS isolation):

  1. python3 solution.py — a reachability + QoS policy engine; see who can reach whom and the bandwidth shares.
  2. Try to make a tenant reach the management VLAN — the invariant blocks it.
  3. Extensions: build it for real with ip netns/veth/bridge/VLAN sub-interfaces + tc HTB, and test isolation/QoS with iperf.

Lab 02 (secure/measured boot + attestation):

  1. python3 solution.py — boot a trusted node (verifies + attests OK) and a tampered node (secure boot rejects / measured boot detects / attestation refuses).
  2. Read the PCR-extend chain and the quote verification against known-good values.
  3. Extensions: use real swtpm + tpm2-tools (tpm2_pcrextend, tpm2_quote); seal a secret to a PCR state; gate Phase 05 provisioning on attestation.

Phase capstone question: "Design multi-tenant isolation for a rack serving paid and free tenants plus a management plane, in a regulated/sovereign environment." (Answer in system-design/04: VLAN-per-tenant + management VLAN as a hard boundary; namespaces/NetworkPolicy/RBAC/quota; QoS via HTB with a reserved management class; no co-placement of competing tenants on a physical accelerator; secure boot + measured boot + attestation gating node trust; defense in depth with a stated threat model.)

Success Criteria

You're done with this phase when — without notes:

  • You can explain the isolation mandate and defense in depth (Ch. 1, 8)
  • You can explain VLANs/802.1Q and Linux namespace/veth/bridge isolation (Ch. 2–3)
  • You can state and enforce the no-cross-tenant / no-tenant→management invariants (Ch. 2, 8)
  • You can explain the noisy-neighbor problem and HTB QoS with a reserved management class (Ch. 4)
  • You can explain Secure Boot, the PK/KEK/db/dbx hierarchy, and what it does/doesn't protect (Ch. 5)
  • You can explain measured boot, TPM PCRs (extend-only), and secure-vs-measured boot (Ch. 6)
  • You can explain remote attestation and why it gates trust before tenant work (Ch. 7)
  • You can lay out a defense-in-depth tenant-isolation design with a threat model (Ch. 8)

Interview Q&A

Q1: How do you isolate tenants on a shared rack at the network level, and why is the management plane special? A: VLAN-per-tenant (802.1Q) puts each tenant in its own broadcast domain; on the host, network namespaces + veth + bridges + VLAN sub-interfaces implement the isolation, with firewall ACLs denying cross-tenant flows. Crossing VLANs requires an explicit router/ACL, so cross-tenant traffic is a misconfiguration, not a default — and my software asserts it never happens. The management plane (BMCs/PDUs/CDU) is special because those devices have total power/firmware control, so it's a hard boundary: its own VLAN, reachable only by the control plane, never from a tenant or the internet — an exposed BMC is a full compromise. It's defense in depth: VLAN + namespace + NetworkPolicy/RBAC (Phase 06) + QoS so no single failure breaches the boundary.

Q2: What's the noisy-neighbor problem and how does QoS solve it? A: On a shared link, one tenant saturating bandwidth degrades every other tenant — and dangerously the management plane, so you can't even manage the rack during the incident. QoS fixes it with Linux tc using HTB (Hierarchical Token Bucket): a class tree where each tenant gets a guaranteed rate and a ceil, with a reserved guaranteed class for management. Idle bandwidth can be borrowed up to the ceiling, so it's fair without waste. The invariant: the sum of guarantees can't exceed link capacity (ceilings can oversubscribe), and management always keeps its reserved guarantee. fq_codel as the leaf qdisc keeps latency low under load.

Q3: Explain Secure Boot — what it protects against and what it doesn't. A: UEFI Secure Boot ensures the machine only runs signed, trusted boot software: firmware verifies the shim, shim verifies GRUB, GRUB verifies the kernel, each checking a signature against the allowed key database (db), with a forbidden list (dbx) for revoked components, all under the PK/KEK key hierarchy. It defends against bootkits/rootkits that load below the OS. What it doesn't do: it doesn't encrypt anything, doesn't protect a running system from runtime attacks, and doesn't prove to a remote party what booted — that's remote attestation. It's enforcement at boot, and it ties to firmware signing/anti-rollback (Phase 08) and PXE (the network boot chain must be signed or Secure Boot blocks it).

Q4: Secure boot vs measured boot — what's the difference, and what's a TPM PCR? A: Secure Boot enforces — it refuses to run unsigned/untrusted code, stopping a bad boot. Measured boot measures — each stage hashes the next component before running it and records the hash in the TPM, building an unforgeable record of exactly what booted (it doesn't stop a boot, it documents it). They're complementary: enforce and record. A TPM PCR (Platform Configuration Register) is a special register you can't write arbitrarily, only extend: PCR = Hash(PCR || measurement). Because it's a one-way hash chain, the final value fingerprints the whole boot sequence in order — any change yields a different PCR. That enables detection, attestation, and sealing (releasing a secret only if PCRs match a known-good state).

Q5: What is remote attestation and why would a rack platform need it? A: Remote attestation lets a node prove its boot state to a remote verifier before being trusted. The verifier sends a nonce; the node's TPM returns a signed quote of its PCR values (plus the nonce); the verifier checks the signature came from a genuine TPM and compares the PCRs against known-good golden measurements. Match → the node booted approved firmware/OS → grant trust (join the cluster, release keys, schedule tenant work); mismatch → quarantine. A rack platform needs it because in multi-tenant or sovereign/air-gapped deployments you can't just assume a node is trustworthy — compromised firmware could exfiltrate tenant data. Attestation makes trust provable and gateable, a natural addition to the provisioning validation gate (Phase 05) and the node-join flow (Phase 06).

Q6 (Staff): Design defense-in-depth multi-tenant isolation for a regulated/sovereign rack. A: Layer it so no single control is load-bearing. Network: VLAN-per-tenant + a management VLAN as a hard boundary (control-plane-only), namespaces/veth/bridges on hosts, NetworkPolicy + RBAC + quotas in Kubernetes (Phase 06), and QoS (HTB) with a reserved management class. Compute/ accelerator: don't co-place competing tenants on one physical accelerator; use MIG-style partitioning isolation (Phase 04) where fractional sharing is needed. Trust: secure boot (won't run untrusted code) + measured boot + remote attestation gating node trust before it gets tenant work or keys, with sealed secrets tied to PCR state. Operational: least-privilege control plane, mTLS, secrets management, audit (Phase 07/12), and RKE2 (Phase 06) for a hardened, air-gappable base. Then I'd write the threat model explicitly — malicious tenant, compromised node, exposed BMC — and show each layer fails safe, because a sovereign customer's security review will probe exactly that, and the Staff deliverable is the documented, defensible design, not just the config.

References

  • IEEE 802.1Q (VLANs); Linux networking: ip netns, ip-link, bridges, VLAN sub-interfaces — https://man7.org/linux/man-pages/
  • Linux Traffic Control (tc, HTB, fq_codel) — https://man7.org/linux/man-pages/man8/tc.8.html , the LARTC HOWTO
  • UEFI Secure Boot & the PK/KEK/db/dbx hierarchy — UEFI spec; https://uefi.org/
  • TPM 2.0 (TCG) — PCRs, quotes, attestation, sealing — https://trustedcomputinggroup.org/
  • tpm2-tools & swtpm (software TPM for labs) — https://github.com/tpm2-software/tpm2-tools
  • Linux IMA/EVM & measured boot; remote attestation frameworks (Keylime) — https://keylime.dev/
  • NIST SP 800-155 (BIOS integrity measurement), 800-193 (firmware resiliency) — https://csrc.nist.gov/
  • Cross-track: Security Engineer track — deeper offensive/defensive security