Hitchhiker's Guide — Hardening Cloud and Kubernetes

The operational companion to the Phase 07 WARMUP. The WARMUP derives cloud security from identity outward (IAM graphs, workload identity, K8s model, admission, detection, compliance); this guide is the how — building the landing zone as a security product, the workload baseline, per-provider attack-path drills, Kubernetes platform validation, detection/recovery drills, and the requirement-to-evidence completion artifact. Use assessment tools only in owned accounts.

Table of Contents

Build the Landing Zone as a Security Product

Use the resource hierarchy to separate security/log archive, shared services, production, non-production, and sandbox. Explicitly define who can: create accounts/projects/subscriptions, attach guardrails, manage identity, read centralized logs, administer keys, and delete recovery data. Baseline:

  • separate prod, non-prod, security/logging, and shared services;
  • federated MFA identities, no daily root/owner use;
  • organization guardrails (SCPs/org policies) and region/service restrictions;
  • central audit logs with separate administration (workload-admin can't tamper);
  • budgets, quotas, ownership tags, automated teardown — established before resources;
  • a break-glass identity that is offline enough to resist daily compromise but tested enough to work, monitored and time-bounded.

IaC is desired state; deployed state is truth. Export normalized identity, network, encryption, logging, and backup state and query the running cloud to catch manual changes, provider defaults, failed applies, and unmanaged resources.

Kubernetes Workload Baseline

securityContext:
  runAsNonRoot: true
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities: {drop: ["ALL"]}
  seccompProfile: {type: RuntimeDefault}

Add: workload identity (Phase 07 WARMUP, Chapter 5), no automounted service-account token unless required, explicit resources, default-deny NetworkPolicy (with a CNI that enforces it), signed images by digest, Pod Security/admission policy, and no host namespaces, hostPath, runtime socket, privileged mode, or broad devices.

Attack Fixtures

  • an identity with a wildcard role;
  • a CI identity able to impersonate a production admin;
  • public object storage;
  • a privileged pod with hostPath/runtime socket;
  • an unsigned image;
  • a secret in Terraform state;
  • deleted primary data with an untested restore;
  • a disabled audit source.

Provider Attack-Path Drills

The IAM graph (Phase 07 WARMUP, Chapter 4) made concrete per provider — prove the minimum path with synthetic assets, collect control-plane event IDs, fix every independent edge, and rerun the graph analyzer:

  • AWS: CI can sts:AssumeRole/iam:PassRole to a stronger runtime role → block with role/resource/service conditions and workload design.
  • GCP: federated CI can impersonate a broadly privileged service account → bind federation attributes and iam.serviceAccountTokenCreator narrowly.
  • Azure: a broad federated subject receives permanent subscription Owner → constrain the subject, scope RBAC, and require eligible/time-bound (PIM) activation.

Kubernetes Platform Validation

Test: API anonymous access, human/workload authentication, RBAC escalation verbs, service-account token mounting, admission-bypass paths, privileged/host settings, image digest/signature policy, node isolation, default-deny network, DNS/metadata access, secret reads, and audit coverage.

Integration tests must query the admitted object and the running pod — mutation, defaulting, or another controller can change effective state, so the manifest you submitted is not proof of what runs. (Namespaces are not a tenant boundary — Phase 07 WARMUP, Chapter 8.)

Policy Tests

Use OPA/Conftest, Gatekeeper, or Kyverno with unit tests and known-good/known-bad manifests. Test exceptions with owner and expiry. Query runtime state to verify admission and configuration did not drift. The bar: block all seeded unsafe manifests, permit documented safe fixtures.

Cloud Detection and Recovery

Detect: trust-policy/federation changes, privilege grants, logging disablement, public exposure, key-policy/disable operations, unusual token creation, admission changes, privileged pods, secret reads, snapshot/share operations, and backup deletion. Monitor source health and retention.

Run quarterly drills for: leaked CI identity, public object storage, compromised cluster service account, destructive administrator, and an unavailable KMS/logging dependency. Measure containment, credential invalidation, scope confidence, rebuild, restore (against declared RPO/RTO), and residual exposure.

Leaked-Identity Incident Drill

For a leaked cloud identity (the capstone scenario): disable/contain → preserve control-plane logs → identify issued sessions and impersonation → enumerate changed resources and accessed data → rotate dependent secrets → rebuild affected workloads → prove old access fails → add graph/detection regression. Rotation isn't done until the old credential is verified rejected (Phase 07 WARMUP, Chapter 10).

Completion Evidence

  • hierarchy, identity, network, data, cluster, and recovery diagrams;
  • an effective-permission/impersonation graph (no unexplained wildcard/trust path);
  • insecure and hardened IaC plus deployed-state queries;
  • policy unit and integration fixtures;
  • control-plane and data-access event evidence;
  • credential-revocation and backup-restore timings;
  • a control map linking requirement → implementation → test → evidence → owner → frequency;
  • teardown and cost report.

Worked Examples

An AWS IAM privilege-escalation path (the graph in action)

A "harmless" role with two permissions becomes admin (Phase 07 WARMUP, Chapter 4):

Role "ci-deployer" can:  iam:PassRole  +  lambda:CreateFunction  +  lambda:InvokeFunction
Path: create a Lambda whose execution role is the high-priv "AdminRole" (PassRole), then invoke it
      → attacker code now runs WITH AdminRole's permissions.

The analyzer must find this transitive path even though each permission "looks fine" alone. Fixes attack every independent edge: scope PassRole with a Condition to only low-priv roles; deny lambda:CreateFunction for this identity; and detect iam:PassRole to privileged roles. "Each role is least-privilege" does not prevent composition — analyze the graph, not the nodes.

Probing Kubernetes RBAC

kubectl auth can-i --list --as=system:serviceaccount:app:default   # what can this SA do?
kubectl auth can-i create pods --as=system:serviceaccount:app:default -n kube-system

If a pod's default service account can create pods in kube-system or read cluster secrets, a single pod compromise becomes cluster compromise. And recall namespaces are not a tenant boundary (WARMUP Chapter 8) — pods share nodes/kernel/network; hard multi-tenancy needs runtime isolation, NetworkPolicy with a real CNI, and tight RBAC.

An admission policy that actually blocks (Rego sketch)

package kubernetes.admission
deny[msg] {
  c := input.request.object.spec.containers[_]
  c.securityContext.privileged == true
  msg := sprintf("privileged container %q denied", [c.name])
}

Test it with known-bad (privileged pod → denied) and known-good (compliant pod → admitted) fixtures, and verify the admitted object (mutation/defaulting can change effective state).

Interview Drills

  1. "Design a secure landing zone." Org hierarchy with org-level guardrails (SCPs) even admins can't exceed; federated IdP + MFA, no local users, least-priv custom roles; separated accounts/projects per env/sensitivity; network zones + private endpoints + controlled egress; centralized tamper-resistant logging; KMS + managed secrets; org-wide public-storage deny; budgets; tested break-glass. All as reviewed IaC with policy-as-code in CI.
  2. "Respond to leaked CI credentials." Blast radius via the IAM graph → revoke/rotate and verify old access fails → audit what it did while live → contain touched resources → close the root cause by moving CI to OIDC federation (no stored secret). Document timeline + evidence.
  3. "Prevent cross-tenant access in multi-tenant K8s." Don't rely on namespaces: per-tenant RBAC, no cluster-admin SAs in pods, default-deny NetworkPolicy with a real CNI enforcer, Pod Security (non-root/read-only/no host ns), strong runtime isolation (gVisor/Kata or dedicated node pools) for untrusted tenants, per-tenant workload identity, quotas — or separate clusters.
  4. "Compare workload-identity approaches." All replace stored static keys with attested, short-lived, audience-scoped creds. Instance/pod identity (token via metadata — protect the endpoint); OIDC federation (external system presents a trusted token to mint creds — zero stored secret); SPIFFE/mesh (per-workload mTLS certs). Differ in who attests; same goal: kill the long-lived secret.
  5. "Map a PCI/FedRAMP requirement to implementation and evidence." Take "encrypt data in transit": implement TLS 1.2+ enforced by config + admission; test = an automated check rejecting cleartext endpoints; evidence = the recurring result + config, owned by platform, run continuously. Deliver requirement→control→test→evidence→owner→frequency, not a scan screenshot.

References

  • AWS Well-Architected Security Pillar; Google security foundations; Azure/Microsoft CAF security.
  • Kubernetes docs (RBAC, admission, Pod Security Standards, NetworkPolicy, audit); NSA/CISA Kubernetes Hardening Guidance.
  • OPA/Gatekeeper and Kyverno docs; Falco (runtime detection); CIS cloud/Kubernetes Benchmarks.
  • IAM privilege-escalation research (e.g. Rhino Security Labs); SPIFFE/SPIRE; GitHub OIDC federation.
  • NIST SP 800-53 / 800-207; FedRAMP; PCI DSS v4.0; SOC 2 TSC; ISO/IEC 27001; CSA CCM.
  • Prowler / ScoutSuite (owned-account assessment); Checkov / tfsec (IaC).