🛸 Hitchhiker's Guide — Phase 14: Security, Privacy & Governance

Read this if: you want the security/governance controls fast — IAM, KMS, Lake Formation, PII, erasure, IaC, DR. Skim, read WARMUP, build the kit.

0. The 30-second mental model

A platform for millions must be compliant by construction. You control access with IAM (Deny wins), encrypt with KMS (who can decrypt ≠ who can read), isolate with VPC/ PrivateLink, enforce column/row access with Lake Formation, tag PII at the schema and mask it, audit everything, govern as Terraform, and design multi-region/DR with RPO/RTO. One sentence: least privilege + encryption + fine-grained masking + audit, all as reviewed code.

1. IAM precedence (memorize)

explicit DENY  >  matching ALLOW  >  implicit DENY (default)
a broad Allow CANNOT override a targeted Deny
least privilege: grant only what's used; scope resources (no Resource:*); roles > keys
cross-account: AssumeRole + trust policy ; ABAC: grant by tags

2. KMS (the "permissions look right but denied" trap)

SSE-KMS envelope: data key encrypts object; CMK encrypts the data key
who can READ = needs s3:GetObject AND kms:Decrypt (KEY POLICY, not just IAM)
separate keys per classification/tenant → revoking a key revokes a slice
in transit: TLS, enforce aws:SecureTransport

3. Network isolation

data stores in PRIVATE subnets (no public IP); security groups restrict reach
VPC endpoints / PrivateLink → AWS/SaaS traffic never hits the public internet
(combine with bucket policy requiring the endpoint = anti-exfiltration)

4. Lake Formation = fine-grained access

table / column / row permissions + COLUMN MASKING over Glue Catalog + S3
enforced by the query engine at scan time (Athena/Trino/Spark)
analyst queries the table but NEVER sees PII columns — no copy, no forked lineage

5. PII & erasure

tag PII at the SCHEMA (P03) → drives masking, access, retention downstream
masking/tokenization for uncleared principals
GDPR erasure on a lakehouse = DELETE current + EXPIRE snapshots + remove orphans (P09)
  (DELETE alone → time travel resurrects it = compliance bug)

6. Audit & certification

log who/what/when/decision (CloudTrail + data-access log)
dataset certification: owner + SLA + lineage + PII class + quality policy (a JD success metric)
lineage (P13) proves where PII flowed → erasure follows lineage, not one table

7. Terraform / IaC

infra as versioned, REVIEWED code → a public bucket shows up in a PR diff
plan (preview diff) → apply ; state = source of truth ; drift detection
governance-as-code: bucket policies, KMS, LF grants, IAM, Athena limits in TF
policy-as-code (tfsec/checkov/OPA) gates insecure changes in CI

8. Multi-region / DR

replicate: S3 CRR · Kafka MirrorMaker/MSK · Cassandra NetworkTopology + LOCAL_QUORUM · DDB Global
RPO (data you can lose) + RTO (time to recover) = the two numbers that define DR + its cost
active-passive (cheaper, slower) vs active-active (costly, instant, multi-write conflicts P01)
DR is only real if you GAME-DAY test the failover

9. Beginner mistakes that mark you

  1. Assuming a broad Allow overrides a Deny (it never does).
  2. Granting s3:GetObject and forgetting kms:Decrypt → mysterious denial.
  3. Copying data into "no-PII" tables instead of column masking (forks lineage).
  4. DELETE and calling GDPR erasure done (snapshots resurrect it, P09).
  5. Click-ops security (no review trail) instead of Terraform.
  6. Data stores with public IPs / no VPC endpoints.
  7. A DR plan never tested with a real failover.

10. How this phase pays off later

  • Enforces P03 PII tags; completes P09 compliant deletion; enforces P10 query masking.
  • Multi-region/DR → P15 architecture + P16 capstone.
  • Audit/certification → P13 observability + P00 data products.

Read WARMUP, build the kit, then P15: principal-level architecture & strategy — ADRs, migrations, build-vs-buy, design reviews, leadership.