Phase 14 — Security, Privacy & Governance
Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 2 weeks (28–36 hours) Prerequisites: Phase 03 (PII tags), Phase 09 (deletion vs time travel), Phase 10 (Lake Formation)
Why This Phase Exists
A platform serving millions must be compliant and secure by construction, not bolted on after an audit. The JD's Problem 5 is an enterprise governance platform, and its cloud responsibility lists "IAM least privilege, cross-account access, encryption at rest and in transit, KMS key policies, network isolation, private endpoints, S3 bucket policies, Lake Formation permissions, Glue catalog governance, Athena workgroup controls, … multi-region architecture, disaster recovery, infrastructure as code." Plus schema/contract governance (P03), PII classification, retention, audit, and breaking-change prevention. This phase builds the access-control, PII, audit, and IaC controls — and the multi-region/DR thinking — that make the platform trustworthy at scale.
Concepts
- IAM: principals, policies, roles, the evaluation logic (explicit Deny > Allow >
implicit Deny — the lab), least privilege, cross-account
AssumeRole+ trust policies, instance profiles, ABAC (tag-based access). - Encryption: at rest (SSE-S3 / SSE-KMS, envelope encryption, CMKs, key policies, grants) and in transit (TLS); who can decrypt is a key-policy question, not just an IAM one.
- Network isolation: VPC, subnets, security groups, VPC endpoints / PrivateLink (keep traffic off the public internet), private subnets for data stores.
- Lake Formation & Glue governance: fine-grained table/column/row permissions and column masking over the Glue Catalog + S3 (P10), enforced at query time; centralized grants; cross-account data sharing.
- PII classification & data privacy: tag fields by sensitivity at the schema (P03), drive masking/tokenization, access, and retention; the right to erasure and the lakehouse deletion-vs-time-travel tension (P09).
- Audit & lineage: CloudTrail-style audit of access and changes; who-accessed-what; dataset certification with an owner, SLA, lineage, and quality policy (P13).
- Athena workgroup controls / cost guardrails: per-query data-scan limits, result encryption, cost attribution (P07/P10).
- Terraform / IaC: declare infra as versioned, reviewed code; promotion across environments; drift detection; secrets management.
- Multi-region & DR: cross-region replication (S3, Cassandra, Kafka MirrorMaker), RPO/RTO, failover, active-active vs active-passive (P15).
Labs
Lab 01 — Governance & Security Kit (flagship, implemented)
| Field | Value |
|---|---|
| Goal | Build an IAM evaluator (Deny-wins), least-privilege analysis, PII column masking by clearance, row-level security, an audit log, and retention/deletion |
| Concepts | IAM precedence, least privilege, column/row security, audit, retention |
| How to Test | pytest test_lab.py -v — 14 tests |
| Talking Points | Why does Deny always win? Why tag PII at the schema? Why isn't DELETE a complete erasure on a lakehouse? |
| Resume bullet | Built a governance kit: IAM policy evaluation, least-privilege analysis, PII column masking + row-level security, audit logging, and retention/erasure |
→ Lab folder: lab-01-governance-kit/
Extension Project A — Terraform a governed data lake (spec)
Write (and terraform plan) an S3 bucket with SSE-KMS + bucket policy, a Glue database, a
Lake Formation permission grant (column-masked), and an Athena workgroup with a scan limit —
governance as code.
Extension Project B — Multi-region DR design (spec; → P15)
RPO/RTO for the capstone platform: S3 cross-region replication, Kafka MirrorMaker/MSK replication, Cassandra NetworkTopologyStrategy + LOCAL_QUORUM, failover runbook, and the cost/consistency tradeoffs.
Integrated-Scenario Hooks
- This phase enforces P03's PII tags and completes P09's compliant deletion (snapshot expiry).
- Its Lake Formation masking is the enforcement of P10's query layer.
- Its multi-region/DR thinking feeds P15's architecture and the P16 capstone.
Guides in This Phase
Key Takeaways
- IAM: explicit Deny always wins; grant least privilege and audit the unused gap.
- Encryption is who-can-decrypt (KMS key policy), not just who-can-read (IAM).
- PII is tagged at the schema (P03) and enforced as masking/row-security at query time.
- Real erasure on a lakehouse = delete + expire snapshots (P09), or time travel resurrects it.
- Govern as code (Terraform); design multi-region/DR with explicit RPO/RTO.
Deliverables Checklist
- Lab 01 implemented; all 14 tests pass
- You can explain IAM Deny precedence and least privilege
- You can explain KMS envelope encryption and key policies
- You can design column/row-level access + a compliant erasure
- Extension A Terraform or Extension B DR design