Phase 09 — Lakehouse Table Formats
Difficulty: ⭐⭐⭐⭐⭐ Estimated Time: 2 weeks (30–40 hours) Prerequisites: Phase 08 (the files these formats sit on), Phase 01 (snapshot isolation)
Why This Phase Exists
The lakehouse is the JD's center of gravity for storage: "Iceberg snapshots … partition evolution and hidden partitioning … compact small files … table layouts for Athena and Trino … streaming writes and batch reads safely … schema evolution … time travel … recover from bad writes … retention and deletion policies." Table formats — Iceberg, Delta Lake, Hudi — are what turn a pile of Parquet files (P08) into a real database: ACID transactions, MVCC snapshots, time travel, safe concurrent writers, and the maintenance (compaction, expiry) that keeps it fast. This is also the destination of the Round-5 migration (Hive → lakehouse).
Concepts
- Why table formats exist: S3 has no atomic rename (P08), so "which files are the table right now?" needs an explicit, atomically-swapped metadata pointer. Table formats provide it → ACID on object storage.
- Iceberg architecture:
metadata.json→ manifest list → manifests → data files; every commit writes a new metadata file and atomically swaps the catalog pointer (a new snapshot). Hidden partitioning, partition evolution, time travel, snapshot isolation via optimistic concurrency. - Delta Lake: a transaction log (
_delta_log/ordered JSON commits + periodic Parquet checkpoints); ACID, time travel,MERGE,OPTIMIZE/Z-ORDER,VACUUM. - Apache Hudi: Copy-on-Write (rewrite files on update — read-optimized) vs Merge-on-Read (write delta logs, merge at read — write-optimized); a timeline of actions; record-level indexes for fast upserts; the upsert/incremental specialist.
- Snapshot isolation & optimistic concurrency: readers see a consistent snapshot; writers commit optimistically and retry on conflict — how Spark and Flink write the same table without a lock (the lab).
- Maintenance: compaction (
rewrite_data_files/OPTIMIZE) for small files; expire_snapshots /VACUUMto reclaim storage; remove_orphan_files. - Streaming + batch on one table: exactly-once streaming writes (the P04 two-phase-commit sink commits an Iceberg snapshot) coexisting with batch reads/backfills.
- Schema & partition evolution: add/rename/reorder columns safely; change partitioning without rewriting history (hidden partitioning).
- Deletes & GDPR: row-level deletes (copy-on-write vs merge-on-read), and the time-travel-vs-deletion tension (a deleted row can be resurrected by time travel until snapshots expire — a real privacy gotcha, P14).
Labs
Lab 01 — Iceberg-Style Table Format (flagship, implemented)
| Field | Value |
|---|---|
| Goal | Build snapshots (MVCC), optimistic-concurrency commits, time travel, partition pruning, compaction, and snapshot expiry |
| Concepts | Snapshot isolation, optimistic concurrency, time travel, compaction, expiry |
| How to Test | pytest test_lab.py -v — 12 tests |
| Talking Points | How do DELETE and time travel coexist? Why optimistic concurrency over locking? What does expiry cost you? |
| Resume bullet | Implemented an Iceberg-style table format: MVCC snapshots, optimistic-concurrency commits, time travel, compaction, and snapshot expiry |
→ Lab folder: lab-01-iceberg-table/
Lab 02 — Iceberg-Style Table Commits in Scala (implemented, sbt test)
| Field | Value |
|---|---|
| Goal | The Scala twin of Lab 01: immutable snapshots, optimistic-concurrency commits (CAS on current snapshot → conflict + retry), time travel, compaction, and snapshot expiry (the GDPR-erasure caveat) |
| Concepts | Snapshot isolation, OCC, time travel, compaction, compliant deletion — in real Scala |
| How to Test | sbt test — 6 ScalaTest specs (runs here) |
| Resume bullet | Built an Iceberg-style table-commit library in Scala (optimistic concurrency, time travel, compaction, snapshot expiry) with ScalaTest coverage |
→ Lab folder: lab-02-scala-iceberg/
Extension Project A — Real Iceberg/Delta hands-on (spec)
With Spark + Iceberg (or Delta): create a table, append, MERGE (upsert), time-travel query,
OPTIMIZE/compact, expire_snapshots/VACUUM; inspect the metadata/manifests (or
_delta_log); evolve the schema and partition spec.
Extension Project B — Format selection memo (spec)
Iceberg vs Delta vs Hudi for a stated workload (append-heavy analytics vs upsert-heavy CDC vs multi-engine openness) with the deciding dial and an ADR (P15).
Integrated-Scenario Hooks
- This table format is the sink of the P04 Flink job (two-phase-commit → snapshot) and the P05 CDC apply (upserts).
- Its partition pruning + P08 layout is what makes Athena/Trino cheap (P10).
- Its time-travel-vs-deletion tension is a P14 privacy problem.
- It is the storage core of capstone Problems 1 and 3 (P16).
Guides in This Phase
Key Takeaways
- Table formats add ACID + snapshots + time travel over Parquet because S3 has no atomic rename.
- Snapshots are immutable (MVCC); writers use optimistic concurrency and retry on conflict.
- Compaction and snapshot expiry are mandatory maintenance, not optional tidiness.
- Iceberg = open multi-engine; Delta = Spark-native rich; Hudi = upsert/incremental specialist.
- Time travel and deletion are in tension — expiry is how you actually delete (P14).
Deliverables Checklist
- Lab 01 implemented; all 12 tests pass
- You can draw Iceberg's metadata→manifest list→manifest→data-file hierarchy
- You can explain optimistic concurrency and the conflict-retry loop
- You can choose Iceberg vs Delta vs Hudi for a workload
- Extension A hands-on or Extension B selection memo