🛸 Hitchhiker's Guide — Phase 09: Lakehouse Table Formats

Read this if: you want Iceberg/Delta/Hudi in your head fast — why they exist, how commits and time travel work, and how to choose. Skim, read WARMUP, build the table.

0. The 30-second mental model

A table format turns the Parquet files of P08 into a real database: ACID, snapshots, time travel. The trick — since S3 has no atomic rename — is an atomically-swapped metadata pointer: the table is "whatever the current metadata says," and a commit writes new metadata

  • swaps the pointer (a new immutable snapshot). One sentence: the lakehouse = atomic metadata swap over immutable file snapshots → ACID + MVCC + time travel on a data lake.

1. Why table formats exist

S3: no atomic rename, "directory" = prefix → can't safely say "the table is these files now"
fix: explicit metadata pointer, atomically swapped on commit → ACID
everything else (time travel, concurrent writers, evolution) follows from immutable snapshots

2. Iceberg hierarchy

catalog → metadata.json (current snapshot, schema, partition spec)
            └── snapshot → manifest list → manifests → data files (+ per-file stats)
manifests carry partition values + min/max → prune files WITHOUT listing S3 (vs Hive)

3. Concurrency = optimistic, not locked

writer: read base snapshot → stage files → commit IF current == base (atomic swap)
        else CONFLICT → re-read new snapshot, RETRY
readers: pin a snapshot (MVCC), never see half-written commits
→ Spark AND Flink AND compaction write one table, no global lock

4. Time travel + rollback

snapshots are immutable → query AS OF snapshot/timestamp (reproducibility, audit)
bad write? roll back = swap pointer to previous metadata (instant, no restore)
cost: snapshots accumulate → must EXPIRE them (bounds time travel, enables real delete)

5. Partitioning that doesn't betray you

HIDDEN partitioning: partition by transform days(ts)/bucket(id); query filters the COLUMN,
  Iceberg derives the prune → no "forgot the partition filter" footgun
partition EVOLUTION: change spec for NEW data, no rewrite of old
schema evolution by FIELD ID → safe rename/reorder

6. Maintenance is mandatory

compaction (rewrite_data_files / OPTIMIZE) → fix small files (P08), same data, new snapshot
clustering / Z-ORDER → tight per-file stats → pushdown actually skips (P08)
expire_snapshots / VACUUM → reclaim storage, bound time travel, enable real deletes
remove_orphan_files → clean failed commits
a table nobody compacts/expires becomes a swamp

7. Iceberg vs Delta vs Hudi

Iceberg → open, multi-engine (Spark/Flink/Trino/Athena), append-heavy analytics, evolution → default
Delta   → Spark/Databricks-centric; mature MERGE / OPTIMIZE / Z-ORDER
Hudi    → upsert/CDC-heavy; CoW (read-fast) vs MoR (write-fast, merge at read); record indexes
all three: ACID + time travel + compaction. Differences = ecosystem + upsert model.

8. Deletion vs time travel (the privacy trap)

DELETE removes a row from CURRENT snapshot, but old snapshots still reference its files
→ time travel can RESURRECT it → not GDPR-compliant until you EXPIRE those snapshots + vacuum
compliant erase = delete current → expire referencing snapshots → remove orphan files

9. Beginner mistakes that mark you

  1. Treating a lakehouse table as set-and-forget (no compaction/expiry schedule).
  2. Thinking DELETE = erased (time travel resurrects until expiry).
  3. Using a global lock instead of optimistic concurrency for concurrent writers.
  4. Hive-style "all files under the path" thinking (no atomic commit).
  5. Picking a format by hype instead of ecosystem + upsert pattern.
  6. Streaming tiny commits with no compaction → small-file swamp.

10. How this phase pays off later

  • 2PC → snapshot commit ← P04 exactly-once sink.
  • Partition pruning + clustering → P10 Athena/Trino cost.
  • Deletion vs time travel → P14 privacy/GDPR.
  • Compaction/expiry schedule → P13 automated maintenance.
  • Storage core of capstone Problems 1 & 3 (P16).

Read WARMUP, build the table, make the conflict-retry test pass, then P10: the query engines (Trino/Athena) that read all this — and how to optimize them.