Warmup Guide — Lakehouse Table Formats

Zero-to-principal primer for Phase 09: why table formats exist, Iceberg/Delta/Hudi internals, snapshot isolation and optimistic concurrency, time travel, compaction and expiry, streaming+batch on one table, and the deletion-vs-time-travel tension.

Table of Contents

Chapter 1: Why Table Formats Exist

Recall P08: S3 is an object store with no atomic rename and "directories" that are just key prefixes. So "what files make up this table right now?" has no safe answer if you just ls a prefix — a concurrent writer mid-commit leaves you reading half-written state, and the old Hive convention ("the table is all files under this path") can't do atomic multi-file commits, row-level deletes, or time travel.

A table format fixes this with an explicit, atomically-swapped metadata pointer: the table is "whatever the current metadata file says," and a commit is "write new metadata, then atomically swap the pointer." That single atomic swap is what gives you ACID on a data lake — the defining lakehouse capability. Everything else (time travel, safe concurrent writers, schema evolution) follows from "the table is a versioned, atomically-updated pointer to a set of files," which the lab implements as a list of immutable snapshots.

Chapter 2: Iceberg Architecture

Iceberg's metadata hierarchy (the lab is a simplification of this):

catalog → table metadata.json (current snapshot id, schema, partition spec, snapshot list)
            └── snapshot → manifest list → manifest(s) → data files (+ per-file stats)
  • A snapshot is the table's state at a point in time: which data files are live. Each commit produces a new snapshot (immutable) and atomically updates the metadata pointer.
  • Manifests list data files with per-file stats (partition values, record counts, column min/max — P08's stats lifted to the table level), so the planner prunes files without listing S3 (a huge speedup over Hive's directory listing).
  • The catalog (Glue, Hive, Nessie, REST) holds the current-metadata pointer and provides the atomic swap.
  • Because snapshots are immutable and additive, you get time travel for free (just read an old snapshot's manifests).

Chapter 3: Snapshot Isolation and Optimistic Concurrency

How do Spark and Flink and a compaction job write the same table without corrupting it, and without a global lock?

  • Snapshot isolation (MVCC): every reader pins a snapshot and sees a consistent view; a concurrent writer creating a new snapshot doesn't affect in-flight reads (P01's snapshot isolation, applied).
  • Optimistic concurrency: a writer reads the current snapshot as its base, prepares its changes, and at commit checks "is the current snapshot still my base?" If yes, it atomically swaps in the new snapshot. If no (someone else committed first), it gets a conflict and must re-read the new snapshot and retry its changes against it. The lab's commit raises ConflictException on a stale base — exactly this.
  • Why optimistic, not locking? Locks across distributed writers on object storage are slow and fragile; optimistic concurrency is lock-free and works because most commits don't actually conflict (different partitions/files). Conflicts that do matter (two writers deleting the same file) are caught and retried. This is the mechanism behind "multiple engines, one table."

Chapter 4: Time Travel and Recovery

Because snapshots are immutable:

  • Time travel: query the table AS OF a snapshot id or timestamp — read that snapshot's manifests. Used for reproducibility ("the exact data the model trained on"), auditing, and debugging ("what did this table look like before the bad job?").
  • Recovery from bad writes: if a job corrupts the table, you roll back to the previous snapshot (atomic pointer swap to the old metadata) — instant recovery, no restore-from-backup. This is one of the lakehouse's best operational properties (the JD's "recover from bad writes").
  • The cost: snapshots and their data files accumulate → storage grows → you must expire old snapshots (Ch. 6), which is what bounds time travel (and enables real deletion, Ch. 8).

Chapter 5: Partitioning, Hidden Partitioning, and Evolution

  • Hidden partitioning (Iceberg): you partition by a transform of a column (days(ts), bucket(16, id)) and Iceberg records the partition value in metadata — queries filter on the column (WHERE ts > ...) and Iceberg derives the partition prune automatically. No more "you forgot to also filter on the partition column" (Hive's classic footgun) and no extra partition column to maintain.
  • Partition evolution: change the partition spec (e.g. days → hours) for new data without rewriting old data — old files keep their old spec, new files use the new one, and the planner handles the mix. Hive couldn't do this; it required a full rewrite.
  • Schema evolution: add/drop/rename/reorder columns by field id (not name/position), so renames and reorders are safe and old data reads correctly (P03's evolution discipline, applied to tables).

Chapter 6: Compaction, Clustering, and Expiry

The maintenance that keeps a lakehouse fast (and the JD's explicit asks):

  • Compaction (rewrite_data_files / Delta OPTIMIZE): merge many small files into fewer target-sized ones (P08's small-file fix), producing a new snapshot with the same data. Essential for streaming-written tables (many tiny commits) and for keeping scan cost down (P10). The lab's compact collapses per partition.
  • Clustering / Z-ordering / sorting: lay data out so the columns people filter on have tight per-file stats (P08 Ch. 6) — so pushdown/file-pruning actually skips. Z-order multi-dimensionally clusters several columns at once.
  • Expire snapshots (expire_snapshots / Delta VACUUM): delete old snapshots and the data files only they referenced — reclaim storage. This bounds time travel (you can't travel to an expired snapshot — the lab tests this) and is the mechanism that makes deletion real (Ch. 8).
  • Remove orphan files: delete files no snapshot references (e.g. from failed commits).

The principal point: a lakehouse table is not "set and forget" — it needs a maintenance schedule (compaction + expiry + clustering), usually automated (P13). A table nobody compacts becomes a small-file swamp; a table nobody expires grows storage and can't truly delete.

Chapter 7: Delta Lake and Hudi

The other two major formats — know the differences:

  • Delta Lake: the table state is a transaction log — an ordered series of JSON commit files in _delta_log/ (each adds/removes files), with periodic Parquet checkpoints to avoid replaying the whole log. Rich Spark-native features: MERGE (upsert), OPTIMIZE + Z-ORDER, VACUUM, time travel, change data feed. Historically Databricks/Spark-centric; now open with growing multi-engine support.
  • Apache Hudi: built for upserts/incremental from day one.
    • Copy-on-Write (CoW): an update rewrites the whole data file → reads are fast (read-optimized), writes are heavier.
    • Merge-on-Read (MoR): an update writes a small delta log file → writes are fast, reads merge base + deltas (slower reads until compaction). The write-optimized choice for high-frequency CDC.
    • A timeline of actions (commits, compactions, cleans) + record-level indexes for fast upserts.

The selection heuristic (Extension B):

  • Iceberg → open, multi-engine (Spark/Flink/Trino/Athena all first-class), big append-heavy analytical tables, partition/schema evolution — the safe "open lakehouse" default.
  • Delta → you're Spark/Databricks-centric and want its mature MERGE/OPTIMIZE/Z-ORDER.
  • Hudi → upsert/CDC-heavy with low-latency write needs (MoR), record-level updates.

All three give ACID + time travel + compaction; the differences are ecosystem, upsert model, and engineering culture — not "which is correct."

Chapter 8: Streaming + Batch, and Deletion vs Time Travel

  • Streaming + batch on one table: a Flink/Spark streaming job writes via an exactly-once sink (P04's two-phase commit → commit an Iceberg/Delta snapshot atomically), while batch jobs read consistent snapshots and backfill — one table, no Lambda dual-system. This is the lakehouse's headline promise: end the streaming/batch storage split.
  • Deletion vs time travel — the privacy gotcha: a DELETE (or GDPR erasure) removes a row from the current snapshot, but old snapshots still reference the data files containing it, so time travel can resurrect it. The deletion is only real once you expire the snapshots (and remove the orphaned files) that still point at it. So a compliant "delete this user's data" is: delete from current + ensure no retained snapshot references it + run expiry/vacuum. Forgetting the second half is a genuine compliance bug (P14). The same immutability that gives you time travel is the thing that makes deletion a two-step process.

Lab Walkthrough Guidance

Lab 01 — Iceberg-Style Table Format, suggested order:

  1. _commit + append — snapshots accumulate the live file set; parent links.
  2. snapshot + scan + read_records — time travel + partition pruning.
  3. delete_where — current loses the files; an old snapshot still sees them (MVCC).
  4. commit (optimistic) — stale base → ConflictException; retry against current succeeds.
  5. compact — one file per partition, records preserved.
  6. expire_snapshots — drop old (keep current); time travel to expired raises.

Success Criteria

You are ready for Phase 10 when you can, from memory:

  1. Explain why table formats exist (no atomic rename → atomic metadata swap → ACID).
  2. Draw Iceberg's metadata→manifest-list→manifest→data-file hierarchy.
  3. Explain snapshot isolation + optimistic concurrency + the conflict-retry loop.
  4. Explain time travel and snapshot-based rollback recovery.
  5. Explain hidden partitioning and partition/schema evolution without rewrite.
  6. Explain compaction, clustering, and snapshot expiry — and why they're mandatory.
  7. Contrast Iceberg/Delta/Hudi (incl. Hudi CoW vs MoR) and pick one.
  8. Explain the deletion-vs-time-travel tension and a compliant delete.

Interview Q&A

Q: How do Spark and Flink write the same table at the same time without corruption? Optimistic concurrency over immutable snapshots. Each writer reads the current snapshot as its base, stages new data files, and at commit checks that the current snapshot is still its base; if so it atomically swaps in a new snapshot, if not it gets a conflict and re-reads + retries against the new snapshot. Readers always pin a consistent snapshot (MVCC), so they never see a half-written commit. There's no global lock — it works because most commits touch different files and genuine conflicts are caught and retried. That's the whole "multiple engines, one table" story.

Q: A user invokes GDPR erasure. You ran DELETE. Are you compliant? Not yet. DELETE removes the row from the current snapshot, but the data files containing it are still referenced by older snapshots, so time travel can resurrect the data. Real erasure requires the second step: ensure no retained snapshot references those files and run snapshot expiry + orphan-file removal so the underlying files are actually deleted. So a compliant flow is delete-from-current → expire the snapshots that still reference the data → vacuum. The same immutability that gives time travel makes deletion a two-phase operation — and forgetting phase two is a real compliance bug.

Q: Iceberg vs Delta vs Hudi — how do you choose? By ecosystem and write pattern, since all three give ACID + time travel + compaction. Iceberg is the open, multi-engine default — Spark, Flink, Trino, Athena are all first-class, and its hidden partitioning + partition evolution are excellent for big analytical tables. Delta if we're Spark/Databricks-centric and want its mature MERGE/OPTIMIZE/Z-ORDER. Hudi if the workload is upsert/CDC-heavy with low write latency — its Merge-on-Read writes deltas fast and merges at read, and record-level indexes make upserts cheap. I'd write the ADR on those axes, not on "which is best."

Q: Your streaming job writes a tiny file every few seconds and queries got slow. Fix? That's the small-file problem from frequent streaming commits. Two parts: (1) batch the writes more (larger commit intervals / buffering) so each commit is fewer, bigger files; (2) run compaction (rewrite_data_files / OPTIMIZE) on a schedule to merge the small files into target-sized ones — a new snapshot with the same data. Also expire old snapshots so the rewritten-away files get reclaimed. Pair compaction with clustering/Z-order on the filter columns so the compacted files also prune well (P08). Compaction + expiry on a schedule is mandatory maintenance for any streaming-written lakehouse table.

References

  • Apache Iceberg spec and Apache Iceberg: The Definitive Guide (Aghajanyan et al.)
  • Delta Lake docs and the Delta Lake VLDB paper (Armbrust et al., 2020)
  • Apache Hudi docs — CoW vs MoR, timeline, indexes
  • Armbrust et al., Lakehouse: A New Generation of Open Platforms (CIDR 2021)
  • Kleppmann, DDIA Ch. 7 (snapshot isolation, MVCC)