Lab 01 — Iceberg-Style Table Format

Phase: 09 — Lakehouse Table Formats | Difficulty: ⭐⭐⭐⭐⭐ | Time: 6–8 hours

A "lakehouse" is just ACID + snapshots + time travel layered over the Parquet files of P08. This lab builds that layer: immutable snapshots (MVCC), optimistic-concurrency commits (so two writers don't corrupt the table), time travel, partition pruning, compaction, and snapshot expiry. After this, Iceberg/Delta/Hudi stop being magic.

What you build

  • IcebergTable with append / delete_where — every write creates a new immutable snapshot pointing at the live file set
  • commit — optimistic concurrency: a commit against a stale base snapshot raises ConflictException; the writer must re-read and retry (how concurrent engines stay correct without a lock)
  • snapshot / scan / read_records — time travel (read as of any snapshot) and manifest-based partition pruning
  • compact — rewrite many small files into one per partition, preserving the data (a new snapshot)
  • expire_snapshots — reclaim old snapshots (and break time travel to them)

Key concepts

ConceptWhat to understand
Snapshot / MVCCeach commit is immutable; readers see a consistent snapshot
Optimistic concurrencyconflicting writers detected at commit → retry (no lock)
Time travelquery AS OF a snapshot; delete doesn't erase history
Partition pruningskip files via manifest partition values (P08 at table level)
Compactionfix the small-file problem without changing the data
Snapshot expiryreclaim storage; the cost of unbounded time travel

Run

pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v

Success criteria

  • All 12 tests pass — test_commit_conflict_on_stale_base and test_delete_where_removes_then_time_travel_keeps certify the ACID/MVCC model.
  • You can explain how a DELETE coexists with time travel (old snapshots still reference the removed files).
  • You can explain why optimistic concurrency (not locking) is how Iceberg lets Spark and Flink write the same table.

Extensions

  • Add partition evolution: change the partition spec for new snapshots without rewriting old data (Iceberg's hidden partitioning) — scan must handle mixed specs.
  • Add merge-on-read vs copy-on-write deletes (Hudi/Delta): position/equality delete files vs rewriting data files — and the read-time merge cost tradeoff.
  • Add remove_orphan_files (files no snapshot references) and a MERGE (upsert) operation.
  • Track manifest stats (per-file min/max like P08) and prune at file level inside scan.