Lab 02 — Iceberg-Style Table Commits in Scala

Phase: 09 — Lakehouse Table Formats | Difficulty: ⭐⭐⭐⭐⭐ | Time: 5–6 hours

The Scala twin of Lab 01's Python table — the lakehouse commit protocol as a JVM library. Immutable snapshots, optimistic-concurrency commits (compare-and-swap on the current snapshot → serializable, atomic, conflict-detecting), time travel, compaction, and snapshot expiry (the GDPR-erasure caveat). ScalaTest-verified.

What you build (IcebergTable)

  • commit / append / overwrite — each produces a new immutable Snapshot; OCC via expectedBase (stale base → ConcurrentCommitException, caller refreshes and retries)
  • scan / scanAsOf — read current, or time-travel to any snapshot
  • compact — rewrite many data files into one (same rows) → fixes small files
  • expireSnapshots — prune history; required for a compliant delete (else time travel resurrects deleted rows — P14)

Run

sbt test

Expected: 6 tests, all green in IcebergTableSpec — including the OCC conflict+retry and the overwrite→time-travel→expiry erasure sequence.

Key concepts (Scala expression)

ConceptScala
Snapshot isolationa table is a list of immutable Snapshots
Optimistic concurrencycommit(expectedBase, ...) CAS; conflict → retry
Time travelscanAsOf(snapshotId)
Compactionmerge currentFiles → one DataFile, same rows
Compliant erasureoverwrite + expireSnapshots (delete alone isn't enough)

Success criteria

  • sbt test → all 6 specs pass.
  • You can explain why two writers off the same base can't both commit (OCC), and the retry.
  • You can explain why a DELETE/overwrite isn't a complete GDPR erasure until snapshots expire.

Extensions

  • Add partition/hidden-partitioning metadata to DataFile and prune in scan by predicate.
  • Add manifest files (a metadata layer between snapshot and data files) — Iceberg's real structure — and stats-based file skipping (P08 pushdown).
  • Add MERGE/upsert (key-based) producing position/equality deletes.