Lab 02 — Storage Format Migration Planner
What You'll Build
A planner that computes the capability delta when migrating a Hive/Spark table from one storage format to another, and determines whether the migration step is rollback-safe.
Apache Context
Hive projects managing petabyte-scale tables face a recurring decision: migrate from older formats (Text, SequenceFile) to columnar formats (ORC, Parquet) for query performance, while preserving operational safety. The key insight is that not all capability gains are free — migrating from ORC to Parquet loses ACID transaction support, which in a Hive 3.x ACID table means you lose INSERT/UPDATE/DELETE semantics and must rewrite your ETL pipeline.
This lab models the static analysis that would precede running ALTER TABLE ... STORED AS PARQUET.
Running
mvn test
Expected: 11 tests, 0 failures.
Key Classes
| Class | Role |
|---|---|
StorageFormat | Enum: TEXT, SEQUENCE_FILE, ORC, PARQUET, AVRO |
FormatCapability | Enum: VECTORIZED_READ, PREDICATE_PUSHDOWN, SCHEMA_EVOLUTION, ACID_TRANSACTIONS, COLUMN_STATISTICS |
FormatCapabilityRegistry | Maps each StorageFormat to its capability set |
MigrationStep | from → to + gained + lost + isRollbackSafe() |
FormatMigrationPlanner | plan(from, to) — computes capability delta |
Capability Matrix
| Capability | TEXT | ORC | PARQUET | AVRO |
|---|---|---|---|---|
| Vectorized read | — | ✓ | ✓ | — |
| Predicate pushdown | — | ✓ | ✓ | — |
| Schema evolution | — | — | ✓ | ✓ |
| ACID transactions | — | ✓ | — | — |
| Column statistics | — | ✓ | ✓ | — |
Interview Angle
Walk through the ORC → Parquet migration trade-off. Parquet is preferred in the Spark ecosystem; ORC is preferred in the Hive ecosystem, especially when ACID transactions are required. Choosing wrong requires a rewrite. This is the kind of decision that lands in a PMC discussion and stays in the release notes.