Phase 08 — Storage & Columnar Formats
Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 1.5 weeks (24–32 hours) Prerequisites: Phase 06 (small files), Phase 07 (partitions, EMRFS)
Why This Phase Exists
This is the byte level the JD demands you debug at: "Parquet internals: row groups, column chunks, pages, encodings, compression, statistics, predicate pushdown, dictionary encoding, and file sizing … Protobuf schema evolution … compare Avro, Protobuf, JSON, Parquet, and ORC … S3 consistency, object layout, request costs, prefix scaling, lifecycle policies, encryption, and cross-region replication." Everything above storage — query speed, scan cost, lakehouse performance — is determined by how the bytes are laid out here. The single most leveraged skill of the whole track is understanding why a columnar file with good stats and partitioning turns a 1 TB scan into a 10 GB one (a 100× cost difference).
Concepts
- S3 as the lake substrate: object storage (not a filesystem); strong read-after-write consistency (since 2020); prefix-based request scaling (~3,500 PUT/s, 5,500 GET/s per prefix); request costs; multipart upload; lifecycle policies (tiering to IA/Glacier); encryption (SSE-S3/KMS, P14); cross-region replication; the small-file & listing cost.
- HDFS (the on-prem ancestor): NameNode (metadata) + DataNodes (blocks), block size, replication factor — and why S3-as-lake replaced it (decoupled storage/compute, elasticity).
- Parquet internals: file → row groups → column chunks → pages; per-chunk statistics (min/max/null) enabling predicate pushdown; dictionary + RLE/ bit-packing encodings; compression (snappy/zstd/gzip); file sizing (128 MB–1 GB).
- ORC: stripes → row index → streams; built-in indexes + bloom filters; Hive-native; Parquet vs ORC tradeoffs.
- Avro (P03 revisited as storage): row-oriented, schema-carrying — great for ingestion/ write-heavy and schema evolution, poor for column-pruned analytics.
- Format selection: row vs columnar; Parquet/ORC (analytics) vs Avro (ingestion) vs JSON/ CSV (interchange) — costed, not by taste.
- Layout for skipping: sorting/clustering on filter columns to tighten row-group stats (the basis of Z-ordering, P09); partitioning vs the small-file/over-partitioning trap.
Labs
Lab 01 — Parquet Reader & Predicate Pushdown (flagship, implemented)
| Field | Value |
|---|---|
| Goal | Build a columnar format: row groups with min/max/null stats, predicate pushdown that skips groups, column projection, and dictionary/RLE encoding |
| Concepts | Columnar layout, row-group stats, pushdown asymmetry (proves absence), encodings |
| How to Test | pytest test_lab.py -v — 14 tests |
| Talking Points | Why do stats only prove absence? Why does sorting boost pushdown? Why is projection a cost feature? |
| Resume bullet | Implemented a columnar reader with row-group statistics, predicate pushdown, projection, and dictionary/RLE encoding |
→ Lab folder: lab-01-parquet-reader/
Extension Project A — Real Parquet inspection (spec)
Write Parquet with pyarrow; inspect the footer (parquet-tools/pyarrow): row-group
count, column stats, encodings, sizes. Sort on a column, rewrite, and observe tighter stats
and better skipping. Measure bytes read for a projected, filtered query.
Extension Project B — Format & layout bake-off (spec)
Same dataset as Parquet/ORC/Avro/JSON: compare file size, write time, and a column-projected filtered read time. Vary row-group size and file size; chart the small-file penalty.
Integrated-Scenario Hooks
- This phase's pushdown + sorting is what Iceberg/Delta exploit (P09) and what makes Athena/Trino cheap (P10).
- Its small-file lens closes the loop with P06 (Spark writes) and P07 (Hive partitions).
- Its S3 layout (prefixes, lifecycle, encryption) feeds P14 governance and P16 capstone.
Guides in This Phase
Key Takeaways
- The bytes' layout decides every downstream cost; this is the highest-leverage phase.
- Columnar = read only the columns and row groups you need; stats enable skipping.
- Stats prove absence, so sorting/clustering on filter columns is what makes skipping work.
- S3 is object storage with prefix scaling and request costs — small files hurt even idle.
- Pick formats by cost: Parquet/ORC for analytics, Avro for ingestion, JSON/CSV for interchange.
Deliverables Checklist
- Lab 01 implemented; all 14 tests pass
- You can draw Parquet's file→row-group→column-chunk→page hierarchy
- You can explain predicate pushdown and why sorting amplifies it
- You can state S3 prefix scaling limits and the small-file cost
- Extension A inspection or Extension B bake-off