Lab 01 — Parquet Reader & Predicate Pushdown

Phase: 08 — Storage & Columnar Formats | Difficulty: ⭐⭐⭐⭐☆ | Time: 5–7 hours

Parquet's magic — "it only reads the data it needs" — is just row-group statistics + predicate pushdown + column projection. This lab builds a working columnar format so those words become code: a file of row groups, each with per-column min/max/null stats, a reader that skips groups whose stats prove no match, and dictionary/RLE encoding.

What you build

  • dictionary_encode / rle_encode / rle_decode — the encodings that shrink columnar data (dictionary for low-cardinality, RLE for runs)
  • build_parquet — rows → row groups → columnar chunks with min/max/null stats (the footer metadata real Parquet keeps)
  • row_group_can_match — the pushdown predicate: skip a group whose stats prove no row matches (stats prove absence, never presence)
  • scan — projection + pushdown + row-level filter, reporting row_groups_scanned vs row_groups_skipped so you see the I/O saved

Key concepts

ConceptWhat to understand
Columnar layoutfile → row groups → column chunks → pages; read only needed columns
Row-group statsmin/max/null per chunk → skip chunks that can't match
Pushdown asymmetrystats prove absence, not presence → passing groups still filter
Dictionary/RLElow-cardinality + runs compress hugely; the columnar advantage
Projectionreading 2 of 50 columns reads ~4% of the bytes

Run

pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v

Success criteria

  • All 14 tests pass — test_scan_skips_groups_via_pushdown is the headline (9 of 10 groups skipped for id == 55).
  • You can explain why a group passing row_group_can_match still needs row-level filtering.
  • You can explain why sorting data on the filter column makes pushdown dramatically more effective (tight min/max per group).

Extensions

  • Add bloom filters per row group for high-cardinality equality (where min/max can't prune) — ORC's trick.
  • Add page-level stats (a finer skip granularity) and measure the extra pruning.
  • Sort the data on the predicate column before build_parquet and show row_groups_skipped jump — the data-layout lesson behind Z-ordering/clustering (P09).
  • Estimate bytes read (projected columns × scanned groups) vs a row-store full read — the columnar cost win (P10 Athena $/TB).