🛸 Hitchhiker's Guide — Phase 08: Storage & Columnar Formats

Read this if: you want S3 + Parquet internals fast, and to understand why one query scans 1 TB and another scans 10 GB of the same data. Skim, read WARMUP, build the reader.

0. The 30-second mental model

The lake lives in S3 (object store: prefix scaling, request costs, no atomic rename). Data is stored columnar (Parquet/ORC): read only the columns you need, and skip row groups whose min/max stats prove they can't match your filter. One sentence: the byte layout — columnar + tight stats from sorting + right file sizes — decides every downstream query's speed and cost.

1. S3 truths

object store, NOT a filesystem → no cheap rename (copy+delete) → table formats exist (P09)
strong read-after-write consistency since 2020 (the old "eventual" footguns are gone)
scaling ≈ 3,500 PUT/s + 5,500 GET/s PER PREFIX → spread prefixes, few large files
costs: storage $/GB-mo + REQUESTS per 1000 (small files cost even idle) + egress
lifecycle policies → auto-tier cold data to IA/Glacier

2. Why S3 beat HDFS

HDFS: NameNode (metadata SPOF, dies on millions of small files) + DataNodes; storage+compute COUPLED
S3:   decoupled storage/compute → scale independently, spin clusters up/down, ~infinite, 11 nines
trade: no data locality (network reads), no atomic rename

3. Parquet hierarchy

File → Row Group (~128MB) → Column Chunk → Page (~1MB)
+ per-chunk stats: min, max, null_count   ← the skip metadata
footer = schema + row-group metadata + stats + offsets
reader: read footer → project columns → skip row groups via stats → decode pages

4. Why columnar wins analytics

projection: read 2 of 50 columns ≈ 4% of bytes
compression: a homogeneous column compresses 10× (mixed rows barely)
skipping: per-group stats skip non-matching data

5. Encodings (the compression magic)

dictionary encode (low-cardinality → int codes) → RLE + bit-packing (runs/sorted) → snappy/zstd
SORTING the data first makes runs form → multiplies compression AND pushdown

6. Pushdown — the asymmetry that matters

stats prove ABSENCE, never presence:
  group min=60,max=69, WHERE id=55 → SKIP (proven absent)
  group min=50,max=59, WHERE id=55 → SCAN + row-filter (can't rule out)
effectiveness ∝ how TIGHT per-group min/max are ∝ DATA LAYOUT (sorting!)
random data → every group spans full range → pushdown skips NOTHING

7. Format picker

Parquet  → analytics lake (Spark/Trino/Athena)
ORC      → Hive-centric analytics (bloom filters, strong indexes)
Avro     → ingestion/streaming, schema evolution (row, schema-carrying)
JSON/CSV → interchange/debug/external; costly at analytics scale
pattern: Avro on the wire/landing → Parquet (sorted, right-sized) on the lake

8. File sizing

target 128 MB – 1 GB
too small → small-file tax (metadata, S3 requests, listing, NameNode, scan cost)
too big   → poor parallelism, memory pressure

9. Beginner mistakes that mark you

  1. Expecting pushdown to work on randomly-ordered data (it can't skip).
  2. Millions of tiny files under one prefix → throttling + request cost.
  3. JSON/CSV for big analytics → 10× the scan bill.
  4. Avro for column-pruned analytics / Parquet for row-by-row ingestion (against the grain).
  5. Assuming S3 rename is cheap (it's copy+delete → use table formats).
  6. Citing "S3 is eventually consistent" (it's strongly consistent since 2020).

10. War stories

  • "Filter on indexed column, full scan anyway." → data not sorted on it; stats overlap.
  • "Athena bill huge for a 'small' query." → millions of small files + no partitioning.
  • "Spark commit corrupted on S3." → no atomic rename; use a table format (P09).

11. How this phase pays off later

  • Sorting → tight stats → Z-order/clustering (P09), Athena cost (P10).
  • No atomic rename → why Iceberg/Delta commits exist (P09).
  • Small files → compaction (P09), cost program (P13).
  • Encryption/prefixes/lifecycle → governance (P14).

Read WARMUP, build the reader, sort the data and watch skips jump, then P09: table formats that add ACID, snapshots, and time travel on top of these files.