Warmup Guide — Storage & Columnar Formats

Zero-to-principal primer for Phase 08: S3/HDFS as the lake substrate, Parquet and ORC internals (row groups, pages, stats, encodings), Avro vs columnar, and the data-layout decisions that determine every downstream query's speed and cost.

Table of Contents

Chapter 1: S3 — Object Storage as a Lake

S3 is the substrate of the modern lake, and it is object storage, not a filesystem — a distinction that explains many gotchas:

  • Objects, not files: a flat key→bytes store. "Directories" are a fiction (key prefixes); there's no cheap rename (a rename is a copy + delete — which is why naive Spark/Hadoop commit protocols are slow/unsafe on S3 and why table formats, P09, exist).
  • Consistency: since Dec 2020, S3 has strong read-after-write consistency for all operations — the old "eventual consistency" footguns (S3Guard, etc.) are gone. Know this is current; interviewers test whether you're up to date.
  • Request scaling: throughput scales per prefix — roughly 3,500 PUT/s and 5,500 GET/s per prefix, and S3 auto-scales as you spread keys across prefixes. A million tiny files under one prefix → throttling; spreading prefixes (and fewer, larger files) → speed.
  • Costs: storage ($/GB-month by tier), requests (PUT/GET per 1,000 — why small files cost money even idle), and data transfer (cross-region/egress). Lifecycle policies tier cold data to IA/Glacier automatically.
  • Security (P14): SSE-S3 / SSE-KMS encryption at rest, bucket policies, VPC endpoints, cross-region replication for DR.

Chapter 2: HDFS and Why S3 Won

HDFS was the original lake storage: a NameNode holds the filesystem metadata (the directory tree, block→DataNode map) and DataNodes store fixed-size blocks (default 128 MB), replicated (default 3×) for durability. It's fast for colocated compute (data locality) but has two fatal-at-cloud-scale problems: the NameNode is a memory-bound single point (millions of small files = NameNode heap death — the small-file problem's origin) and storage is coupled to compute (scaling one means scaling the other).

S3 won the lake because it decouples storage from compute (scale independently; spin clusters up/down against durable storage), is effectively infinite and 11-nines durable, and is cheaper. The cost: no data locality (compute reads over the network) and no atomic rename (→ table formats). Know HDFS to understand what EMR/Spark inherited and what you're migrating from (P07/P15).

Chapter 3: Row vs Columnar

The fundamental storage choice:

  • Row-oriented (CSV, JSON, Avro): all of row 1, then all of row 2… Great for writing whole records and reading whole rows (OLTP, ingestion). Bad for analytics: to sum one column you read every column of every row.
  • Columnar (Parquet, ORC): all of column A, then all of column B… Great for analytics:
    • Projection: reading 2 of 50 columns reads ~4% of the bytes.
    • Compression: a column is homogeneous (all ints, all of one enum) → compresses far better than mixed rows.
    • Skipping: per-chunk stats let you skip data that can't match (Ch. 6). Analytics is overwhelmingly "aggregate a few columns over many rows," so columnar wins — which is why the lakehouse is built on Parquet/ORC.

Chapter 4: Parquet Internals

Parquet's hierarchy (the lab builds it):

File
└── Row Group (≈128 MB of rows — the unit of parallelism & skipping)
    └── Column Chunk (one column's data for this row group)
        └── Page (≈1 MB — the unit of encoding/compression/IO)
    + Column statistics per chunk: min, max, null_count, (distinct_count)
File footer: schema + row-group metadata + column stats + offsets
  • A reader reads the footer first (metadata), then reads only the column chunks it needs (projection) from only the row groups whose stats can match (pushdown, Ch. 6) — often a tiny fraction of the file.
  • Row-group size trades parallelism vs skipping granularity vs memory; ~128 MB is typical.
  • Repetition/definition levels encode nested/optional fields (Dremel encoding) — how Parquet stores arrays/maps/structs and nulls compactly.

Chapter 5: Encodings and Compression

Two layers, and the order matters:

  1. Encoding (logical, type-aware): dictionary encoding maps distinct values → small integer codes (huge win for low-cardinality columns — the lab's dictionary_encode), then RLE + bit-packing on the codes (huge win for runs/sorted data — the lab's rle_encode). Delta encoding for sorted integers/timestamps.
  2. Compression (byte-level): snappy (fast, default), zstd (better ratio, now often preferred), gzip (smaller, slower). Applied to the already-encoded pages.

The lesson: columnar + dictionary + RLE is why a column compresses 10× where the same data as JSON rows barely compresses — and why sorting the data first (so runs form) multiplies the win. This is also a cost lever (fewer bytes = cheaper scans, P10).

Chapter 6: Predicate Pushdown and Why Sorting Matters

Predicate pushdown = the reader uses per-row-group statistics to skip groups that can't contain a matching row. For WHERE id = 55, a group with min=60, max=69 is skipped without reading its data. The lab's row_group_can_match is exactly this.

The crucial asymmetry: statistics prove ABSENCE, not presence. A group with min=50, max=59 might contain id=55, so it's scanned and row-filtered; a group with min=60 can't, so it's skipped. This is why pushdown's effectiveness depends entirely on how tight the per-group min/max ranges are — and that depends on data layout:

  • If the data is sorted by id, each row group covers a tight, disjoint id range, so a point/range query skips almost everything. Massive win.
  • If id is randomly distributed, every group's [min, max] spans the whole range, so nothing can be skipped. Pushdown does nothing.

This is the entire motivation for sorting / clustering / Z-ordering (P09): you lay data out so the columns people filter on have tight per-group stats. "Why is my Parquet query still slow despite pushdown?" is almost always "your data isn't sorted on the filter column."

Chapter 7: ORC, Avro, JSON, CSV — Choosing a Format

FormatOrientationBest forAvoid for
Parquetcolumnaranalytics on the lake; Spark/Trino/Athenatiny row-by-row writes
ORCcolumnarHive-centric analytics; has bloom filters + strong indexesnon-Hadoop ecosystems sometimes
Avrorowingestion/streaming, schema evolution, write-heavycolumn-pruned analytics
JSONrow, textinterchange, debugging, external APIshigh-volume analytics (cost)
CSVrow, texthuman/legacy interchangeanything typed or large
  • Parquet vs ORC: both excellent columnar formats; ORC has historically stronger built-in indexes/bloom filters and is Hive-native; Parquet has the broader ecosystem. Pick by ecosystem; don't agonize.
  • Avro's niche: row-oriented + schema-carrying makes it ideal for the ingestion side (Kafka, write-heavy, evolving schemas — P03) where you write whole records and rarely column-prune. The pattern: Avro on the wire/landing, Parquet on the lake.

Chapter 8: File Sizing and the Small-File Problem

The recurring villain, now at its source:

  • Target file size: 128 MB – 1 GB. Too small → the small-file tax (metadata blowup, S3 request costs, listing slowness, poor scan throughput, HDFS NameNode pressure). Too large → poor read parallelism, memory pressure, slow single-file operations.
  • Where small files come from: Spark writing one file per partition per task (P06), over-partitioning a Hive table (P07), streaming sinks committing frequently (P04), and per-record/per-minute micro-writes.
  • Fixes: repartition/coalesce to target before writing (P06); compaction jobs (P09 table formats automate this); right-sized partitioning; batching streaming writes.
  • The cost framing: small files inflate scan cost (Athena reads/opens more, $/TB up), request cost (more GETs), and operational cost (slow listing, metastore pressure) — so compaction is a cost program (a JD deliverable), not just tidiness.

Lab Walkthrough Guidance

Lab 01 — Parquet Reader & Predicate Pushdown, suggested order:

  1. dictionary_encode / rle_encode / rle_decode — the encodings (and roundtrip).
  2. _column_stats + build_parquet — chunk into row groups, compute min/max/null.
  3. row_group_can_match — the pushdown logic; remember stats prove absence; all-null group skipped; != not prunable.
  4. scan — skip via pushdown, then row-filter + project; report scanned/skipped.
  5. Then the extension: sort the data on the predicate column, rebuild, and watch row_groups_skipped jump — Ch. 6 made tangible.

Success Criteria

You are ready for Phase 09 when you can, from memory:

  1. Explain S3 as object storage: prefix scaling, request costs, no atomic rename, strong consistency.
  2. Explain why S3 (decoupled storage/compute) replaced HDFS.
  3. Contrast row vs columnar and why analytics wants columnar (projection, compression, skipping).
  4. Draw Parquet's file→row-group→column-chunk→page hierarchy and the role of stats.
  5. Explain dictionary + RLE encoding and why sorting amplifies compression and pushdown.
  6. Explain why pushdown proves absence and why data layout (sorting) determines its value.
  7. Pick a format for ingestion vs analytics vs interchange, and state the file-size target.

Interview Q&A

Q: Your Athena query has a WHERE on an indexed column but still scans the whole table. Why? Almost certainly the data isn't sorted/clustered on that column, so every Parquet row group's [min,max] for it spans the full range and predicate pushdown can skip nothing — stats only prove absence, and with overlapping ranges they can't. Confirm by inspecting row-group stats; the fix is to rewrite the data sorted (or Z-ordered, P09) on the filter column so each group covers a tight, disjoint range, plus partition on a coarse column (date) for directory-level pruning. Also check it's not a small-file problem inflating per-file overhead. The point: columnar skipping is a layout property, not an automatic one.

Q: When would you store data as Avro instead of Parquet? On the ingestion/write side. Avro is row-oriented and schema-carrying, which suits whole- record writes, streaming, and schema evolution (P03) — you rarely column-prune a Kafka topic. Parquet is columnar and shines on the read/analytics side where you project a few columns over many rows. The common architecture is Avro on the wire and at landing, then a job rewrites to Parquet (sorted, right-sized) for the analytical lake. Using Parquet for high-frequency row-by-row ingestion, or Avro for big analytical scans, is using each against its grain.

Q: A pipeline writes millions of 1 MB Parquet files. What's wrong and what does it cost? That's the small-file problem: each file carries footer/metadata overhead, S3 charges per GET/PUT (so even idle they cost money), listing is slow, and scans can't reach good throughput — and downstream Athena/Trino pay more per query ($/TB plus per-file overhead). It usually comes from too many Spark output partitions/tasks or over-partitioning. The fix is to target 128 MB–1 GB files (coalesce/repartition before write), partition sensibly (date, not user_id), and run compaction — ideally via a table format (P09) that automates it. It's a cost program, not housekeeping.

References