👨🏻 Brother Talk — Phase 08, Off the Record

The bytes. The least sexy phase, the highest leverage. Let me convince you why this is where the real money and the real speed live.

Brother, I'm going to make a claim that sounds like exaggeration and isn't: this phase — the boring one about file formats and storage — is the single highest-leverage thing in the entire track. Here's why. Everything above it (Spark, Trino, Athena, the lakehouse) is just reading these bytes. If the bytes are laid out well, every query above is fast and cheap. If they're laid out badly, no amount of cluster tuning saves you. I've watched teams throw bigger warehouses and more nodes at slow queries for months, when the actual fix was "your data isn't sorted and you have a million tiny files." Layout beats hardware. Every time.

Let me give you the one insight that, if you truly get it, makes you dangerous: predicate pushdown only works if the data is sorted on what you filter. Everyone learns "Parquet has statistics and does pushdown" and assumes that's automatic magic. It isn't. The statistics are per-row-group min/max, and they can only help you skip a group if that group's range doesn't overlap your filter. If your data is randomly ordered, every single row group spans the entire range of values, every group's min is near the global min and max near the global max, and so nothing can ever be skipped. Pushdown does literally nothing. The exact same data sorted on the filter column gives each group a tight, disjoint range, and suddenly a point query skips 99% of the file. Same data. Same format. Same query. 100× difference, purely from sort order. When you internalize this — when "why is this query slow?" reflexively makes you ask "is the data sorted on the predicate?" — you've gained an instinct most senior engineers don't have.

The small-file thing I keep hammering across phases lives here, at its source, so let me say the quiet part: small files are a tax you pay forever, levied by past-you on future-everyone. A Spark job spits out a million 1 MB files because nobody set the output partitioning, the job finishes fine, everyone moves on — and then for months, every Athena query against that table pays extra (more files to open, more requests, $/TB inflated by overhead), the metastore struggles to list them, and the S3 request bill quietly accumulates. The person who created the mess felt nothing; everyone downstream pays rent. Be the person who controls file size on write, and be the person who notices a small-file problem and schedules compaction. It's unglamorous and it saves real money.

Here's a thing that'll date you in an interview if you get it wrong: S3 is strongly consistent now. Has been since late 2020. For years we built elaborate workarounds (S3Guard, consistency layers) because S3 was eventually consistent for listings, and a lot of older engineers still carry that mental model. If you say "well, S3 is eventually consistent so we need to be careful about..." in 2026, you've just signaled your knowledge is five years stale. It's strongly read-after-write consistent. Know it. (And conversely: S3 still has no atomic rename — that hasn't changed, and it's the reason table formats exist, which is the whole next phase.)

The format choice — Avro vs Parquet — trips people up because they think one must be "better." They're not competitors; they're for different sides of the pipeline. Avro on the wire and at landing (row-oriented, schema-carrying, write-whole-records, evolves gracefully — perfect for Kafka and ingestion). Parquet on the lake (columnar, projects and skips, compresses hugely — perfect for analytics). The architecture that confuses these — Parquet for high-frequency row writes, or Avro for big column-pruned scans — fights the grain of each format and pays for it. The clean pattern is: ingest as Avro, then a job rewrites to sorted, right-sized Parquet for the analytical lake. Say that in a design review and you sound like you've built this before.

One more, because it's a quiet superpower: learn to read a Parquet footer. pyarrow or parquet-tools will show you row-group count, per-column min/max stats, encodings, and sizes. When a query is slow, open the file and look. Are the row groups huge or tiny? Do the stats for the filter column have tight ranges or do they all span the full range (= not sorted = no skipping)? Is it dictionary-encoded? Most engineers debug query performance by staring at the query plan and guessing; the ones who open the actual files find the real problem in minutes. That habit — go to the bytes — is what "debug at the byte and file level" (the JD's exact words) means in practice.

Career note: storage and format expertise is invisible until it isn't. Nobody throws a parade for "good file layout." But the engineer who can cut a query from 90 seconds to 3 by rewriting a table sorted and compacted, and explain exactly why with the footer stats, is doing something most people treat as black magic. Being the person who demystifies the bytes makes you the person other engineers bring their performance problems to — and that's authority.

Build the reader. Sort the data and watch the skipped-row-group count jump in the test. Feel the layout lesson in your hands. Then come to P09, where we put ACID, snapshots, and time travel on top of these files — the lakehouse, the thing that makes the lake a real database.

— your brother 👨🏻