👨🏻 Brother Talk — Phase 12, Off the Record
My favorite phase. Scala and functional programming scare people off, and that's exactly why being good at it is a moat. Let me get you over the hump.
Brother, I'll be straight: this is the phase people avoid, and that's precisely why it's worth your time. Functional Scala has a reputation — monads, "category theory," cryptic operators — that makes a lot of competent engineers quietly decide it's not for them. So the pool of data engineers who can actually build a clean, type-safe, resource-safe Scala platform library is small, and the JD is explicitly hunting for them ("more than basic Scala syntax… build reusable frameworks… prevent entire classes of bugs"). Push through the intimidation and you've got a skill most of your competition skipped. The moat is real.
Here's the reframe that dissolves the fear: you don't need category theory; you need a few practical patterns. Forget the scary words. The whole thing reduces to a handful of moves you can actually feel:
- "I want to combine validations and see all the errors" → that's
Validated, and the reason it accumulates is the only thing you need to know about "applicative." - "I want to chain steps where each needs the last, and stop on the first failure" → that's
flatMap/Either, and "it short-circuits" is the only thing you need to know about "monad." - "I want this connection to always close, even if everything explodes" → that's
Resource. - "I want to describe an effect now and run it later, and be able to test it without running
it" → that's
IO.
That's it. That's 90% of functional data engineering in practice. The lab makes you build
each one, and once you've built Resource and watched it release in LIFO order even when an
exception is thrown, the magic evaporates and it's just a clever, useful pattern. Build them.
The understanding-by-building is the whole point.
Now the single most powerful idea in this phase, the one I want tattooed on your brain: make
illegal states unrepresentable. Most engineering is "write code, then write tests/checks to
catch the bad cases." Functional type-driven design flips it: make the bad cases impossible to
even write. An OrderId that's a distinct type from UserId means you can never swap them —
not "you'll catch it in review," but it won't compile. An Amount with a private constructor
and a validating factory means a negative amount cannot exist anywhere in your program — ever,
by construction. When you build a platform library this way, you're not asking the hundreds of
engineers who use it to be careful. You're making carelessness impossible. That is the
highest form of the "prevent entire classes of bugs" mandate, and it's what separates a senior
who writes correct code from a principal who makes everyone's code correct by design.
The effect-system thing — IO/ZIO — trips people up, so let me give you the click. The
confusion is "why would I want my code to do nothing when I call it?" Because an effect that's
a value instead of an action is something you can hold, pass around, retry, combine, and
test. When database.write() runs immediately, you can't test the logic around it without
hitting a database. When database.write() returns an IO describing the write, you can
build the whole program as a value, inspect it, wrap it in retry, run it in tests with a fake,
and only actually perform it at the very edge. The laziness isn't a quirk — it's what makes
effectful code as composable and testable as pure code. The lab's "build an IO, prove it does
nothing until run, prove it runs each time" is that click in miniature. Feel it.
A practical warning that'll save you real pain: resource leaks and blocking-the-wrong-pool
are the two FP-in-production bugs that bite hardest. Use Resource for everything with a
lifecycle (connections, files, clients, the Iceberg writer) so cleanup is guaranteed — the day
you have a "we ran out of connections in prod" incident, it's because someone didn't. And never
block the compute thread pool with I/O; effect systems give you a separate blocking pool for a
reason, and mixing them up gives you the world's most confusing deadlock. These aren't theory;
they're the 2 a.m. pages of functional Scala.
On JVM tuning: don't glaze over it because it's not as elegant as the FP. The day a Flink job's checkpoints start timing out and you realize it's GC pauses from too much on-heap state — and you fix it by moving state off-heap and switching to ZGC — you'll have done something most "Scala developers" can't, because they treat the JVM as a black box. The role wants someone who debugs down to the GC and serialization layer. Knowing that Kryo-with-registered-classes is 10× faster than Java serialization on a shuffle, or that a low-pause collector saves your checkpoint SLA, is the kind of byte-level fluency that defines the seniority bar.
Career truth: Scala/FP fluency is rare and sticky. Rare because most people avoid it; sticky because once a company builds its platform on Cats/ZIO, they desperately need people who can maintain and extend it. Being the person who can build the SDK and explain it to the team so they can use it is a genuinely defensible, well-paid position. The intimidation factor that keeps others out is your advantage — walk through the door they won't.
Build the SDK. Make Resource release LIFO under an exception. Make Validated collect every
error. Make IO stay lazy. Read reference.scala and see the real Cats. Then come to P13,
where we make all of this observable and reliable in production — SLOs, lineage, data quality,
incident response.
— your brother 👨🏻