👨🏻 Brother Talk — Phase 05, Off the Record

The engine wars, CDC, and joins — where I'll save you from two expensive mistakes: over-engineering with Flink, and the leakage bug that quietly poisons your data.

Brother, let me start with a confession that'll save you grief: for a long time I reached for Flink for everything, because it was the powerful, impressive choice. And I paid for it — in operational toil, in 3 a.m. pages for a cluster that was overkill for the job, in onboarding pain for teammates who just wanted to enrich a stream. The lesson took too long: the most powerful tool is not the best tool; the best tool is the one that fits the job with the least operational weight. A principal who reaches for Flink reflexively is signaling the same immaturity as a junior who reaches for microservices reflexively.

So here's the engine maturity ladder, the way I think about it now. If you're already on Kafka and the job fits "run more app instances," Kafka Streams lets you delete an entire cluster from your life — your streaming logic lives in a normal app you deploy normally. If you already run Spark and your latency budget is seconds, Spark Structured Streaming means one engine, one API, one set of skills for batch and stream. If it's a single service's internal pipeline, Akka Streams or FS2 keeps it in-process where it belongs. And then, when you genuinely need millisecond latency with huge keyed state and the strongest exactly-once — then Flink earns its keep. Being able to say "we don't need Flink here, and here's the lighter thing that fits" is a more senior move than knowing every Flink knob.

Now, the bug. I'm going to plant this so deep you'll never commit it: temporal leakage in joins. It's the most innocent-looking, most damaging mistake in stream enrichment. You have events, you have a dimension table (users, products, accounts), you join them — and the obvious thing, the thing every tutorial shows, is to look up the dimension's current value. And it's wrong. Because the event happened in the past, and the dimension's value then may differ from now. A user who's "pro" today was "free" when they generated last month's events. If you label those old events "pro," every time-based analysis is corrupted, every model trained on that data learns a lie, and — the cruel part — nothing crashes. The dashboard looks fine. The model trains fine. It's just subtly, silently wrong, and you find out months later when a number doesn't reconcile. The fix is the as-of join: look up the value that was valid at the event's timestamp. Build it in this lab, feel why the point-in-time lookup matters, and carry that paranoia forever. (It's the same bug as feature-store leakage — the data world keeps re-discovering it under different names.)

Third thing: CDC is magic, and like all magic it has rules. Change Data Capture is one of the most powerful patterns you'll wield — it turns any database into an event stream without the dual-write problem, because you're capturing what the DB already committed rather than asking the app to write twice. But the apply side has to be correct or you get corruption that's miserable to debug. The rules: apply in LSN order, gate by version so duplicates and reordering are no-ops, and — the one people forget — keep tombstone memory so that after you delete a row, a stale older insert can't wander in and resurrect it. I've seen "deleted" customers reappear in a downstream table because someone's CDC apply forgot that last rule. It looks like a ghost. It's an ordering bug. Build the idempotent apply in the lab and you'll never ship the ghost.

A word on the functional libraries — Akka Streams and FS2. They'll feel out of place next to Flink and Spark, and I want you to not skip them, because they're a preview of the most intellectually rewarding phase in this whole track (P12). When you build the platform's in-service pieces — a validating ingestion sidecar, a CDC connector, an enrichment service — you'll often reach for exactly these, and they bring something the cluster engines don't: resource safety by construction. A stream that opens a connection cannot forget to close it, even on failure, because the type system and the runtime guarantee it. That's a different kind of correctness than checkpoints — it's correctness you get at compile time — and once you taste it, sloppy resource handling in other languages starts to feel barbaric. Hold that feeling; P12 turns it into a superpower.

Career note: the person who can look at a problem and say "this doesn't need a cluster" saves the org real money and real toil, and that gets noticed by the people who pay the cloud bill and carry the pager. Restraint reads as seniority. The flashy move is building the impressive distributed thing; the principal move is building the right-sized thing and being able to defend why it's right-sized with the workload's numbers. Practice saying no to Flink as confidently as you say yes.

Build the lab. Make CDC idempotent under a shuffled, duplicated stream. Make the as-of join refuse to leak. Then come to P06, where we cross the bridge from streaming to batch and take apart Spark — the engine that, when it misbehaves, generates the most expensive bills and the longest job runtimes in the business.

— your brother 👨🏻