👨🏻 Brother Talk — Phase 02, Off the Record

Kafka and Kinesis, the stuff the docs won't tell you because the docs are written by people who want you to succeed and aren't on your pager.

Brother, ingestion is where I've seen the most careers quietly dented, and it's because of one brutal property: you cannot fix ingestion downstream. If you lose an event at the front door, it's gone — no Flink job, no Spark backfill, no clever SQL resurrects it. If you let a garbage schema through, it poisons every table that reads from it. So the stakes here are higher than they look, and the work is less glamorous than Flink or the lakehouse. Do it right anyway. The people who skip the boring durability config are the people who later write the postmortem titled "How We Lost Six Hours of Payments."

Let me give you the hard-won stuff.

Config the durability knobs on day one, in writing. RF=3, acks=all, min.insync.replicas=2, unclean.leader.election=false. Memorise that line like your phone number. The default acks in some clients has changed over the years, and "we used the default" is not a sentence you want in an incident review. And unclean.leader.election — that innocent-looking boolean — is literally the CAP theorem as a config flag. If it's true on a topic that matters, you've signed up to silently lose acknowledged data when the ISR empties. I've seen it default-on bite a team hard. Check it. Write down why it's set the way it is. That's a 10-minute ADR that saves a 6-hour outage.

The poison message will come for you. Not might. Will. Some upstream team will ship a malformed payload, or a null where you expected a value, or an emoji in a field that your deserializer chokes on, and if your consumer does the naive thing — try, fail, retry forever — that one record wedges the whole partition and everything behind it starves. I've watched lag climb into the millions because of one bad record nobody could see. The fix is a mindset: every consumer assumes its input is hostile. Validate, and on failure, shove it in the DLQ with the reason attached and keep moving. Then — and this is the part people forget — alert on the DLQ rate, because a sudden spike in quarantined records is the earliest, cheapest signal that an upstream producer broke their contract. You find out in minutes from a clean alert instead of in days from a confused executive asking why a dashboard looks wrong.

Respect the partition key like it's load-bearing, because it is. Choosing the key feels like a throwaway decision and it's actually one of the most consequential things you'll do. It decides your ordering guarantee (all of one key's events stay ordered) and your skew (pick a low-cardinality key like country or event_type and you've built yourself a hot partition that no amount of hardware fixes). I once inherited a pipeline keyed by event_type with about eight types — eight hot partitions doing all the work while the others idled. The fix wasn't more brokers; it was re-keying. When someone proposes a partition key, immediately ask: "how many distinct values, and is the distribution skewed?" That one question marks you as someone who's been burned.

"Exactly-once" will be on a vendor slide in a meeting you're in. Be ready. Someone will say "we don't need to worry about duplicates, Kafka/Kinesis gives us exactly-once." The junior move is to nod. The senior move is to ask, gently, "exactly-once into where?" Because the guarantee is real inside Kafka (transactions, read_committed) and evaporates the moment you write to S3 or Cassandra or a feature store — which is, you know, the entire point of the pipeline. You re-earn it at the sink with idempotency. Saying this calmly in a meeting, without making anyone feel dumb, is exactly the kind of thing that gets you promoted, because you just prevented a class of bug the whole room was about to wave away.

Keep a raw copy. Always. The single most valuable insurance policy in data engineering is teeing your raw events to cheap storage (S3) the instant they arrive, before any processing, with long retention. Kafka retention is finite (often 7 days), and the day you have a bug in your processing logic that's been running for two weeks, the only thing that saves you is being able to replay from a raw copy that goes back far enough. "We can't reproduce last month's data" is the most demoralizing sentence in this job, and it's almost always caused by someone trusting the stream's retention as their source of truth. The stream is transport. S3 is memory. Don't confuse them.

Now the career note. Ingestion is unsexy, and that is exactly why owning it is a power move. Everyone wants to build the cool Flink job; nobody wants to own the schema-validation gateway and the DLQ redrive tooling and the partition-key standards. So if you become the person who owns the front door — the one who can say "no event is lost, every bad payload is quarantined and visible, and we can replay anything" — you become infrastructure in the org's mind. You're not "the person who built feature X," you're "the person the platform depends on." That's a much harder position to be laid off from, and a much easier one to be promoted from. Boring, load-bearing, and owned by you. That's the play.

Build the broker by hand. When you make the replay test work — seek back to zero and watch the lag jump and the events flow again — you'll feel why replay is the platform's superpower. That feeling is worth more than reading about it ten times.

Next stop, P03: we put a real contract on this front door, because an unvalidated topic is just an outage with a delay timer.

— your brother 👨🏻