👨🏻 Brother Talk — Phase 03, Off the Record

Schemas and contracts — the least glamorous phase, the one that quietly prevents more 2 a.m. pages than any other. Let me tell you why to care.

Brother, I'm going to be honest: schema governance sounds like the most boring possible topic. "Compatibility modes." "Field numbers." Your eyes want to glaze. Mine did too. And then I lived through the alternative, and now I evangelize this stuff like a convert, because unversioned schemas are the single most common cause of self-inflicted data outages I've ever seen, and they're 100% preventable.

Here's the scene that converted me. A producer team — good engineers, well-meaning — renamed a field and shipped it on a Tuesday afternoon. They tested it. It worked for them. What they didn't know is that under the hood a field number got shuffled, and forty downstream consumers — analytics jobs, ML features, a fraud model, a finance report — started silently deserializing garbage. Not crashing. Silently wrong. The fraud model degraded. The finance numbers drifted. It took days to trace it back, because the failure was nowhere near the cause. That entire week of pain, across a dozen teams, would have been a single red X on a pull request if a compatibility check had been wired into CI. One red X versus a week of cross-team archaeology. That's the whole pitch for this phase.

So let me give you the stuff that matters.

"It worked when I tested it" is the most dangerous sentence in data engineering. A producer tests against the current consumer, in the current deploy, with fresh data. But the real world has consumers on six different versions, historical data written months ago, and replay jobs reading last year's bytes. The schema contract is the only thing that reasons about all of those at once. When you internalize that producer and consumer are never on the same version — that they literally cannot be, at any real org — the whole idea of compatibility modes stops being academic and becomes obviously necessary.

Make the breaking change impossible to merge, not merely discouraged. Here's the thing about humans: if a rule is a wiki page, it gets violated. If a rule is a failing CI check, it gets followed. The entire value of a schema registry isn't the storage — it's that register() rejects incompatible changes, and when you wire that into the producer's CI, breaking the contract becomes literally un-shippable. You're not asking people to be careful. You're making carelessness impossible. That's the principal move: don't write a guideline, build a guardrail.

Field numbers are sacred. Treat retiring one like retiring a jersey. In Protobuf, the number is the identity. When you remove a field, reserve its number so nobody ever reuses it. I cannot tell you how many corruptions trace to "oh, number 7 was free, I'll use it" when number 7 was actually a retired field still present in old data and replay streams. Reserve it. Comment it. Never reuse it. This one discipline prevents a whole category of silent corruption.

Tag PII at the schema, today, even if nobody's asked. This is the unglamorous favor you do your future self. When you define an event, mark which fields are PII right there in the contract. Why now, when there's no governance system yet to consume the tags? Because P14's governance platform — masking, access control, GDPR deletion — is infinitely easier to build when the PII is already labeled at the source than when you have to go back and classify ten thousand fields across a thousand schemas under a compliance deadline. The tag costs you ten seconds at definition time and saves a future team a quarter of misery. Do it.

Accumulate your errors, because the human on the other end is a person fixing data. When you validate a record and it has six problems, tell them all six. The fail-fast pattern — report one error, stop — is fine for a compiler but cruel for a data producer, who then fixes one thing, resubmits, finds the next, and slowly loses their mind. The Validated pattern in this lab (and for real in Scala in P12) exists because someone realized validation is about helping a human fix everything at once. Small thing. Huge difference in how much people hate or love your platform.

Now the career angle, because it's real. The person who owns schema governance becomes the person who prevents the org's most embarrassing outages, and that's a reputation worth having. It's not flashy — nobody throws a party because a breaking change got caught in CI instead of in prod — but the senior people notice. They notice that since you built the gate, the "mystery data corruption" incidents dropped to near zero. They notice that onboarding a new consumer is now safe because contracts are explicit. Quiet competence in the load-bearing, unglamorous places is exactly what gets you trusted with the architecture decisions. The flashy stuff gets you applause; the boring guardrails get you authority.

Build the registry gate. Make a breaking change fail the test. Feel how good it is to know — not hope — that the bad change can't ship. Then come to P04, where we take these clean, validated, contract-checked events and do the genuinely hard thing: stateful, exactly-once, event-time stream processing with Flink.

— your brother 👨🏻