🛸 Hitchhiker's Guide — Phase 03: Serialization, Schema & Data Contracts

Read this if: you want Protobuf/Avro/JSON and the compatibility rules in your head fast. Skim, then read the WARMUP, then build the registry.

0. The 30-second mental model

The bytes on the wire are a contract between a producer and a consumer who are always on different versions. Schema governance makes a breaking change un-shippable: you encode in Protobuf/Avro (compact, typed), register the schema in a registry, and a compatibility gate rejects changes that would break the other side. One sentence: version the contract, pick the compatibility mode from your deploy order, and enforce it in CI.

1. Format cheat sheet

2. Protobuf in 6 bullets

• The number is the wire identity; the name is not. Rename freely; never reuse or change a number's type → silent corruption.
• Add fields with new numbers → old readers skip them (safe).
• Unknown-field preservation: old readers keep & re-emit fields they don't know → safe multi-hop.
• proto3 scalars have no presence (missing == 0); use optional for presence; repeated defaults empty.
• Enums: reserve 0 = UNKNOWN; add symbols safely; don't remove/renumber.
• Wire types: varint (small ints cheap), zig-zag (sint for negatives), length-delimited (strings/messages/packed).

3. Avro in 4 bullets

• Data is written with a writer schema, read with a reader schema; Avro resolves them.
• Reader field not in writer → needs a default (so adds are safe only with defaults).
• Writer field not in reader → ignored.
• In Kafka you ship a schema ID (magic byte + 4 bytes), not the whole schema.

4. Compatibility — the symmetry to memorise

backward  = new READER reads OLD data   → consumers upgrade first
            ✅ add-with-default, remove        ❌ add-required, change type
forward   = old READER reads NEW data   → producers upgrade first
            ✅ add, remove-optional            ❌ remove-required, change type
full      = both → only add/remove OPTIONAL fields; never change types

Mnemonic: remove is backward-safe / forward-risky; add-required is forward-safe / backward-risky. Pick the mode from who deploys first (can't coordinate → full).

5. Registry = the CI gate

producer ships:  [magic byte][4-byte schema ID][payload]
consumer:        fetch+cache schema by ID → resolve
register(new):   check vs latest under mode → REJECT if breaking  ← the gate

Wired into CI, this makes "no breaking schema change reaches prod" literally true.

6. A data contract is more than a schema

schema (types) + compatibility policy
+ semantics (units, ranges, invariants)   ← the Validated semantic rules
+ SLA / freshness                          → P13 SLOs
+ ownership (a team on a pager)
+ PII classification (per field)           → P14 masking/access/retention
+ deprecation lifecycle (announce, deprecate, remove — never yank)

7. Validate by ACCUMULATING

Fail-fast (Either/exceptions) → "fix one error, resubmit, find the next." Accumulating (Validated) → "here are all 6, fix them at once." For contracts, accumulate. (You build Validated for real in Scala in P12.)

8. Beginner mistakes that mark you

1. Reusing/renumbering a Protobuf field → silent corruption.
2. Adding a required field and calling it backward-compatible.
3. Thinking "rename broke it" when it was a number change.
4. Shipping JSON at billions/day without costing it (key repetition + parse).
5. No registry / no CI gate → breaking changes discovered in prod.
6. A schema with no owner, no SLA, no PII tags → not a contract, just a shape.
7. Fail-fast validation that makes producers fix errors one at a time.

9. War stories

• "Half the events deserialize as garbage after a deploy." → a field number got reused. Reserve retired numbers; add buf breaking to CI.
• "New consumer can't read last year's data." → someone added a required field (backward break) and backfilled nothing. Use optional-with-default.
• "A PII column leaked into an open table." → no PII tags on the schema; governance had nothing to enforce on (P14).

10. How this phase pays off later

• Validation = the P02 DLQ's reason-for-rejection.
• Validated = real Cats code in P12.
• PII tags = the input to P14 governance.
• Compatibility matrix = the heart of the P15 migration ADR.

Read the WARMUP, build the registry gate, then P04: Flink turns these validated events into stateful, exactly-once results.