Phase 03 — Serialization, Schema & Data Contracts
Difficulty: ⭐⭐⭐⭐☆ Estimated Time: 2 weeks (28–36 hours) Prerequisites: Phase 02 (the topics these schemas govern)
Why This Phase Exists
The JD dedicates an entire core responsibility to this: "Own schema governance across event streams, lakehouse tables, and analytical datasets … manage Protobuf, Avro, and JSON schema evolution … build compatibility checks … integrate schema registry workflows into CI/CD … prevent breaking changes in event contracts." And the deliverables include a "data contract system integrated into CI/CD" and a "schema governance platform."
Here's the truth that makes this phase matter: the format on the wire is a contract, and an unversioned contract is an outage waiting for a deploy. When a producer and a consumer disagree about the bytes, you get silent corruption, deserialization storms, or a poisoned table — and the producer and consumer are always on different versions, because they're owned by different teams who deploy at different times. Schema governance is how you make that disagreement impossible to ship.
Concepts
- Why encoding matters: text (JSON/CSV) vs binary (Avro/Protobuf); size, speed, type-safety, and self-description tradeoffs; why analytics wants columnar (Parquet/ORC, P08) but ingestion wants row-oriented schema-carrying formats.
- Protobuf deeply: field numbers as wire identity, varint/zig-zag/length-delimited
wire types,
optional/repeated/oneof/map, unknown-field preservation, packed encoding, and the evolution rules (never reuse/repurpose a number; add, don't mutate). - Avro deeply: schema-travels-with-(or-near)-the-data, reader vs writer schema resolution, defaults, unions/nullability, and why Avro + a registry is the classic Kafka pairing.
- JSON Schema: ubiquity and self-description vs cost and weak typing; when "just JSON" is fine and when it's a liability.
- Compatibility: backward (new reader ↔ old data), forward (old reader ↔ new data), full, transitive variants; which one to pick based on deploy order.
- Schema registry design: subjects, versions, IDs, the magic-byte wire envelope, and the registry as the source of truth.
- Data contracts: schema + semantics + SLA + ownership + PII classification; contract testing; producer/consumer compatibility matrices; safe deprecation.
- CI/CD integration: the registry compatibility check as a pre-merge gate (the lab).
Labs
Lab 01 — Schema Registry & Data-Contract Engine (flagship, implemented)
| Field | Value |
|---|---|
| Goal | Build compatibility checking (backward/forward/full), Protobuf field-number rules, an error-accumulating Validated validator, and a registry that rejects breaking changes on register — the CI gate |
| Concepts | Schema evolution rules, Protobuf wire identity, applicative validation, registry-as-gate |
| How to Test | pytest test_lab.py -v — 17 tests |
| Talking Points | Why is "remove field" backward-OK but forward-broken? Why accumulate errors? Which mode for consumer-first vs producer-first deploys? |
| Resume bullet | Built a schema-registry compatibility engine (backward/forward/full + Protobuf number rules) and an accumulating data-contract validator wired as a pre-merge CI gate |
→ Lab folder: lab-01-schema-registry/
Lab 02 — Schema Compatibility & Contracts in Scala (implemented, sbt test)
| Field | Value |
|---|---|
| Goal | The JVM-library form of Lab 01: backward/forward/full compatibility + Protobuf field-number rules, plus Cats Validated record validation that accumulates every violation |
| Concepts | Compatibility modes, Protobuf tag safety, applicative (accumulating) validation — in real Scala |
| How to Test | sbt test — 11 ScalaTest specs (runs here) |
| Resume bullet | Built a Scala schema-governance library: compatibility checks, Protobuf tag rules, and Cats-Validated accumulating record validation |
→ Lab folder: lab-02-scala-contracts/
Extension Project A — Real Protobuf round-trip (spec)
Define a .proto, generate code, serialize/deserialize, then evolve it (add a field, change
a number — observe the break) and verify unknown-field preservation across versions.
Extension Project B — Contract test in CI (spec)
Wire buf breaking (Protobuf) or the Confluent registry's compatibility check into a GitHub
Action that fails a PR introducing an incompatible change. Produce the
producer/consumer compatibility matrix as a build artifact.
Integrated-Scenario Hooks
- This phase's validation is the ingress check the P02 DLQ routes failures from.
- Its PII classification field-tags feed the governance platform (P14).
- Its compatibility matrix is the centerpiece of the Hive/Flume → modern migration (P15).
- Its
Validatedis the real Cats abstraction you implement in Scala (P12).
Guides in This Phase
Key Takeaways
- The wire format is a contract; version it or ship outages on a delay timer.
- Backward vs forward is decided by who deploys first, not by taste.
- Protobuf identity is the number; Avro leans on names + defaults + reader/writer resolution.
- Enforce contracts in CI, accumulate all violations, and tag PII at the schema.
Deliverables Checklist
- Lab 01 implemented; all 17 tests pass
- You can state the add/remove/type-change rules for backward, forward, and full
- You can explain Protobuf number reuse danger and unknown-field preservation
- Extension A round-trip or Extension B CI gate