Lab 01 — Schema Registry & Data-Contract Engine

Phase: 03 — Serialization, Schema & Data Contracts | Difficulty: ⭐⭐⭐⭐☆ | Time: 5–7 hours

"A topic without a schema is an outage on a delay timer." This lab builds the machinery that keeps a thousand producers from breaking a thousand consumers: compatibility checking (backward / forward / full), Protobuf field-number rules, an error-accumulating Validated validator, and a registry that rejects breaking changes on register() — i.e. the CI gate that the JD requires.

What you build

  • Schema / Field — fields with Protobuf-style numbers and Avro-style names/defaults/required (so you learn both evolution models in one model)
  • backward_violations / forward_violations / compatibility_violations — the exact rules: add-with-default is safe; add-required breaks backward; remove-required breaks forward; type change breaks both
  • protobuf_violations — wire identity is the number; a number whose type changed corrupts existing bytes
  • Validated — applicative, error-accumulating validation (collect every contract violation in one pass, not fail-fast) — the Cats Validated you build for real in P12
  • SchemaRegistry — versioned schemas + a compatibility mode; register is the pre-merge CI check

Key concepts

ConceptWhat to understand
Backwardnew readers read old data → can't add a no-default field
Forwardold readers read new data → can't remove a required field
Fullboth → only add/remove optional fields, never change types
Protobuf numbersrenaming is free; changing a number's type breaks the wire
Validatedaccumulate ALL errors so producers fix everything at once
Registry = CI gatebreaking changes are rejected before merge, not in prod

Run

pip install -r requirements.txt
pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v

Success criteria

  • All 17 tests pass.
  • You can explain why "remove a field" is backward-compatible but forward-incompatible.
  • You can explain why Validated accumulates instead of failing on the first error, and why that's the right shape for a data contract.
  • You can state which compatibility mode you'd set for an event topic where consumers deploy before producers (forward) vs after (backward).

Extensions

  • Add transitive compatibility (check the new schema against all prior versions, not just the latest) — the stricter BACKWARD_TRANSITIVE mode.
  • Add enum evolution rules (adding a symbol is forward-safe with a default; removing one isn't) and nullable vs optional distinctions.
  • Generate a producer/consumer compatibility matrix (which writer versions which reader versions can read) — the artifact you put in a migration ADR (P15).
  • Wire it as a real pre-commit hook: fail the build if a .proto/.avsc change is incompatible with the registered latest.