Warmup Guide — Serialization, Schema & Data Contracts

Zero-to-principal primer for Phase 03: how bytes carry meaning (Protobuf, Avro, JSON), why field numbers and reader/writer resolution exist, the precise compatibility rules, the schema registry, and how a data contract becomes a CI gate that makes breaking changes un-shippable.

Table of Contents

Chapter 1: Encoding — Turning Structure into Bytes

A record in memory is a graph of typed values; the wire is a flat sequence of bytes. Encoding is the rule that maps between them, and the choice has consequences:

  • Text formats (JSON, CSV): self-describing (field names travel with every record), human-readable, ubiquitous — but large (field names repeated on every record), slow to parse, and weakly typed (is "42" a number? is the date a string?). Great for config and debugging; expensive at billions of events/day.
  • Binary, schema-on-write (Avro, Protobuf): compact, fast, strongly typed — but you need the schema to read the bytes. This is the right default for high-volume ingestion.
  • Columnar (Parquet, ORC — P08): a storage layout for analytics (read few columns over many rows), not an ingestion wire format. Different job.

The principal framing: ingestion wants row-oriented schema-carrying binary (Avro/Protobuf); analytics wants columnar (Parquet); JSON is the convenient default you grow out of. Picking the wrong one shows up as cost (JSON at scale) or pain (CSV's no-types nightmare).

Chapter 2: Protobuf in Working Detail

Protocol Buffers is the JD's preferred event payload, and its design is its evolution story. A message is a set of fields, each with a number (the tag), a type, and a name:

message Order {
  string order_id = 1;       // the "1" is the wire identity, not the name
  double amount   = 2;
  optional string currency = 3;
  repeated string tags = 4;
}

The wire format: each field is encoded as (field_number << 3 | wire_type) then the value. Wire types: varint (int32/64, bool, enum — 7 bits/byte, smaller numbers = fewer bytes), zig-zag (sint, so negatives don't cost 10 bytes), 64-bit/32-bit fixed, and length-delimited (strings, bytes, embedded messages, packed repeated). Consequences a principal knows:

  • The number is everything. Renaming a field is free (names aren't on the wire); changing or reusing a number re-interprets old bytes as the wrong field — silent corruption. The cardinal rule: never reuse or repurpose a field number; reserve retired ones.
  • Adding a field is safe if you give it a new number; old readers don't know it and skip it.
  • Unknown-field preservation: a Protobuf reader keeps fields it doesn't recognise and can re-serialize them intact — so an intermediary on an old schema doesn't drop data added by a newer producer. This is huge for multi-hop pipelines.
  • optional vs implicit (proto3): proto3 scalars have no "was it set?" by default (a missing int reads as 0, indistinguishable from an explicit 0); optional brings back presence tracking. repeated defaults to empty (never "missing"). oneof models mutually exclusive fields. map is sugar for repeated key/value.
  • Enums: add symbols safely (with a known default at 0); removing a symbol or changing a number is breaking. Always reserve a 0 = UNKNOWN.

The lab models the number-as-identity rule (protobuf_violations): a number whose type changed is flagged because it breaks every byte ever written with it.

Chapter 3: Avro and Reader/Writer Schema Resolution

Avro takes a different tack: the data is encoded against a writer schema, and the reader supplies its own reader schema; Avro resolves the two at read time:

  • Fields present in the writer but not the reader → ignored. Fields in the reader but not the writer → filled from the reader's default (so a reader field must have a default to be added safely). This resolution is exactly why Avro compatibility is defaults-driven.
  • Avro is fully self-describing if the schema travels with the file (object-container files embed it) — but in Kafka the schema is too big to ship per-record, so you ship a tiny schema ID and resolve it via a registry (Ch. 6).
  • Unions (["null", "string"]) model nullability; the order matters (the first is the default branch).

Avro vs Protobuf, the honest comparison: Avro's reader/writer resolution + defaults are elegant for data files and the Kafka+registry world; Protobuf's number-based identity + codegen + unknown-field preservation are excellent for RPC and multi-language services and multi-hop event pipelines. Both are correct choices; the JD names Protobuf as primary, so know it cold and know Avro well.

Chapter 4: JSON, JSON Schema, and When Text Is Fine

JSON's superpower is that everything speaks it and a human can read it. JSON Schema adds a validation contract (types, required, ranges, patterns, enums). It's the right call when: volume is modest, consumers are heterogeneous/external, debuggability matters more than bytes, or you're prototyping. It's a liability when: you're at billions of events/day (the repeated keys and parse cost dominate), you need strong cross-language typing, or you need compact binary. Principals don't sneer at JSON — they cost it out and switch to Avro/Protobuf when the numbers say so, keeping a JSON Schema contract either way.

Chapter 5: Compatibility — Backward, Forward, Full

The single most important operational idea in this phase, because producers and consumers are always on different versions. Definitions (precise, the way the lab implements them):

  • Backward compatible: a reader on the new schema can read data written with the old schema. → Consumers can upgrade first. You may add fields with defaults and remove fields; you may not add a required (no-default) field (old data lacks it) or change a field's type.
  • Forward compatible: a reader on the old schema can read data written with the new schema. → Producers can upgrade first. You may add fields (old readers ignore them) and remove optional fields; you may not remove a required field (old reader needs it) or change a type.
  • Full: both. → Only add or remove optional fields, never change types. The safest, and what you want for a contract many independent teams share.
  • Transitive variants check against all prior versions, not just the latest — stricter, prevents "compatible with N but not N−2" drift.

The decision rule: pick the mode from your deploy order. If consumers roll out before producers, you need backward; if producers roll first, forward; if you can't coordinate (the usual case at scale), full. Note the pleasing symmetry: remove is backward-safe/forward-dangerous; add-required is forward-safe/backward-dangerous. The lab's tests pin exactly this.

Chapter 6: The Schema Registry

A registry is a service that makes schemas first-class:

  • Subjects & versions: schemas are registered under a subject (often <topic>-value); each registration is a new version with a global ID.
  • The wire envelope: instead of shipping the schema per message, the producer ships a magic byte + the 4-byte schema ID, then the payload; the consumer fetches & caches the schema by ID. Tiny overhead, full resolution.
  • The compatibility gate: when you register a new version, the registry checks it against the subject's compatibility mode and rejects a breaking change. This is the lab's register() — and wired into CI, it's how "no breaking schema change reaches production without compatibility checks" (a JD SLO) becomes literally true.
  • Confluent Schema Registry, AWS Glue Schema Registry, and Buf Schema Registry are the market implementations; the concept is what you must own.

Chapter 7: Data Contracts — Beyond the Schema

A schema says what shape; a data contract says everything a consumer needs to depend on the producer safely:

  • Schema (types, fields) + compatibility policy.
  • Semantics: what the fields mean — units, ranges, invariants (an amount is in minor currency units and is non-negative; a status is one of these enum values). These are the semantic rules the lab's Validated enforces beyond mere types.
  • SLA / freshness: how often, how late, how complete (handoff to P13's SLOs).
  • Ownership: a named team, on a pager, accountable for the contract.
  • PII classification: each field tagged by sensitivity (none/pii/sensitive) to drive masking, access, and retention downstream (handoff to P14).
  • Lifecycle / deprecation: how a field or version is announced, deprecated, and removed over versions — never yanked.

The shift: a data product (P00) is a dataset with a contract. Contracts turn "that table broke and nobody knew who owned it" into "the contract test failed in the producer's CI before merge."

Chapter 8: Validation, Accumulation, and Contract Testing in CI

Two distinct enforcement points:

  • Schema-evolution checks (build time): does the new schema break the old one? → the registry/buf breaking gate (Ch. 5–6). Catches contract regressions before merge.
  • Record validation (runtime, at ingress — P02): does this record satisfy the schema + semantics? Failures → DLQ. Here the accumulation property matters: a fail-fast validator (Either/exceptions) reports the first error and stops, so the producer fixes one thing, re-runs, finds the next — a slow loop. An accumulating validator (Cats Validated, the lab's) reports all violations at once, so the producer fixes everything in one pass. For data contracts, accumulation is the correct shape, and it's why P12 builds Validated for real.

Contract testing closes the loop: the consumer publishes the contract it depends on; the producer's CI runs the consumer's expectations against the producer's output (Pact-style), so a breaking change fails the producer's build — shifting the failure left from prod to PR.

Lab Walkthrough Guidance

Lab 01 — Schema Registry & Contract Engine, suggested order:

  1. backward_violations — add-no-default breaks; remove is fine; type change breaks.
  2. forward_violations — remove-required breaks; add is fine; type change breaks (note the symmetry with backward).
  3. compatibility_violations — dispatch on mode; full = both.
  4. protobuf_violations — by number; type change on a shared number is wire-breaking.
  5. Validated.combine + _type_ok (bool is not int!) + validate_record — accumulate all errors; unknown fields allowed.
  6. SchemaRegistry.register — the gate: compatible → version bump; breaking → raise and don't store.

Success Criteria

You are ready for Phase 04 when you can, from memory:

  1. Contrast Protobuf, Avro, and JSON and say which you'd pick for ingestion vs analytics vs external APIs.
  2. State the Protobuf number rule and what unknown-field preservation buys you.
  3. Explain Avro reader/writer resolution and why defaults drive its compatibility.
  4. Give the add/remove/type-change rules for backward, forward, and full — and pick a mode from a deploy order.
  5. Describe the registry's wire envelope and the compatibility gate.
  6. List the parts of a data contract beyond the schema (semantics, SLA, ownership, PII).
  7. Explain why a contract validator should accumulate errors.

Interview Q&A

Q: You must add a mandatory tenant_id to an event consumed by 30 teams. How? Adding a required field is backward-incompatible — old data lacks it, so new consumers can't read history and the registry (rightly) rejects it under backward/full. The safe path: add it as optional with a default (or a sentinel like UNKNOWN), deploy producers to start populating it, let consumers adopt it at their pace, monitor coverage, and only later — if ever — tighten validation to "must be present going forward" via a new major version or a runtime contract rule rather than a schema-required flag. The principal point: "mandatory" is a rollout, not a flag flip, when 30 teams and historical data are involved.

Q: A producer renamed a Protobuf field and consumers broke. Why, if names aren't on the wire? They didn't just rename — almost certainly they reused or shifted a number, or regenerated code such that a number now maps to a different field/type. On the wire Protobuf is keyed by number; a rename alone is invisible and safe, but if currency (number 3) became region (number 3) with a different type, every old byte for "3" now deserializes as the wrong thing. The fix is the cardinal rule: never reuse a number — reserve the old one and give the new field a fresh number.

Q: Backward vs forward — how do you choose? By deploy order. If I roll consumers out before producers (so new readers must handle old data), I need backward. If producers roll first (old readers must tolerate new data), forward. At scale I usually can't coordinate the order across many teams, so I default to full (only add/remove optional fields, never change types) and use transitive mode to stop drift against older versions. The mnemonic: remove is backward-safe but forward-risky; add-required is forward-safe but backward-risky.

Q: Why should a data-contract validator accumulate errors instead of failing fast? Because the consumer of the result is a producer fixing their data, and the humane, fast loop is "here are all 6 things wrong, fix them and resubmit" — not "fix one, resubmit, find the next, repeat 6 times." Fail-fast (Either/exceptions) is right for dependent steps where step 2 is meaningless if step 1 failed; validation steps are independent, so the applicative Validated that accumulates is the correct abstraction (I build it in Scala in P12).

References