Warmup Guide — Release Engineering

Zero-to-expert primer for Phase 10: turning Phase 04's release process knowledge into release tooling — checklist engines, API-compatibility analyzers, and vote tallying — the automation a release manager actually relies on.

Table of Contents

• Chapter 1: From Process to Tooling — Why RMs Automate
• Chapter 2: Release Checklists as Executable Specifications
• Chapter 3: Designing a Checklist Engine
• Chapter 4: API Compatibility Checking — The Analyzer's View
• Chapter 5: Classifying Changes — The Rules Table
• Chapter 6: Vote Threads as Parseable Data
• Chapter 7: Binding-ness, Tallies, and Edge Cases
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: From Process to Tooling — Why RMs Automate

Phase 04 taught the release process as ceremony; this phase builds the machines that make the ceremony reliable. The motivating fact: an RC involves dozens of verifications (build, sigs, licenses, compat, docs, JIRA hygiene), each individually trivial, and a missed one sinks the RC days later in the vote thread — the vote round-trip (72h minimum, plus re-spin) is the most expensive unit of time in the process. Tooling collapses human-checklist error rates and turns "RM" from a heroic role into a repeatable one. Every mature project converges on the same three tools, which are exactly the three labs: a checklist runner, a compat checker, and vote-thread automation.

Chapter 2: Release Checklists as Executable Specifications

The conceptual move: a checklist item like "all artifacts are signed" is not prose — it's a predicate over artifacts (∀ artifact: ∃ artifact.asc that verifies against KEYS). Re-reading Phase 04's checklist as predicates:

Some predicates are cheap and local (version string), some expensive (full build), some advisory rather than blocking (docs freshness). That classification — blocking vs advisory, cheap vs expensive — drives the engine design in Chapter 3 and is Lab 01's core modeling decision.

Chapter 3: Designing a Checklist Engine

The design (Lab 01's ReleaseValidator), which generalizes to any policy engine:

• Item = (id, description, severity, check function) — severity ∈ {BLOCKER, WARNING}: blockers fail the release; warnings annotate the report. Severity is data, not code, because projects re-tune it.
• Result ≠ boolean: PASS / FAIL(reason, evidence) / SKIPPED(why) — a failed check must say what it observed, because the report's audience is the vote thread (people re-verify your claims).
• Independence and order: checks must be order-independent (each takes the artifact set, returns a result) so they parallelize and so one failure doesn't mask others — run all checks, report all failures: an RM wants the complete defect list before re-spinning, not one-failure-per-cycle.
• The report is the product: machine-readable (JSON for CI) + human-readable (markdown for the vote thread). The vote email's "here's what I verified" section is generated, not typed — which is also why voters can trust-but-verify it.
• The pattern's reach: swap "release artifacts" for "PR" and severity for gate policy and you have Phase 09's accuracy-regression CI from the model-accuracy track — policy engines over evidence are one design, many domains.

Chapter 4: API Compatibility Checking — The Analyzer's View

Phase 04 ran japicmp; Lab 02 builds a simplified one — because implementing the analyzer is what makes binary compatibility precise instead of folklore:

• Input model: two versions of an API = two sets of class descriptions; per class, the member signatures (name, parameter types, return type, modifiers). Real tools read bytecode (ASM) for exactly the reason SpotBugs does — bytecode is the truth the JVM links against; the lab's MethodSignature model abstracts that step.
• The comparison: align classes/members across versions, classify every difference (Chapter 5's table), aggregate to a report with the worst severity surfaced.
• The JVM linkage rules that make this non-obvious — what binary compatibility actually means (JLS ch. 13): the JVM resolves method calls by exact descriptor, so a return-type change breaks linkage even when source recompiles (covariant returns are a compiler fiction implemented with bridge methods); removing a method breaks callers at run time with NoSuchMethodError (not compile time — the scariest category); widening a parameter type (ArrayList → List) breaks binary callers too (descriptor changed) even though it's source-compatible and "obviously safer."
• Severity ≠ symmetry: additions are mostly safe (new method: fine on classes; on interfaces only safe post-default-methods, and behaviorally risky regardless); removals and signature changes are breaks; modifier changes vary (final-izing a method breaks overriders; widening visibility is safe, narrowing isn't).

Chapter 5: Classifying Changes — The Rules Table

The decision table Lab 02 encodes (and the one to know cold in reviews — it's the mechanical half of interview-prep/04 Scenario 2):

Scoping rule on top: the table applies to annotated public surface only (@InterfaceAudience.Public etc. — Phase 04 Ch. 6); @Private surface is exempt, which is why annotation hygiene is itself release engineering.

Chapter 6: Vote Threads as Parseable Data

Lab 03 parses RC vote threads — semi-structured email with real-world mess:

• The grammar in the wild: votes appear as +1 (binding), +1 binding, -1, +0.5 (yes, fractional non-votes happen), buried in reply quoting (> +1 is a quote, not a vote — strip quoted lines first), with later messages superseding earlier ones from the same voter ("changing my -1 to +1 after RC2 testing").
• Identity: the voter is the From: header, normalized (the same person mails from work and personal addresses; projects maintain informal alias knowledge — the lab uses an explicit alias map, which is also the honest design).
• The parser's contract: extract (voter, vote, binding-claim, timestamp), apply supersession (last vote per voter wins), and never guess silently — ambiguous lines go to an UNPARSED bucket for human eyes. A vote tallier that misreads a -1 is worse than no tallier; the failure mode is governance, not formatting.

Chapter 7: Binding-ness, Tallies, and Edge Cases

• Binding-ness is not claimed, it's checked: a vote is binding iff the voter is on the PMC roster at vote time — the tool joins against the roster; "(binding)" annotations in email are convention, not authority. (Mismatches — someone marking binding who isn't — are flagged, not silently corrected; that's a human conversation.)
• The pass rule (Phase 04 Ch. 2, now as code): binding_plus >= 3 AND binding_plus > binding_minus AND now >= start + 72h. Note what's not in the rule: non-binding votes (persuasive, never counted), and -1s as vetoes (release votes are majority, not consensus — encode the rule correctly and you've internalized the governance distinction the interview always probes).
• Edge cases the tests must cover (Lab 03): vote changed across messages; vote in a quoted block; voter not on roster claiming binding; thread spanning RC cancellations ([CANCEL][VOTE] resets the tally); the 72h boundary itself (timezone discipline: compare instants, not local times).

Lab Walkthrough Guidance

Order: Lab 01 (checklist) → Lab 02 (compat checker) → Lab 03 (vote parser) — each lab automates the next stage of the release timeline.

• Lab 01: model Item/Result/severity first (Chapter 3); implement three cheap checks end-to-end including the report, then add expensive ones. Test the engine properties (all checks run despite failures; report contains evidence), not just the checks.
• Lab 02: encode Chapter 5's table as data (a rules map), not as a switch-pyramid; test each row with a minimal v1/v2 pair; make the report cite the JLS-level reason ("descriptor changed") — the why is what a vote thread needs.
• Lab 03: build the line-level vote matcher with a quoted-line stripper first; then supersession; then the roster join and pass rule. Feed it a real vote thread from a project's archive as an integration test — reality is the best fuzzer.

Success Criteria

You are ready for Phase 11 when you can, from memory:

1. Re-state five checklist items as predicates with their evidence and severity.
2. Explain why a policy engine runs all checks and reports evidence (vote-thread audience).
3. Recite five rows of the compatibility table including the return-type and interface-method rows, with the JVM-linkage reasons.
4. State what makes a vote binding and write the release pass-rule as one boolean expression.
5. Name four vote-parsing edge cases and the parser's never-guess contract.

Interview Q&A

Q: What would you automate first as a new release manager, and why? The verification checklist (Lab 01's shape): it's the highest error-rate × highest cost-of-error step — a missed license or signature check costs a 72h+ vote round-trip. Then the compat report generation (it's already mechanical), then vote tallying (lowest risk-reduction, nice ergonomics). Order by (human error rate × cost of the miss), which is the general automation-prioritization answer.

Q: Why does changing a return type from List to ArrayList break binary compat when every caller's source still compiles? The JVM resolves calls by exact method descriptor — ()Ljava/util/List; and ()Ljava/util/ArrayList; are different methods at link time. Old client bytecode references the old descriptor and dies with NoSuchMethodError at runtime. Source recompilation regenerates descriptors, which is why the break is invisible in development and fatal in deployment — the precise reason compat checking runs on bytecode in CI rather than relying on "it compiles."

Q: Your vote tallier says an RC passed; a PMC member disputes it. What does your tool need to win that conversation? Evidence, not conclusions: per-vote provenance (message-id, quoted raw line, timestamp), the roster snapshot used for binding-ness, the supersession chain for any changed votes, and the UNPARSED bucket showing nothing material was dropped. A governance tool's output must be auditable — the design requirement that distinguishes it from a text scraper.

References

• Apache Release Policy — the checklist's source of truth
• JLS Chapter 13, Binary Compatibility — read 13.4.15 (method result type) and 13.5 (interfaces)
• japicmp — compare its change classifications against your Lab 02 table
• ASM bytecode library — how real analyzers read signatures
• Hadoop InterfaceClassification — the annotation scoping
• A real vote thread with an RC re-spin: search lists.apache.org for [VOTE] Release Spark and read one RC1→RC3 sequence with Chapter 6's eyes