Warmup Guide — ASF Governance & Release Engineering

Zero-to-expert primer for Phase 04: how an Apache release actually ships — votes and vetoes, GPG signing and verification, license hygiene, and binary-compatibility checking — assuming no prior release-engineering background.

Table of Contents

Chapter 1: What a "Release" Legally Is

From zero: an ASF release is the source tarball that the PMC voted on — not the git tag, not the binaries, not the Docker image. The foundation legally distributes source code under the Apache License; the voting, signing, and distribution rules all protect that artifact. Binaries are "convenience artifacts": built from the source, nice to have, not the thing.

Why this matters practically: release-blocking review focuses on the tarball — does it build from scratch, are LICENSE/NOTICE correct for what's inside it, does it contain no compiled jars or category-X code (Chapter 5), does it match the tag. A broken convenience binary is an embarrassment; a tainted source tarball is a legal problem for the foundation. This is also why every PMC vote email says "I checked: signatures, build, license files" — the votes are attestations, and casting one without checking is the governance sin.

Chapter 2: Votes, Vetoes, and Consensus — The Decision Mechanics

The precise rules (commonly half-known; know them fully):

  • +1 / 0 / -1 are conventions for approve / abstain / object, but their force depends on the question:
  • Code modification votes: a committer/PMC -1 with technical justification is a veto — it cannot be outvoted, only resolved (address the objection or persuade the vetoer). An unjustified -1 is invalid. Vetoes exist so that majority enthusiasm can't override one person who sees the data-loss bug.
  • Procedural votes (releases, adding committers): majority rules — a -1 on a release is NOT a veto. A release passes with ≥3 binding +1s and more binding +1 than -1, after a minimum 72-hour window. ("Binding" = PMC members; everyone may vote, and non-binding votes carry real persuasive weight.)
  • Lazy consensus: announce intent, wait 72h, proceed absent objection — the default for low-controversy work; never for API/compat changes.
  • Where: on the dev list, in a [VOTE] thread, with a [RESULT] posted. Votes in chat or PRs bind nobody.

Chapter 3: The Release Lifecycle End to End

The pipeline the RM (release manager — any committer/PMC volunteer) drives:

  1. Branch & stabilize: cut branch-X.Y; only blockers (tracked in JIRA against the version) land; master moves on.
  2. Produce RC: versions set (X.Y.Z no SNAPSHOT), build from a clean checkout, produce the source tarball + convenience binaries, sign (Ch. 4), checksum, stage — Maven artifacts to a Nexus staging repo, tarballs to dist.apache.org/repos/dist/dev.
  3. [VOTE] thread: links to artifacts, tag, staging repo; voters verify and report what they checked. 72h minimum.
  4. Fail → RC2: any real defect sinks the RC; fix, re-spin, re-vote (release threads routinely reach RC4 — normal, not failure).
  5. Pass → publish: move artifacts to dist/release (mirrors + downloads.apache.org), release the staging repo to Maven Central, push the final tag, update the site, [ANNOUNCE].
  6. Aftercare: track upgrade reports; severe regressions trigger an X.Y.Z+1 patch release — the RM role rotates but the discipline persists.

Lab 01 (release-mechanics) walks a toy project through 1–2 and 5's mechanics; Phase 10's labs automate pieces of 3–4.

Chapter 4: Cryptographic Trust — GPG, Checksums, KEYS

The threat model: artifacts travel through mirrors the ASF doesn't control. Users must be able to verify (a) integrity — bytes unmodified — and (b) authenticity — built by the RM the PMC trusts.

  • Checksums (.sha512): integrity only. Anyone who can replace the tarball can replace its checksum — necessary, insufficient.
  • GPG signature (.asc): the RM signs with their private key; gpg --verify artifact.asc artifact proves the holder of that key produced these bytes. Authenticity now reduces to: is this key really the RM's?
  • The KEYS file: each project publishes committer public keys at a canonical ASF URL (fetched over TLS from apache.org, not from the mirror). Stronger: keys are cross-signed at key-signing events (web of trust). Your verification script (Lab 02) imports KEYS, verifies the sig, checks the checksum, and — the step people skip — confirms the signing key's fingerprint appears in KEYS.
  • RM hygiene: 4096-bit RSA, apache.org uid, key published before the vote; signing happens on a trusted machine (supply-chain 101).

Chapter 5: License Hygiene — LICENSE, NOTICE, and Category X

The part of release review that surprises engineers:

  • LICENSE: the Apache-2.0 text plus the licenses of everything bundled in the artifact. If the source tarball vendors a BSD-licensed file, its license text appears here. Per-artifact accuracy — the binary's LICENSE differs from the source's because it bundles more.
  • NOTICE: required attributions only (a legal instrument, not a thank-you list) — copyright notices that licenses require be preserved. Over-stuffing NOTICE is itself a review comment.
  • Category A (BSD, MIT, ...): may be bundled. Category B (EPL, MPL, ...): binary-only bundling with conditions. Category X (GPL/LGPL, SSPL, JSON license, ...): may NOT be bundled in an ASF artifact at all — a GPL dependency in the tarball is an automatic release blocker. ("Optional runtime dependency the user installs" is the standard escape hatch.)
  • This is checkable mechanically (mvn license:check, rat plugin for headers) and socially (the recurring "why is this jar in the source release?" vote -1).

Chapter 6: Compatibility Checking — Semver and japicmp

The governance promise "no breaking changes in a minor release" is enforceable only by tooling:

  • The dimensions (Phase 03 Ch. 5, now mechanized): source, binary (the one tools check — and the one with surprises: changing a return type to a subtype breaks binary linkage even though sources recompile; adding a method to an interface broke all implementors pre-default-methods), behavioral (only tests catch), serialized forms.
  • japicmp compares two jars and classifies every change (METHOD_REMOVED, METHOD_RETURN_TYPE_CHANGED, ...) against binary/source compat rules; run as a Maven plugin it fails the build on undeclared breakage; declared exceptions live in a reviewed exclusion file. Spark's MiMa is the Scala equivalent; the pattern — compat promises as CI gates with sign-off-required exclusions — is the institutional lesson.
  • Scope control: the check applies to the annotated public surface (@InterfaceAudience.Public/@Stable in Hadoop; everything else is fair game — which is why the annotations exist (Phase 11 builds the migration tooling on top of exactly this).
  • Lab 03 runs japicmp over a deliberately broken v1→v2 pair; Phase 10 Lab 02 then makes you implement a simplified checker — read the report here, build the analyzer there.

Chapter 7: Maven Release Mechanics

Enough mechanics to execute Lab 01 credibly:

  • Versioning: X.Y.Z-SNAPSHOT = mutable dev version (re-resolved on each build); releases strip SNAPSHOT and are immutable forever (Central never lets you replace a released GAV — hence RC re-spins bump -rcN, not reuse).
  • The release dance (whether via maven-release-plugin or scripts): set release version → commit → tag → build+sign+stage (gpg + deploy to a staging repo) → bump to next SNAPSHOT → commit. Staging repos are the key safety: artifacts are visible for vote-time verification but not yet public; "release" or "drop" is one button after the vote.
  • Reproducibility: pin plugin versions, build from a clean clone in a container, record the JDK — "builds for the RM but not for voters" is a classic RC sinker. Modern projects aim for bit-reproducible builds (project.build.outputTimestamp).

Lab Walkthrough Guidance

Order: Lab 01 (release mechanics) → Lab 02 (signing) → Lab 03 (api compatibility).

  • Lab 01: run the version dance on the toy project; produce a source tarball that builds from scratch in a clean directory; write the [VOTE] email you would send, with the checklist of what a voter should verify.
  • Lab 02: generate a key, sign, verify; then break things deliberately — corrupt a byte (checksum catches), re-sign with a wrong key (KEYS-fingerprint check catches) — your script must catch both. The negative tests are the lab.
  • Lab 03: run japicmp on the provided v1/v2; for each reported incompatibility, classify it source-vs-binary and write the one-line release-notes entry it would require; decide which changes could be made compatible (and how) vs which need a major version.

Success Criteria

You are ready for Phase 05 when you can, from memory:

  1. State what artifact a release vote covers and why; recite the 3×+1/72h rule and the veto-vs-majority distinction with one example each.
  2. Walk the six-stage release lifecycle naming the RM's action and the failure mode at each stage.
  3. Explain the three-step verification (checksum, signature, KEYS fingerprint) and what attack each step closes.
  4. Distinguish LICENSE from NOTICE, and categorize MIT/EPL/GPL bundling.
  5. Give two binary-incompatible changes that are source-compatible, and describe how japicmp-as-CI-gate institutionalizes the promise.

Interview Q&A

Q: You're RM and a PMC member -1s your RC for a flaky test. Proceed? A release -1 is not a veto — formally I could pass with 3 binding +1s. Practically: investigate first (flaky test or real race?), respond with findings in-thread. If it's benign flakiness with evidence, say so and let the vote run; if there's any doubt about a correctness issue, re-spin — shipping over a PMC member's documented concern spends trust that outvalues any schedule. The formal rule and the social rule differ; a good RM knows both.

Q: Why can't the ASF just rely on HTTPS instead of GPG signatures? TLS protects the channel to a mirror, not the artifact's provenance — mirrors are third-party, and a compromised mirror serves valid-TLS poison. The GPG chain (signature

  • KEYS fetched from apache.org) binds artifacts to a person the PMC vouches for, end-to-end, independent of any transport or host. Checksums alone fail the same way: co-located with the artifact, they're replaced together.

Q: A minor release accidentally shipped a binary-incompatible change. What now? First, scope: japicmp diff to inventory exactly what broke and for whom. Then the options, in order of preference: restore the old signature in a fast patch release (delegating to the new path — additive fix); if impossible, document loudly + provide a shim artifact. And the institutional fix: why didn't the gate catch it — missing annotation, exclusion-file rot, or the check not running on that module. The incident isn't closed until the gate gap is.

References