🛸 Hitchhiker's Guide — Phase 04: ASF Governance and Release Engineering

Read this if: You can write good code but you've never shipped a release — or you've pushed to npm/PyPI and think Apache releases are similar. They are not. An Apache release is a legally binding vote by PMC members who are personally attesting to the quality of the artifact. By the end you should understand every step from JIRA planning to the announcement email, and know exactly what you are agreeing to when you write +1 on a vote thread.

0. The 30-second mental model

An Apache release is three things simultaneously:

1. A legal act: The PMC votes to release software under the Apache License 2.0. Each +1 vote is a personal attestation that the voter checked the artifacts.
2. A cryptographic chain of custody: GPG signatures link each artifact back to a committer whose key is in the project's KEYS file. Anyone can verify the chain.
3. A distribution process: Artifacts flow from dist.apache.org/dev/ (staging) → dist.apache.org/release/ (official) → Maven Central / mirror network (convenience). The Apache infrastructure is always the source of truth.

Committer machine
  │  mvn release:perform
  ▼
dist.apache.org/dev/[project]/[version]-RC1/
  │  [VOTE] email sent to dev@project.apache.org
  │  72-hour voting period
  │  ≥ 3 binding +1 votes, more +1 than -1
  ▼
dist.apache.org/release/[project]/[version]/
  │  Maven repository sync (24–48h)
  ▼
Maven Central / mirrors
  │  Announcement email to announce@apache.org
  ▼
End users

1. The Release Lifecycle

1.1 Planning Phase

Before a line of release prep is done, the release manager (RM) does:

1. JIRA release version: Create a version in JIRA (e.g., 4.0.0). Move unresolved issues out of scope.
2. Announce intent: Email dev@ list with the proposed release timeline. This is not optional — it gives the community a chance to land in-progress work.
3. Feature freeze: Announce a date after which only bug fixes and documentation are merged to the release branch.

1.2 Stabilization Branch

# Cut the stabilization branch from main (or the previous release)
git checkout -b branch-4.0 origin/main
git push origin branch-4.0

# From this point, main gets new features; branch-4.0 gets only targeted fixes
# Fixes are cherry-picked: git cherry-pick <commit-hash>

A stabilization branch serves two purposes:

• It isolates the release from new feature churn on main
• It makes it possible to patch a released version (e.g., 4.0.1) without including unrelated work from main

1.3 RC Numbering

Release Candidates are numbered RC0, RC1, RC2, etc.

There is no release-labeled artifact until a vote passes. Anything labeled RC is not a release.

1.4 Building the RC

The Maven Release Plugin (mvn release:prepare + mvn release:perform) handles the mechanical parts:

mvn release:prepare
  ├── Prompts for release version (e.g., 4.0.0) and next dev version (4.0.1-SNAPSHOT)
  ├── Updates pom.xml version strings
  ├── Commits the updated pom.xml
  ├── Creates a Git tag (e.g., v4.0.0-rc1)
  ├── Updates pom.xml to next dev version
  └── Commits the next dev version pom.xml

mvn release:perform
  ├── Checks out the tagged commit
  ├── Runs the full build (mvn deploy)
  └── Uploads artifacts to the staging repository

After release:perform, the RM signs the artifacts and uploads to dist.apache.org/dev/.

2. The Vote Thread

2.1 Vote Email Format

A vote email must contain, at minimum:

To: dev@[project].apache.org
Subject: [VOTE] Release Apache [Project] [Version] RC[N]

I propose we release Apache [Project] [Version] based on RC[N].

The RC artifacts are available at:
  https://dist.apache.org/repos/dist/dev/[project]/[version]-RC[N]/

Release artifacts:
  [project]-[version]-src.tar.gz
  [project]-[version]-src.tar.gz.asc
  [project]-[version]-src.tar.gz.sha256

Keys used to sign this release are available at:
  https://www.apache.org/dist/[project]/KEYS

The tag to vote on:
  https://github.com/apache/[project]/releases/tag/v[version]-rc[N]

Please vote:
  [ ] +1 Release this package as Apache [Project] [Version]
  [ ] +0 No strong preference
  [ ] -1 Do not release this package because: (state reason)

This vote is open for 72 hours.

2.2 Vote Arithmetic

A release passes when:

1. At least 3 binding +1 votes (from PMC members)
2. More binding +1 than binding -1 votes
3. The 72-hour minimum voting period has elapsed

A single PMC member's -1 (veto) does not automatically block a release but must be addressed. If the veto identifies a genuine legal or technical problem, it holds.

2.3 Result Email

To: dev@[project].apache.org
Subject: [RESULT][VOTE] Release Apache [Project] [Version] RC[N]

The vote has PASSED.

Binding votes:
  +1 Alice (PMC)
  +1 Bob (PMC)
  +1 Carol (PMC)

Non-binding votes:
  +1 Dave (committer)

Vote thread: [link to archive]

3. GPG Signing and the Chain of Trust

3.1 Why GPG Signing?

A SHA-256 checksum proves an artifact hasn't been corrupted in transit. It does not prove who created it. Anyone who can modify dist.apache.org could also update the checksum.

A GPG signature proves the artifact was created by the holder of a specific private key. If that key is listed in the project's KEYS file and the KEYS file is hosted on www.apache.org (which is controlled by Apache infrastructure, not by the project), the chain of trust is:

Your trust in ASF infrastructure
    → trust in KEYS file
    → trust in committer's public key
    → verified signature on artifact
    → trust in artifact

3.2 The Project KEYS File

Every Apache project maintains a KEYS file at https://www.apache.org/dist/[project]/KEYS. It contains ASCII-armored public keys for every committer who has ever signed a release. When a new committer joins and signs releases, they add their public key to KEYS via an SVN commit.

# Importing the KEYS file
curl https://www.apache.org/dist/spark/KEYS | gpg --import

# Verifying a release signature
gpg --verify spark-3.5.1.tgz.asc spark-3.5.1.tgz

3.3 Creating a GPG Key for Releases

# Generate a key (use RSA 4096 or ed25519)
gpg --full-generate-key
# Choose: RSA and RSA → 4096 bits → does not expire → your Apache email

# List your keys
gpg --list-secret-keys --keyid-format LONG

# Export your public key for KEYS file
gpg --armor --export your@apache-email.org

# Sign an artifact (detached, armored signature)
gpg --batch --yes --detach-sign --armor --output artifact.jar.asc artifact.jar

# Verify your own signature
gpg --verify artifact.jar.asc artifact.jar

3.4 Checksums

# Generate SHA-256 checksum
shasum -a 256 artifact.jar > artifact.jar.sha256    # macOS
sha256sum artifact.jar > artifact.jar.sha256        # Linux

# Verify (the file must have the exact format: "<hash>  <filename>")
shasum -a 256 --check artifact.jar.sha256  # macOS
sha256sum --check artifact.jar.sha256      # Linux

Note: Apache previously used MD5 and SHA-1. Both are now prohibited. SHA-256 or SHA-512 only.

4. API Compatibility Validation

4.1 Binary vs Source Compatibility

Apache projects generally guarantee binary compatibility within a major version (e.g., 3.x). Breaking binary compatibility requires a major version bump and a migration guide.

4.2 What Breaks Binary Compatibility

// v1 — stable API
public class DataProcessor {
    public void process(String input) { ... }      // callers compile against this
    public void setConfig(BatchConfig cfg) { ... }
}

// v2 — breaking changes
public class DataProcessor {
    public void process(String input, boolean strict) { ... }  // BREAKS callers of process(String)
    // setConfig removed                                        // BREAKS callers of setConfig
    public DataProcessor withConfig(BatchConfig cfg) { ... }   // new — non-breaking
}

Callers compiled against v1 will get NoSuchMethodError at runtime when run against v2.

4.3 Using japicmp

japicmp is the standard Java API compatibility checker used by many Apache projects.

# Run via Maven plugin (see lab-03)
mvn install   # builds both v1 and v2, then runs japicmp comparison

# Or via CLI
java -jar japicmp.jar --old v1.jar --new v2.jar --html-file report.html

japicmp classifies each change as:

• BINARY_INCOMPATIBLE: Will break compiled callers
• SOURCE_INCOMPATIBLE: Will break source compilation
• NON_BREAKING: New method/class added; safe
• EXCLUDED: Filtered out (e.g., internal classes)

5. The Distribution Flow and Legal Requirements

5.1 dist.apache.org

All Apache releases must be available on Apache infrastructure before they appear anywhere else. The flow:

1. RM uploads to dist.apache.org/repos/dist/dev/[project]/ (SVN)
2. Vote passes
3. RM moves to dist.apache.org/repos/dist/release/[project]/ (another SVN commit)
4. Apache mirrors sync within 24 hours
5. Maven Central sync happens via Nexus staging repository

Maven Central is a convenience copy. The official release is on dist.apache.org.

5.2 LICENSE and NOTICE Requirements

Every release archive must contain:

• LICENSE file: the Apache License 2.0 text, plus any third-party licenses for bundled code
• NOTICE file: copyright notice for the project, plus required notices from third-party dependencies (e.g., attribution notices from MIT/BSD code)

Missing or incorrect LICENSE/NOTICE is the most common cause of a -1 vote on legal grounds. The apache-rat Maven plugin checks for license headers:

mvn apache-rat:check

6. Common Release Failures

7. Interview Questions — Concepts You Must Explain Out Loud

8. References

• Apache Release Policy — the authoritative policy document
• How to Publish a Release (Apache guide)
• Apache Spark Release Process (community docs) — a real project's release runbook
• Maven Release Plugin
• japicmp — Java API Compatibility Checker
• The GNU Privacy Handbook — GPG key management
• Apache RAT (Release Audit Tool)
• Apache Legal — Third Party Licenses