Hitchhiker's Guide — Release Engineering

(0) 30-Second Mental Model

JIRA: fixVersion set     RC1 cut             [VOTE] thread           Distribution
──────────────────────   ──────────────────  ──────────────────────  ──────────────
Committers merge         Tag in git          PMC members cast:       Mirrored to
patches with             Sign artifacts      +1 (binding)            apache.org
fixVersion=3.5.0         Upload to           +1 (non-binding)        mirrors network
                         dist/dev/           0  (abstain)
                         Generate:           -1 (binding VETO!)      72-hour minimum
                           .tar.gz                                    vote window
                           .tar.gz.asc       Result:
                           .tar.gz.sha512    ≥ 3 binding +1 AND
                           LICENSE           no binding -1 → PASS
                           NOTICE

(1) RC Checklist Anatomy

Every Apache source release must pass this checklist before the vote passes. PMC members vote +1 only after personally verifying these items.

Verifying GPG locally:

gpg --verify apache-spark-3.5.0.tgz.asc apache-spark-3.5.0.tgz
# GOOD signature from "Release Manager <rm@apache.org>"

Verifying SHA-512:

sha512sum -c apache-spark-3.5.0.tgz.sha512
# apache-spark-3.5.0.tgz: OK

(2) GPG Trust Chain

Apache Root CA
  └── Committer GPG key (published at https://people.apache.org/keys/)
        └── Detached signature (.asc) for each release artifact

Key points:

• Committer keys are published at https://people.apache.org/keys/committer/<username>.asc
• The project KEYS file in dist/ aggregates all signing keys for that project
• Votes that say "+1 but I didn't verify the signature" are non-binding for practical purposes

(3) japicmp — Reading API Compatibility Output

japicmp compares two JARs and reports compatibility issues:

@@@ MODIFIED CLASS: org.apache.spark.sql.Dataset
  === REMOVED METHOD: public Dataset filter(String)
      Binary incompatible!
  === MODIFIED METHOD: public Dataset select(Column[])
      Return type changed: Dataset -> RelationalGroupedDataset
      Binary incompatible!
  +++ NEW METHOD: public Dataset filter(Column)
      Binary compatible.

Classification rules:

SemVer implication: Any binary-breaking change requires a MAJOR version bump.

(4) Vote Thread Grammar

Apache votes follow a strict format. The release manager sends a [VOTE] email; community members reply.

Subject: [VOTE] Release Apache Spark 3.5.0 RC1

+1 (binding) Alice PMC Chair — verified GPG, built from source, ran TCK
+1 (binding) Bob PMC member — signature OK, license headers checked
+1 (binding) Carol Committer — smoke tested on cluster
+1             Dave contributor — built OK on ARM
0  (binding) Eve — abstaining, on vacation
-1 (binding) Frank — SHA-512 mismatch on the .tar.gz artifact

Quorum rules (Apache Release Policy §6.3):

• ≥ 3 binding +1 votes required
• No binding -1 votes (any binding -1 is a veto; release cannot proceed)
• Minimum 72-hour open window to allow all time zones to participate
• Non-binding votes count for morale but not quorum

After a -1 veto:

1. Fix the issue (e.g., regenerate artifact if checksum mismatch)
2. Cut RC2
3. Start a new vote thread with [VOTE] Release Apache Spark 3.5.0 RC2

(5) Lazy Consensus vs Full Vote

Most day-to-day Apache work runs on lazy consensus — if nobody objects in 72 hours, the change is approved.

(6) Interview Q&As

(7) References

• Apache Release Policy
• Apache Software Distribution
• How to Verify a Release
• japicmp
• ASF Category A/X dependency policy