« All Roles

Apache Committer / PMC Engineer — Complete Apprenticeship Curriculum

Target Roles:

• Apache Committer — Spark, Hadoop, Hive, Tez, Flink, and related ASF projects
• PMC Member — Apache Software Foundation
• Staff / Principal Engineer — Data Infrastructure (Databricks, Cloudera, LinkedIn, Uber, Netflix, Meta, Apple)
• Distributed Systems Lead — JVM ecosystem
• Principal Engineer — Open Source Infrastructure

Duration: 24–36 weeks core — extendable to 18 months for full committer readiness
Goal: Develop the technical depth, community understanding, and engineering discipline required to contribute at the committer and PMC level in major Apache Software Foundation projects.

Why This Curriculum Exists

Most engineers who want to contribute to Apache projects approach it backwards: they submit a patch, wait, get confused by the silence or feedback, and quietly give up.

The reason is not lack of technical skill. The reason is that Apache projects run on a trust model that is almost never explicitly documented. Committers and PMC members are not evaluating your code alone. They are evaluating whether you are the kind of engineer they would want to maintain this system with for the next decade.

This curriculum is built backward from what actually makes someone a trusted Apache committer:

• Deep familiarity with JVM distributed systems and their failure modes
• The discipline to write patches that other engineers can review and maintain
• The ability to navigate unfamiliar codebases at production scale
• An understanding of how Apache releases, governance, and consensus actually work
• Experience reviewing code, writing design docs, and operating these systems — not just using them

Reference Apache Projects and Ecosystems

• Apache Spark → Phases 01–02, 06–07, 09, 11
• Apache Hadoop + YARN → Phases 03, 08, 10
• Apache Hive → Phases 03, 09, 10
• Apache Tez → Phases 03, 09
• ASF Release Process → Phase 04
• Cross-project → Phases 05, 11–12

What You Will Build

By the end of this curriculum you will have produced:

• Contributor landscape maps for real Apache projects with committer journey analysis
• Trust gap audits — structured assessments of what separates contributors from committers
• Annotated codebase traces through real Apache Spark and Hadoop source trees
• Post-mortems of documented Apache ecosystem design failures
• A full release checklist and simulated RC vote thread for a production Apache release
• Code review annotations — reviewer-quality comments on real and simulated patches
• Refactored Java code — taking "clever" production code to maintainable quality with tests
• A graduated build-and-review lab ladder — Raft election and Kafka partitioning, JMH/GC/lock-contention performance work, contribution tooling (diff parser, semver engine, issue state machine), release tooling (checklist engine, compat checker, RC vote parser), up to PMC-level API evolution and consensus labs
• A complete design document at subsystem level with alternatives, tradeoffs, and rollout analysis
• A final committer-simulation package — patch, tests, review response, release note, and reflection

Folder Structure

PMC-Apache-Comitter-Engineer/
├── README.md                                          ← You are here
├── jd.md                                              ← Target role profile
├── phase-01-apache-contributor-foundations/           ← Contributor ladder, trust, stewardship
├── phase-02-jvm-codebase-navigation/                  ← Navigate and trace Apache codebases
├── phase-03-lessons-from-large-systems/               ← Technical and human lessons from Hadoop/Hive/Tez/Spark
├── phase-04-asf-governance-release/                   ← Release lifecycle, signing, japicmp compatibility
├── phase-05-code-quality/                             ← Checkstyle, SpotBugs, refactoring "clever" code
├── phase-06-review-culture/                           ← Reviewing flawed patches, design tradeoffs, patch archaeology
├── phase-07-production-systems/                       ← Raft election, Kafka partitioning, multi-module Maven
├── phase-08-performance-engineering/                  ← JMH, GC/allocation profiling, lock contention
├── phase-09-apache-contribution-mechanics/            ← Diff parsing, semver, issue state machines
├── phase-10-release-engineering/                      ← Release checklist engine, API compat checker, RC vote parser
├── phase-11-issue-journey-pmc/                        ← API migration, format migration, consensus tracking
├── phase-12-final-apprenticeship-milestone/           ← Full contribution cycle capstone
├── interview-prep/                                    ← Committer-track interviews, systems design, behavioral
└── system-design/                                     ← Scheduler, metastore, API evolution walkthroughs

24-Week Core Schedule

Lab Structure

Every lab follows a consistent format:

Code labs ship as complete, tested reference implementations rather than TODO stubs: the committer skill being trained is reading, reviewing, and extending real code under a test harness — run mvn test, then do the extension exercises in each lab README.

Prerequisites

• Java 11+ fluency (read and write production Java; understand generics, concurrency basics, exceptions)
• Maven fundamentals (multi-module builds, mvn compile test package)
• Git (branch, rebase, cherry-pick, bisect)
• Basic distributed systems concepts (RPC, consensus, replication, fault tolerance)
• Comfortable reading JIRA issues and GitHub pull requests
• Familiarity with at least one of: Spark, Hadoop, or Hive as a user

Target Role — Apache Committer / PMC Member / Principal Distributed Systems Engineer

Primary Target Positions

Apache Committer / PMC Member

Apache Software Foundation — Open Source (Volunteer / Community)

These are not salaried positions. They represent technical recognition and governance trust granted by the community after sustained, high-quality contribution.

• Committer: Elected by PMC after demonstrated technical credibility, review engagement, and community participation. Granted write access to project repositories.
• PMC Member: Elected by existing PMC members. Holds binding governance authority over releases, project direction, and new committer elections.

Staff / Principal Engineer — Data Infrastructure

Sample companies: Databricks, Cloudera, Palantir, LinkedIn, Uber, Netflix, Apple, Meta, Stripe, Snowflake, Google (Cloud Dataproc / BigQuery), Amazon (EMR / Redshift), Microsoft (Azure HDInsight)

Representative job description components:

Responsibilities:

• Design and implement core components of large-scale distributed data processing systems
• Contribute to and maintain open-source Apache projects (Spark, Hadoop, Hive, Tez, Flink, Iceberg)
• Review code from other engineers; maintain codebase quality and long-term health
• Author design documents and lead technical discussions across engineering teams
• Identify and resolve performance bottlenecks, correctness bugs, and operational issues at scale
• Participate in release engineering, compatibility validation, and upgrade strategy
• Mentor engineers; elevate team-wide engineering standards
• Represent the company in the Apache open-source community

Requirements:

• Deep Java expertise (concurrent programming, JVM internals, GC behavior, heap analysis)
• Direct experience contributing to Apache Spark, Hadoop, Hive, or equivalent at the patch level
• Ability to read, trace, and debug unfamiliar distributed systems codebases
• Strong understanding of distributed systems fundamentals: consensus, replication, scheduling, fault tolerance
• Experience with large-scale production deployments of Apache ecosystem tools
• Demonstrable history of code review, design discussion, and technical writing

Nice to have:

• Apache Committer or PMC status
• Prior experience as a release manager or release engineer for an open-source project
• Published post-mortems, design RFCs, or conference talks on distributed systems

Principal / Staff Software Engineer — Open Source Infrastructure

Sample companies: Confluent (Kafka), Elastic (Elasticsearch), Grafana Labs, HashiCorp, MongoDB, Redis (active open-source infrastructure companies)

Very similar profile to the above: deep JVM systems expertise, community credibility, ability to operate at the intersection of open-source governance and production engineering.

What This Curriculum Prepares You For

By completion, you will have the experience to:

1. Navigate any large Apache JVM codebase without a guide
2. Identify, reproduce, and fix issues in distributed systems under review scrutiny
3. Write patches that pass the Apache review bar — correct, tested, documented, backward-compatible
4. Review others' patches at the level a committer would
5. Understand and participate in ASF governance: releases, votes, and consensus
6. Write design documents that subsystem-level changes require
7. Operate and reason about Spark, YARN, Hive, and Tez at production scale
8. Demonstrate the stewardship mindset required for committer and PMC candidacy

Salary Benchmarks (2025–2026, US market)

Apache Committer / PMC status is a significant differentiator at the Staff and Principal level, particularly at companies whose products are built on ASF projects.

Phase 01 — Apache Contributor Foundations

Difficulty: ⭐☆☆☆☆ | Estimated Time: 1–2 weeks
Roles supported: All tracks — foundational mental model for every phase that follows.

Why This Phase Exists

Every engineer who has ever contributed a patch and heard nothing back, or submitted a PR and been asked for a complete redesign, has encountered the same invisible barrier: the Apache trust model. It is never written in one place. It is learned by observing and participating over months and years.

This phase makes the implicit explicit. You will understand the Apache hierarchy, how trust is built and evaluated, why technically excellent engineers fail to become committers, and what the stewardship mindset actually requires. Without this foundation, every technical skill in the phases that follow will be applied in the wrong direction.

Concepts

• Apache contributor → committer → PMC member hierarchy
• The mechanics of committer elections
• Trust dimensions: technical credibility, review quality, reliability, communication, stewardship
• Why large-batch contribution fails; why incremental sustained participation wins
• Apache mailing list culture: dev@, user@, private@
• Consensus and voting (+1 / 0 / -1, binding vs non-binding)
• Lazy consensus and when it applies
• Stewardship behaviors: backward compatibility, operational empathy, incremental change
• How to read a contributor's public history (JIRA, GitHub, mailing list archives)
• Constructing a "trust gap" analysis: what separates a strong contributor from a committer candidate

Labs

Lab 01 — Contributor Landscape Mapping

Lab 02 — Trust Audit

Deliverables Checklist

• Five contributor profile documents completed (lab-01)
• At least one 200-word journey summary (lab-01)
• Trust audit completed with all six dimension scores (lab-02)
• Top two gaps identified with evidence (lab-02)
• Three recommended actions written (lab-02)
• Nomination draft or blocker explanation written (lab-02)

Interview Relevance

• "How does someone become an Apache committer?" (you can answer with the full election mechanics)
• "What makes a good open-source contributor?" (you can answer with the six trust dimensions)
• "Why do technically strong engineers sometimes fail to get traction in open source?" (you can answer with five failure modes)
• "What is lazy consensus?" (you can answer precisely)
• "What is the difference between a committer and a PMC member?" (you can answer with governance authority details)

Warmup Guide — Apache Contributor Foundations

Zero-to-expert primer for Phase 01. Explains, from nothing, what the Apache Software Foundation is, how its trust model actually works, and how careers inside it are built — so the two labs (contributor landscape, trust audit) have full context.

Table of Contents

• Chapter 1: What the ASF Is
• Chapter 2: The Apache Way — Five Operating Principles
• Chapter 3: The Role Ladder
• Chapter 4: The Trust Model — What Is Actually Being Evaluated
• Chapter 5: Where Apache Work Happens — Lists, JIRA, GitHub
• Chapter 6: Reading a Project's Social Topology
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: What the ASF Is

From zero: the Apache Software Foundation is a US 501(c)(3) charity (founded 1999, descended from the Apache HTTP Server group) that provides legal, infrastructural, and governance shelter for ~300 open-source projects — Hadoop, Spark, Kafka, Hive, Flink, Iceberg among them. It owns the trademarks, holds the licenses, runs the infrastructure, and — crucially — defines the governance template every project follows.

Why a foundation at all: corporate-owned open source can be relicensed, abandoned, or weaponized by its owner. ASF projects can't be: the license (Apache 2.0) is fixed, the governance is by individuals on their own merit (not their employers), and the Board audits each project's independence. For companies betting infrastructure on these projects, that neutrality is the product. This is also why committer status is personal and portable — it survives changing employers, which is precisely what makes it a career asset.

The hierarchy above a project: Board → elected by ASF Members (individuals) → each project's PMC reports quarterly to the Board. Below that, everything is per-project.

Chapter 2: The Apache Way — Five Operating Principles

These phrases carry precise operational meaning; the labs ask you to observe them in the wild:

1. Community over code: a project with healthy reviewers and a mediocre codebase outlives the reverse. Decisions optimize for the decade, not the sprint.
2. Merit, earned and remembered: status is granted retroactively for sustained contribution. You cannot apply to be a committer; you are noticed into it.
3. Open communications — "if it didn't happen on the list, it didn't happen": decisions made in private (a company Slack, a hallway) are not decisions. This rule protects contributors who aren't in the room — including you.
4. Consensus before votes: voting is the fallback when discussion fails, not the default. Lazy consensus ("I'll do this in 72h absent objections") keeps velocity.
5. Independence: hats matter — a committer employed by Cloudera speaks as an individual; PMC composition spanning employers is a health metric the Board watches (and your Lab 01 measures).

Chapter 3: The Role Ladder

The numbers that calibrate expectations (your Lab 01 will measure real ones): typical contributor→committer journeys in major data projects run 9–24 months and 20–60 merged patches plus visible review/list activity. Spark has had ~2,000 contributors and ~75 committers — a ~4% conversion. The ladder is real but steep, and each project's culture sets a different grade.

Chapter 4: The Trust Model — What Is Actually Being Evaluated

The non-obvious core of this phase: a committer vote is a risk assessment, not a reward. Write access means your judgment executes without prior review. The PMC is asking: if this person merges something at 2am, do we sleep? Concretely, they weight:

• Calibration: do you know what you don't know? (Asking good questions scores positively; overconfident patches in unfamiliar subsystems score very negatively.)
• Follow-through: do your merged patches stay healthy? Do you respond when your change breaks CI a week later? Abandoned messes are remembered for years.
• Review behavior: reviewing others' patches is the single most legible trust signal — it is literally practicing the committer job in public.
• Conflict behavior: how you took your first harsh review; how you handled being wrong. One graceless thread can outweigh twenty good patches.
• Sustained presence: monthly-for-a-year beats daily-for-six-weeks-then-gone. The fear behind the criterion: committers who vanish leave unmaintainable code with write access.

The strategic consequence: optimizing for merged-PR count is optimizing the wrong variable. The Lab 02 trust audit has you score yourself against these axes and design the gap-closing plan.

Chapter 5: Where Apache Work Happens — Lists, JIRA, GitHub

The tooling map (projects vary; Spark/Hadoop/Hive conventions):

• dev@ mailing list: design discussions, votes, release threads. Slow, archived, authoritative. Reading six months of archives is the fastest culture download available — Lab 01 requires a slice of this.
• user@: support. Answering questions here is an underrated trust builder (it proves knowledge and patience).
• JIRA (or GitHub Issues): the work ledger. Issue quality — reproduction, version, stack trace, minimal case — is your first writing sample.
• GitHub PRs: code review happens here now for most projects, but decisions still belong to the list (principle 3).
• Slack/Discord where they exist: ephemeral, convenience-only; nothing binding.

Etiquette constants: search before filing; one topic per thread; patches small; silence ≠ rejection (ping politely after a week); never DM-pressure a reviewer.

Chapter 6: Reading a Project's Social Topology

Lab 01's skill — given a project, map who actually moves it:

• git shortlog -sn --since="1 year ago" — the live contributor distribution (expect a power law; note where the elbow is).
• Match top authors against the project's committer/PMC roster (every project publishes one) — the gap between "high committer" and "high author" reveals who reviews vs who writes.
• Read 10 recently merged PRs and 10 stalled ones: who reviewed, how long to first response, what stalled the stalled ones (no reviewer? scope? contributor vanished?).
• Employer mapping (public affiliations): single-employer dominance is a risk flag — and an opportunity flag for an outsider with sustained presence.
• Identify the 2–3 people whose review unblocks the subsystem you care about. Your eventual first patches should land in their area of attention, scoped to be easy to say yes to.

Lab Walkthrough Guidance

Order: Lab 01 (landscape) → Lab 02 (trust audit) — measure the territory, then yourself against it.

• Lab 01: pick ONE project you plausibly intend to contribute to (the analysis pays off later phases). Fill the contributor-profile template with measured numbers (shortlog, roster, PR latencies), not impressions. Deliverable test: could someone else plan their first patch from your map alone?
• Lab 02: score yourself on Chapter 4's five axes with evidence (links, dates), not self-ratings. The output is a 6-month plan whose units are observable acts ("review one open PR per week in module X"), not virtues ("get better at Java").

Success Criteria

You are ready for Phase 02 when you can, from memory:

1. Explain committer vs PMC vs Chair, and who grants each.
2. State the five Apache Way principles with one operational consequence each.
3. Describe the trust model as a risk assessment and name its five evaluation axes.
4. Name where decisions are binding (the list) and why that rule protects you.
5. Produce, for your chosen project: the contributor power-law shape, the 2–3 key reviewers in your target subsystem, and the median first-response latency.

Interview Q&A

Q: Why do companies pay Staff salaries for committer status? Because it is non-purchasable influence over infrastructure the company depends on: patches land faster, the roadmap hears their needs, releases get unblocked, and the trademark of trust transfers to the company's hiring and credibility. A committer on staff is a seat at a table money can't otherwise buy — which is also why the status being personal (it leaves with you) makes it a career asset.

Q: How would you earn committership in a project where you know nobody? Months 0–2: build, run tests, fix the onboarding doc gaps I hit (cheap, genuinely valued); triage and reproduce others' bug reports. Months 2–6: small correct patches in one subsystem until the same reviewers see my name repeatedly; answer user@ questions in that area. Months 6+: review others' PRs in that subsystem, take on a stuck-but-wanted issue, stay present. The plan optimizes reviewer recognition and trust, not patch count.

Q: What's the biggest mistake new contributors make? Submitting a large, multi-concern first patch into a subsystem they haven't read, then going silent when review stalls. It maximizes reviewer cost exactly when trust is zero. The inverse strategy — small, test-covered, in an active reviewer's area, with fast responses — is mechanically easy and rare enough to be noticed.

References

• How the ASF Works
• The Apache Way
• ASF: New Committers guide
• Karl Fogel, Producing Open Source Software — producingoss.com (read ch. 2, 6, 8)
• Spark: Contributing / Hadoop: How to Contribute
• Nadia Eghbal, Working in Public — the contributor-funnel sociology

🛸 Hitchhiker's Guide — Phase 01: Apache Contributor Foundations

Read this if: You want to contribute to an Apache project but you've never shipped a patch that got merged — or you've shipped one and still feel invisible. By the end you should be able to draw the Apache trust hierarchy from memory, explain the six trust dimensions that committers use to evaluate candidates, and place any engineer (including yourself) on the contributor ladder with a clear picture of what it would take to move up.

0. The 30-second mental model

Apache projects are maintained by a trust-weighted graph of engineers, not a company org chart.

Users
└── Contributors    (send patches, file bugs, vote non-bindingly)
      └── Committers    (write access; can merge; trusted with the code)
           └── PMC Members    (binding governance; vote on releases, new committers)
                └── PMC Chair    (reports to ASF Board; administrative, not technical authority)
                     └── ASF Board    (elected by ASF Members; governs all projects)

Three questions drive every major decision in an Apache project:

1. Can we trust this person with the codebase long-term?
2. Will this change help or hurt future maintainers?
3. Does the community support this direction?

By the end of Phase 01 you should be able to place any engineer on this graph and explain what it would actually take to move them up one level.

1. The Apache Hierarchy in Depth

1.1 Users

Users file bugs, ask questions on user@, and sometimes vote (+1/0/-1) on mailing list decisions. Their votes are not binding on release or governance matters. Users build on the project but have no formal standing in its governance.

1.2 Contributors

A contributor has submitted at least one accepted patch, reported and helped diagnose a confirmed bug, or contributed documentation, tests, or design review. "Contributor" is informal — there is no ceremony for it and no roster entry.

What separates contributors from non-contributors in the Apache model is visible, trackable participation. The project's JIRA, GitHub, and mailing list archives are permanent. Maintainers remember — and search — who did what.

1.3 Committers

Committers have been explicitly elected by the PMC. They have write access to the codebase. They can merge patches (their own and others'). They can vote on releases and governance matters — but only PMC members' votes are binding.

"A committer is not someone who can write good code. A committer is someone the current committers trust to make good decisions about the project without supervision." — paraphrased from Apache Contributor Guide

The election process:

1. A PMC member nominates a contributor (private@ PMC mailing list)
2. PMC discusses — can take days to weeks
3. Formal vote is held
   - Requires 3+ binding +1 votes
   - Zero binding -1 vetoes
4. If approved, ASF Infrastructure grants repository access
5. New committer is announced on dev@ mailing list

1.4 PMC Members

PMC = Project Management Committee. PMC members:

• Cast binding votes on releases (3 binding +1, no vetoes required)
• Vote on adding new committers and new PMC members
• Steer the project's roadmap and policies
• Are the legal stewards (with the ASF) of the project's releases

A committer can be invited to the PMC. Most PMC members are also committers, but the two roles are technically independent.

1.5 PMC Chair

Each Apache project has a PMC Chair who acts as VP for that project and reports quarterly to the ASF Board. The Chair is elected by the PMC — it is an administrative and communications role, not a technical authority role. The Chair has no veto power that regular PMC members lack.

1.6 ASF Members and the Board

The ASF Board is elected by ASF Members (individuals recognized for significant ASF contributions across projects). The Board delegates all project decisions to PMCs and intervenes only when a project fails to follow ASF policies. Day-to-day technical decisions are never made at the Board level.

2. How Trust Is Actually Built

This is the section most engineers miss. Trust in Apache is compound interest: small deposits over a long time outperform large deposits made once.

2.1 The Six Trust Dimensions

2.2 Why Technically Strong Engineers Never Become Committers

This happens more than most people expect. Five failure modes:

Failure Mode 1 — Low review engagement

The engineer submits good patches but never reviews others' code. Reviewers feel like the engineer is extracting value from the project — consuming the review bandwidth of others without contributing to it. Trust stagnates regardless of patch quality.

Failure Mode 2 — Abandoning patches mid-cycle

Filing a JIRA, producing a draft patch, receiving review comments, then going silent is one of the fastest ways to damage trust. It creates work for maintainers who must re-triage the issue. One abandoned patch is forgiven. Two is a pattern.

Failure Mode 3 — Design without community alignment

Building a large feature privately and then submitting a 5,000-line patch is a reliable path to rejection — not because the code is bad, but because:

• The PMC had no chance to redirect the design early
• The review burden is enormous
• There may be better alternatives nobody got to suggest
• The author has sunk cost and will resist changes

Apache favors incremental changes discussed openly on dev@ before significant implementation begins.

Failure Mode 4 — Adversarial communication

A single dismissive or hostile exchange on a mailing list can set back a committer candidacy by years. The Apache community is small enough that everyone reads the archives. A reputation for poor communication persists.

Failure Mode 5 — Treating Apache as a vendor

Some contributors engage only when they need a bug fixed or a feature for their employer. This is obvious to maintainers. Community members who contribute durably regardless of immediate self-interest are the ones who get elected.

2.3 What Senior Maintainers Actually Evaluate

When a PMC member considers nominating a contributor, the mental checklist looks like this:

1. Have I (and other committers) seen this person's patches over multiple months?
2. Do they find issues we wouldn't have found ourselves?
3. Do their patches arrive already well-tested and documented?
4. When we pushed back, did they understand why and improve — or argue defensively?
5. Have they reviewed others' patches, and were those reviews genuinely helpful?
6. Do they participate on the dev list or just submit code?
7. Would I be comfortable if they had write access and I was on vacation for two weeks?

Question 7 is the real one. Committer access means merge access without review. PMC members are trusting the nominee with the production quality of a system used by thousands of companies and millions of users.

3. The Stewardship Mindset

Stewardship is the quality that separates committers from contributors, and PMC members from committers. It is also the quality most difficult to fake and most rewarding to genuinely develop.

A steward asks:

"What is the right long-term decision for the project, even if it is harder for me right now?"

3.1 Stewardship Behaviors in Practice

Backward compatibility by default: You do not break APIs unless the benefit is overwhelming and a documented migration path exists. You assume the person upgrading from version N to N+1 has 50 production jobs depending on the behavior you are about to change.

Incremental change: You break large improvements into reviewable patches, each of which leaves the project in a better state than it was before. A reviewer should be able to understand each patch independently.

Operational empathy: You think about the operators who run this system in production, not just the developers using the API. "What happens when this node fails during the migration?" is an operational question that most feature implementors don't ask.

Documentation as a first-class artifact: Design docs, migration guides, release notes, and JIRA descriptions are not afterthoughts — they are part of the contribution. A patch without a test is suspicious. A design doc without a migration section is incomplete.

Saying no as a service: Rejecting a well-intentioned patch that would create long-term maintenance burden is a contribution to the project, not a rejection of the person. The hardest part of stewardship is learning to say "this is not the right approach" kindly and clearly.

3.2 The Five-Year Test

A useful mental exercise: imagine the project in five years, with 20 new committers who weren't there when this change was made.

• Will they understand why it was done?
• Will they be able to maintain it?
• Will it have created problems they'll need to untangle?
• Did the documentation survive?
• Did the compatibility guarantee hold?

If the answer to any of these is "probably not," the change needs more work.

4. Apache Mailing Lists — The Permanent Record

Apache projects operate on mailing lists, not Slack or Discord. This is intentional — mailing lists create a permanent, searchable, asynchronous record of every technical and governance decision.

4.1 The Three Primary Lists

All dev@ and user@ archives are public. Everything you write becomes part of the project's permanent technical history.

4.2 Communication Norms

Be specific: "This may have a performance issue" is not a useful review comment. "In TaskSchedulerImpl.submitTasks(), the taskIdToTaskSetManager map is accessed without synchronization on the CoarseGrainedSchedulerBackend thread; this can cause a ConcurrentModificationException under concurrent task submission" is.

Be patient: Apache contributors span dozens of time zones and most are volunteers or split their time between employer work and community work. A week of silence is not rejection.

Start a thread before writing code: For non-trivial changes, send a [DISCUSS] thread to dev@ before writing a patch. This surfaces design concerns early and builds community alignment before you have sunk cost in an implementation.

Understand consensus vs majority: Apache uses consensus, not majority rule. A single binding -1 (veto) on a release or governance motion blocks it. Vetoes must be accompanied by a technical reason, but the veto-holder is not required to accept a rebuttal. The right response to a veto is to address the technical concern — not to outvote.

4.3 Vote Notation

+1  : I actively support this
+0  : I don't object but won't advocate
 0  : Neutral
-0  : I have concerns but won't block
-1  : I veto this (must include technical justification)

For releases: a binding -1 blocks the release until the technical concern is resolved.
For committer elections: a binding -1 with technical justification blocks the election.
For most code changes: consensus is assumed if no binding -1 appears within a reasonable period (lazy consensus).

4.4 Searching Archives

Every mailing list archive is publicly searchable. Before asking a question on dev@:

1. Search the archive at lists.apache.org — the question may have been answered
2. Search JIRA — there may be an open or closed issue
3. Search the project's documentation

Asking a question that was answered three months ago on the list is a small negative signal. Referencing the prior thread and asking a follow-up is a positive one.

5. Reading a Contributor's Public History

5.1 Where the Evidence Lives

5.2 What a Strong Contributor Profile Looks Like

Timeline: 18+ months of visible activity
JIRA:
  - 20+ issues filed with good reproduction cases
  - 15+ patches submitted, most accepted
  - Comments on 30+ issues (not just "any update?")
  - No abandoned patches without explanation
GitHub:
  - Consistent PR review activity across multiple components
  - Review comments that identify real problems
dev@ list:
  - Participated in at least 3 DISCUSS threads
  - Responded constructively when challenged
  - Answered 5+ user@ questions
No red flags:
  - No hostile exchanges
  - No pattern of claiming issues then going silent
  - Activity spread over time, not burst-and-disappear

5.3 What a Weak Contributor Profile Looks Like (Despite Technical Skill)

Activity: 3–6 months, driven by a single employer need
JIRA: 5 patches submitted, 3 accepted — but only for one subsystem
GitHub: 0 reviews on others' PRs
dev@ list: 0 DISCUSS thread participation; 0 user@ responses
Pattern: contributor got what they needed and disengaged

This profile is common and almost always results in stalled committer candidacy. The engineer is technically capable but has not demonstrated the durability and breadth the community requires.

6. Interview Questions — Concepts You Must Explain Out Loud

7. References

• How the ASF Works — authoritative overview of the governance model
• Apache Voting Process — binding vs non-binding votes, lazy consensus
• Apache Community Guide for Developers
• Apache Committer FAQ — what committer access means in practice
• Apache Release Policy
• Producing OSS — Karl Fogel — free online book; chapters 4 and 8 are essential
• Apache Whimsy Roster — official PMC and committer rosters
• Apache Spark Contribution Guide
• Apache Hadoop Contribution Guide
• Apache Mailing List Archives — search dev@ history for any project

Lab 01 — Contributor Landscape Mapping

Phase: 01 — Apache Contributor Foundations
Difficulty: ⭐☆☆☆☆ | Estimated Time: 4–6 hours
Type: Research / Document

Goal

Build a concrete mental model of what the contributor → committer → PMC progression actually looks like by researching real engineers in a live Apache project. You are not reading theory — you are reading evidence.

Background

Apache project rosters and contribution histories are entirely public. JIRA archives, GitHub activity, and mailing list archives create a permanent, searchable record of every contribution ever made. You can reconstruct the journey of any Apache committer from their first patch to their election.

Most engineers have never done this. After this lab you will be able to look at any contributor profile and immediately recognize where they are on the ladder — and what is blocking them from moving up.

Setup

Primary project: Apache Spark
(You may substitute Apache Hadoop if you have a stronger background there.)

Tools you will use:

Steps

Step 1 — Orient Yourself on the Roster (30 min)

1. Open Whimsy Spark Roster.
2. Note the total number of PMC members and committers.
3. Pick two PMC members whose names you recognize (e.g., from conference talks or blogs).
4. Search their names in the JIRA archive and GitHub. Find their first ever Spark-related commit or merged PR. Note the date.
5. Note the approximate date they were added to the committer list (check git log of the CHANGES file or mailing list announcement).

This gives you a rough sense of how long the journey took for established names.

Step 2 — Profile Five Engineers (2–3 hours)

Select five engineers to profile:

For each, complete one contributor-profile-template.md (copy the template below into a new file named contributor-profile-[handle].md).

Step 3 — Write a Journey Summary (30–45 min)

For Profile A or B (your most historically rich profile), write a 200–300 word "journey narrative" answering:

• What was their first contribution?
• What types of contributions built their reputation (fixes? reviews? design work? documentation)?
• Were there visible inflection points — a period of significantly increased activity?
• Is there any evidence of the committer election announcement on dev@?
• What can you infer about why they were trusted?

Save this as journey-summary-[handle].md.

Step 4 — Synthesize (30 min)

After completing all five profiles, answer these three questions in a section at the bottom of your journey summary file:

1. What patterns appear across all committer profiles that are absent in Profile D (the non-committer)?
2. What is the minimum visible track record you observed before someone was elected committer (in time and contribution breadth)?
3. If Profile D were nominated today, what is the single most important thing they are missing?

Template

Copy this template into a new file for each engineer you profile:

→ contributor-profile-template.md

Expected Output

lab-01-contributor-landscape/
├── README.md                              ← this file
├── contributor-profile-template.md        ← blank template (do not edit)
├── contributor-profile-[handle-a].md      ← your completed Profile A
├── contributor-profile-[handle-b].md      ← your completed Profile B
├── contributor-profile-[handle-c].md      ← your completed Profile C
├── contributor-profile-[handle-d].md      ← your completed Profile D
├── contributor-profile-[handle-e].md      ← your completed Profile E
└── journey-summary-[handle].md            ← 200–300 word narrative + synthesis

Talking Points

Be prepared to explain:

• Why is the Apache contributor history fully public? (Legal, governance, and trust model reasons)
• What is the difference between a committer vote and a PMC vote? (Advisory vs binding)
• Why does the election process use a private mailing list? (Personnel discussion norms; protects nominees from public rejection)
• What would you change about your own public contribution history after seeing these profiles?

Resume Bullet

"Analyzed contributor-to-committer progression for Apache Spark: mapped five public contribution histories across JIRA, GitHub, and mailing list archives to identify the trust-building patterns and minimum track record that precede committer elections."

Lab 02 — Trust Audit

Phase: 01 — Apache Contributor Foundations
Difficulty: ⭐⭐☆☆☆ | Estimated Time: 3–4 hours
Type: Analysis / Document

Goal

Apply the Apache trust model as an evaluator — not a contributor. Given a fictional contributor profile, score each of the six trust dimensions, identify what is blocking committer candidacy, produce concrete actions, and write a PMC nomination draft or explain precisely why you cannot write one yet.

Background

When a PMC member considers nominating someone, they do not run a checklist. They have accumulated an impression over months or years of observation. But that impression can be decomposed into the six trust dimensions from the HITCHHIKERS-GUIDE.

This lab makes you practice that decomposition on a profile you haven't lived through — the way you would evaluate a contributor you don't know personally when their name comes up in a private@ nomination thread.

The Fictional Profile: Alex Kowalczyk

Alex is a backend engineer at a mid-size data platform company. Their company runs Apache Spark on YARN at scale. Alex has been contributing to Apache Spark for 14 months. Here is what the public record shows:

JIRA activity:

• Filed 12 issues, 9 of which were confirmed as real bugs. Reproduction cases are detailed and reliable.
• Submitted 8 patches. 6 were merged. 1 is still open with review comments from 4 months ago that Alex has not responded to. 1 was closed as "won't fix" without explanation.
• Left 7 comments on other people's issues — all were "+1 to reproduce, I see the same" type comments with no technical depth.

GitHub (apache/spark) activity:

• Authored 6 merged PRs. All are in the sql/ and core/ modules.
• Reviewed 3 PRs in the last 14 months. Two reviews were "LGTM" with no comments. One review had a single comment: "Can you add a test for the null case?"
• No reviews outside the two subsystems where Alex's own patches live.

dev@ mailing list:

• Replied to one DISCUSS thread 8 months ago with a brief technical opinion. No follow-up when challenged.
• Zero posts on user@.
• Sent one "I'd like to work on this" email that was not followed up.

General observation:

• Alex attended Spark Summit and gave a lightning talk about the company's use case.
• Alex's patches are well-written: unit tests included, Javadoc updated, no obvious style violations.
• The one open PR with 4-month-old review comments is a non-trivial change to TaskSchedulerImpl that the reviewer said had a subtle thread-safety issue.

Steps

Step 1 — Score the Six Trust Dimensions (1 hour)

Using only the evidence in the profile above, complete the trust audit template.

For each dimension:

• Assign a score of 1–5 (1 = no evidence, 5 = strong consistent evidence)
• Write 2–3 sentences of specific evidence citing details from the profile (not general statements)
• Note any red flags for the dimension

→ trust-audit-template.md

Step 2 — Identify the Top Two Gaps (30 min)

After scoring all six dimensions:

1. Which two dimensions have the lowest scores?
2. For each, write a paragraph explaining:
   • Why this gap matters to the Apache community specifically
   • What a committer's private reaction would likely be when this gap comes up in the nomination discussion
   • Whether this gap is recoverable, and in what timeframe

Save this in a section called ## Gap Analysis in your trust-audit file.

Step 3 — Produce Three Concrete Actions (30 min)

Write three specific, actionable recommendations for Alex to strengthen their committer candidacy over the next 6 months.

Each action must be:

• Specific (not "contribute more reviews" — but "review 2 PRs per week outside the sql/ module, focusing on components Alex knows from reading the issue tracker")
• Time-bounded (achievable in 6 months)
• Directly addressing a gap from Step 2

Save this in a section called ## Recommended Actions in your trust-audit file.

Step 4 — Write the Nomination Draft or Blocker Statement (30 min)

Option A — If you believe Alex is ready for nomination today:

Write a 150–200 word PMC nomination email for Alex. Apache nomination emails follow a rough format:

1. State the nominee's name and Apache ID
2. Summarize their contributions (specific patches, reviews, community work)
3. State your assessment of their trustworthiness
4. Explicitly invite discussion

Option B — If you believe Alex is NOT ready for nomination today:

Write a 150–200 word explanation of:

1. The single most important blocker
2. What specifically Alex would need to demonstrate before you could write Option A
3. A realistic timeline

Save this in a section called ## Nomination Draft or Blocker Statement in your trust-audit file.

Step 5 — Self-Audit Extension (Optional, 45 min)

Apply the same template to yourself. This is the most valuable part of the lab.

Be ruthless. The PMC member evaluating your profile will be.

Template

→ trust-audit-template.md

Expected Output

lab-02-trust-audit/
├── README.md                              ← this file
├── trust-audit-template.md               ← blank template (do not edit)
├── trust-audit-alex-kowalczyk.md          ← your completed audit
└── trust-audit-self.md                   ← (optional) self-audit

Talking Points

Be prepared to explain:

• Why is review engagement a trust signal more important than patch volume? (asymmetric signal: anyone can submit patches; willingness to review benefits others with no personal upside)
• What does an abandoned patch with review comments signal to a PMC member? (unreliability under pressure; the hardest patches to respond to are the ones where the reviewer is right)
• Why is a single "LGTM" review noise rather than signal? (no evidence of understanding; maintainers need to know Alex can catch real problems)
• What is the most dangerous failure mode in Alex's profile? (the 4-month-old open PR with unresolved thread-safety concern — an in-progress change to a core scheduling component that is now blocking the maintainer)

Resume Bullet

"Applied the Apache committer trust framework to structured contributor evaluation: scored six trust dimensions from public evidence, identified candidacy blockers, produced a 6-month development plan, and drafted a PMC nomination or blocker analysis."

Phase 02 — JVM Codebase Navigation

Difficulty: ⭐⭐☆☆☆ | Estimated Time: 1–2 weeks
Roles supported: All tracks — mandatory prerequisite for Phases 07–11.
Primary project: Apache Spark

Why This Phase Exists

Becoming an Apache committer requires the ability to navigate codebases you did not write, in subsystems you have never touched, under time pressure, without a guide. This is a skill almost never taught explicitly and almost always assumed.

The engineers who become committers are the ones who can open a 2 million-line Java/Scala codebase, find the relevant entry point in 10 minutes, trace the call path through four modules, understand the failure mode, and write a correct fix — all before the issue gets stale. This phase teaches that skill methodically.

You will set up Apache Spark for local development, learn to read its Maven module hierarchy, trace a complete execution path from user-facing API to internal executor, and then perform a full issue trace on a real historical bug: reading the JIRA, locating the broken code path, understanding the fix, and explaining why it was correct.

Concepts

• Maven multi-module project structure in large Apache codebases
• Reading pom.xml hierarchies: parent POMs, dependency management, module declarations
• IntelliJ IDEA project import for Apache Spark (Scala + Java mixed modules)
• JVM codebase navigation strategies: API-surface-first, interface-following, test-reading
• Apache Spark module topology: core/, sql/catalyst/, sql/core/, resource-managers/yarn/, network/, common/
• The Spark execution model: SparkSession → Logical Plan → Physical Plan → RDD → Task → Executor
• Key Spark classes and their responsibilities: SparkContext, DAGScheduler, TaskScheduler, ShuffleManager, BlockManager
• Reading git history to understand a bug fix: git log, git show, git blame
• Connecting a JIRA issue to a commit: PR description, JIRA link, release note entry
• Writing a "bug journey" document: problem, trigger, code path, fix, correctness argument

Labs

Lab 01 — Codebase Spelunking

Lab 02 — Issue Trace

Deliverables Checklist

• Apache Spark cloned and project index built in IntelliJ
• Top-level pom.xml module map documented (lab-01)
• Annotated execution trace: SparkSession.read().csv() → FileScanRDD.compute() (lab-01)
• Module dependency diagram drawn (lab-01)
• Logical→physical plan boundary explained in 3 sentences (lab-01)
• JIRA issue selected and fully read including all comments (lab-02)
• Fix commit located and diff read (lab-02)
• Root cause located to specific class/method/line (lab-02)
• bug-journey-*.md completed (lab-02)

Interview Relevance

• "Walk me through how a spark.read.csv() call executes." (you can answer end-to-end)
• "What is the Catalyst optimizer?" (you can explain its role in the logical→physical boundary)
• "How do you approach debugging an unfamiliar distributed codebase?" (you can describe the API-surface-first strategy)
• "What is a DAGScheduler?" (you can describe its role relative to TaskScheduler)
• "How do you find the root cause of a bug in a 2-million-line codebase you didn't write?" (you can describe the git blame → class trace → JIRA strategy)

Warmup Guide — JVM Codebase Navigation

Zero-to-expert primer for Phase 02: how to land in a million-line Apache JVM codebase (Spark, Hadoop, Kafka) and become productive without a guide — build systems, module anatomy, IDE archaeology, and execution-path tracing.

Table of Contents

• Chapter 1: The Shape of a Large Apache JVM Project
• Chapter 2: Maven (and SBT/Gradle) — Enough to Navigate
• Chapter 3: Setting Up the Archaeology Lab — IDE and Tools
• Chapter 4: Top-Down Navigation — From Public API Inward
• Chapter 5: Bottom-Up Navigation — From Symptom to Cause
• Chapter 6: git Archaeology — History as Documentation
• Chapter 7: A Worked Example — Tracing a Spark Action
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: The Shape of a Large Apache JVM Project

From zero: Spark is ~1.5M lines across ~30 Maven/SBT modules; Hadoop ~3M. Nobody reads it linearly — you build a map of intent, then descend only where needed. The recurring module anatomy (learn it once, recognize it everywhere):

• common/core: shared utilities, config machinery, RPC — everything depends on it.
• API modules (sql/api, hadoop-client-api): the public surface, deliberately thin, annotated with audience/stability markers (Phase 04's compat machinery).
• Engine modules (sql/core, sql/catalyst, core/): the actual machinery.
• Connector/integration modules: optional, depend outward, good first-patch terrain (small blast radius).
• *-tests / test-jars: shared test fixtures — also where you learn how to test in this codebase, which matters more than how to code in it.

The 80/20 of any module: README/package-info → the 2–3 interfaces everything implements → the config keys it registers → its largest test class. That ordering — docs, contracts, knobs, behaviors — beats reading implementation files in alphabetical order by an order of magnitude.

Chapter 2: Maven (and SBT/Gradle) — Enough to Navigate

You need build-system literacy to navigate, not to administer:

• The reactor: mvn -pl sql/core -am package builds one module And its upstream deps (-am) — the difference between a 5-minute and a 2-hour loop. Learn each project's documented fast paths (Spark: ./build/sbt ~compile for incremental; -DskipTests everywhere).
• dependency:tree: where "which jar does this class come from / why two versions" questions die. Shaded jars (relocated packages — org.sparkproject.guava) explain classes whose package names don't exist in any source tree.
• Profiles (-Phadoop-3.4, -Pyarn): the same tree builds different artifact sets; when code seems unused, check whether a profile compiles it.
• Where generated code lives (target/generated-sources — ANTLR parsers, protobuf): "I can't find this class's source" usually ends here.
• Run a single test fast: mvn test -pl core -Dtest=DAGSchedulerSuite (or ./build/sbt "core/testOnly *DAGSchedulerSuite -- -z \"stage failure\"").

Chapter 3: Setting Up the Archaeology Lab — IDE and Tools

• IntelliJ IDEA opens the Maven/SBT model directly; give it heap (-Xmx4g+ in the VM options) for Spark-sized indexes. The navigation muscle memory that matters: go-to-implementation (⌥⌘B — interfaces have many), find-usages (⌥F7), type hierarchy (⌃H), and structural search for patterns.
• Decompiled views: IDEA decompiles dependency bytecode on click — how you read shaded/generated classes.
• ripgrep beside the IDE: rg -t scala "spark.sql.shuffle.partitions" across the repo answers "where is this config consumed" faster than any index. Config-key strings are the single best entry points into unfamiliar engines — they appear at exactly the decision sites.
• Debugger over print: a conditional breakpoint inside a real test run (-Dmaven.surefire.debug / IDEA's test runner) teaches call structure faster than a day of reading. Capture stacks: a breakpoint's stack trace is the call-path diagram for your notes.

Chapter 4: Top-Down Navigation — From Public API Inward

Use when the question is "how does feature X work":

1. Find the public entry point (the method users call — df.collect(), producer.send()).
2. Follow it through interfaces, not implementations — note each interface and ask "what are the other implementations and why do they exist?" (the answer encodes the design space).
3. At each hop, record only: component name, responsibility in one phrase, the handoff. Resist reading bodies — you are drawing the subway map, not the streets.
4. Stop at process/thread boundaries (RPC, event loops, queues) and mark them — they're where stack traces will later be discontinuous (Chapter 5's problem).

The deliverable form (Lab 01 uses it): a numbered hop list, API → ComponentA(role) → [event queue] → ComponentB(role) → …, each hop with file:line. Ten hops of this beats fifty pages of prose notes.

Chapter 5: Bottom-Up Navigation — From Symptom to Cause

Use when the question is "why did this break" (Lab 02's mode):

1. Start from exact text: error messages and log lines are grep-able anchors — rg "Container killed on request" lands you at the throw/log site immediately.
2. Read the stack trace as a path, but know its gaps: executor-side traces don't show the driver decision that scheduled them; async boundaries (Netty handlers, event loops, Future callbacks) restart stacks. At each gap, find the enqueue site for the dequeue you're looking at.
3. Reproduce in a test: find the nearest existing test of the failing component (test class names mirror source names) and bend it toward the bug. A failing test is both your understanding made checkable and 80% of your eventual patch.
4. Then bisect versions if needed: git bisect with that test as the oracle.

Chapter 6: git Archaeology — History as Documentation

In mature projects, why lives in history, not comments:

• git log --follow -p -- path/File.scala — evolution of one file, renames included.
• git blame -w -C (ignore whitespace, track copies) on the exact lines you don't understand → commit message → JIRA id (Apache commit discipline: [SPARK-12345] ...) → the JIRA thread holds the design discussion, the rejected alternatives, and the reviewers (= the experts to tag someday).
• git log -S"methodName" (pickaxe) — when was this introduced/removed, even if blame is obscured by refactors.
• The compound move you'll use weekly: blame line → JIRA → linked PR → review comments. That chain is the project's institutional memory, fully public.

Chapter 7: A Worked Example — Tracing a Spark Action

The Lab-01-shaped trace, as it should read when done (Spark ~3.5, names stable for years):

1. RDD.collect() → SparkContext.runJob — the action entry; everything funnels here.
2. → DAGScheduler.submitJob — posts JobSubmitted onto an event loop (DAGSchedulerEventProcessLoop) — first async boundary; driver-side.
3. handleJobSubmitted → stage graph built by walking RDD lineage backward, cutting at shuffle dependencies (wide deps = stage boundaries — the Phase 07/interview staple).
4. → submitMissingTasks → TaskScheduler.submitTasks → SchedulerBackend — tasks serialized, offered to executors via RPC (second boundary: process hop).
5. Executor: Executor.TaskRunner.run → deserialize → Task.runTask → ShuffleMapTask/ResultTask — the closure finally executes; results travel back via block manager / direct result depending on size.

Every hop is findable by the Chapter 4 method in under an hour; the asymmetry between "impossible wall of code" and "ten findable hops" is the entire lesson of this phase.

Lab Walkthrough Guidance

Order: Lab 01 (spelunking) → Lab 02 (issue trace).

• Lab 01: build the project first (the build always teaches: profiles, generated code, module graph). Produce the hop-list trace with file:line anchors; timebox each hop to 15 minutes — if stuck, you're reading bodies instead of following interfaces.
• Lab 02: pick a resolved bug with a linked PR (you'll have ground truth). Do the full bottom-up: symptom → anchor grep → path → reproducing test idea → then compare against the actual fix. Score your diagnosis honestly in the template; the deltas are the curriculum.

Success Criteria

You are ready for Phase 03 when you can, from memory:

1. Name the recurring module anatomy and the 80/20 reading order for a new module.
2. Build one module with upstream deps and run one named test, in one command each.
3. Execute the blame→JIRA→PR→review chain on an arbitrary confusing line.
4. Reproduce the five-hop Spark action trace (or your project's equivalent) on a whiteboard.
5. State the two navigation modes and which evidence each starts from.

Interview Q&A

Q: You've joined a team owning a 1M-line service you've never seen. First week? Build it and run its tests (the build reveals the module graph and the lies in the README); read the top 3 interfaces of the core module; trace ONE main request path top-down into a hop list; then take a real ticket bottom-up — navigation is learned by need, not by survey. Deliverable by Friday: the hop-list map and one small merged fix.

Q: How do you find why a config flag exists and whether it's safe to remove? rg the key for registration + consumption sites; blame the registration → JIRA → original motivating incident; check usage telemetry/defaults history (git log -S on the default value); then the Phase 04 compat lens — a public config is API surface, so removal is a deprecation cycle, not a delete.

Q: A stack trace ends inside an event loop. How do you keep tracing? Find the dequeue site's corresponding enqueue: search for the event class's constructor/post call. Async frameworks make causality discontinuous in stacks but explicit in event types — the event class name is the thread-crossing breadcrumb. (Same skill for Future callbacks: find where the future was created/completed.)

References

• Spark: Useful Developer Tools — builds, single tests, IDEA setup
• Maven: Guide to Working with Multiple Modules
• Spark Internals (Mastering Apache Spark notes) — companion for Chapter 7
• Feathers, Working Effectively with Legacy Code — the characterization-test mindset
• ripgrep — learn -t, -g, -S
• Hadoop/Kafka equivalents: Kafka coding guide, Hadoop wiki "How To Contribute"

🛸 Hitchhiker's Guide — Phase 02: JVM Codebase Navigation

Read this if: You can write Java/Scala but you've never opened a 2-million-line production codebase and felt confident navigating it. By the end you should be able to clone Apache Spark, find any class from a user-facing symptom, trace it to its implementation, and explain the execution model from SparkSession to Executor without consulting the documentation.

0. The 30-second mental model

Apache Spark is a multi-module Maven project. Each module has a clear responsibility, and the modules form a dependency DAG (not a tree). Understanding this DAG is the first step to navigating the codebase.

          User API
             │
     SparkSession / SparkContext
             │
     ┌───────┴───────┐
     │               │
  SQL Path       RDD Path
     │               │
  Catalyst       DAGScheduler
  (optimizer)        │
     │          TaskScheduler
  Physical Plan       │
     │           TaskSetManager
  FileScanRDD         │
     │            Executor
  compute()           │
     │         TaskRunner.run()
  Iterator[Row]       │
                 task.run()

Three invariants to keep in mind throughout:

1. The boundary between logical and physical plans is the optimizer. Anything above FileSourceScanExec is logical. Anything from it downward is physical/runtime.
2. Scheduling is separate from execution. DAGScheduler breaks the job into stages. TaskScheduler assigns tasks to executors. The executor runs them.
3. Storage is abstracted behind RDDs. Whether you read CSV, Parquet, or JDBC, you eventually get an RDD[InternalRow] with a compute() method.

1. Apache Spark's Maven Module Structure

1.1 Reading the Top-Level pom.xml

Every Maven project has a pom.xml. In a multi-module project, the top-level pom.xml is the parent POM: it declares all child modules, sets shared properties (Java version, Scala version, plugin versions), and defines the dependency management section — but does not itself produce a JAR.

<!-- Simplified excerpt from Spark's top-level pom.xml -->
<modules>
  <module>core</module>
  <module>sql/catalyst</module>
  <module>sql/core</module>
  <module>sql/hive</module>
  <module>streaming</module>
  <module>mllib</module>
  <module>resource-managers/yarn</module>
  <module>resource-managers/kubernetes</module>
  <module>launcher</module>
  <module>network/common</module>
  <module>network/shuffle</module>
  <module>common/unsafe</module>
  <module>common/sketch</module>
  <module>common/kvstore</module>
  ...
</modules>

Each <module> is a subdirectory containing its own pom.xml with a <parent> pointing back to the root. This is the standard Maven multi-module pattern.

1.2 The Primary Spark Modules

1.3 The Dependency DAG

Modules depend on each other. The key rule: core has no dependency on sql/. The SQL layer depends on core, not the other way around. This means:

• You can read an RDD bug purely in core/ without touching SQL
• A DataFrame bug almost always involves both sql/catalyst/ and sql/core/
• A shuffle bug may touch core/, network/shuffle/, and resource-managers/yarn/

                         ┌─────────────┐
                         │  sql/hive   │
                         └──────┬──────┘
              ┌──────────────────┘
              ▼
         ┌─────────┐    ┌──────────────┐
         │sql/core │───▶│ sql/catalyst │
         └────┬────┘    └──────┬───────┘
              └────────────────┘
                       │
                       ▼
              ┌────────────────┐
              │      core      │◀── resource-managers/yarn
              └────────┬───────┘◀── resource-managers/k8s
                       │
              ┌────────┴──────────────┐
              │                       │
              ▼                       ▼
     ┌─────────────────┐    ┌──────────────────┐
     │  network/common │    │  common/unsafe   │
     └─────────────────┘    └──────────────────┘

2. Setting Up Apache Spark for Local Development

2.1 Prerequisites

# Required tools
java --version    # Java 11 or 17 (not 21+ for older branches)
mvn --version     # Maven 3.8+
scala -version    # Scala 2.12 or 2.13 (match your Spark branch)
git --version

# IntelliJ IDEA (Community edition is sufficient)
# Plugin required: Scala plugin

2.2 Clone and Build

# Clone the repository
git clone https://github.com/apache/spark.git
cd spark

# Check out a stable release tag (faster than building main)
git checkout v3.5.1

# Build only the modules you need (skip tests and docs to save time)
./build/mvn -DskipTests -Phive -Pyarn package -pl core,sql/catalyst,sql/core \
    -am 2>&1 | tail -50
# -pl : only these modules
# -am : also build modules they depend on
# This takes 8–15 minutes on a modern laptop

# Full build (takes 30–60 minutes; only do this once):
./build/mvn -DskipTests package

2.3 Import into IntelliJ IDEA

1. File → Open → select the spark/ directory
2. IntelliJ detects the Maven project and auto-imports modules
3. Wait for indexing to complete (5–15 minutes for initial index)
4. File → Project Structure → SDKs → confirm Java 11 or 17 is selected
5. Install Scala plugin if not already present (IntelliJ will prompt)

After indexing, you have:

• Go to Declaration (Cmd+B / Ctrl+B): jump from usage to definition
• Find Usages (Opt+F7 / Alt+F7): find every call site for a method
• Navigate to Class (Cmd+O / Ctrl+N): open any class by name
• Type Hierarchy (Ctrl+H): see all implementations of an interface

2.4 Running a Single Test

# Run one test class without building the whole project
./build/mvn test -pl sql/core -Dtest=DataFrameReaderSuite -DfailIfNoTests=false

3. JVM Codebase Navigation Strategies

3.1 The API-Surface-First Strategy

When you receive a bug report like "reading a CSV file with a null value in column 3 throws NPE," start at the public API, not at the crash stack trace.

1. Find the public API entry point
   └── SparkSession.read().csv("path")

2. Follow the method chain until you reach an interface or abstract class
   └── DataFrameReader.csv() → DataFrameReader.load() → DataSource.apply()

3. Find all implementations of the interface/abstract class
   └── CSVFileFormat implements FileFormat
   └── Use Type Hierarchy (Ctrl+H) on FileFormat to see all implementations

4. Narrow to the implementation your bug is in
   └── The NPE points at CSVFileFormat.buildScan() line 174

5. Now read the implementation in context

This strategy gets you to the right place faster than starting from a stack trace, because stack traces only show one path — but the real problem may be that the path shouldn't have been taken at all.

3.2 The Test-Reading Strategy

Before reading implementation code, read the tests for the class you're investigating. Tests tell you:

• What the expected behavior is (the passing tests)
• What edge cases were considered (the named test methods)
• What the error handling contract is (tests with assertThrows)
• What assumptions the implementor made (what is NOT tested)

# Find tests for a class
find . -name "*DataFrameReader*Test*" -o -name "*DataFrameReaderSuite*"
# Apache Spark uses ScalaTest, so test files end in "Suite"

3.3 The Interface-Following Strategy

Apache projects are heavily interface-driven. When tracing a call path:

SparkSession.sessionState.executePlan()
         │
         ▼
  QueryExecution.executedPlan   ← this is a SparkPlan
                                  SparkPlan is an abstract class
                                  Use Type Hierarchy to see all physical plans

Every time you hit an abstract method or interface, stop and ask: "Which implementation is actually running for my use case?" The answer is usually in:

• The factory method that constructs the object
• A pattern-match in a Rule class
• An injection point in SparkContext or SparkSession.Builder

3.4 The git Blame Strategy

When you find the specific line that causes a bug, use git blame to understand its history:

# Who last changed this line, and why?
git blame sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala

# Show the full commit that changed a specific line
git show <commit-hash>

# Find all commits that touched a file
git log --oneline sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala

# Find when a specific string was introduced
git log -S "nullValue" --oneline

git blame is underused by engineers who are new to large codebases. It tells you: the person who wrote this line, the commit that introduced it, and usually the JIRA number in the commit message.

4. The Spark Execution Model in Depth

4.1 The Full Call Path: spark.read.csv("path")

SparkSession.read()
    └── Returns a new DataFrameReader instance
            │
DataFrameReader.csv(path: String)
    └── Calls format("csv").load(path)
            │
DataFrameReader.load(path)
    └── DataSource.apply(sparkSession, "csv", paths = Seq(path))
    └── df = DataSource(sparkSession, className, paths).resolveRelation()
            │
DataSource.resolveRelation()
    └── fileFormat = DataSource.lookupDataSource("csv")  → CSVFileFormat
    └── Returns: HadoopFsRelation(location, ..., fileFormat = CSVFileFormat)
    └── Wraps in: LogicalRelation(HadoopFsRelation)
            │
LogicalRelation  ← this is the LOGICAL PLAN node
            │
            ▼  (when an Action is called, e.g., .show() or .count())
            │
QueryExecution.executedPlan
    └── Calls optimizer rules on the logical plan
    └── Produces: FileSourceScanExec  ← PHYSICAL PLAN node
            │
FileSourceScanExec.inputRDD
    └── Returns: FileScanRDD
            │
FileScanRDD.compute(partition, context)
    └── Opens file via Hadoop InputFormat
    └── Returns: Iterator[InternalRow]
            │
            ▼
  Iterator consumed by downstream operators
  (Project, Filter, Aggregate, etc.)

4.2 The Logical → Physical Plan Boundary

The Catalyst optimizer sits between the analyzed logical plan and the physical plan. It applies a sequence of rules (each a pattern-matching transform over the plan tree) to produce a more efficient plan before planning begins.

4.3 The Scheduling Model

Once an action triggers execution, this is what happens:

SparkContext.runJob()
    │
    ▼
DAGScheduler.runJob()
    └── Builds DAG of stages from the RDD dependency chain
    └── Determines stage boundaries (shuffle boundaries = stage boundaries)
    └── Submits stages in topological order
            │
TaskScheduler.submitTasks(taskSet)
    └── Assigns tasks to executors based on locality (PROCESS_LOCAL > NODE_LOCAL > RACK_LOCAL > ANY)
    └── Manages task retries on failure
            │
CoarseGrainedSchedulerBackend
    └── Sends task binary to executor via Netty transport
            │
Executor.launchTask()
    └── Deserializes the task
    └── Runs TaskRunner in a thread pool
            │
TaskRunner.run()
    └── task.run(taskAttemptId, attemptNumber, metricsSystem)
    └── Task is either a ShuffleMapTask or a ResultTask
            │
ShuffleMapTask.runTask()  OR  ResultTask.runTask()
    └── Calls rdd.iterator(partition, context)
    └── Writes output to shuffle files or returns result to driver

Key insight: DAGScheduler operates on RDD lineages. TaskScheduler operates on task sets. They communicate through a clean interface (DAGScheduler.taskStarted(), taskEnded(), etc.). Bugs often live at this boundary.

4.4 The BlockManager

Every executor has a BlockManager. It is responsible for:

• Caching RDD partitions in memory or disk (persist() / cache())
• Broadcasting variables (driver sends to executors via the block manager)
• Shuffle write (ShuffleMapTask writes shuffle blocks)
• Shuffle read (tasks fetch shuffle blocks from peer executors)

The BlockManager is the part of Spark most involved in out-of-memory errors, shuffle fetch failures, and broadcast issues. Many production bugs live here.

5. Reading a JIRA Issue and Connecting It to Code

5.1 Anatomy of a JIRA Issue

SPARK-XXXXX
├── Summary          ← one-line description (the symptom, not the cause)
├── Description      ← reproducer, stack trace, context
├── Comments         ← discussion, investigation, design debate
├── Attachments      ← patches (older workflow) or PR links (modern)
├── Fix Version/s    ← which release included the fix
└── Linked Issues    ← related JIRAs (parent, blocker, duplicate)

The comments section is the most valuable. It often contains:

• Stack traces with line numbers (now stale after the fix, but useful for locating the area)
• Bisect results ("the regression was introduced in commit X")
• Design debate ("Option A vs Option B — here's why we chose B")
• Reviewer concerns that shaped the final patch

5.2 Finding the Fix Commit

# Method 1: From the JIRA, click "Commits" tab → links to GitHub PR
# Method 2: Search git log for the JIRA number
git log --oneline --all | grep "SPARK-XXXXX"

# Method 3: Search merged PRs on GitHub
# github.com/apache/spark/pulls?q=SPARK-XXXXX

# Method 4: If you know the fix version (e.g., 3.2.0):
git log v3.1.3..v3.2.0 --oneline | grep "SPARK-XXXXX"

5.3 Reading the Diff

When you find the commit, read the diff with context:

git show <commit-hash>          # full diff
git show <commit-hash> --stat   # which files changed
git show <commit-hash> -U10     # 10 lines of context per change

Look for:

• What was added: new null checks, new synchronized blocks, new early-exit conditions
• What was removed: the broken code path
• What was not changed: the calling code — which tells you the fix is internal
• Test changes: what new case was added to the test suite

5.4 The Root Cause vs Symptom Distinction

This is what committers care about most in a patch review:

Example: A NullPointerException in CSVFileFormat.buildScan() when schema inference finds a column of all nulls.

• Symptom fix: Add if (field == null) continue; at the crash site
• Root cause fix: Understand why field can be null (schema inference returns null for all-null columns in some code paths), fix the schema inference to return NullType instead of null, add a test for all-null columns

Committers will always ask: "Why is this field null? Should it be? What other code paths hit this same condition?"

6. Interview Questions — Concepts You Must Explain Out Loud

7. References

• Apache Spark GitHub — apache/spark
• Spark Internals (Databricks slides) — overview of the execution model
• Deep Dive: Apache Spark Memory Management (Databricks)
• Mastering Apache Spark (Jacek Laskowski — online, free) — the most detailed public Spark internals reference
• Spark SQL Paper — Armbrust et al., SIGMOD 2015 — original Catalyst/DataFrame paper
• Effective Java, 3rd ed. — Joshua Bloch — Java idioms used throughout Spark's Java interop layers
• Maven: Guide to Multi-Module Projects
• IntelliJ IDEA: Navigating Source Code

Lab 01 — Codebase Spelunking

Phase: 02 — JVM Codebase Navigation
Difficulty: ⭐⭐☆☆☆ | Estimated Time: 4–6 hours
Type: Code Navigation / Document
Repository: apache/spark — tag v3.5.1

Goal

Produce an annotated execution trace of spark.read.csv("path") from the first line of user-facing API code down to FileScanRDD.compute(). Map the Maven module dependencies touched along the way. Explain the logical→physical plan boundary in your own words.

After this lab you will be able to open any Apache Spark bug report and navigate to the relevant code in under 10 minutes.

Setup (Do This Once)

# 1. Clone the repository
git clone https://github.com/apache/spark.git
cd spark
git checkout v3.5.1

# 2. Build the modules used in this lab (skip tests to save time)
./build/mvn -DskipTests -Phive -Pyarn package \
    -pl core,sql/catalyst,sql/core -am \
    2>&1 | tail -30

# 3. Open in IntelliJ IDEA
# File → Open → select the spark/ directory
# Wait for Maven import and indexing (5–20 minutes first time)
# Confirm: File → Project Structure → SDKs → Java 11 or 17

# 4. Verify navigation works:
# Press Cmd+O (Mac) or Ctrl+N (Linux/Windows)
# Type: SparkSession
# You should see org.apache.spark.sql.SparkSession

Steps

Step 1 — Map the Module Structure (45 min)

Open spark/pom.xml (the root). Find the <modules> block.

1. List every <module> entry. There are approximately 30–40.

2. For each of the following 8 modules, write one sentence describing its responsibility:

   • core
   • sql/catalyst
   • sql/core
   • sql/hive
   • resource-managers/yarn
   • network/common
   • network/shuffle
   • common/unsafe

3. Draw a simple dependency diagram (ASCII is fine) showing which of these 8 modules depend on which others.

Hint: Open each module's pom.xml and look at its <dependencies>. The <artifactId> values show you the module names.

Save this in execution-trace.md under ## Module Map.

Step 2 — Trace the Call Path (2–3 hours)

Starting at SparkSession.read(), trace the code path to FileScanRDD.compute(). For each class in the chain, record:

• The module it lives in (e.g., sql/core)
• The fully qualified class name
• One sentence describing its responsibility
• The specific method where control transfers to the next class in the chain

Use IntelliJ's navigation tools:

• Cmd+B / Ctrl+B — Go to Declaration
• Opt+F7 / Alt+F7 — Find Usages
• Ctrl+H — Type Hierarchy (for abstract classes and interfaces)
• Cmd+Shift+F / Ctrl+Shift+F — Full-text search across the project

Starting point:

// In IntelliJ: Cmd+O → "SparkSession"
// Find the method: def read: DataFrameReader
val df = spark.read.csv("path/to/file.csv")

Waypoints to find and record (you fill in the details):

1. SparkSession.read()                          → sql/core
2. DataFrameReader.csv(path)                    → sql/core
3. DataFrameReader.load()                       → sql/core
4. DataSource.apply() / DataSource.resolveRelation()  → sql/core
5. CSVFileFormat  (or TextInputCSVDataSource)   → sql/core
6. LogicalRelation                              → sql/catalyst
7. FileSourceScanExec                           → sql/core
8. FileScanRDD                                  → sql/core
9. FileScanRDD.compute()                        → sql/core

When does the physical plan get built? The logical plan is built immediately when you call csv(). The physical plan is built lazily when you call an action (.show(), .count(), .collect()). Find where QueryExecution.executedPlan is called — this is the boundary.

Save this as a table in execution-trace.md under ## Execution Trace.

Step 3 — The Logical → Physical Plan Boundary (30 min)

Find and read these two classes:

1. QueryExecution (in sql/core/)
2. SparkPlanner (in sql/core/)

Answer these questions in execution-trace.md under ## Plan Boundary:

1. What method in QueryExecution triggers physical plan creation? What does it call?
2. What is SparkPlanner's role? What does it do with an optimized logical plan?
3. In 3 sentences or fewer: explain the logical → physical plan transition in your own words, as if explaining it to an engineer who has never seen Spark internals.

Step 4 — Draw the Module Dependency Diagram (30 min)

For the classes you touched in Step 2, draw an ASCII diagram showing:

• Which module each class belongs to
• Which module depends on which (draw an arrow from dependent → dependency)

Example format:

sql/core ──depends on──▶ sql/catalyst
sql/core ──depends on──▶ core
sql/catalyst ──depends on──▶ core

Save this in execution-trace.md under ## Module Dependencies.

Expected Output

lab-01-codebase-spelunking/
├── README.md                   ← this file
└── execution-trace.md          ← your completed trace document

execution-trace.md structure:

## Module Map
[8 module descriptions + dependency diagram]

## Execution Trace
[Table: class name | module | responsibility | handoff method]

## Plan Boundary
[3-question answers + 3-sentence explanation]

## Module Dependencies
[ASCII dependency diagram for classes touched]

Verification

Your trace is correct if:

1. Every class in your table is a real class you can open in IntelliJ
2. The handoff methods connect correctly (output of one is input to next)
3. LogicalRelation appears before FileSourceScanExec in your list
4. Your plan boundary explanation correctly identifies that the physical plan is built lazily

Optional verification: set a breakpoint in FileScanRDD.compute() and run a small Spark app locally. Confirm the call stack matches your trace.

Talking Points

• Why is DataSource a pivot point? It is the registry that maps a format name ("csv") to its FileFormat implementation. This is the extensibility point for custom data sources.
• Why is the plan built lazily? Lazy evaluation lets the optimizer see the entire plan before committing. It also means you can build a plan in a driver application and serialize it to executors without running anything.
• What is InternalRow vs Row? InternalRow is the unboxed binary format used inside the execution engine. Row is the user-facing boxed API. The conversion happens at the API boundary (e.g., Dataset.collect()).

Resume Bullet

"Navigated Apache Spark's multi-module Maven codebase from scratch; produced an annotated execution trace from SparkSession.read().csv() through Catalyst optimization to FileScanRDD.compute(), documenting module boundaries, class responsibilities, and the logical→physical plan transition."

Lab 02 — Issue Trace

Phase: 02 — JVM Codebase Navigation
Difficulty: ⭐⭐⭐☆☆ | Estimated Time: 4–6 hours
Type: Code Investigation / Document
Repository: apache/spark

Goal

Perform an end-to-end trace of a real closed Apache Spark JIRA issue: read the full issue, locate the root cause in the source tree using git blame and git log, read and understand the fix commit, and produce a structured "bug journey" document.

This lab trains the most important skill in open-source contribution: the ability to take an unfamiliar bug report and navigate a large codebase to its root cause — independently, without guidance.

Pre-Selected Issues

Choose one of the following closed issues. All are real SPARK bugs with clean, single-commit fixes and complete JIRA descriptions. Start with Level 1 if you are new to Spark internals.

If a JIRA link is unavailable: The Apache JIRA is at https://issues.apache.org/jira/browse/SPARK-XXXXX. The corresponding GitHub PR can be found by searching github.com/apache/spark/pulls?q=SPARK-XXXXX.

Steps

Step 1 — Read the Issue Completely (45 min)

Open your chosen JIRA. Read:

1. The Summary line — this is the symptom
2. The Description — this is the reproducer and context
3. Every comment from top to bottom — this is where the diagnosis happens
4. The Fix Version/s field — which Spark release included the fix
5. Any linked issues — blockers, duplicates, parents

While reading, take notes on:

• What is the observable symptom?
• What conditions trigger it?
• Is there a stack trace? What line does it point to?
• What was the first hypothesis? Was it correct?
• Where did the investigation narrow down to?

Save your notes in bug-journey-[SPARK-XXXXX].md under ## Issue Analysis.

Step 2 — Find the Fix Commit (20 min)

cd spark

# Method 1: JIRA "Commits" tab (most reliable)
# JIRA → Commits tab → click GitHub PR link

# Method 2: Search git log
git log --oneline --all | grep "SPARK-XXXXX"
# Replace XXXXX with your issue number

# Method 3: GitHub PR search
# github.com/apache/spark/pulls?q=SPARK-XXXXX+is%3Amerged

# Once you have the commit hash or PR:
git show <commit-hash> --stat     # which files changed
git show <commit-hash> -U5        # diff with 5 lines of context

Record in your bug journey document:

• The commit hash
• The PR number (if applicable)
• Which files were changed
• How many lines were changed (added/removed)

Step 3 — Locate the Root Cause (1–2 hours)

This is the core skill.

Do NOT start by reading the fix. First, try to locate the root cause independently:

1. Use the stack trace or description to identify the first suspicious class/method
2. Open that class in IntelliJ. Read the method in question.
3. Use git blame on the specific lines — who wrote this, and when?
4. Read 50 lines above and below the crash site for context
5. Ask: "Under what condition does this code misbehave?"

# git blame with line range
git blame -L 120,160 sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala

# Show what a specific commit changed in context
git show <commit-hash> -- path/to/File.scala

Then, after you've formed your own hypothesis, read the fix commit and compare.

Record in ## Root Cause Analysis:

• The specific class, method, and approximate line where the bug lives
• The exact condition that triggers incorrect behavior
• Your hypothesis (before reading the fix): what you thought the fix would do
• Whether your hypothesis matched the actual fix

Step 4 — Trace the Broken Code Path (30 min)

Reconstruct the execution path from user-facing API call to the point of failure. This is the same skill as Lab 01, but starting from a bug instead of a known entry point.

For example, if the bug is in CSVFileFormat.buildScan(), trace backward:

• Who calls buildScan()?
• Under what conditions is the null input passed?
• Could the caller have validated earlier?

Record a simplified call chain (5–8 entries is enough) in ## Broken Code Path.

Step 5 — Read and Understand the Fix (30 min)

Read the fix diff carefully:

git show <commit-hash> -U10   # 10 lines of context

For every change in the diff, answer:

1. What was the broken behavior?
2. What does the fix do instead?
3. Is this fixing the root cause or the symptom? How can you tell?
4. What test was added? What boundary condition does it cover?
5. Are there related conditions the fix does NOT cover? (Look for TODOs or known limitations in the commit message)

Record this in ## Fix Analysis.

Step 6 — Write the Correctness Argument (20 min)

A correctness argument answers: "Why does this fix work? Why doesn't it break anything else?"

This is exactly what a reviewer writing "+1" on the fix PR would say. Write yours in ## Correctness Argument:

• Why the fix resolves the root cause (not just the symptom)
• Why the fix doesn't introduce regressions in the non-bug case
• What the test added in the fix covers and why that's sufficient

Template: bug-journey-[SPARK-XXXXX].md

# Bug Journey — SPARK-XXXXX

## Issue Summary
- **JIRA**: [SPARK-XXXXX](https://issues.apache.org/jira/browse/SPARK-XXXXX)
- **Symptom**: (one sentence)
- **Trigger conditions**: (what user action / data / configuration causes this)
- **Affected versions**: 
- **Fix version**: 
- **Severity** (your assessment): Blocker / Critical / Major / Minor

## Issue Analysis
[Notes from reading the full JIRA: investigation timeline, key comments]

## Root Cause Analysis
- **Location**: `ClassName.methodName()` in module `xxx`
- **Root cause**: (one paragraph — what is wrong with the code)
- **Your hypothesis before reading the fix**:
- **Did your hypothesis match?**: Yes / No / Partially — explain

## Broken Code Path
[Simplified call chain from user API to point of failure]
1. `SparkSession.X()` → ...
2. ...
N. `BuggyClass.method()` ← bug lives here

## Fix Analysis
- **Commit**: [link to GitHub commit]
- **Files changed**: 
- **Lines changed**: +X / -Y
- **What changed**: (per-file summary)
- **Root cause fix or symptom fix**: (your assessment + reasoning)
- **Test added**: (describe the new test case)
- **Known limitations**: (what cases does this NOT fix)

## Correctness Argument
[Why the fix works and why it doesn't break anything else — 2–4 paragraphs]

Expected Output

lab-02-issue-trace/
├── README.md                              ← this file
└── bug-journey-SPARK-XXXXX.md            ← your completed bug journey

Verification

Your bug journey is complete when:

1. Your root cause location matches the lines changed in the fix commit
2. Your correctness argument explains why the fix works — not just what it does
3. Your broken code path connects to the root cause location
4. You can explain the bug and fix out loud in 2 minutes without notes

Talking Points

• What is the difference between root cause and symptom? The symptom is the NPE. The root cause is that the schema inference returned null for an all-null column when it should have returned NullType.
• Why do reviewers push back on "add a null check here"? Because it papers over the symptom. The real question is: should null be reachable here at all? If not, why is it reachable?
• How do you know when a bug is "yours" to fix? Any committer can fix any bug. But fixing a bug in a subsystem you don't know well requires more review scrutiny. The more you understand the surrounding context, the more confidently you can fix root causes rather than symptoms.

Resume Bullet

"Performed end-to-end issue trace on a closed Apache Spark JIRA: located root cause in Spark internals via git blame and diff analysis, reconstructed the broken execution path across modules, and produced a structured correctness argument for the accepted fix."

Phase 03 — Lessons from Large Apache Systems

Why this phase exists: Before you contribute to a large system, you need to understand how large systems fail — not to be pessimistic, but because the committers and PMC members you'll be working with carry this institutional memory. They've seen the same mistakes made across generations of distributed systems projects, and they'll hold your patches to a higher standard because of it.

What You'll Learn

• How Hadoop 1.x's single NameNode became a systemic SPOF and why fixing it took years
• How Hive's metastore became a metadata bottleneck at scale and the architectural decisions that caused it
• How Tez was designed to fix MapReduce's DAG limitations — and what it got right and wrong
• Why Spark originally avoided Hadoop's abstractions, and how compatibility pressure brought some of them back
• The recurring patterns of distributed systems failure: over-abstraction, centralization, backward-compatibility debt, coordination overhead
• How to read a technical design failure as a learning document, not a post-mortem

Concepts Covered

Labs

Deliverables Checklist

• Post-mortem document for one chosen design failure (Hadoop NameNode SPOF or Hive metastore bottleneck)
• Compatibility burden analysis for one chosen interface (Hadoop FileSystem API or Hive SerDe)
• You can explain the chosen failure from first principles — what was built, why it failed, what the fix required, what lessons generalize

Interview Relevance

Staff/Principal Engineer interviews at companies running Spark/Hadoop at scale (Databricks, Cloudera, LinkedIn, Uber, Netflix, Meta) will ask:

• "What are the biggest architectural mistakes you've seen in distributed systems?"
• "How do you think about backward compatibility when designing an API?"
• "What's the cost of making a distributed system highly available? What are the tradeoffs?"
• "Why do metadata services become bottlenecks? How do you avoid it?"

The engineers interviewing you have often been personally involved in these failures. This phase gives you the vocabulary to engage at their level.

Warmup Guide — Lessons from Large Apache Systems

Zero-to-expert primer for Phase 03: the design failures and compatibility burdens of Hadoop, Hive, Tez, and Spark as case law — the background needed to write the post-mortem and compatibility-analysis labs at PMC altitude.

Table of Contents

• Chapter 1: Why Study Failures — Systems as Case Law
• Chapter 2: Case — the HDFS NameNode SPOF
• Chapter 3: Case — MapReduce's Rigidity and the YARN Split
• Chapter 4: Case — Hive's SerDe and Metastore Burdens
• Chapter 5: The Compatibility Burden — Why Old Decisions Are Forever
• Chapter 6: Extracting Lessons — The Post-Mortem Method
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Why Study Failures — Systems as Case Law

Senior engineers reason by precedent. "We shouldn't make the coordinator stateful-and- singular" is an opinion; "that's the NameNode SPOF pattern, and fixing it post-hoc took Hadoop five years and three mechanisms" is an argument. This phase builds your case-law library: each case is studied as decision → context that made it reasonable → consequence → cost of the fix → transferable principle. The discipline matters: hindsight without the original context teaches arrogance, not judgment. Every design you'll later review (Phases 06, 11) or write (Phase 12) gets tested against these precedents.

Chapter 2: Case — the HDFS NameNode SPOF

The decision (2005–06): one NameNode holds the entire filesystem namespace — in memory, single process. Block locations are soft state (rebuilt from DataNode reports); mutations go to an edit log.

Why it was reasonable: GFS — the design template — did the same; a single in-memory namespace made every metadata op a lock-free-ish RAM lookup (performance), and consistency is trivial when one process owns truth. For a batch system in 2006, hours of downtime were tolerable.

The consequence: as clusters became multi-tenant infrastructure, the NameNode became (a) an availability SPOF — restart = cluster outage, with painful edit-log replay times; (b) a scalability wall — namespace bounded by one heap (files ≈ RAM), and GC pauses on 100GB+ heaps became availability events themselves (your Phase 08 JVM knowledge, manifesting as system design).

The fix's price tag: Secondary NameNode (checkpointing only — widely misunderstood as HA, fixing nothing) → QJM + Standby NameNode + ZKFC automatic failover (true HA, ~2012, 5+ years after the problem was evident) → HDFS Federation for namespace scale. Three mechanisms, enormous operational complexity, all retrofitted around a single early decision.

Transferable principles: centralizing metadata in one process trades scale/ availability ceilings for simplicity — fine if you state the ceiling; "we'll add HA later" means "we'll redesign state management later"; and memory-resident metadata means JVM behavior is your availability story.

Chapter 3: Case — MapReduce's Rigidity and the YARN Split

The decision: Hadoop 1's JobTracker fused two jobs — cluster resource management and per-job execution logic — and the only programming model was map→shuffle→reduce.

Why reasonable: one abstraction, aggressively simple, matched the dominant workload (batch scans) and the hardware (disks). Fault tolerance via re-execution of idempotent tasks was — and remains — a brilliant simplification.

The consequence: every non-batch pattern (iterative ML, interactive SQL, graphs) had to be contorted into MR jobs with disk round-trips between every stage; JobTracker fused concerns meant scheduler bugs took down job bookkeeping and vice versa, and scale capped around ~4K nodes.

The fix: YARN (Hadoop 2, ~2012) split RM (resources) from per-app AMs (execution) — the two-level scheduler you study in system-design/01 — making MR just one framework and clearing the runway for Tez (DAGs of stages, no forced disk barriers) and Spark (DAG + memory residency). Lesson pattern: fusing the platform with its first application is the classic v1 sin; the unbundling is always a major-version event with a years-long migration tail.

Chapter 4: Case — Hive's SerDe and Metastore Burdens

SerDe: Hive let any table declare a serializer/deserializer class — maximal format flexibility. Consequence: the SerDe interface became load-bearing public API for an ecosystem of third-party formats; every engine touching Hive tables (Spark, Presto, Impala) had to reimplement or shim SerDe semantics, and behavioral quirks (null handling, type coercions) became de-facto standards no one can fix without breaking someone (Hyrum's Law at format scale).

Metastore: thousands of jobs speak the Thrift metastore API; partition metadata as RDBMS rows hit the wall at millions of partitions. The escape (Iceberg/Delta-style table formats — metadata in the storage layer) took a new project generation rather than an in-place fix — the full anatomy is system-design/02.

Transferable principles: an extension interface is a bigger forever-commitment than any implementation; behavioral quirks of a popular system become its real spec; and when the data model is the bottleneck, the fix often can't be incremental — plan for the strangler migration (Phase 11's format-migration lab rehearses exactly this).

Chapter 5: The Compatibility Burden — Why Old Decisions Are Forever

The connective tissue of all three cases — what "public surface" really includes once thousands of deployments depend on you:

• APIs (source/binary/behavioral — the Phase 04/10 taxonomy), but also:
• Wire protocols (a Hadoop RPC change must interop across rolling-upgrade version skew), file formats (data outlives code by a decade — SequenceFile readers must exist forever), config keys and their defaults (changing a default is a behavioral break someone will hit at 2am), CLI output (someone's cron job parses it), and timing/ordering quirks nobody promised.
• The asymmetry that explains maintainer conservatism: a break costs every downstream user simultaneously; a workaround costs only the project. Multiply by ten thousand clusters and "just fix it properly" loses to "carry the shim" almost every time.
• The institutional response you'll study in Phase 04: annotate the surface (audience/stability), gate releases on compat tooling (japicmp), and make deprecation a multi-release ceremony rather than an event.

Chapter 6: Extracting Lessons — The Post-Mortem Method

The lab's template, and why each section exists:

1. Decision & context — what was decided and what was true at the time (workloads, team size, precedents). Forces steel-manning; no hindsight allowed yet.
2. Mechanism of failure — the technical chain from decision to pain, specific enough to be falsifiable (heap growth → GC pause → failover storm, with numbers where findable).
3. Detection lag — when did evidence appear vs when was it acted on? (NameNode HA: evident ~2008, shipped ~2012 — the lag itself is a finding about incentives.)
4. Cost of the fix — engineering years, migration burden, complexity added. This is the number that prices future shortcuts.
5. Transferable principle — one sentence, stated so it could guide a different system's review. If it only describes the past, it's a summary, not a lesson.

The test of a good post-mortem: a reader could apply section 5 in a design review next week and cite sections 2–4 when challenged.

Lab Walkthrough Guidance

Order: Lab 01 (post-mortem) → Lab 02 (compatibility burden).

• Lab 01: pick ONE case (Chapter 2's NameNode is the best-documented). Source from primary materials — JIRAs (HDFS-1064, HDFS-1623 for HA), design docs, conference talks — not blog summaries. Fill the template in Chapter 6's order; spend the most effort on context (steel-man) and cost (numbers).
• Lab 02: pick one API/format (Hadoop FileSystem and Hive SerDe are the canonical choices); inventory what compatibility dimensions it carries (Chapter 5's list); find two real JIRAs where a desired change was constrained or abandoned for compat reasons; estimate the carried cost (shims, docs, support) per year.

Success Criteria

You are ready for Phase 04 when you can, from memory:

1. Tell each of the three cases as decision → context → mechanism → fix-cost → principle, in two minutes each.
2. Explain why the Secondary NameNode is not HA, and what QJM/standby actually added.
3. State the v1 fusion sin (platform = first app) and name two non-Hadoop systems that repeated it.
4. List six kinds of compatibility surface beyond method signatures.
5. Apply one transferable principle, with citation, to a hypothetical new design in one paragraph.

Interview Q&A

Q: Tell me about a system design failure you've studied and what you took from it. Use the Lab 01 case in the five-part structure — context first ("with 2006 workloads and GFS as precedent, the single NameNode was defensible"), then mechanism, the five-year fix lag and its cost, ending with the principle you now apply ("any singular stateful coordinator gets asked: what's the namespace ceiling, what's the failover story, before v1 ships"). The structure itself signals seniority.

Q: Why do mature projects move so slowly on obvious improvements? The asymmetry of Chapter 5: a break is paid by every downstream simultaneously and erodes the trust that is the project's product; a carried wart is paid only by maintainers. Add governance (consensus, vetoes) designed to protect decade-scale stability over quarter-scale velocity. Slow is the feature, priced deliberately.

Q: When is a breaking redesign the right call despite all that? When the carried cost compounds (every new feature pays the wart tax), when the fix cannot be incremental (data-model walls, Hive metastore-style), or when security/correctness is unfixable in place. Then: major version, strangler migration with both paths alive, executable migration tooling, and the Phase 11/12 consensus process — the manner of breaking is what separates Hadoop 2 from Python 3.

References

• Shvachko et al., The Hadoop Distributed File System (MSST 2010) — the NameNode design from the source
• HDFS-1623: High Availability Framework for HDFS NN — read the design doc and discussion
• Vavilapalli et al., Apache Hadoop YARN: Yet Another Resource Negotiator (SoCC 2013) — §2 "History and rationale" is the MR post-mortem in the authors' own words
• Saha et al., Apache Tez: A Unifying Framework for Modeling and Building Data Processing Applications (SIGMOD 2015)
• Zaharia et al., Resilient Distributed Datasets (NSDI 2012) — read against the MR limitations it answers
• Hive SerDe docs and Iceberg spec — the burden and the escape
• Hyrum's Law
• Hadoop Compatibility Specification

🛸 Hitchhiker's Guide — Phase 03: Lessons from Large Apache Systems

Read this if: You can navigate a JVM codebase but you've never studied why large distributed systems fail — not in theory, but by tracing real architectural decisions from motivation to failure to recovery. By the end you should be able to explain Hadoop's NameNode SPOF, Hive's metastore bottleneck, and the MapReduce DAG limitation, and connect each failure to a pattern that still applies in systems you'll work on today.

0. The 30-second mental model

Every large Apache system failed in a predictable way. The failure wasn't random — it followed from the original design constraints being violated by scale:

Original Design Constraint         What Violated It
────────────────────────────────   ──────────────────────────────────────
NameNode manages all metadata      Namespace grew to billions of files
Metadata fits in JVM heap          It didn't (at 100M+ files)
MapReduce is general enough        Iterative algorithms = absurd disk I/O
HDFS is the only storage           It became a bottleneck; S3 arrived
Hive runs on HDFS + MapReduce      It was too slow for interactive use
Hive metastore is a SQL database   Became serialization bottleneck at scale

The lesson is not that these were bad engineers. The lesson is that scale changes the dominant constraint. The dominant constraint at design time (correctness, simplicity, getting it to work at all) is replaced at scale by a new dominant constraint (throughput, availability, coordination overhead). Systems that handle this well (YARN, Spark's Catalyst) separate concerns early. Systems that handle it badly become legacy.

1. Hadoop 1.x: The NameNode SPOF

1.1 What the NameNode Does

In HDFS, the NameNode is a single JVM process that maintains the entire filesystem namespace in memory. It stores:

• The full directory tree (all files and directories)
• For every file: its block list, replication factor, and owner/permissions
• Block-to-DataNode mapping (which DataNode hosts each replica)

The design is elegant: all metadata in one place, all operations O(1) or O(path depth). The DataNodes are dumb storage nodes that just hold blocks and report heartbeats.

               NameNode (single JVM)
              ┌──────────────────────────────────┐
              │  Namespace: /user/alice/data.csv  │
              │  Blocks: [blk_001, blk_002]       │
              │  Locations: [DN1, DN3], [DN2, DN4]│
              └───────────┬──────────────────────┘
                          │ Heartbeat every 3s
         ┌────────────────┼────────────────┐
         ▼                ▼                ▼
    DataNode 1       DataNode 2       DataNode 3
    [blk_001,        [blk_002,        [blk_001,
     blk_005, ...]    blk_007, ...]    blk_009, ...]

1.2 Why It Became a SPOF

The NameNode is a single point of failure in multiple dimensions:

1. Process failure: If the NameNode JVM crashes, the entire HDFS cluster is unavailable — zero reads, zero writes. All running MapReduce jobs fail.
2. GC pressure: As namespaces grew to 100M+ files, the NameNode JVM heap grew to 50–100GB. Full GC pauses of seconds caused cluster-wide stalls. The NameNode heap was the hard ceiling on cluster scale.
3. Software maintenance: Any NameNode upgrade required a cluster outage. At Yahoo's scale (2010), an upgrade of the 4,000-node production cluster required hours of downtime.

1.3 Why Fixing It Took Years

Fixing a SPOF in a production distributed system is not a weekend project. The Hadoop NameNode HA solution (shipped in Hadoop 2.0, 2012) required:

• Primary/Standby NameNode: Two NameNode processes in active-standby mode
• JournalNodes: A quorum (3+) of JournalNode processes that store the edit log; both NameNodes read from JournalNodes so the standby stays in sync
• ZooKeeper Fencing: ZooKeeper elects the active NameNode; fencing ensures the old active cannot corrupt the namespace after a failover
• DataNode dual-reporting: DataNodes send block reports and heartbeats to both NameNodes

ZooKeeper (3 nodes)
     │ leader election
     ▼
Active NameNode ──writes edits──▶ JournalNode 1
Standby NameNode ◀─reads edits─── JournalNode 2
                                   JournalNode 3
     │
     └── Both receive heartbeats from all DataNodes

The time cost wasn't the engineering — it was the risk management. Replacing the metadata layer of a production system serving tens of thousands of jobs per day while maintaining zero data loss requires years of testing, staged rollout, and fallback planning.

1.4 HDFS Federation: The Namespace Scalability Fix

HA solved availability but not namespace scale. Even two NameNodes, each managing the full namespace, still hit the JVM heap ceiling.

HDFS Federation (Hadoop 2.x) partitions the namespace: different NameNodes manage different subtrees of the directory hierarchy. DataNodes belong to multiple "block pools," one per NameNode.

The lesson: centralized state does not scale horizontally. You can replicate it (HA), but to scale throughput you must partition it. This lesson recurs in every distributed database, every distributed queue, and every distributed filesystem.

2. MapReduce's Two-Phase Limitation and the Rise of Tez

2.1 The MapReduce Programming Model

MapReduce forces every computation into two phases: Map and Reduce, with a mandatory shuffle between them. For simple batch jobs this is elegant. For complex algorithms, it is a strait-jacket.

Input ──▶ Map ──▶ [Shuffle + Sort] ──▶ Reduce ──▶ Output
                        │
              Disk write on every shuffle
              Disk read on every reduce input

Key cost: Every shuffle writes to disk. Every reduce input reads from disk. This is intentional (fault tolerance via materialization) but catastrophic for iterative algorithms.

2.2 Why Iterative Algorithms Are Expensive in MapReduce

Machine learning, graph algorithms, and many analytics pipelines are iterative: run the algorithm, update a state, run it again, converge.

In MapReduce, each iteration is a separate job:

Iteration 1: Input → M1 → shuffle → R1 → HDFS write
Iteration 2: HDFS read → M2 → shuffle → R2 → HDFS write
Iteration 3: HDFS read → M3 → shuffle → R3 → HDFS write
...

100 iterations of PageRank = 200 HDFS writes and 200 HDFS reads. At 10TB/iteration = 2PB of I/O for an algorithm that could run in memory in 10 passes.

This is why Apache Spark was created: in-memory RDD lineage eliminates the intermediate HDFS writes. The DAGScheduler pipelines transformations and only writes to disk at shuffle boundaries.

2.3 Apache Tez: The DAG Solution Within the Hadoop Ecosystem

Rather than replacing MapReduce entirely, Tez (Apache project, emerged from Hortonworks) generalized it to a DAG of tasks:

MapReduce model:   Input ──▶ M ──▶ R ──▶ Output

Tez model:         Input ──▶ Vertex ──▶ Vertex ──▶ Vertex ──▶ Output
                                 │                    ▲
                                 └────────────────────┘
                                   (direct connection, no mandatory disk)

Each Vertex in Tez is an arbitrary processing unit. Edges between vertices define data movement (memory, disk, or network). Crucially: intermediate results do not need to be materialized to HDFS.

Hive on Tez (Hive 0.13+) replaced the Hive-on-MapReduce execution engine and significantly improved multi-stage query performance.

2.4 What Tez Got Right and Wrong

Right:

• Generalized the execution model beyond two phases
• Stayed within the YARN/Hadoop ecosystem (easier adoption)
• Hive on Tez became production-ready and widely deployed

Wrong:

• Still JVM-process-per-task overhead (compared to Spark's in-memory pipelines)
• The DAG API is low-level; most users interact through Hive, not Tez directly
• Spark's simpler programming model (RDD/DataFrame) won the developer mindshare

The lesson: a generalization that stays compatible with the existing ecosystem will be adopted; a generalization that requires abandoning the ecosystem will only be adopted if the performance gap is extreme. Spark's performance gap was extreme enough.

3. Hive: The Metastore Bottleneck

3.1 What the Hive Metastore Is

Hive is a SQL-on-Hadoop system. It stores table metadata in a relational database (Derby, MySQL, or PostgreSQL) called the Hive Metastore. The metastore holds:

• Table schemas (column names, types, comments)
• Partition metadata (for partitioned tables: which partitions exist and where)
• Storage descriptors (HDFS location, input format, output format, SerDe)
• Statistics (table/column row counts, for query planning)

3.2 Why It Became a Bottleneck

HiveServer2 (multiple instances, accepts JDBC connections)
        │
        ▼
Hive Metastore Service (single process, or a few)
        │
        ▼
Relational Database (MySQL or PostgreSQL, single primary)
        │
        Tables: TBLS, SDS, COLUMNS_V2, PARTITIONS, PARTITION_KEYS, ...

Every Hive query requires at least one metastore call to resolve table schemas. Large tables with thousands of partitions require thousands of metastore calls to enumerate and prune partitions. Every ALTER TABLE, MSCK REPAIR, and CREATE TABLE is a metastore write.

At scale:

• A table with 1 million partitions (e.g., hourly partitions for 5 years × 100 tenants) makes partition enumeration extremely slow
• The metastore becomes a serialization bottleneck: all writers contend for the same relational database rows
• Metastore latency directly degrades query latency for the entire cluster

3.3 The Fixes (and Why They Were Slow to Arrive)

The slowness of fixes reflects a fundamental truth: the metastore is the integration point for dozens of tools (Hive, Spark, Presto, Flink, Impala, Pig). Replacing it requires all these tools to migrate simultaneously, which is a coordination problem, not an engineering problem.

4. The SerDe Interface and the Cost of Backward Compatibility

4.1 What a SerDe Is

In Hive, a SerDe (Serializer/Deserializer) defines how to read bytes from HDFS and deserialize them into Hive rows — and how to serialize Hive rows back to bytes for writing.

// Simplified SerDe interface
public interface Deserializer {
    void initialize(Configuration conf, Properties tbl) throws SerDeException;
    Object deserialize(Writable blob) throws SerDeException;
    ObjectInspector getObjectInspector() throws SerDeException;
}

This interface has been part of Hive since 2008. It is used by hundreds of production deployments and third-party tools. Every custom file format (proprietary binary formats, legacy log formats, XML, JSON) has a SerDe implementation.

4.2 The Backward Compatibility Trap

Once an interface is used in production by external code you don't control, you cannot change its method signatures without breaking that code. This is the backward compatibility constraint.

For the SerDe interface, this means:

• You cannot add a new method without providing a default implementation (Java 8+ default methods)
• You cannot change the semantics of existing methods without silently breaking implementations that rely on the old semantics
• You cannot replace Writable with a more modern serialization format without rewriting every SerDe implementation

The result: the SerDe interface from 2008 is still in use in 2025. The Writable type (from Hadoop's own legacy serialization framework) is still its parameter type, even though every modern Hadoop component has moved away from Writable to Protobuf or Avro.

4.3 The Recurring Pattern

This pattern appears in every long-lived Apache project:

The lesson: the first API you ship publicly is the one you will maintain forever. Design it for the constraints you expect at 10x your current scale, because you will not get a chance to redesign it once it has users.

5. The Cross-Cutting Patterns

Reading these four failure modes together, you see the same patterns recurring:

5.1 Centralization Doesn't Scale

• NameNode: one process for all metadata → SPOF + heap bottleneck
• Hive Metastore: one RDBMS for all table metadata → write contention bottleneck
• MapReduce's single shuffle coordinator: single sort stage handles all data → disk I/O bottleneck

Pattern: Any single process that all other processes must talk to will eventually become the bottleneck. The fix is always partitioning, caching, or eliminating the coordination point.

5.2 Interfaces Outlive Their Context

• SerDe was designed for Hadoop 1.x's Writable-based world. It still exists in Hadoop 4.x.
• Hadoop's FileSystem API was designed for HDFS. It now has implementations for S3, GCS, Azure Blob Storage, and a dozen other backends — all constrained by an API that was never designed for eventually-consistent object stores.

Pattern: Every public interface you ship will be used in ways you did not anticipate, and will need to support backends that do not exist when you write it. Design interfaces for the abstraction, not the implementation.

5.3 Compatibility Prevents Correctness Fixes

Hadoop's FileSystem.rename() is supposed to be atomic on POSIX filesystems. On S3, rename is a copy + delete — not atomic. But the API contract says rename is atomic, so Hadoop code that relies on atomic rename is silently broken on S3. Fixing this requires changing the semantics of a public API used by thousands of programs — impossible without breaking them.

Pattern: Once a correctness guarantee is documented in a public API, you cannot weaken it without a major version bump and a migration guide. Build only the guarantees you can actually enforce.

5.4 The "Good Enough" Trap

Hadoop 1.x's NameNode was good enough for 2006-scale HDFS (tens of millions of files). MapReduce was good enough for 2008-scale batch jobs. Hive's metastore was good enough for 2010-scale tables.

All three became technical debt when scale increased by 100x. The engineering work to fix them was not hard — it was risky and slow because of the coordination required to migrate production systems.

Pattern: Technical debt is not about bad code. It is about constraints that were reasonable at design time becoming wrong at scale. The cost of paying the debt grows with the number of dependent systems, not with the complexity of the fix.

6. Interview Questions — Concepts You Must Explain Out Loud

7. References

• HDFS Architecture Guide — Apache docs — the authoritative description of NameNode design
• HDFS High Availability (Apache docs) — HA with JournalNodes
• HDFS Federation Design Doc (Apache JIRA: HADOOP-7684)
• Apache Tez — Bikas Saha, Hortonworks — original design presentation
• Hive Metastore documentation
• The Hadoop Distributed File System — Shvachko et al., MSST 2010 — the original HDFS paper
• MapReduce: Simplified Data Processing on Large Clusters — Dean & Ghemawat, OSDI 2004 — the original MapReduce paper
• Designing Data-Intensive Applications — Martin Kleppmann, O'Reilly 2017 — Chapter 10 (Batch Processing) and Chapter 11 (Stream Processing) for context

Lab 01 — Post-Mortem: A Known Design Failure

Phase: 03 — Lessons from Large Apache Systems
Difficulty: ⭐⭐☆☆☆ | Estimated Time: 3–4 hours
Type: Research / Document
Output: post-mortem-[failure-name].md

Goal

Produce a structured post-mortem for one real design failure in the Apache Hadoop ecosystem. The post-mortem must explain: what was built, why it was built that way, at what scale it failed, what the failure cost, how it was fixed, and what engineering lessons generalize beyond this specific case.

This is not a blame exercise. It is a learning exercise. The engineers who built these systems were excellent. The failures are interesting precisely because they were not caused by incompetence — they were caused by constraints that seemed reasonable at design time becoming wrong at scale.

Choose One Failure

Recommendation: If you are less familiar with Hadoop internals, choose Option A. The NameNode SPOF is well-documented, the fix (HA) is concrete, and the failure mode is easy to reason about. Option B requires understanding more of the query execution stack.

Research Sources

Option A — Hadoop NameNode SPOF:

• HDFS Architecture (Apache docs)
• HDFS High Availability (Apache docs)
• HADOOP-7684 (HDFS Federation design JIRA)
• The Hadoop Distributed File System — Shvachko et al., MSST 2010
• HDFS HA at Yahoo (engineering blog, 2012)

Option B — Hive Metastore Bottleneck:

• Hive Metastore Administration (Apache wiki)
• HIVE-7155 (Metastore performance JIRA)
• Facebook's Hive at scale post (original blog, 2011)
• Search for "hive metastore bottleneck" on the Hive dev mailing list archives: lists.apache.org/list.html?dev@hive.apache.org

Steps

Step 1 — Read the Primary Sources (60–90 min)

Read at least 2 primary sources for your chosen failure. Take notes on:

1. When was this system built, and what problem did it solve?
2. What was the original scale assumption?
3. At what scale did the failure become apparent?
4. What were the symptoms that users experienced?
5. How long did it take to fix? What was the fix?

Step 2 — Understand the Technical Root Cause (30 min)

Go beyond "the NameNode was a SPOF." Understand:

• For Option A: Why does a single JVM process create both availability and scale problems? What is the heap ceiling? What does JournalNode quorum buy you?
• For Option B: Why does partition enumeration hit the metastore so hard? What SQL queries does Hive run against the metastore schema? What is the contention point?

Step 3 — Analyze the Fix (30 min)

Read the fix (HA for Option A, or partition pruning/LLAP/table formats for Option B):

• What architectural change was required?
• What was the migration cost for existing clusters?
• Did the fix fully solve the problem, or introduce new constraints?

Step 4 — Extract the Generalizable Lessons (20 min)

What does this failure teach you that applies beyond this specific system? Think about:

• Centralization and its limits
• Scale changes the dominant constraint
• Backward compatibility and migration cost
• The cost of tight coupling between components

Step 5 — Write the Post-Mortem (30–45 min)

Fill out post-mortem-template.md for your chosen failure. Rename it to post-mortem-namenode-spof.md or post-mortem-hive-metastore.md.

Expected Output

lab-01-post-mortem/
├── README.md                           ← this file
└── post-mortem-[failure-name].md       ← your completed post-mortem

Verification

Your post-mortem is complete when:

1. You can explain the failure from first principles in 3 minutes without notes
2. Your technical root cause section contains a specific class, process, or data structure — not just "it didn't scale"
3. Your generalizable lessons are stated as patterns applicable to other systems — not just observations about Hadoop
4. You can answer: "What would you have done differently at design time, given the information available then?"

Talking Points

• "Why did it take so long to fix the NameNode SPOF?" The HA fix required a new JournalNode quorum service, ZooKeeper integration, DataNode dual-reporting, and fencing — all while maintaining backward compatibility with existing clients. The engineering was not hard; the risk management was.
• "Why not just use a distributed metadata store from the start?" In 2006, Chubby (Google's distributed lock service) was not open source and ZooKeeper didn't exist yet. The single-node design was reasonable given what was available.
• "What replaced the Hive Metastore?" Modern table formats (Apache Iceberg, Delta Lake, Apache Hudi) store table metadata alongside the data files, eliminating the central metastore dependency for reads. But the metastore is still used for discoverability and catalog integration.

Resume Bullet

"Conducted structured post-mortem analysis of a production-scale Apache system design failure (Hadoop NameNode SPOF or Hive Metastore bottleneck); documented root cause, failure timeline, fix architecture, and generalizable distributed systems lessons."

Lab 02 — Compatibility Burden Analysis

Phase: 03 — Lessons from Large Apache Systems
Difficulty: ⭐⭐⭐☆☆ | Estimated Time: 4–5 hours
Type: Code Reading / Document
Repository: apache/hadoop or apache/hive
Output: compatibility-analysis-[interface-name].md

Goal

Analyze a real, long-lived Apache interface and document the compatibility burden it has accumulated — the constraints it imposes on new implementations, the semantic guarantees it cannot enforce on all backends, and the design choices that would have reduced this burden.

After this lab you will understand why committers are conservative about API changes and why "just add a method" is rarely simple in a project with thousands of downstream users.

Choose One Interface

Recommendation: Option A (FileSystem) is better for engineers who know POSIX filesystems. Option B (SerDe) is better for engineers who've worked with Hive or custom data formats.

Setup

# Option A — Hadoop FileSystem
git clone https://github.com/apache/hadoop.git
cd hadoop
git checkout rel/release-3.3.6   # stable recent release

# The interface file:
# hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java

# Option B — Hive SerDe
git clone https://github.com/apache/hive.git
cd hive
git checkout rel/release-3.1.3

# The interface file:
# serde/src/java/org/apache/hadoop/hive/serde2/Deserializer.java
# Related: AbstractSerDe.java, ObjectInspector.java

Steps

Step 1 — Read the Interface in Full (45 min)

Open your chosen interface file. Read every method:

1. For each abstract/non-default method, write one sentence describing what it is supposed to do
2. Note the parameter types and return types
3. Identify which methods are the "core contract" (must be implemented) vs which are optional extensions
4. Note the @Deprecated annotations — each one is a story

# Count abstract methods (Java)
grep -c "abstract" FileSystem.java

# Find deprecated methods
grep -n "@Deprecated" FileSystem.java

# Find methods added in specific Hadoop versions (look for @since in Javadoc)
grep -n "@since" FileSystem.java

Save an annotated method list in ## Interface Methods.

Step 2 — Count and Categorize the Implementations (30 min)

For FileSystem: find all production implementations in the Hadoop ecosystem:

# In the Hadoop repo:
find . -name "*.java" -exec grep -l "extends FileSystem" {} \;

# Key implementations to find:
# - LocalFileSystem (POSIX local disk)
# - DistributedFileSystem (HDFS)
# - S3AFileSystem (AWS S3)
# - GCSFileSystem (Google Cloud Storage — in hadoop-connectors repo)
# - AzureBlobFileSystem (Azure — in hadoop-azure module)
# - FTPFileSystem
# - ViewFileSystem (federation namespace)

For each implementation, note:

• Which methods are overridden
• Whether the implementation fully honors the documented semantics

Save this in ## Implementations.

Step 3 — Find the Semantic Gaps (60–90 min)

This is the core task. Find methods where the documented semantics cannot be fully honored by some implementations.

For FileSystem — key semantic gaps to find:

1. rename(Path src, Path dst): Documented as atomic (POSIX semantics). S3 implements as copy + delete. Find the Hadoop JIRA where this was debated.
2. delete(Path f, boolean recursive): Find the consistency semantics on S3 vs HDFS.
3. append(Path f, ...): Find which implementations return UnsupportedOperationException.
4. listStatus(Path f): On S3, there's no such thing as a "directory" — listing is eventually consistent. Find how S3AFileSystem handles this.

For each semantic gap:

• What does the Javadoc promise?
• What does the S3 (or other) implementation actually do?
• What breaks if caller code relies on the promised behavior?
• Is there a JIRA or mailing list thread about this?

For SerDe — key semantic gaps to find:

1. Writable input type: The deserialize(Writable blob) signature assumes the data arrives as a Hadoop Writable. How does this constrain column-oriented formats like ORC and Parquet?
2. ObjectInspector pattern: The getObjectInspector() method returns an ObjectInspector tree describing the schema. How does this interact with schema evolution?
3. initialize(Configuration conf, Properties tbl): All configuration is passed as a flat Properties map. What is lost?

Save your findings in ## Semantic Gaps.

Step 4 — Find the Compatibility Constraints (45 min)

Look at the interface's git history to understand the evolution:

# Interface evolution over time
git log --oneline -- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java | head -40

# What changed in a specific commit
git show <commit-hash> -- [file]

# Find when a specific method was added
git log -S "public abstract FileStatus getFileStatus" --oneline

For each meaningful change in the last 5–10 years:

• Was the change backward-compatible?
• Was it a new abstract method (breaking), a default method (safe), or a deprecation?
• What required a major version bump?

Save in ## Evolution History.

Step 5 — Write the Analysis (30–45 min)

Fill out compatibility-analysis-template.md. Your analysis should answer:

1. What is this interface's core abstraction?
2. What semantic guarantees does it make that cannot be honored by all backends?
3. What would you change if you could start fresh, with only the knowledge available when it was designed?
4. What is the cost of those semantic gaps in production systems?

Expected Output

lab-02-compatibility-burden/
├── README.md                                        ← this file
└── compatibility-analysis-[interface-name].md       ← your completed analysis

Verification

Your analysis is complete when:

1. You can name at least 3 methods with semantic gaps and explain the gap precisely
2. You can explain why adding a new abstract method to FileSystem would break thousands of programs
3. You can describe an alternative interface design that would reduce the compatibility burden
4. You can answer: "Why doesn't Apache just release a v2 of this interface?"

Talking Points

• "Why not just release FileSystem v2?" S3AFileSystem is in production at AWS, Google, Netflix, Databricks, and every major cloud vendor. Each has their own patched version. A v2 that breaks the interface means all of them have to migrate simultaneously — coordination that's effectively impossible to organize.
• "What's wrong with Writable?" It's a Hadoop-specific serialization format from 2003. Every modern system (Arrow, Protobuf, Avro, ORC, Parquet) uses a different format. A SerDe interface that takes Writable cannot efficiently read Parquet without converting through Writable first.
• "What would a better interface design look like?" Apache Arrow's RecordBatch as the exchange format; backends implement RecordBatchReader. No Writable. Schema as a first-class object, not Properties. Default no-op implementations for optional capabilities. This is essentially what Apache Iceberg's FileIO interface does for storage.

Resume Bullet

"Analyzed backward compatibility constraints in a long-lived Apache interface (Hadoop FileSystem or Hive SerDe); documented semantic gaps between the API contract and real-world backend behavior, evolution history, and alternative design approaches for a lower-maintenance interface."

Phase 04 — ASF Governance and Release Engineering

Why this phase exists: A release is a legally binding act. When you vote +1 on an Apache release you are personally attesting that you checked the artifacts, verified the signatures, and believe the code meets the ASF's legal and technical standards. This phase teaches the mechanics of how that actually works.

What You'll Learn

• How Apache releases are planned, branched, and built
• What an RC (Release Candidate) is, how it is built and distributed, and how the vote works
• How GPG signing and SHA-256 checksums create the verifiable chain of custody from committer to end user
• How the Maven Release Plugin automates the mechanical parts of a release
• How to use japicmp to detect binary and source compatibility regressions before a release ships
• Common release failures and how mature projects prevent them

Concepts Covered

Labs

Deliverables Checklist

• mvn release:prepare -DdryRun=true output shows correct version bump and tag name
• mvn release:prepare produces a Git tag and a bumped SNAPSHOT version
• sign-artifact.sh produces .asc and .sha256 files for a given input artifact
• verify-release.sh correctly verifies both signature and checksum, prints clear pass/fail
• mvn install in lab-03-api-compatibility/ produces a japicmp report identifying the planted breaking changes
• You can identify all breaking changes in the v1→v2 diff from the japicmp output

Interview Relevance

Every Staff+ distributed systems role at a company that ships open-source software (Databricks, Cloudera, LinkedIn, Uber, Confluent, Elastic) will ask some version of:

• "Walk me through how you would cut a release for an open-source library."
• "What does binary compatibility mean, and how do you validate it before a release?"
• "What's your release checklist look like? What's the worst release failure you've seen?"
• "Why does the Apache release process require GPG signatures? What does it prove?"

This phase gives you concrete, hands-on answers to all four.

Warmup Guide — ASF Governance & Release Engineering

Zero-to-expert primer for Phase 04: how an Apache release actually ships — votes and vetoes, GPG signing and verification, license hygiene, and binary-compatibility checking — assuming no prior release-engineering background.

Table of Contents

• Chapter 1: What a "Release" Legally Is
• Chapter 2: Votes, Vetoes, and Consensus — The Decision Mechanics
• Chapter 3: The Release Lifecycle End to End
• Chapter 4: Cryptographic Trust — GPG, Checksums, KEYS
• Chapter 5: License Hygiene — LICENSE, NOTICE, and Category X
• Chapter 6: Compatibility Checking — Semver and japicmp
• Chapter 7: Maven Release Mechanics
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: What a "Release" Legally Is

From zero: an ASF release is the source tarball that the PMC voted on — not the git tag, not the binaries, not the Docker image. The foundation legally distributes source code under the Apache License; the voting, signing, and distribution rules all protect that artifact. Binaries are "convenience artifacts": built from the source, nice to have, not the thing.

Why this matters practically: release-blocking review focuses on the tarball — does it build from scratch, are LICENSE/NOTICE correct for what's inside it, does it contain no compiled jars or category-X code (Chapter 5), does it match the tag. A broken convenience binary is an embarrassment; a tainted source tarball is a legal problem for the foundation. This is also why every PMC vote email says "I checked: signatures, build, license files" — the votes are attestations, and casting one without checking is the governance sin.

Chapter 2: Votes, Vetoes, and Consensus — The Decision Mechanics

The precise rules (commonly half-known; know them fully):

• +1 / 0 / -1 are conventions for approve / abstain / object, but their force depends on the question:
• Code modification votes: a committer/PMC -1 with technical justification is a veto — it cannot be outvoted, only resolved (address the objection or persuade the vetoer). An unjustified -1 is invalid. Vetoes exist so that majority enthusiasm can't override one person who sees the data-loss bug.
• Procedural votes (releases, adding committers): majority rules — a -1 on a release is NOT a veto. A release passes with ≥3 binding +1s and more binding +1 than -1, after a minimum 72-hour window. ("Binding" = PMC members; everyone may vote, and non-binding votes carry real persuasive weight.)
• Lazy consensus: announce intent, wait 72h, proceed absent objection — the default for low-controversy work; never for API/compat changes.
• Where: on the dev list, in a [VOTE] thread, with a [RESULT] posted. Votes in chat or PRs bind nobody.

Chapter 3: The Release Lifecycle End to End

The pipeline the RM (release manager — any committer/PMC volunteer) drives:

1. Branch & stabilize: cut branch-X.Y; only blockers (tracked in JIRA against the version) land; master moves on.
2. Produce RC: versions set (X.Y.Z no SNAPSHOT), build from a clean checkout, produce the source tarball + convenience binaries, sign (Ch. 4), checksum, stage — Maven artifacts to a Nexus staging repo, tarballs to dist.apache.org/repos/dist/dev.
3. [VOTE] thread: links to artifacts, tag, staging repo; voters verify and report what they checked. 72h minimum.
4. Fail → RC2: any real defect sinks the RC; fix, re-spin, re-vote (release threads routinely reach RC4 — normal, not failure).
5. Pass → publish: move artifacts to dist/release (mirrors + downloads.apache.org), release the staging repo to Maven Central, push the final tag, update the site, [ANNOUNCE].
6. Aftercare: track upgrade reports; severe regressions trigger an X.Y.Z+1 patch release — the RM role rotates but the discipline persists.

Lab 01 (release-mechanics) walks a toy project through 1–2 and 5's mechanics; Phase 10's labs automate pieces of 3–4.

Chapter 4: Cryptographic Trust — GPG, Checksums, KEYS

The threat model: artifacts travel through mirrors the ASF doesn't control. Users must be able to verify (a) integrity — bytes unmodified — and (b) authenticity — built by the RM the PMC trusts.

• Checksums (.sha512): integrity only. Anyone who can replace the tarball can replace its checksum — necessary, insufficient.
• GPG signature (.asc): the RM signs with their private key; gpg --verify artifact.asc artifact proves the holder of that key produced these bytes. Authenticity now reduces to: is this key really the RM's?
• The KEYS file: each project publishes committer public keys at a canonical ASF URL (fetched over TLS from apache.org, not from the mirror). Stronger: keys are cross-signed at key-signing events (web of trust). Your verification script (Lab 02) imports KEYS, verifies the sig, checks the checksum, and — the step people skip — confirms the signing key's fingerprint appears in KEYS.
• RM hygiene: 4096-bit RSA, apache.org uid, key published before the vote; signing happens on a trusted machine (supply-chain 101).

Chapter 5: License Hygiene — LICENSE, NOTICE, and Category X

The part of release review that surprises engineers:

• LICENSE: the Apache-2.0 text plus the licenses of everything bundled in the artifact. If the source tarball vendors a BSD-licensed file, its license text appears here. Per-artifact accuracy — the binary's LICENSE differs from the source's because it bundles more.
• NOTICE: required attributions only (a legal instrument, not a thank-you list) — copyright notices that licenses require be preserved. Over-stuffing NOTICE is itself a review comment.
• Category A (BSD, MIT, ...): may be bundled. Category B (EPL, MPL, ...): binary-only bundling with conditions. Category X (GPL/LGPL, SSPL, JSON license, ...): may NOT be bundled in an ASF artifact at all — a GPL dependency in the tarball is an automatic release blocker. ("Optional runtime dependency the user installs" is the standard escape hatch.)
• This is checkable mechanically (mvn license:check, rat plugin for headers) and socially (the recurring "why is this jar in the source release?" vote -1).

Chapter 6: Compatibility Checking — Semver and japicmp

The governance promise "no breaking changes in a minor release" is enforceable only by tooling:

• The dimensions (Phase 03 Ch. 5, now mechanized): source, binary (the one tools check — and the one with surprises: changing a return type to a subtype breaks binary linkage even though sources recompile; adding a method to an interface broke all implementors pre-default-methods), behavioral (only tests catch), serialized forms.
• japicmp compares two jars and classifies every change (METHOD_REMOVED, METHOD_RETURN_TYPE_CHANGED, ...) against binary/source compat rules; run as a Maven plugin it fails the build on undeclared breakage; declared exceptions live in a reviewed exclusion file. Spark's MiMa is the Scala equivalent; the pattern — compat promises as CI gates with sign-off-required exclusions — is the institutional lesson.
• Scope control: the check applies to the annotated public surface (@InterfaceAudience.Public/@Stable in Hadoop; everything else is fair game — which is why the annotations exist (Phase 11 builds the migration tooling on top of exactly this).
• Lab 03 runs japicmp over a deliberately broken v1→v2 pair; Phase 10 Lab 02 then makes you implement a simplified checker — read the report here, build the analyzer there.

Chapter 7: Maven Release Mechanics

Enough mechanics to execute Lab 01 credibly:

• Versioning: X.Y.Z-SNAPSHOT = mutable dev version (re-resolved on each build); releases strip SNAPSHOT and are immutable forever (Central never lets you replace a released GAV — hence RC re-spins bump -rcN, not reuse).
• The release dance (whether via maven-release-plugin or scripts): set release version → commit → tag → build+sign+stage (gpg + deploy to a staging repo) → bump to next SNAPSHOT → commit. Staging repos are the key safety: artifacts are visible for vote-time verification but not yet public; "release" or "drop" is one button after the vote.
• Reproducibility: pin plugin versions, build from a clean clone in a container, record the JDK — "builds for the RM but not for voters" is a classic RC sinker. Modern projects aim for bit-reproducible builds (project.build.outputTimestamp).

Lab Walkthrough Guidance

Order: Lab 01 (release mechanics) → Lab 02 (signing) → Lab 03 (api compatibility).

• Lab 01: run the version dance on the toy project; produce a source tarball that builds from scratch in a clean directory; write the [VOTE] email you would send, with the checklist of what a voter should verify.
• Lab 02: generate a key, sign, verify; then break things deliberately — corrupt a byte (checksum catches), re-sign with a wrong key (KEYS-fingerprint check catches) — your script must catch both. The negative tests are the lab.
• Lab 03: run japicmp on the provided v1/v2; for each reported incompatibility, classify it source-vs-binary and write the one-line release-notes entry it would require; decide which changes could be made compatible (and how) vs which need a major version.

Success Criteria

You are ready for Phase 05 when you can, from memory:

1. State what artifact a release vote covers and why; recite the 3×+1/72h rule and the veto-vs-majority distinction with one example each.
2. Walk the six-stage release lifecycle naming the RM's action and the failure mode at each stage.
3. Explain the three-step verification (checksum, signature, KEYS fingerprint) and what attack each step closes.
4. Distinguish LICENSE from NOTICE, and categorize MIT/EPL/GPL bundling.
5. Give two binary-incompatible changes that are source-compatible, and describe how japicmp-as-CI-gate institutionalizes the promise.

Interview Q&A

Q: You're RM and a PMC member -1s your RC for a flaky test. Proceed? A release -1 is not a veto — formally I could pass with 3 binding +1s. Practically: investigate first (flaky test or real race?), respond with findings in-thread. If it's benign flakiness with evidence, say so and let the vote run; if there's any doubt about a correctness issue, re-spin — shipping over a PMC member's documented concern spends trust that outvalues any schedule. The formal rule and the social rule differ; a good RM knows both.

Q: Why can't the ASF just rely on HTTPS instead of GPG signatures? TLS protects the channel to a mirror, not the artifact's provenance — mirrors are third-party, and a compromised mirror serves valid-TLS poison. The GPG chain (signature

• KEYS fetched from apache.org) binds artifacts to a person the PMC vouches for, end-to-end, independent of any transport or host. Checksums alone fail the same way: co-located with the artifact, they're replaced together.

Q: A minor release accidentally shipped a binary-incompatible change. What now? First, scope: japicmp diff to inventory exactly what broke and for whom. Then the options, in order of preference: restore the old signature in a fast patch release (delegating to the new path — additive fix); if impossible, document loudly + provide a shim artifact. And the institutional fix: why didn't the gate catch it — missing annotation, exclusion-file rot, or the check not running on that module. The incident isn't closed until the gate gap is.

References

• Apache Release Policy and Release Distribution Policy
• Apache Voting Process
• ASF Third-Party License Policy (Category A/B/X)
• Apache Release Signing — the GPG/KEYS mechanics
• japicmp documentation and MiMa
• Hadoop Compatibility Specification
• Browse a real vote: search any project's dev-list archive for "[VOTE] Release" and read an RC1→RC3 saga end to end

🛸 Hitchhiker's Guide — Phase 04: ASF Governance and Release Engineering

Read this if: You can write good code but you've never shipped a release — or you've pushed to npm/PyPI and think Apache releases are similar. They are not. An Apache release is a legally binding vote by PMC members who are personally attesting to the quality of the artifact. By the end you should understand every step from JIRA planning to the announcement email, and know exactly what you are agreeing to when you write +1 on a vote thread.

0. The 30-second mental model

An Apache release is three things simultaneously:

1. A legal act: The PMC votes to release software under the Apache License 2.0. Each +1 vote is a personal attestation that the voter checked the artifacts.
2. A cryptographic chain of custody: GPG signatures link each artifact back to a committer whose key is in the project's KEYS file. Anyone can verify the chain.
3. A distribution process: Artifacts flow from dist.apache.org/dev/ (staging) → dist.apache.org/release/ (official) → Maven Central / mirror network (convenience). The Apache infrastructure is always the source of truth.

Committer machine
  │  mvn release:perform
  ▼
dist.apache.org/dev/[project]/[version]-RC1/
  │  [VOTE] email sent to dev@project.apache.org
  │  72-hour voting period
  │  ≥ 3 binding +1 votes, more +1 than -1
  ▼
dist.apache.org/release/[project]/[version]/
  │  Maven repository sync (24–48h)
  ▼
Maven Central / mirrors
  │  Announcement email to announce@apache.org
  ▼
End users

1. The Release Lifecycle

1.1 Planning Phase

Before a line of release prep is done, the release manager (RM) does:

1. JIRA release version: Create a version in JIRA (e.g., 4.0.0). Move unresolved issues out of scope.
2. Announce intent: Email dev@ list with the proposed release timeline. This is not optional — it gives the community a chance to land in-progress work.
3. Feature freeze: Announce a date after which only bug fixes and documentation are merged to the release branch.

1.2 Stabilization Branch

# Cut the stabilization branch from main (or the previous release)
git checkout -b branch-4.0 origin/main
git push origin branch-4.0

# From this point, main gets new features; branch-4.0 gets only targeted fixes
# Fixes are cherry-picked: git cherry-pick <commit-hash>

A stabilization branch serves two purposes:

• It isolates the release from new feature churn on main
• It makes it possible to patch a released version (e.g., 4.0.1) without including unrelated work from main

1.3 RC Numbering

Release Candidates are numbered RC0, RC1, RC2, etc.

There is no release-labeled artifact until a vote passes. Anything labeled RC is not a release.

1.4 Building the RC

The Maven Release Plugin (mvn release:prepare + mvn release:perform) handles the mechanical parts:

mvn release:prepare
  ├── Prompts for release version (e.g., 4.0.0) and next dev version (4.0.1-SNAPSHOT)
  ├── Updates pom.xml version strings
  ├── Commits the updated pom.xml
  ├── Creates a Git tag (e.g., v4.0.0-rc1)
  ├── Updates pom.xml to next dev version
  └── Commits the next dev version pom.xml

mvn release:perform
  ├── Checks out the tagged commit
  ├── Runs the full build (mvn deploy)
  └── Uploads artifacts to the staging repository

After release:perform, the RM signs the artifacts and uploads to dist.apache.org/dev/.

2. The Vote Thread

2.1 Vote Email Format

A vote email must contain, at minimum:

To: dev@[project].apache.org
Subject: [VOTE] Release Apache [Project] [Version] RC[N]

I propose we release Apache [Project] [Version] based on RC[N].

The RC artifacts are available at:
  https://dist.apache.org/repos/dist/dev/[project]/[version]-RC[N]/

Release artifacts:
  [project]-[version]-src.tar.gz
  [project]-[version]-src.tar.gz.asc
  [project]-[version]-src.tar.gz.sha256

Keys used to sign this release are available at:
  https://www.apache.org/dist/[project]/KEYS

The tag to vote on:
  https://github.com/apache/[project]/releases/tag/v[version]-rc[N]

Please vote:
  [ ] +1 Release this package as Apache [Project] [Version]
  [ ] +0 No strong preference
  [ ] -1 Do not release this package because: (state reason)

This vote is open for 72 hours.

2.2 Vote Arithmetic

A release passes when:

1. At least 3 binding +1 votes (from PMC members)
2. More binding +1 than binding -1 votes
3. The 72-hour minimum voting period has elapsed

A single PMC member's -1 (veto) does not automatically block a release but must be addressed. If the veto identifies a genuine legal or technical problem, it holds.

2.3 Result Email

To: dev@[project].apache.org
Subject: [RESULT][VOTE] Release Apache [Project] [Version] RC[N]

The vote has PASSED.

Binding votes:
  +1 Alice (PMC)
  +1 Bob (PMC)
  +1 Carol (PMC)

Non-binding votes:
  +1 Dave (committer)

Vote thread: [link to archive]

3. GPG Signing and the Chain of Trust

3.1 Why GPG Signing?

A SHA-256 checksum proves an artifact hasn't been corrupted in transit. It does not prove who created it. Anyone who can modify dist.apache.org could also update the checksum.

A GPG signature proves the artifact was created by the holder of a specific private key. If that key is listed in the project's KEYS file and the KEYS file is hosted on www.apache.org (which is controlled by Apache infrastructure, not by the project), the chain of trust is:

Your trust in ASF infrastructure
    → trust in KEYS file
    → trust in committer's public key
    → verified signature on artifact
    → trust in artifact

3.2 The Project KEYS File

Every Apache project maintains a KEYS file at https://www.apache.org/dist/[project]/KEYS. It contains ASCII-armored public keys for every committer who has ever signed a release. When a new committer joins and signs releases, they add their public key to KEYS via an SVN commit.

# Importing the KEYS file
curl https://www.apache.org/dist/spark/KEYS | gpg --import

# Verifying a release signature
gpg --verify spark-3.5.1.tgz.asc spark-3.5.1.tgz

3.3 Creating a GPG Key for Releases

# Generate a key (use RSA 4096 or ed25519)
gpg --full-generate-key
# Choose: RSA and RSA → 4096 bits → does not expire → your Apache email

# List your keys
gpg --list-secret-keys --keyid-format LONG

# Export your public key for KEYS file
gpg --armor --export your@apache-email.org

# Sign an artifact (detached, armored signature)
gpg --batch --yes --detach-sign --armor --output artifact.jar.asc artifact.jar

# Verify your own signature
gpg --verify artifact.jar.asc artifact.jar

3.4 Checksums

# Generate SHA-256 checksum
shasum -a 256 artifact.jar > artifact.jar.sha256    # macOS
sha256sum artifact.jar > artifact.jar.sha256        # Linux

# Verify (the file must have the exact format: "<hash>  <filename>")
shasum -a 256 --check artifact.jar.sha256  # macOS
sha256sum --check artifact.jar.sha256      # Linux

Note: Apache previously used MD5 and SHA-1. Both are now prohibited. SHA-256 or SHA-512 only.

4. API Compatibility Validation

4.1 Binary vs Source Compatibility

Apache projects generally guarantee binary compatibility within a major version (e.g., 3.x). Breaking binary compatibility requires a major version bump and a migration guide.

4.2 What Breaks Binary Compatibility

// v1 — stable API
public class DataProcessor {
    public void process(String input) { ... }      // callers compile against this
    public void setConfig(BatchConfig cfg) { ... }
}

// v2 — breaking changes
public class DataProcessor {
    public void process(String input, boolean strict) { ... }  // BREAKS callers of process(String)
    // setConfig removed                                        // BREAKS callers of setConfig
    public DataProcessor withConfig(BatchConfig cfg) { ... }   // new — non-breaking
}

Callers compiled against v1 will get NoSuchMethodError at runtime when run against v2.

4.3 Using japicmp

japicmp is the standard Java API compatibility checker used by many Apache projects.

# Run via Maven plugin (see lab-03)
mvn install   # builds both v1 and v2, then runs japicmp comparison

# Or via CLI
java -jar japicmp.jar --old v1.jar --new v2.jar --html-file report.html

japicmp classifies each change as:

• BINARY_INCOMPATIBLE: Will break compiled callers
• SOURCE_INCOMPATIBLE: Will break source compilation
• NON_BREAKING: New method/class added; safe
• EXCLUDED: Filtered out (e.g., internal classes)

5. The Distribution Flow and Legal Requirements

5.1 dist.apache.org

All Apache releases must be available on Apache infrastructure before they appear anywhere else. The flow:

1. RM uploads to dist.apache.org/repos/dist/dev/[project]/ (SVN)
2. Vote passes
3. RM moves to dist.apache.org/repos/dist/release/[project]/ (another SVN commit)
4. Apache mirrors sync within 24 hours
5. Maven Central sync happens via Nexus staging repository

Maven Central is a convenience copy. The official release is on dist.apache.org.

5.2 LICENSE and NOTICE Requirements

Every release archive must contain:

• LICENSE file: the Apache License 2.0 text, plus any third-party licenses for bundled code
• NOTICE file: copyright notice for the project, plus required notices from third-party dependencies (e.g., attribution notices from MIT/BSD code)

Missing or incorrect LICENSE/NOTICE is the most common cause of a -1 vote on legal grounds. The apache-rat Maven plugin checks for license headers:

mvn apache-rat:check

6. Common Release Failures

7. Interview Questions — Concepts You Must Explain Out Loud

8. References

• Apache Release Policy — the authoritative policy document
• How to Publish a Release (Apache guide)
• Apache Spark Release Process (community docs) — a real project's release runbook
• Maven Release Plugin
• japicmp — Java API Compatibility Checker
• The GNU Privacy Handbook — GPG key management
• Apache RAT (Release Audit Tool)
• Apache Legal — Third Party Licenses

Lab 01 — Release Mechanics

Phase: 04 — ASF Governance and Release Engineering
Difficulty: ⭐⭐⭐☆☆ | Estimated Time: 2–3 hours
Primary Artifact: Working mvn release:prepare run with Git tag
Verify: git log --oneline --decorate shows a v0.1.0 tag; pom.xml version is 0.2.0-SNAPSHOT

Goal

Run the full Maven Release Plugin workflow against a real local Git repository. By the end you will understand exactly what mvn release:prepare does, what commits it makes, what it tags, and why each step matters for a reproducible, traceable release.

The project ships with a small Java library (Calculator) and JUnit 5 tests that already pass. Your job is to wire up the release infrastructure and execute the release.

Setup

# 1. Enter the lab directory
cd phase-04-asf-governance-release/lab-01-release-mechanics

# 2. Initialize a local bare Git repository to act as the "remote"
#    (This simulates dist.apache.org or a GitHub remote without needing network access)
git init --bare ../release-remote.git

# 3. Initialize the lab project as a Git repo and push to the fake remote
git init
git add .
git commit -m "initial: add Calculator library"
git remote add origin ../release-remote.git
git push -u origin main

# 4. Verify the tests pass before touching the release plugin
mvn test

Expected output from Step 4:

[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS

Steps

Step 1 — Dry Run (read the plan before executing)

mvn release:prepare -DdryRun=true

Answer the prompts:

• What version do you want to release? → 0.1.0
• What is the SCM release tag? → v0.1.0
• What is the next development version? → 0.2.0-SNAPSHOT

After the dry run, inspect:

# What pom.xml would look like after release:prepare
cat pom.xml.tag           # the to-be-tagged version
cat pom.xml.next          # the next snapshot version
cat release.properties    # everything release:prepare recorded

Answer before continuing: What 3 commits would release:prepare make to Git?

# Clean up dry run artifacts before the real run
mvn release:clean

Step 2 — Execute the Release

mvn release:prepare

Use the same version answers as Step 1. Watch what happens. After it completes:

# Inspect the Git log
git log --oneline --decorate

# Inspect the tag
git show v0.1.0

# Inspect what changed between the two release commits
git diff HEAD~2 HEAD~1    # the "prepare release" commit
git diff HEAD~1 HEAD      # the "prepare for next development" commit

Questions to answer:

1. What is the version string in pom.xml on the tag v0.1.0? What is it on main now?
2. What does release.properties contain? Why does Maven save this?
3. What happens if you run mvn release:prepare again on the same version? Try it with -DdryRun=true.

Step 3 — Simulate release:perform

release:perform checks out the tagged commit, builds it, and deploys the artifacts. Since we don't have a staging repository, we'll simulate just the build step:

# Check out the tagged commit in a temp directory
git clone --branch v0.1.0 ../release-remote.git /tmp/release-check
cd /tmp/release-check

# Build and verify the release artifacts
mvn verify

# What SHA-256 is the produced JAR?
shasum -a 256 target/release-lab-0.1.0.jar      # macOS
sha256sum target/release-lab-0.1.0.jar           # Linux

Record the SHA-256. You will sign this artifact in Lab 02.

Step 4 — Write the Release Checklist

In release-checklist.md (create it), document the complete release steps you just performed as a numbered checklist with checkboxes. Format it as if you were writing the release runbook for a future release manager who has never done this before.

The checklist must include: branch cut, version bump, tag, build, artifact verification, GPG signing (placeholder), dist upload (placeholder), vote (placeholder), announcement (placeholder).

Expected Output

lab-01-release-mechanics/
  pom.xml
  src/
  release-checklist.md        ← your created checklist

And in Git:

$ git log --oneline --decorate
abc1234 (HEAD -> main) [maven-release-plugin] prepare for next development iteration
def5678 (tag: v0.1.0) [maven-release-plugin] prepare release v0.1.0
ghi9012 initial: add Calculator library

Verification

# Tag exists
git tag | grep v0.1.0

# Version on tag is 0.1.0 (not SNAPSHOT)
git show v0.1.0:pom.xml | grep '<version>'

# Version on main is next SNAPSHOT
grep '<version>' pom.xml | head -1

# Tests pass on the tagged commit
git stash
git checkout v0.1.0
mvn test
git checkout main

All four checks must pass.

Talking Points

• Why tag a release in Git? The tag is the only way to reproducibly check out exactly the code that was shipped. Without it, "the release build" means nothing — any commit could have been used.
• Why does release:prepare make two commits? The first commit sets the version to 0.1.0 (no SNAPSHOT suffix) — this is what gets tagged. The second commit bumps to 0.2.0-SNAPSHOT — this is what developers continue working from.
• What is release.properties? It stores the state of an in-progress release so release:perform (and release:rollback) know what to do. If you lose this file mid-release, you have to release:rollback and start over.

Source Files

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example.release</groupId>
    <artifactId>release-lab</artifactId>
    <!-- maven-release-plugin will rewrite this version during release:prepare -->
    <version>0.1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>Release Lab — Calculator</name>
    <description>
    A minimal Maven library used to practice the Maven Release Plugin workflow.
    The Calculator class provides basic arithmetic operations.
  </description>

    <!--
    SCM section: Required by maven-release-plugin.

    For this lab, we point to a local bare git repo you create with:
      git init --bare ../release-remote.git

    The ${project.basedir} resolves to the directory containing this pom.xml,
    so ../release-remote.git is one level up from the lab directory.

    In a real Apache project this would be:
      scm:git:https://gitbox.apache.org/repos/asf/[project].git
  -->
    <scm>
        <connection>scm:git:file://${project.basedir}/../release-remote.git</connection>
        <developerConnection>scm:git:file://${project.basedir}/../release-remote.git</developerConnection>
        <tag>HEAD</tag>
    </scm>

    <properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <!-- Pins the exact plugin versions so the release is reproducible -->
        <maven-release-plugin.version>3.0.1</maven-release-plugin.version>
        <maven-surefire-plugin.version>3.2.5</maven-surefire-plugin.version>
        <junit.version>5.10.2</junit.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <!--
        maven-surefire-plugin: runs JUnit 5 tests.
        Version 3.x supports JUnit Platform natively (no extra providers needed).
      -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>${maven-surefire-plugin.version}</version>
            </plugin>

            <!--
        maven-release-plugin: automates the release workflow.

        Key lifecycle:
          mvn release:prepare  — version bump, tag, push
          mvn release:perform  — checkout tag, build, deploy
          mvn release:rollback — undo a prepare (if something went wrong)
          mvn release:clean    — remove release.properties and *.backup files

        tagNameFormat: The format of the Git tag.
          @{project.version} resolves to the version being released (e.g., 0.1.0)
          We prefix with "v" (v0.1.0) — standard for most Apache projects.

        preparationGoals: Goals run during release:prepare (default: install).
          We run verify to ensure tests pass before the tag is created.

        autoVersionSubmodules: For multi-module projects, version all modules
          together. Setting to true prevents version drift between sibling modules.
      -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-release-plugin</artifactId>
                <version>${maven-release-plugin.version}</version>
                <configuration>
                    <tagNameFormat>v@{project.version}</tagNameFormat>
                    <autoVersionSubmodules>true</autoVersionSubmodules>
                    <preparationGoals>verify</preparationGoals>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <!--
    distributionManagement: Where mvn release:perform deploys artifacts.
    For this lab we use a local file-system repo so no network access is needed.
    Replace these paths with your Apache Nexus staging repository in a real project.
  -->
    <distributionManagement>
        <repository>
            <id>local-release</id>
            <url>file://${project.basedir}/../local-maven-repo</url>
        </repository>
        <snapshotRepository>
            <id>local-snapshots</id>
            <url>file://${project.basedir}/../local-maven-repo-snapshots</url>
        </snapshotRepository>
    </distributionManagement>

</project>

Calculator.java

package org.example.release;

/**
 * A basic four-function calculator.
 *
 * <p>
 * This class is the release artifact for Lab 01. Its purpose is to give the
 * Maven Release
 * Plugin something real to build, version, and tag — not to demonstrate complex
 * logic.
 *
 * <p>
 * Version history (maintained as release notes in this comment block):
 * <ul>
 * <li>0.1.0 — Initial release: add, subtract, multiply, divide</li>
 * </ul>
 */
public class Calculator {

    /**
     * Returns the sum of {@code a} and {@code b}.
     */
    public int add(int a, int b) {
        return a + b;
    }

    /**
     * Returns the difference {@code a - b}.
     */
    public int subtract(int a, int b) {
        return a - b;
    }

    /**
     * Returns the product of {@code a} and {@code b}.
     */
    public int multiply(int a, int b) {
        return a * b;
    }

    /**
     * Returns the integer quotient of {@code a / b}.
     *
     * @throws ArithmeticException if {@code b} is zero — not silently swallowed,
     *                             because hiding errors in a library violates the
     *                             principle of least surprise.
     *                             Callers that want a default value must catch this
     *                             explicitly.
     */
    public int divide(int a, int b) {
        if (b == 0) {
            throw new ArithmeticException("Division by zero: cannot divide " + a + " by 0");
        }
        return a / b;
    }
}

CalculatorTest.java

package org.example.release;

import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;

class CalculatorTest {

    private Calculator calc;

    @BeforeEach
    void setUp() {
        calc = new Calculator();
    }

    // -------------------------------------------------------------------------
    // Add
    // -------------------------------------------------------------------------

    @Test
    void add_positives() {
        assertEquals(5, calc.add(2, 3));
    }

    @Test
    void add_negativeAndPositive() {
        assertEquals(-1, calc.add(-4, 3));
    }

    @Test
    void add_zeros() {
        assertEquals(0, calc.add(0, 0));
    }

    // -------------------------------------------------------------------------
    // Subtract
    // -------------------------------------------------------------------------

    @Test
    void subtract_largerFromSmaller() {
        assertEquals(-1, calc.subtract(2, 3));
    }

    @Test
    void subtract_sameValue() {
        assertEquals(0, calc.subtract(7, 7));
    }

    // -------------------------------------------------------------------------
    // Multiply
    // -------------------------------------------------------------------------

    @ParameterizedTest(name = "{0} × {1} = {2}")
    @CsvSource({
            "2,  3,  6",
            "0,  9,  0",
            "-2, 3, -6",
            "-2,-3,  6"
    })
    void multiply_tableOfValues(int a, int b, int expected) {
        assertEquals(expected, calc.multiply(a, b));
    }

    // -------------------------------------------------------------------------
    // Divide
    // -------------------------------------------------------------------------

    @Test
    void divide_evenDivision() {
        assertEquals(3, calc.divide(9, 3));
    }

    @Test
    void divide_integerTruncation() {
        // Integer division truncates toward zero — important to document for callers
        assertEquals(2, calc.divide(7, 3));
    }

    @Test
    void divide_byZero_throwsArithmeticException() {
        ArithmeticException ex = assertThrows(
                ArithmeticException.class,
                () -> calc.divide(1, 0));

        // The message must mention "zero" — callers deserve a readable error
        assertTrue_messageContains(ex.getMessage(), "zero");
    }

    // -------------------------------------------------------------------------
    // Private helper — avoid importing Hamcrest just for one assertion
    // -------------------------------------------------------------------------

    private static void assertTrue_messageContains(String message, String substring) {
        if (!message.toLowerCase().contains(substring.toLowerCase())) {
            throw new AssertionError(
                    "Expected message to contain '" + substring + "' but was: " + message);
        }
    }
}

Lab 02 — Signing and Verification

Phase: 04 — ASF Governance and Release Engineering
Difficulty: ⭐⭐☆☆☆ | Estimated Time: 1–2 hours
Primary Artifact: Working sign-artifact.sh and verify-release.sh
Verify: bash verify-release.sh release-lab-0.1.0.jar <your-key-id> prints RELEASE VERIFIED

Goal

Implement two shell scripts that together reproduce the signing and verification steps every Apache release manager performs before uploading artifacts to dist.apache.org:

1. sign-artifact.sh — sign an artifact with your GPG key and generate a SHA-256 checksum
2. verify-release.sh — verify both the signature and checksum and print a clear pass/fail verdict

Both scripts ship with # TODO stubs. Your job is to fill them in.

Setup

1. Generate a GPG key (if you don't already have one)

# Generate a key — use your real name and email
gpg --full-generate-key

# Select: (1) RSA and RSA
# Key size: 4096
# Expiry: 0 (does not expire — Apache convention)
# Name: Your Name
# Email: your@email.com

2. Find your key ID

gpg --list-secret-keys --keyid-format LONG

Output looks like:

sec   rsa4096/ABCDEF1234567890 2024-01-01 [SC]
      ABC1234567890DEF...
uid                 [ultimate] Your Name <your@email.com>

Your key ID is the hex string after the slash (ABCDEF1234567890 in the example).

3. Build the artifact from Lab 01

cd ../lab-01-release-mechanics
git checkout v0.1.0
mvn package -DskipTests
cp target/release-lab-0.1.0.jar ../../lab-02-signing-verification/
cd ../../lab-02-signing-verification

Steps

Step 1 — Implement sign-artifact.sh

Open sign-artifact.sh. Fill in the four # TODO sections:

1. Check file exists: Fail clearly if the artifact path is not a regular file
2. GPG detached signature: Produce <artifact>.asc using gpg --detach-sign --armor
3. SHA-256 checksum: Produce <artifact>.sha256 using shasum (macOS) or sha256sum (Linux)
4. Print summary: Print the paths to the .asc and .sha256 files

Test it:

chmod +x sign-artifact.sh
./sign-artifact.sh release-lab-0.1.0.jar

Expected output (key ID will differ):

Signed artifact  : release-lab-0.1.0.jar
Signature file   : release-lab-0.1.0.jar.asc
Checksum file    : release-lab-0.1.0.jar.sha256

Inspect what was generated:

cat release-lab-0.1.0.jar.sha256    # should show hex hash + filename
cat release-lab-0.1.0.jar.asc       # ASCII-armored GPG signature

Step 2 — Implement verify-release.sh

Open verify-release.sh. Fill in the four # TODO sections:

1. Check required files exist: .asc, .sha256 — fail clearly for each missing file
2. GPG verify: Run gpg --verify and capture success/failure
3. Checksum verify: Run OS-appropriate checksum verify and capture success/failure
4. Final verdict: Print RELEASE VERIFIED or RELEASE FAILED with a clear overall result

Test it:

chmod +x verify-release.sh
./verify-release.sh release-lab-0.1.0.jar <YOUR-KEY-ID>

Expected output:

GPG signature: VALID
Checksum: VALID
RELEASE VERIFIED: release-lab-0.1.0.jar is authentic.

Step 3 — Tamper Test

Verify that your script correctly detects tampering:

# Tamper with the JAR (flip one byte)
python3 -c "
data = open('release-lab-0.1.0.jar', 'rb').read()
tampered = bytes([data[0] ^ 0xFF]) + data[1:]
open('release-lab-0.1.0.jar', 'wb').write(tampered)
"

# Now verify — should FAIL
./verify-release.sh release-lab-0.1.0.jar <YOUR-KEY-ID>

Expected output:

GPG signature: INVALID
Checksum: INVALID
RELEASE FAILED: release-lab-0.1.0.jar could not be verified.

Restore the original JAR by rebuilding from Lab 01 before continuing.

Step 4 — Vote Email Draft (bonus, no script required)

Draft the [VOTE] email you would send if you were releasing release-lab-0.1.0.jar as an Apache project. Save it to vote-email.txt in this directory.

Your email must include:

• Subject line in the correct format
• RC artifact location (use a fake dist.apache.org URL)
• KEYS file URL
• Git tag URL
• The three vote options with checkboxes
• This vote is open for 72 hours.

Reference the HITCHHIKERS-GUIDE vote format.

Verification

# Both scripts are executable
test -x sign-artifact.sh && echo "sign-artifact.sh: OK"
test -x verify-release.sh && echo "verify-release.sh: OK"

# Sign produces expected files
./sign-artifact.sh release-lab-0.1.0.jar
test -f release-lab-0.1.0.jar.asc && echo "asc: OK"
test -f release-lab-0.1.0.jar.sha256 && echo "sha256: OK"

# Verify passes on untampered artifact
./verify-release.sh release-lab-0.1.0.jar <YOUR-KEY-ID> | grep "RELEASE VERIFIED"

Common Errors

Source Files

sign-artifact.sh

#!/usr/bin/env bash
# ==============================================================================
# sign-artifact.sh — Apache Release Signing Script
#
# Lab 02, Phase 04: ASF Governance and Release Engineering
#
# Usage:
#   ./sign-artifact.sh <path-to-artifact>
#
# Produces:
#   <artifact>.asc    — GPG detached armored signature
#   <artifact>.sha256 — SHA-256 checksum
#
# Fill in the four TODO sections below. When complete, running this script
# against any file should produce the two output files and print a summary.
#
# Reference:
#   Apache Release Policy: https://www.apache.org/legal/release-policy.html
#   GPG sign command:  gpg --batch --yes --detach-sign --armor --output <out> <in>
#   macOS checksum:    shasum -a 256 <file> > <file>.sha256
#   Linux checksum:    sha256sum <file> > <file>.sha256
# ==============================================================================
set -euo pipefail

ARTIFACT="${1:-}"

if [[ -z "$ARTIFACT" ]]; then
    echo "Usage: $0 <path-to-artifact>" >&2
    exit 1
fi

SIG_FILE="${ARTIFACT}.asc"
CHECKSUM_FILE="${ARTIFACT}.sha256"

# ------------------------------------------------------------------------------
# Step 1: Verify the artifact exists
# ------------------------------------------------------------------------------
# TODO: Check that $ARTIFACT is a regular file (-f).
#       If it is not, print an error message to stderr and exit with code 1.
#       Example message: "Error: file not found: $ARTIFACT"
#
# Hint: Use:  if [[ ! -f "$ARTIFACT" ]]; then ...
#


# ------------------------------------------------------------------------------
# Step 2: Sign with GPG (detached armored signature)
# ------------------------------------------------------------------------------
# TODO: Run the GPG sign command to produce $SIG_FILE.
#
# The command signature is:
#   gpg --batch --yes --detach-sign --armor --output "$SIG_FILE" "$ARTIFACT"
#
# Flags explained:
#   --batch           non-interactive (no prompts)
#   --yes             overwrite $SIG_FILE if it already exists
#   --detach-sign     produce a detached signature (not embedded in the file)
#   --armor           ASCII-armor the output (human-readable, safe to email)
#   --output <file>   write the signature to this file
#
# Note: If you have multiple GPG keys, add --local-user <key-id> to pick the
#       right one. For this lab any key is fine.
#


# ------------------------------------------------------------------------------
# Step 3: Generate SHA-256 checksum
# ------------------------------------------------------------------------------
# TODO: Detect the operating system and run the appropriate checksum command.
#
# macOS: shasum -a 256 "$ARTIFACT" > "$CHECKSUM_FILE"
# Linux: sha256sum "$ARTIFACT" > "$CHECKSUM_FILE"
#
# Hint for OS detection:
#   OS="$(uname -s)"
#   case "$OS" in
#     Darwin) ... ;;
#     Linux)  ... ;;
#     *) echo "Unsupported OS: $OS" >&2; exit 1 ;;
#   esac
#
# The output file format produced by both commands is the same:
#   <hex-hash>  <filename>   (two spaces between hash and name)
# This format is understood by --check on both macOS and Linux.
#


# ------------------------------------------------------------------------------
# Step 4: Print a summary
# ------------------------------------------------------------------------------
echo "Signed artifact  : $ARTIFACT"
# TODO: Print the path to the signature file (SIG_FILE) and checksum file (CHECKSUM_FILE).
#       Format them like the line above, aligned with spaces for readability.
#       Example:
#         Signature file   : release-lab-0.1.0.jar.asc
#         Checksum file    : release-lab-0.1.0.jar.sha256
#

verify-release.sh

#!/usr/bin/env bash
# ==============================================================================
# verify-release.sh — Apache Release Verification Script
#
# Lab 02, Phase 04: ASF Governance and Release Engineering
#
# Usage:
#   ./verify-release.sh <artifact> <signer-key-id>
#
# Example:
#   ./verify-release.sh release-lab-0.1.0.jar ABCDEF1234567890
#
# This script verifies that an Apache release artifact:
#   1. Has a valid GPG detached signature from a known signer
#   2. Matches its published SHA-256 checksum
#
# Fill in the four TODO sections below. The script must:
#   - Exit 0 and print "RELEASE VERIFIED: <artifact> is authentic." on success
#   - Exit 1 and print "RELEASE FAILED: <artifact> could not be verified." on failure
#
# Reference:
#   gpg --verify <sig-file> <data-file>
#   macOS checksum verify: shasum -a 256 --check <checksum-file>
#   Linux checksum verify: sha256sum --check <checksum-file>
# ==============================================================================
set -uo pipefail
# Note: we do NOT set -e here because we want to capture individual command
# exit codes rather than failing immediately on any non-zero exit.

ARTIFACT="${1:-}"
SIGNER_KEY="${2:-}"

if [[ -z "$ARTIFACT" || -z "$SIGNER_KEY" ]]; then
    echo "Usage: $0 <artifact> <signer-key-id>" >&2
    exit 1
fi

SIG_FILE="${ARTIFACT}.asc"
CHECKSUM_FILE="${ARTIFACT}.sha256"

# We'll track overall pass/fail in these variables.
GPG_OK=false
CHECKSUM_OK=false

# ------------------------------------------------------------------------------
# Step 1: Check that all required files exist
# ------------------------------------------------------------------------------
# TODO: Verify that each of these three files exists as a regular file:
#         $ARTIFACT
#         $SIG_FILE
#         $CHECKSUM_FILE
#
#       For each missing file, print a message to stderr like:
#         "Error: missing artifact: $ARTIFACT"
#         "Error: missing signature: $SIG_FILE"
#         "Error: missing checksum: $CHECKSUM_FILE"
#
#       If ANY file is missing, exit 1 after printing all errors.
#       This way the user sees all missing files at once, not just the first one.
#
# Hint:
#   MISSING=0
#   [[ ! -f "$ARTIFACT" ]] && { echo "Error: ..." >&2; MISSING=1; }
#   ...
#   [[ $MISSING -eq 1 ]] && exit 1
#


# ------------------------------------------------------------------------------
# Step 2: Verify GPG signature
# ------------------------------------------------------------------------------
# TODO: Run:  gpg --verify "$SIG_FILE" "$ARTIFACT"
#
# IMPORTANT: The argument order matters.
#   Correct:    gpg --verify <signature-file> <data-file>
#   Wrong:      gpg --verify <data-file> <signature-file>  ← common mistake
#
# Capture the exit code:
#   If exit code is 0 → set GPG_OK=true and print "GPG signature: VALID"
#   If exit code is non-zero → print "GPG signature: INVALID"
#
# You may redirect gpg's own output to /dev/null to keep the output clean,
# or let it print (useful for debugging).
#
# Hint:
#   gpg --verify "$SIG_FILE" "$ARTIFACT" 2>/dev/null
#   if [[ $? -eq 0 ]]; then ...
#


# ------------------------------------------------------------------------------
# Step 3: Verify SHA-256 checksum
# ------------------------------------------------------------------------------
# TODO: Detect the OS and run the correct checksum verification command.
#
# The checksum file produced by sign-artifact.sh has format:
#   <hex>  <filename>    (two spaces — the default output format)
#
# macOS verify: shasum -a 256 --check "$CHECKSUM_FILE"
# Linux verify: sha256sum --check "$CHECKSUM_FILE"
#
# Both commands must be run from the directory containing the artifact because
# the filename in the checksum file is a relative path.
#
# Capture the exit code the same way as Step 2:
#   If exit code is 0 → set CHECKSUM_OK=true and print "Checksum: VALID"
#   If exit code is non-zero → print "Checksum: INVALID"
#
# Hint for OS detection: see sign-artifact.sh Step 3.
#


# ------------------------------------------------------------------------------
# Step 4: Final verdict
# ------------------------------------------------------------------------------
# TODO: If both GPG_OK and CHECKSUM_OK are true, print:
#         "RELEASE VERIFIED: $ARTIFACT is authentic."
#       and exit 0.
#
#       Otherwise, print:
#         "RELEASE FAILED: $ARTIFACT could not be verified."
#       and exit 1.
#
# Hint:
#   if [[ "$GPG_OK" == "true" && "$CHECKSUM_OK" == "true" ]]; then
#     ...
#   fi
#

Lab 03 — API Compatibility

Phase: 04 — ASF Governance and Release Engineering
Difficulty: ⭐⭐⭐⭐☆ | Estimated Time: 2–3 hours
Primary Artifact: Multi-module Maven project producing a japicmp compatibility report
Verify: mvn install from parent directory fails (japicmp detects binary incompatibilities)

Goal

A library maintainer has just released DataProcessor v2 with breaking changes, and you need to catch them before shipping. You will:

1. Build and install a v1 and v2 API module to your local Maven repository
2. Run japicmp (Java API Compatibility Checker) to compare the two
3. Interpret the compatibility report and classify each change
4. Fill out the compatibility matrix below

This is exactly the process Apache libraries like Commons Lang, Jackson, and Spark run before every release.

Project Structure

lab-03-api-compatibility/
  pom.xml                               ← parent: declares api-v1, api-v2, compat-check modules
  api-v1/
    pom.xml
    src/main/java/org/example/compat/
      DataProcessor.java                ← v1 stable API
      BatchConfig.java                  ← v1 mutable config
  api-v2/
    pom.xml
    src/main/java/org/example/compat/
      DataProcessor.java                ← v2 with BREAKING changes
      BatchConfig.java                  ← v2 immutable (BREAKING constructor change)
  compat-check/
    pom.xml                             ← uses japicmp-maven-plugin to compare v1 vs v2

Steps

Step 1 — Inspect the v1 API

Read api-v1/src/main/java/org/example/compat/DataProcessor.java and BatchConfig.java.

Before looking at v2, write down:

• The method signatures of DataProcessor
• The constructor signature of BatchConfig
• Whether BatchConfig is mutable or immutable

This is what your library's users have compiled their code against.

Step 2 — Read the v2 Changes

Read the v2 source files. Annotate each change in the table below (before running japicmp):

Step 3 — Run japicmp

# From the lab-03-api-compatibility/ directory
mvn install

The build will intentionally fail — japicmp is configured with breakBuildOnBinaryIncompatibleModifications=true. Read the build output carefully.

Find the japicmp HTML report:

# The report is in the compat-check module's target directory
open compat-check/target/japicmp/japicmp.html     # macOS
xdg-open compat-check/target/japicmp/japicmp.html # Linux
# Or read the XML
cat compat-check/target/japicmp/japicmp.xml

Step 4 — Fill Out the Compatibility Matrix

After running japicmp, complete this table:

Classifications you will see in the report:

• METHOD_REMOVED — binary and source incompatible
• METHOD_ADDED — non-breaking (cannot break existing callers)
• METHOD_NOW_ABSTRACT — binary incompatible
• CONSTRUCTOR_REMOVED — binary and source incompatible

Step 5 — Fix the Version Number

The breaking changes between v1 and v2 require a major version bump under semantic versioning.

1. Open api-v2/pom.xml and change <version>1.0.0</version> to <version>2.0.0</version>
2. Open compat-check/pom.xml and update the newVersion dependency from 1.0.0 to 2.0.0
3. Run mvn install again — the build still fails (binary incompatibilities are there regardless of version)
4. Add <breakBuildOnBinaryIncompatibleModifications>false</breakBuildOnBinaryIncompatibleModifications> to compat-check/pom.xml and run again

Discuss: Why does setting breakBuildOnBinaryIncompatibleModifications=false not make the problem go away? What does it just change?

Step 6 — Write a Migration Guide

In migration-guide.md (create it in this directory), write a short migration guide from DataProcessor v1 to v2. For each breaking change:

• State what code breaks
• Show the before/after code snippet for the caller
• State the minimum Maven dependency change needed

Verification

# Run from lab-03-api-compatibility/
mvn install 2>&1 | grep -E "BINARY_INCOMPATIBLE|BUILD FAILURE|japicmp"

Expected to see lines like:

[WARNING] BINARY INCOMPATIBLE: ...DataProcessor.process(java.lang.String) has been removed
[WARNING] BINARY INCOMPATIBLE: ...DataProcessor.setConfig(...) has been removed
[INFO] BUILD FAILURE

Key Insight

japicmp gives you a machine-readable, auditable diff of your public API. Running it in CI means:

1. Developers get immediate feedback when they accidentally break binary compatibility
2. PMC members can attach the japicmp report to the [VOTE] email as evidence that compatibility was checked
3. Users can look at the report in a release's GitHub release page to understand exactly what changed

Apache projects that do this well: Apache HttpComponents, Apache Commons Collections, Apache Kafka.

Source Files

pom.xml (parent)

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <!--
    Parent POM for the API compatibility lab.

    Module build order (determined by Maven dependency resolution):
      1. api-v1     — builds and installs data-processor-v1:1.0.0 to local repo
      2. api-v2     — builds and installs data-processor-v2:1.0.0 to local repo
      3. compat-check — depends on both, runs japicmp to compare them

    Run from this directory:
      mvn install

    Expected result: BUILD FAILURE — japicmp detects binary incompatibilities
    between api-v1 and api-v2 and breaks the build as configured.
  -->

    <groupId>org.example.compat</groupId>
    <artifactId>api-compat-lab</artifactId>
    <version>1.0.0</version>
    <packaging>pom</packaging>

    <name>API Compatibility Lab — Parent</name>

    <modules>
        <module>api-v1</module>
        <module>api-v2</module>
        <module>compat-check</module>
    </modules>

    <properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

</project>

api-v1/pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.example.compat</groupId>
        <artifactId>api-compat-lab</artifactId>
        <version>1.0.0</version>
    </parent>

    <!--
    data-processor-v1 — the stable, released API.

    This is what library users have compiled their code against.
    japicmp will use this JAR as the "old version" baseline.
  -->
    <artifactId>data-processor-v1</artifactId>
    <version>1.0.0</version>
    <name>Data Processor v1</name>

</project>

api-v1 — DataProcessor.java

package org.example.compat;

/**
 * Processes a stream of string records according to a {@link BatchConfig}.
 *
 * <p>
 * <strong>This is the v1 stable API.</strong> This is what user
 *  have compiled
 * their application code against. Any incompatible change to this class in v2
 * is 
 * a binary compatibility regression.
 *
 * >Example usage (caller code compiled against v1):
 * 
 * @code
 * hConfig config = new BatchConfig(100);
 * config.setStrict(true);
 * 
 * 
 * Processor processor = new DataProcessor();
 *   processor.setConfig(config);
 *   processor.process("record-001");
 *   S
 * ystem.out.println(processor.getResult());
 * }</pre>
 * 
 * 
pu c class DataProcessor {
 * 

    private BatchConfig config;
    pr
 * ivate String lastResult;
 
 * 
   ublic DataProcessor() {
   
 
    /**
   * Process a single input record.
   *
     * <p>This method signature — {@code process(String)} — is what callers have compiled
     * against. Removing it or changing its parameter list in v2 will produce
     * {@code NoSuchMethodError} at runtime for any code compiled against v1.
     *
     * @param input the record to process; must not be null
     */
    public void process(String input) {
        if (input == null) {
            throw new IllegalArgumentException("input must not be null");
        }
        // Simulate processing — in a real library this would do meaningful work.
        boolean strict = (config != null && config.isStrict());
        this.lastResult = "processed[strict=" + strict + "]: " + input;
    }

    /* 
     * Returns the result of the last {@link #process(String)} call.
     *
     * @return the last processed result, or {@code null} if {@link #process(String)}
     *         has not been called yet
     */
    public
     *  String getResult() {
        return lastResult;
    }

    /**
     * Configure this processor.
     *               
     *
     * <p>This method must be called before {@link #process(String)} if the caller
     * wants non-default batch behaviour.
     *
     * @param config the configuration to apply; may be null to reset to defaults
     */
    public void setConfig(BatchConfig config) {
        this.config = config;
    }

    /**
     * Returns the current configuration, or {@code null} if none has been set.
     */
    public BatchConfig getConfig() {
        return config;
    }
     *         
}
 
    // 
     * 
     * 
     * 
     * 
     * 

api-v1 — BatchConfig.java

package org.example.compat;

/**
 * Configuration for a {@link DataProcessor}.
 *
 * <p><strong>This is the v1 stable API.</strong> The class is mutable: after construction
 * the caller can call {@link #setStrict(boolean)} to toggle strict mode.
 *
 * <p>Example usage (caller code compiled against v1):
 * <pre>{@code
 *   BatchConfig config = new BatchConfig(100);   // single-arg constructor
 *   config.setStrict(true);                       // mutable setter
 * }</pre>
 */
public class BatchConfig {

    private final int batchSize;
    private boolean strict;

    /**
     * Creates a BatchConfig with the given batch size.
     *
     * <p>This is the only constructor in v1. Any caller that instantiates
     * {@code BatchConfig} via this signature ({@code new BatchConfig(int)}) will get
     * {@code NoSuchMethodError} at runtime if v2 removes this constructor.
     *
     * @param batchSize number of records to process per batch; must be positive
     */
    public BatchConfig(int batchSize) {
        if (batchSize <= 0) {
            throw new IllegalArgumentException("batchSize must be positive, was: " + batchSize);
        }
        this.batchSize = batchSize;
        this.strict = false; // default: lenient
    }

    /**
     * Returns the batch size.
     */
    public int getBatchSize() {
        return batchSize;
    }

    /**
     * Returns whether strict mode is enabled.
     */
    public boolean isStrict() {
        return strict;
    }

    /**
     * Enables or disables strict mode.
     *
     * <p>This mutable setter is removed in v2 (the class becomes immutable).
     * Any caller that calls this method will get {@code NoSuchMethodError} at runtime.
     *
     * @param strict {@code true} to throw on any parse error; {@code false} to skip invalid records
     */
    public void setStrict(boolean strict) {
        this.strict = strict;
    }
}

api-v2/pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.example.compat</groupId>
        <artifactId>api-compat-lab</artifactId>
        <version>1.0.0</version>
    </parent>

    <!--
    data-processor-v2 — the new version with BREAKING changes.

    This JAR is intentionally binary-incompatible with data-processor-v1.
    japicmp will use this as the "new version" and report all regressions.

    In a real project, if this were a minor release (1.1.0 instead of 2.0.0),
    these changes would be blocked by the japicmp build failure you are about
    to trigger. That is exactly the point: catch regressions before they ship.
  -->
    <artifactId>data-processor-v2</artifactId>
    <version>1.0.0</version>
    <name>Data Processor v2</name>

</project>

api-v2 — DataProcessor.java

package org.example.compat;

/**
 * Processes a stream of string records according to a {@link BatchConfig}.
 *
 * <p>
 * <strong>This is the v2 API — it contains intentional BREAKING
 * changes.</strong>
 *
 * <p>
 * Breaking changes introduced in this version:
 * <ol>
 * <li>{@code process(String)} REMOVED — replaced by
 * {@code process(String, boolean)}.
 * Any code compiled against v1 calling {@code process(String)} will get
 * {@code NoSuchMethodError} at runtime.</li>
 * <li>{@code setConfig(BatchConfig)} REMOVED — replaced by fluent
 * {@code withConfig}.
 * Any code compiled against v1 calling {@code setConfig} will get
 * {@code NoSuchMethodError} at runtime.</li>
 * </ol>
 *
 * <p>
 * Non-breaking additions in this version:
 * <ul>
 * <li>{@code isConfigured()} added — new method, cannot break existing
 * callers.</li>
 * <li>{@code withConfig(BatchConfig)} added — new method, cannot break existing
 * callers.</li>
 * </ul>
 *
 * <p>
 * Correct migration path for v1 callers:
 * 
 * <pre>{@code
 * // v1 caller (breaks at runtime against v2)
 * processor.setConfig(config);
 * processor.process("record-001");
 *
 * // v2 caller (recompile required)
 * processor.withConfig(config).process("record-001", false);
 * }</pre>
 */
public class DataProcessor {

    private BatchConfig config;
    private String lastResult;

    public DataProcessor() {
    }

    // =========================================================================
    // BREAKING CHANGE 1: process(String) removed
    //
    // The v1 method was: public void process(String input)
    // That method no longer exists. Callers compiled against v1 will get:
    // java.lang.NoSuchMethodError: DataProcessor.process(Ljava/lang/String;)V
    // =========================================================================

    /**
     * Process a single input record with an explicit strict-mode flag.
     *
     * <p>
     * BREAKING: replaces {@code process(String)} from v1.
     * The added {@code strict} parameter makes this a different method signature.
     * Code compiled against v1 cannot call this method without recompilation.
     *
     * @param input  the record to process; must not be null
     * @param strict if {@code true}, throw {@link IllegalStateException} on invalid
     *               input;
     *               if {@code false}, skip silently
     */
    public void process(String input, boolean strict) {
        if (input == null) {
            if (strict) {
                throw new IllegalStateException("strict mode: null input not allowed");
            }
            return; // lenient: skip silently
        }
        this.lastResult = "processed[strict=" + strict + "]: " + input;
    }

    /**
     * Returns the result of the last {@link #process(String, boolean)} call.
     *
     * @return the last processed result, or {@code null} if process has not been
     *         called
     */
    public String getResult() {
        return lastResult;
    }

    // =========================================================================
    // BREAKING CHANGE 2: setConfig(BatchConfig) removed
    //
    // The v1 method was: public void setConfig(BatchConfig config)
    // Callers compiled against v1 calling setConfig will get:
    // java.lang.NoSuchMethodError:
    // DataProcessor.setConfig(Lorg/example/compat/BatchConfig;)V
    // =========================================================================

    /**
     * Returns a new DataProcessor with the given configuration applied.
     *
     * <p>
     * Fluent builder pattern: replaces the v1 {@code setConfig(BatchConfig)} void
     * setter.
     * This is a non-breaking addition — existing callers who never called
     * {@code setConfig} are unaffected.
     *
     * <p>
     * Note: returns {@code this} for chaining convenience (this processor is
     * mutated).
     * A fully immutable design would return a new instance.
     *
     * @param config the configuration to apply
     * @return {@code this}, for method chaining
     */
    public DataProcessor withConfig(BatchConfig config) {
        this.config = config;
        return this;
    }

    /**
     * Returns the current configuration, or {@code null} if none has been set.
     */
    public BatchConfig getConfig() {
        return config;
    }

    // =========================================================================
    // NON-BREAKING ADDITION: isConfigured()
    //
    // New method — cannot break existing callers. Callers compiled against v1
    // simply cannot call this method (compile error if they try).
    // =========================================================================

    /**
     * Returns {@code true} if this processor has been configured via
     * {@link #withConfig}.
     */
    public boolean isConfigured() {
        return config != null;
    }
}

api-v2 — BatchConfig.java

package org.example.compat;

/**
 * Configuration for a {@link DataProcessor}.
 *
 * <p>
 * <strong>This is the v2 API — it contains intentional BREAKING
 * changes.</strong>
 *
 * <p>
 * Breaking changes introduced in this version:
 * <ol>
 * <li>{@code BatchConfig(int)} REMOVED — the single-argument constructor is
 * gone.
 * Any code calling {@code new BatchConfig(100)} will get
 * {@code NoSuchMethodError} at runtime.</li>
 * <li>{@code setStrict(boolean)} REMOVED — the class is now immutable.
 * Any code calling {@code config.setStrict(true)} will get
 * {@code NoSuchMethodError} at runtime.</li>
 * </ol>
 *
 * <p>
 * Correct migration path for v1 callers:
 * 
 * <pre>{@code
 * // v1 caller (breaks at runtime against v2)
 * BatchConfig config = new BatchConfig(100);
 * config.setStrict(true);
 *
 * // v2 caller (recompile required)
 * BatchConfig config = new BatchConfig(100, true);
 * }</pre>
 */
public class BatchConfig {

    private final int batchSize;
    private final boolean strict;

    // =========================================================================
    // BREAKING CHANGE 1: BatchConfig(int) removed
    //
    // v1 constructor: public BatchConfig(int batchSize)
    // v2 requires both batchSize AND strict at construction time.
    // Callers compiled against v1 calling new BatchConfig(100) will get:
    // java.lang.NoSuchMethodError: BatchConfig.<init>(I)V
    // =========================================================================

    /**
     * Creates an immutable BatchConfig.
     *
     * <p>
     * BREAKING: replaces {@code BatchConfig(int)} from v1.
     * The {@code strict} parameter is now required at construction time — the class
     * is immutable and the old {@link #setStrict(boolean)} setter is gone.
     *
     * @param batchSize number of records per batch; must be positive
     * @param strict    if {@code true}, throw on any parse error;
     *                  if {@code false}, skip invalid records silently
     */
    public BatchConfig(int batchSize, boolean strict) {
        if (batchSize <= 0) {
            throw new IllegalArgumentException("batchSize must be positive, was: " + batchSize);
        }
        this.batchSize = batchSize;
        this.strict = strict;
    }

    /**
     * Returns the batch size.
     */
    public int getBatchSize() {
        return batchSize;
    }

    /**
     * Returns whether strict mode is enabled.
     */
    public boolean isStrict() {
        return strict;
    }

    // =========================================================================
    // BREAKING CHANGE 2: setStrict(boolean) removed
    //
    // v1 method: public void setStrict(boolean strict)
    // The class is now immutable — all fields are final and there are no setters.
    // Callers compiled against v1 calling setStrict will get:
    // java.lang.NoSuchMethodError: BatchConfig.setStrict(Z)V
    // =========================================================================
}

compat-check/pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.example.compat</groupId>
        <artifactId>api-compat-lab</artifactId>
        <version>1.0.0</version>
    </parent>

    <!--
    compat-check — runs japicmp to compare data-processor-v1 against data-processor-v2.

    This module has no source code. Its only job is to declare the japicmp plugin
    execution that compares the two API JARs.

    How japicmp resolves artifacts:
      oldVersion: data-processor-v1:1.0.0  (must be installed in local Maven repo)
      newVersion: data-processor-v2:1.0.0  (must be installed in local Maven repo)

    Both are installed to the local repo when you run 'mvn install' from the parent,
    because Maven builds modules in dependency order and installs each one before
    moving to the next.

    breakBuildOnBinaryIncompatibleModifications=true:
      This setting causes the Maven build to FAIL if japicmp detects any binary
      incompatibility between the old and new versions. This is how CI enforces
      the compatibility contract: you cannot accidentally ship a breaking change
      without the build explicitly telling you.
  -->
    <artifactId>compat-check</artifactId>
    <version>1.0.0</version>
    <name>Compatibility Check (japicmp)</name>

    <!--
    Declare a dependency on data-processor-v2 so Maven builds the modules in order:
      api-v1 → api-v2 → compat-check
    Without this, Maven might try to build compat-check before the JARs exist.
  -->
    <dependencies>
        <dependency>
            <groupId>org.example.compat</groupId>
            <artifactId>data-processor-v2</artifactId>
            <version>1.0.0</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <!--
        japicmp-maven-plugin: Java API Compatibility Checker.
        Version 0.20.0 is current as of 2024.

        Plugin documentation: https://siom79.github.io/japicmp/MavenPlugin.html

        Key configuration fields:
          oldVersion: the "baseline" — what users compiled their code against
          newVersion: the "candidate" — what you want to release
          breakBuildOnBinaryIncompatibleModifications: fail if binary compat broken
          onlyModified: only report things that changed (not the whole API)
          htmlOutputFile: where to write the HTML report

        The comparison runs during the 'verify' phase. When you run 'mvn install',
        the verify phase runs before install, so incompatibilities are caught before
        the artifact is installed to the local repo.
      -->
            <plugin>
                <groupId>com.github.siom79.japicmp</groupId>
                <artifactId>japicmp-maven-plugin</artifactId>
                <version>0.20.0</version>
                <executions>
                    <execution>
                        <id>check-api-compatibility</id>
                        <phase>verify</phase>
                        <goals>
                            <goal>cmp</goal>
                        </goals>
                        <configuration>
                            <oldVersion>
                                <dependency>
                                    <groupId>org.example.compat</groupId>
                                    <artifactId>data-processor-v1</artifactId>
                                    <version>1.0.0</version>
                                </dependency>
                            </oldVersion>
                            <newVersion>
                                <dependency>
                                    <groupId>org.example.compat</groupId>
                                    <artifactId>data-processor-v2</artifactId>
                                    <version>1.0.0</version>
                                </dependency>
                            </newVersion>
                            <parameter>
                                <!--
                  BREAK THE BUILD if any binary incompatibilities are found.
                  This is the key setting that makes japicmp a real CI gate
                  rather than just a reporting tool.

                  Lab step 5 asks you to change this to false and observe
                  what happens — the report is still generated, but the build
                  no longer fails. This is useful during a deliberate major
                  version bump where breaking changes are expected and documented.
                -->
                                <breakBuildOnBinaryIncompatibleModifications>true</breakBuildOnBinaryIncompatibleModifications>

                                <!--
                  Only report classes/methods that actually changed.
                  Without this, the report lists every class in the entire JAR.
                -->
                                <onlyModified>true</onlyModified>

                                <!--
                  Include package-private and protected members in the report.
                  For Apache libraries, public and protected APIs are the contract.
                  Package-private is internal.
                -->
                                <accessModifier>public</accessModifier>
                            </parameter>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

Phase 05 — Code Quality and Engineering Discipline

Why this phase exists: Apache projects live for decades and are maintained by rotating casts of contributors who have never met. The only way that scales is with automated quality gates that make the bar explicit and measurable. This phase teaches the three-layer quality stack every serious Apache project runs: style enforcement, static analysis, and human review.

What You'll Learn

• How Checkstyle enforces coding conventions across a multi-contributor codebase and why that matters
• How SpotBugs finds likely correctness bugs by analyzing bytecode, not source code
• How to read and act on a SpotBugs report (confidence levels, bug categories, false positives)
• How to write a structured, useful code review comment in the Apache style
• The difference between a style nit and a correctness concern — and which one to raise first

Concepts Covered

Labs

Deliverables Checklist

• mvn checkstyle:check passes with zero violations in Lab 01
• git diff of FlawedScheduler.java shows exactly the 5 fixed violations
• mvn spotbugs:check passes with zero high-confidence bugs in Lab 02
• You can name the SpotBugs bug category for each of the 4 bugs by memory
• Lab 03 failing tests pass after your fixes to CleverPartitioner.java
• Lab 03 review document identifies every issue with a clear severity label

Interview Relevance

Warmup Guide — Code Quality & Engineering Discipline

Zero-to-expert primer for Phase 05: what "quality" means operationally in Apache codebases, how static analysis (Checkstyle, SpotBugs) actually works, and the refactoring discipline that turns "clever" code into maintainable code.

Table of Contents

• Chapter 1: Quality, Defined Operationally
• Chapter 2: The Maintainer's Time Horizon
• Chapter 3: Style Checking — Checkstyle and Why Style Is Enforced
• Chapter 4: Bug-Pattern Analysis — How SpotBugs Works
• Chapter 5: The Limits of Static Analysis
• Chapter 6: Clever vs Clear — The Refactoring Discipline
• Chapter 7: Tests as the Quality Substrate
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Quality, Defined Operationally

"Quality" in an Apache review is not aesthetics. It decomposes into checkable properties, in priority order:

1. Correct under its contract — including error paths, edge inputs, and concurrent callers (the parts the happy-path test doesn't touch).
2. Comprehensible to a stranger in five years — names that state intent, structure that mirrors the domain, comments that state invariants and why (never what).
3. Testable and tested — a design you can't test in isolation is a design problem, not a testing problem.
4. Consistent with its surroundings — local idiom beats global preference; a "better" pattern that fights the codebase costs more than it pays.
5. Resource-honest — closes what it opens, bounds what it buffers, fails without leaking.

The phase's three labs map onto these: Checkstyle mechanizes (4), SpotBugs hunts (1) and (5), and the review/refactor lab trains (2) and (3).

Chapter 2: The Maintainer's Time Horizon

The economic fact that explains Apache review culture: code is read 10–100× more often than written, and maintained for 10+ years by people who never met the author. Under that horizon:

• A clever trick that saves 5 lines but needs 5 minutes of reader decode time is a net loss by the second reader.
• Every public surface (Phase 03/04) is a forever-commitment, so reviewers challenge additions hardest — the cheapest API is the one never added.
• "Works now" is the weakest property: inputs drift, scale grows, neighbors refactor. Reviewers therefore probe what happens when assumptions break, which reads as pedantry until you've maintained something for a decade.

Internalizing this horizon is what changes your code more than any rule list — it's the difference between writing for the compiler and writing for the 2036 maintainer.

Chapter 3: Style Checking — Checkstyle and Why Style Is Enforced

Mechanism: Checkstyle parses Java source into an AST and runs declarative rules ("checks") over it — line length, import order, naming patterns, Javadoc presence, brace policy. Configured per-project (checkstyle.xml), run as a Maven plugin, usually build-failing in CI.

Why projects enforce what seems trivial: (a) diffs stay semantic — no whitespace churn burying real changes (the Phase 12 mega-patch scenario); (b) review attention is finite — every style nit a machine catches is a correctness comment a human can make instead; (c) consistency is readability at scale — a codebase with one voice loads faster into a stranger's head. The rule of engagement you'll need as a contributor: don't argue with the config in a patch; match it. Style debates go to the dev list, once, with a config change proposal.

Reading a violation productively (Lab 01): the fix is usually mechanical, but each rule encodes a story — VisibilityModifier exists because public mutable fields became un-evolvable API surface; MissingSwitchDefault because a new enum constant silently fell through someone's switch in production.

Chapter 4: Bug-Pattern Analysis — How SpotBugs Works

Mechanism (different layer than Checkstyle — know the distinction): SpotBugs analyzes compiled bytecode, not source. It runs dataflow analyses over methods — null-ness lattices, type states, resource open/close tracking — and matches ~400 known bug patterns, each with an id, category, and confidence. Because it sees bytecode, it catches what generated code and compiler sugar hide, but it loses source-level intent (hence false positives).

The pattern families worth knowing cold (Lab 02 plants several):

• NP_: null-pointer paths — a branch where a value proven-nullable is dereferenced.
• RV_: ignored return values — file.delete(), BigDecimal.add results dropped.
• EI_/EI2_: exposing internal mutable state (returning the array field itself).
• IS_/MT_: inconsistent synchronization — a field guarded by a lock on 9 of 10 accesses (the 10th is the bug).
• DM_: dubious method use — new String(), == on boxed types, default-encoding I/O.
• RCN_: redundant null checks — often evidence of author confusion about which values can be null; the finding is a prompt to fix the contract, not delete the check.

Triage discipline: every finding gets one of three dispositions — fix, suppress with a written justification (@SuppressFBWarnings(justification=...)), or refactor so the pattern can't occur. Silent suppression is the anti-pattern; an unjustified suppression file is where bugs go to hibernate.

Chapter 5: The Limits of Static Analysis

What the tools cannot see — i.e., what human review must cover (the bridge to Phase 06):

• Contract correctness: the code does X flawlessly; the caller needed Y.
• Concurrency beyond patterns: SpotBugs catches inconsistent locking, not a wrong happens-before design or a livelock.
• Performance: an O(n²) loop is style-clean and bug-pattern-free (Phase 08's JMH exists because of this gap).
• Blast radius: whether a change breaks a compat promise (japicmp covers signatures; behavior is human territory).
• Design altitude: whether this abstraction should exist at all.

The mature workflow stacks the layers: formatter → Checkstyle → SpotBugs/ErrorProne → tests → human review, each catching what the previous can't, the human spending attention only where machines are blind.

Chapter 6: Clever vs Clear — The Refactoring Discipline

Lab 03's subject. "Clever" code optimizes for the writer (density, tricks, implicit invariants); maintainable code optimizes for the reader. The refactor procedure that preserves correctness while transferring intent:

1. Characterize first: write tests pinning current behavior — including the weird edges (negative inputs, boundary sizes) — before touching anything. For a bit-twiddled partitioner: property tests against a naive reference implementation over random inputs.
2. Name the invariants the trick depends on (power-of-two table size, non-negative hash) — as assertions or checked preconditions, so violation fails loudly instead of corrupting silently.
3. Extract and name the sub-steps: (h ^ (h >>> 16)) becomes spreadHashBits(h) with a comment citing why (high-bit entropy folding, à la HashMap) — the trick survives, legibly.
4. Keep the performance if it was the point: benchmark before/after (JMH, Phase 08); a refactor that silently costs 15% on a hot path is a regression with good manners.
5. Decide what to keep clever: hot-path tricks earn their density if fenced by tests + named invariants + a reference implementation. That triple is the difference between clever and reckless — and the model answer in interview-prep/04, Scenario 1.

Chapter 7: Tests as the Quality Substrate

The test-shaped habits Apache reviewers check for reflexively:

• A failing test exists for every bug fix — proves you fixed the right thing and pins it forever (Phase 12 Ch. 1's rule).
• Tests document contracts: test names read as specification sentences (returnsEmptyListWhenPartitionCountIsZero); a stranger learns the class from its test file first (Chapter 1's reading order).
• Edge-first: empty, one, many, max, null, concurrent — the five-minute checklist that catches more than an hour of happy-path elaboration.
• Deterministic or quarantined: a flaky test erodes the whole suite's authority (the Phase 04 RC-sinking scenario); time, randomness, and ports get injected/seeded.
• Fast enough to run always: suites that take an hour stop being run; speed is a correctness feature.

Lab Walkthrough Guidance

Order: Lab 01 (Checkstyle) → Lab 02 (SpotBugs) → Lab 03 (review/refactor) — mechanized style, then mechanized bugs, then the human layer.

• Lab 01: run the check, fix every violation, but for three of them write the one-line "story" of why the rule exists (Chapter 3's habit). Note which fixes changed behavior risk vs pure form.
• Lab 02: triage every finding with the three-disposition discipline; at least one should be argued as a justified suppression and one refactored-away. Verify fixes with a test where the pattern was a real bug.
• Lab 03: follow Chapter 6's procedure in order — characterization tests first (the lab's hidden grading axis), then the rename/extract refactor, then the benchmark showing performance held.

Success Criteria

You are ready for Phase 06 when you can, from memory:

1. Recite the five operational quality properties in priority order.
2. Explain the source-AST vs bytecode-dataflow distinction and one consequence of each.
3. Name five SpotBugs pattern families with a one-line example bug each.
4. List four things static analysis cannot catch (and which layer catches each).
5. Execute the five-step clever→clear refactor procedure, and state when cleverness is kept.

Interview Q&A

Q: Your team wants to turn off Checkstyle because it "slows them down." Respond. Separate the costs: if the friction is learning the config, add a formatter that auto-fixes (spotless) — the check becomes free. If it's disagreement with rules, change the config once, by consensus, not per-PR. What we don't do is hand style back to human reviewers — that's paying senior-engineer attention for machine work and re-introducing diff churn. The check's real product is that reviews discuss correctness.

Q: Static analysis is clean but you suspect a concurrency bug. What do you actually do? Tools catch pattern-shaped races only. I'd map the shared state and its guard story (which lock protects which fields — write it down; gaps appear immediately), check publication points (constructor escape, non-final fields crossing threads), then write a stress test with many threads + assertions on invariants (and run with thread-sanitizer class tooling/jcstress for the memory-model cases). The deliverable is the documented guard discipline — undocumented locking is the bug factory.

Q: When is it right to ship the "clever" version? When the hot path measurably needs it (benchmark in hand), and it ships fenced: a naive reference implementation kept for differential testing, property tests over random inputs, named/asserted invariants, and a comment citing the benchmark. Cleverness with that escort is engineering; without it, it's a trap for the 2036 maintainer.

References

• Checkstyle documentation — browse the checks, read the rationale fields
• SpotBugs bug descriptions — the pattern catalog
• Error Prone — Google's compile-time equivalent, worth knowing
• Effective Java (Bloch, 3rd ed.) — items 15–17 (minimize accessibility/mutability) are the EI_ patterns' theory
• Java Concurrency in Practice (Goetz) — ch. 2–4 for the guard-discipline vocabulary
• Fowler, Refactoring (2nd ed.) — extract-function/rename as behavior-preserving moves
• Google Engineering Practices: Code Review Developer Guide — the industry mirror of Phase 06

🛸 Hitchhiker's Guide — Phase 05: Code Quality and Engineering Discipline

Read this if: You write code that passes tests but gets rejected in code review — or you've been on a project where every PR battle is about style rather than correctness. By the end you should understand the three-layer automated quality stack every serious Apache project runs, know what each layer catches (and what it misses), and be able to write a structured code review comment that will actually get addressed instead of ignored.

0. The 30-second mental model

Three quality layers. Each catches a different class of problem:

Layer 1 — Style (Checkstyle)
  "Does the code look consistent across all contributors?"
  Catches: wrong names, long lines, missing Javadoc, magic numbers, unused imports
  Misses: anything about what the code does

Layer 2 — Static Analysis (SpotBugs)
  "Does the bytecode have recognizable bug patterns?"
  Catches: null dereferences, resource leaks, mutable state exposure, deprecated APIs
  Misses: logic bugs, wrong algorithms, performance regressions

Layer 3 — Human Review (dev@ mailing list / GitHub review)
  "Is the design sound? Is the API contract right? Are edge cases handled?"
  Catches: everything layers 1-2 miss
  Misses: nothing — but it's slow and expensive

All three complement each other. A green Checkstyle + SpotBugs build doesn't mean the code is correct. A thorough human review doesn't mean the formatting is consistent. Run all three.

1. Checkstyle

1.1 What it does

Checkstyle parses the Java source into an AST (Abstract Syntax Tree) and walks each node, checking it against a set of configured rules. It runs during the Maven build via maven-checkstyle-plugin.

The configuration file (checkstyle.xml) specifies which rules to enforce and with what parameters. Projects either:

• Extend a well-known style (Google Java Style, Sun Coding Conventions)
• Write project-specific rules, often derived from one of the above

Checkstyle fails the build when failsOnError=true (the default for mvn checkstyle:check).

1.2 Rules You Must Know

1.3 Running in Maven

# Check only — does not compile, just style checks
mvn checkstyle:check

# Check during the verify phase (standard CI mode)
mvn verify

# Check and generate an HTML report (does not fail build)
mvn checkstyle:checkstyle
# Report at: target/reports/checkstyle.html

1.4 Suppressing False Positives

Two mechanisms:

Inline suppression (for one-off exceptions):

// CHECKSTYLE:OFF MagicNumber
private static final int FIBONACCI_MULTIPLIER = 0x9e3779b97;
// CHECKSTYLE:ON MagicNumber

Suppression XML (for systematic exceptions, e.g., test classes):

<!-- suppressions.xml -->
<suppressions>
  <suppress checks="MagicNumber" files=".*Test\.java"/>
</suppressions>

Use suppressions sparingly. A rule with 100 suppressions is not enforced — it's just noise.

2. SpotBugs

2.1 How it works

SpotBugs analyzes compiled bytecode (.class files), not source code. It applies ~400 bug patterns — each is a recognizable sequence of bytecode instructions or a data-flow property that typically indicates a bug.

Because it analyzes bytecode, SpotBugs can catch bugs that exist after inlining, optimization, and type erasure — things Checkstyle never sees.

2.2 Bug Categories

2.3 Confidence Levels

Apache projects typically set CI to fail only on HIGH confidence bugs. MEDIUM is reported but not blocking. LOW is usually suppressed.

2.4 Running SpotBugs in Maven

# Check — fails build on threshold violations
mvn spotbugs:check

# Generate HTML report without failing
mvn spotbugs:spotbugs
# Report: target/spotbugsXml.xml + target/spotbugs.html (with spotbugs-maven-plugin)

# Launch the SpotBugs GUI viewer
mvn spotbugs:gui

2.5 The Four Bugs You Must Know

NP_NULL_ON_SOME_PATH — a code path exists where a method is called on a potentially-null reference:

// BUGGY: Map.get() returns null when the key is absent
return properties.get(key).trim();  // NPE if key not present

// FIX:
String value = properties.get(key);
return (value != null) ? value.trim() : null;

EI_EXPOSE_REP — a getter returns a direct reference to an internal mutable object:

// BUGGY: caller can mutate the internal map
public Map<String, String> getProperties() { return properties; }

// FIX:
public Map<String, String> getProperties() {
    return Collections.unmodifiableMap(properties);
}

MS_MUTABLE_COLLECTION — a static final field holds a mutable collection:

// BUGGY: 'final' prevents reassignment, but the list can still be mutated
public static final List<String> REQUIRED_KEYS = new ArrayList<>();

// FIX:
public static final List<String> REQUIRED_KEYS = List.of("host", "port", "db.name");

DM_NUMBER_CTOR — using a deprecated new Integer(int) constructor:

// BUGGY: deprecated since Java 9, removed in Java 17
Integer boxed = new Integer(value);

// FIX:
Integer boxed = Integer.valueOf(value);  // uses cached instances for -128..127

3. Code Review in Apache Projects

3.1 The Review Workflow

Apache projects do code review on GitHub Pull Requests and (for controversial changes) on the dev@ mailing list. The workflow:

1. Contributor opens a PR with a JIRA issue number in the title
2. Committers review: automated CI runs first, then humans review after CI is green
3. Reviewer leaves comments — questions, suggestions, required fixes
4. Contributor addresses each comment with either a fix or a reasoned disagreement
5. A committer with write access merges when they're satisfied (often signaled by "+1")

3.2 What Makes a Good Review

A good review is:

• Specific: point to the exact line and explain the concern
• Actionable: "consider using Collections.unmodifiableMap" not "this is bad"
• Prioritized: distinguish blockers (correctness, API contract) from suggestions (readability) from nits (style)
• Respectful: the tone is collegial, not dismissive — the contributor wrote this on their own time

A bad review is:

• "This is wrong" without explaining why
• Demanding a full rewrite because the reviewer would have designed it differently
• Mixing 10 style nits with 1 real bug — the author can't tell which matters

3.3 The Review Rubric

Use this checklist when reviewing a patch:

3.4 Severity Labels

BLOCKER  — prevents merge; correctness or API regression
MAJOR    — should be fixed before merge; may not be correctness but is important
MINOR    — should be addressed; not urgent
NIT      — trivial style, can be ignored or fixed in a follow-up
QUESTION — asking for clarification, not requesting a change

4. What Automated Tools Miss

This is what you say in interviews:

• Logic errors: Checkstyle and SpotBugs don't know if your algorithm is wrong
• Missing tests: SpotBugs won't tell you the null path isn't tested
• Wrong abstraction: The API might work but be impossible to extend
• Performance regression: SpotBugs can flag O(n²) patterns in some cases, but not always
• Concurrency bugs: SpotBugs catches some, but subtle races require manual analysis (or tools like ThreadSanitizer)

The conclusion: automated tools shrink the review surface so humans can focus on what matters.

5. Interview Questions — Concepts You Must Explain Out Loud

6. References

• Checkstyle — Available Checks
• maven-checkstyle-plugin
• SpotBugs Bug Descriptions
• spotbugs-maven-plugin
• Google Java Style Guide
• Apache Spark Code Review Guidelines — real-world Apache review culture
• Apache Kafka Contributing Guide — review checklist used by a production Apache project

Lab 01 — Checkstyle Enforcement

Phase: 05 — Code Quality and Engineering Discipline
Difficulty: ⭐⭐☆☆☆ | Estimated Time: 1.5–2 hours
Primary Artifact: FlawedScheduler.java with 5 planted Checkstyle violations
Verify: mvn checkstyle:check reports 0 violations after your fixes

Goal

FlawedScheduler.java is a task scheduler that compiles and runs correctly — but it has 5 Checkstyle violations that would be rejected by Apache CI. Your job is to:

1. Run mvn checkstyle:check and read the violation report
2. Identify each violation from the report (rule name, line number, description)
3. Fix each one so the build goes green
4. Understand why each rule exists

Setup

cd phase-05-code-quality/lab-01-checkstyle
mvn checkstyle:check

The build will fail immediately. Read the output — it lists each violation with:

• The rule name (e.g., ConstantName)
• The file and line number
• A short description of the violation

The 5 Violations

Find and fix each one. Do not look at the answers below until you have found them yourself in the report.

Steps

Step 1 — Baseline: read the report

mvn checkstyle:check 2>&1 | grep '\[ERROR\]'

Count the violations. There should be exactly 5. If you see more or fewer, check that you haven't already modified the file.

Step 2 — Fix one violation at a time

Fix them in this order (each fix is isolated):

1. Unused import — easiest, just delete the line
2. Constant name — rename field + update all references in the same file
3. Magic number — extract to a constant above the field declarations
4. Missing Javadoc — add a complete Javadoc block
5. Line too long — restructure the LOG.warning(...) call

After each fix, run mvn checkstyle:check to confirm the violation count decreases by 1.

Step 3 — Green build

mvn checkstyle:check && echo "ALL CLEAN"

This must print ALL CLEAN.

Step 4 — Confirm the class still compiles and the scheduler still works

mvn test

There are no unit tests in this module (the scheduler is tested manually), but the build must succeed with no compilation errors.

Step 5 — Reflection

Answer these questions in a comment at the top of your fixed FlawedScheduler.java:

1. What would happen if you left maxRetries as a magic number 3 in the sleep condition?
2. Why does Checkstyle flag ArrayList when it compiles fine without being used?
3. What's the downside of the Checkstyle LineLength rule being set to 80 instead of 100?

Verification

# Zero Checkstyle violations
mvn checkstyle:check
echo "Exit code: $?"   # must be 0

# Code still compiles
mvn compile

Source Files

checkstyle.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE module PUBLIC "-//Checkstyle//DTD Checkstyle Configuration 1.3//EN" "https://checkstyle.org/dtds/configuration_1_3.dtd">
<!--
  Checkstyle configuration for Lab 01 — Code Quality.

  This file deliberately enables exactly the rules needed to catch the 5
  violations planted in FlawedScheduler.java. It is a subset of what a real
  Apache project (e.g., Spark, Kafka) would enforce.

  To generate a report without failing the build:
    mvn checkstyle:checkstyle
    open target/reports/checkstyle.html

  To fail the build on violations (CI mode):
    mvn checkstyle:check
-->
<module name="Checker">
    <property name="charset" value="UTF-8" />
    <!--
    severity=error means violations are reported as errors (not warnings)
    and will fail the build when maven-checkstyle-plugin's failsOnError=true.
  -->
    <property name="severity" value="error" />

    <!--
    Rule 5: LineLength
    Maximum line length is 100 characters.
    Placed under Checker (not TreeWalker) because it is a file-level check
    that doesn't require AST parsing.

    Why 100 and not 80? Modern monitors are wide enough that 100 columns
    avoids unnecessary line wrapping without going so far that lines are
    unreadable on a split-screen laptop.
  -->
    <module name="LineLength">
        <property name="max" value="100" />
        <!-- Ignore lines that are only a URL (can't be wrapped sensibly) -->
        <property name="ignorePattern" value="^ *\* *https?://" />
    </module>

    <module name="TreeWalker">

        <!--
      Rule 1: UnusedImports
      Flags any import statement whose type is never referenced in the file.

      Why it matters: unused imports are a readability smell — they suggest the
      code was copy-pasted or a refactor left orphaned imports behind. They also
      cause spurious merge conflicts in files that many contributors touch.
    -->
        <module name="UnusedImports" />

        <!--
      Rule 2: ConstantName
      static final fields (constants) must match UPPER_SNAKE_CASE.

      Default pattern: ^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$
      Violating example: static final int maxRetries = 3;
      Correct example:   static final int MAX_RETRIES = 3;

      Why it matters: UPPER_SNAKE_CASE is the universal Java constant convention.
      A lowercase name for a static final field looks like a mutable field at a
      glance, making it harder to reason about mutability.
    -->
        <module name="ConstantName" />

        <!--
      Rule 3: MagicNumber
      Numeric literals other than the listed "ignoreNumbers" are flagged.

      Ignored by default: -1, 0, 1, 2  (universal, self-documenting)
      We also ignore 100 because it appears in the queue capacity constructor
      and is self-documenting in context.

      Violating example: Thread.sleep(5000)
      Correct example:   Thread.sleep(RETRY_DELAY_MS)

      Why it matters: a magic number tells you the value but not the intent.
      RETRY_DELAY_MS is searchable, changeable in one place, and self-documenting.
    -->
        <module name="MagicNumber">
            <property name="ignoreNumbers" value="-1, 0, 1, 2, 100" />
            <property name="ignoreAnnotation" value="true" />
        </module>

        <!--
      Rule 4: MissingJavadocMethod
      Public methods must have a Javadoc comment (/** ... */).
      Scope=public means package-private, protected, and private methods
      are NOT required to have Javadoc (though it is encouraged).

      Why it matters: public methods are part of the library's API contract.
      Without Javadoc, callers don't know what the method promises, what
      null behavior looks like, or what exceptions can be thrown.
    -->
        <module name="MissingJavadocMethod">
            <property name="scope" value="public" />
        </module>

    </module>
</module>

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example.quality</groupId>
    <artifactId>checkstyle-lab</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <name>Code Quality Lab 01 — Checkstyle</name>

    <properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <build>
        <plugins>
            <!--
        maven-checkstyle-plugin: runs Checkstyle against the source files.

        configLocation: path to checkstyle.xml, relative to the project root.
        failsOnError: if true, the build fails when violations are found.
        violationSeverity: minimum severity level that counts as a violation
          (must match <property name="severity" value="error"/> in checkstyle.xml).
        includeTestSourceDirectory: also check test sources (set false here to
          focus only on main sources for this lab).
        consoleOutput: print violations to the console (not just the report file).

        Run:  mvn checkstyle:check   → fail on violations (CI mode)
              mvn checkstyle:checkstyle → generate report only (no build failure)
      -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-checkstyle-plugin</artifactId>
                <version>3.3.1</version>
                <configuration>
                    <configLocation>checkstyle.xml</configLocation>
                    <failsOnError>true</failsOnError>
                    <violationSeverity>error</violationSeverity>
                    <includeTestSourceDirectory>false</includeTestSourceDirectory>
                    <consoleOutput>true</consoleOutput>
                </configuration>
                <!--
          Bind checkstyle:check to the 'verify' lifecycle phase so that
          running `mvn verify` automatically checks style.
          This is how Apache projects integrate Checkstyle into CI.
        -->
                <executions>
                    <execution>
                        <id>checkstyle-verify</id>
                        <phase>verify</phase>
                        <goals>
                            <goal>check</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

FlawedScheduler.java (before your fixes)

package org.example.quality;

// VIOLATION 1 — UnusedImports
// ArrayList is imported but never used in this file.
// Fix: delete this import.
import java.util.ArrayList;

import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.logging.Logger;

/**
 * A simple retry-capable task scheduler backed by a bounded blocking queue.
 *
 * <p>
 * Tasks submitted via {@link #submit(Runnable)} are queued and executed
 * in order by {@link #runAll()}. If a task throws, it is re-queued for up
 * to {@code MAX_RETRIES} attempts with a delay between each attempt.
 */
public class FlawedScheduler {

    private static final Logger LOG = Logger.getLogger(FlawedScheduler.class.getName());

    // VIOLATION 2 — ConstantName
    // Static final fields (constants) must use UPPER_SNAKE_CASE.
    // 'maxRetries' starts with a lowercase letter and uses camelCase.
    // Fix: rename to MAX_RETRIES and update all references in this file.
    static final int maxRetries = 3;

    private final String name;
    private final BlockingQueue<Runnable> queue;

    /**
     * Creates a scheduler with the given name and queue capacity.
     *
     * @param name     a human-readable label used in log messages
     * @param capacity maximum number of tasks that can be queued at once
     */
    public FlawedScheduler(String name, int capacity) {
        this.name = name;
        this.queue = new LinkedBlockingQueue<>(capacity);
    }

    // VIOLATION 3 — MissingJavadocMethod
    // This public method has no Javadoc comment.
    // Fix: add /** @param task the task to enqueue; ignored if null */ above.
    public void submit(Runnable task) {
        if (task != null) {
            queue.offer(task);
        }
    }

    /**
     * Drains the queue, executing each task in order.
     *
     * <p>
     * If a task throws an exception, it is re-queued and retried after a delay,
     * up to {@code MAX_RETRIES} times. Tasks that exceed the retry limit are
     * permanently dropped with a warning log.
     *
     * @throws InterruptedException if the thread is interrupted while sleeping
     *                              between retries
     */
    public void runAll() throws InterruptedException {
        int attempt = 0;
        while (!queue.isEmpty()) {
            Runnable task = queue.poll();
            if (task != null) {
                try {
                    task.run();
                    attempt = 0;
                } catch (Exception ex) {
                    attempt++;
                    if (attempt <= maxRetries) {
                        // VIOLATION 4 — MagicNumber
                        // 5000 is a magic number. Its meaning (milliseconds? retries?) is unclear.
                        // Fix: extract to a named constant, e.g.:
                        // private static final long RETRY_DELAY_MS = 5000L;
                        Thread.sleep(5000);
                        queue.offer(task); // re-enqueue for retry
                    } else {
                        // VIOLATION 5 — LineLength
                        // This line is longer than 100 characters (count from 'LOG' to the closing
                        // ')').
                        // Fix: split the string concatenation across multiple lines, or use a local
                        // variable.
                        LOG.warning("Task permanently failed after " + maxRetries + " retries on scheduler '" + name
                                + "': " + ex.getMessage());
                        attempt = 0;
                    }
                }
            }
        }
    }
}

Lab 02 — SpotBugs Static Analysis

Phase: 05 — Code Quality and Engineering Discipline
Difficulty: ⭐⭐⭐☆☆ | Estimated Time: 2–2.5 hours
Primary Artifact: ProductionConfig.java with 4 planted SpotBugs bugs
Verify: mvn spotbugs:check reports 0 HIGH-confidence bugs after your fixes

Goal

ProductionConfig.java is a configuration manager that compiles cleanly and passes Checkstyle — but it has 4 SpotBugs bugs that indicate real correctness and safety problems. Your job is to:

1. Run mvn spotbugs:check and read the bug report
2. Look up each bug category in the SpotBugs documentation
3. Understand why each pattern is a bug (not just a style issue)
4. Fix each one so the build goes green

Setup

cd phase-05-code-quality/lab-02-spotbugs
mvn spotbugs:check

The build will fail. Read the output — each bug is reported with:

• The bug category code (e.g., NP_NULL_ON_SOME_PATH)
• The class name and line number
• A one-line description

For the full HTML report with more context:

mvn spotbugs:spotbugs
open target/spotbugs.html       # macOS
xdg-open target/spotbugs.html   # Linux

The 4 Bugs

Find and fix each one. Do not expand the answers until you have attempted to understand the bug category from the SpotBugs description alone.

Steps

Step 1 — Read the report

mvn spotbugs:check 2>&1 | grep -E 'Bug|Priority|Type'

Note the bug code for each reported issue. You should see exactly 4 HIGH-confidence bugs.

Step 2 — Look up each bug code

For each bug code (e.g., NP_NULL_ON_SOME_PATH), look it up at:
https://spotbugs.readthedocs.io/en/latest/bugDescriptions.html

Read the description. Before looking at the fix, answer:

• What runtime failure does this cause?
• Under what inputs does it trigger?
• Is this a correctness bug, a safety bug, or a performance bug?

Step 3 — Fix each bug

Fix them in order from most dangerous to least:

1. NP_NULL_ON_SOME_PATH — will cause a production NPE on any missing config key
2. EI_EXPOSE_REP — silent data corruption when callers modify the returned map
3. MS_MUTABLE_COLLECTION — silent data corruption when callers modify REQUIRED_KEYS
4. DM_NUMBER_CTOR — deprecated API, minor performance concern

After each fix, run mvn spotbugs:check to confirm the count decreases by 1.

Step 4 — Green build

mvn spotbugs:check && echo "ALL CLEAN"

Step 5 — Design discussion

Bug #3 (MS_MUTABLE_COLLECTION) is particularly subtle because the code looks correct — the field is declared static final. Answer in a comment added to the top of ProductionConfig.java:

1. What does final actually guarantee for a reference type?
2. What does List.of(...) guarantee that Collections.unmodifiableList(new ArrayList<>()) does NOT?
3. If REQUIRED_KEYS needed to be built dynamically at class load time (the static {} block pattern), what would be a safe alternative to List.of()?

Verification

# Zero HIGH-confidence SpotBugs bugs
mvn spotbugs:check
echo "Exit code: $?"  # must be 0

# Code still compiles
mvn compile

Source Files

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example.quality</groupId>
    <artifactId>spotbugs-lab</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <name>Code Quality Lab 02 — SpotBugs</name>

    <properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <build>
        <plugins>
            <!--
        spotbugs-maven-plugin: runs SpotBugs against the compiled bytecode.

        IMPORTANT: SpotBugs analyzes .class files, not .java files.
        You must run `mvn compile` before `mvn spotbugs:check` (or use `mvn verify`
        which runs compile → test → package → verify in order).

        Goals:
          mvn spotbugs:check     — fail build if bugs found above threshold
          mvn spotbugs:spotbugs  — generate report only (no build failure)
          mvn spotbugs:gui       — launch the interactive SpotBugs viewer

        effort: analysis depth.
          max = most thorough (slowest) — use this in CI for maximum coverage
          default = balanced (good for dev iteration)
          min = fastest (fewer checks)

        threshold: minimum confidence level that causes a build failure.
          High   = only HIGH confidence bugs fail the build
          Medium = HIGH + MEDIUM confidence bugs fail the build
          Low    = all bugs fail the build (many false positives)

        For this lab we use effort=max and threshold=High so that only the
        4 deliberately planted HIGH-confidence bugs cause the build failure.
      -->
            <plugin>
                <groupId>com.github.spotbugs</groupId>
                <artifactId>spotbugs-maven-plugin</artifactId>
                <version>4.8.3.1</version>
                <configuration>
                    <effort>Max</effort>
                    <threshold>High</threshold>
                    <xmlOutput>true</xmlOutput>
                </configuration>
                <executions>
                    <execution>
                        <id>spotbugs-verify</id>
                        <phase>verify</phase>
                        <goals>
                            <goal>check</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

ProductionConfig.java (before your fixes)

package org.example.quality;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

/**
 * Manages application configuration as a key-value store.
 *
 * <p>
 * Configuration is loaded by the {@link #load(Map)} method and
 * validated against {@link #REQUIRED_KEYS} to ensure all mandatory
 * settings are present before the application starts.
 *
 * <p>
 * This class has 4 SpotBugs bugs. Find and fix them.
 * Run: mvn spotbugs:check
 */
public class ProductionConfig {

    // BUG 3 — MS_MUTABLE_COLLECTION
    // The 'final' keyword on a static field only prevents reassigning the
    // reference.
    // The List object itself is still mutable: callers can call
    // REQUIRED_KEYS.add("evil")
    // and corrupt every downstream validation.
    //
    // Fix: replace with List.of("host", "port", "db.name")
    // which returns a truly unmodifiable list.
    public static final List<String> REQUIRED_KEYS = new ArrayList<>();
    static {
        REQUIRED_KEYS.add("host");
        REQUIRED_KEYS.add("port");
        REQUIRED_KEYS.add("db.name");
    }

    // The internal mutable backing store.
    // Note: HashMap is fine here — the bug is in how we expose it (see BUG 2).
    private final Map<String, String> properties = new HashMap<>();

    /**
     * Loads all entries from the supplied map into the internal store.
     * Validates that all required keys are present.
     *
     * @param source the configuration source map (must not be null)
     * @throws IllegalArgumentException if any required key is missing
     */
    public void load(Map<String, String> source) {
        properties.putAll(source);
        for (String key : REQUIRED_KEYS) {
            if (!properties.containsKey(key)) {
                throw new IllegalArgumentException("Missing required config key: " + key);
            }
        }
    }

    /**
     * Returns the trimmed value associated with the given key.
     *
     * @param key the configuration key
     * @return the trimmed value, or null if the key is not present
     */
    public String get(String key) {
        // BUG 1 — NP_NULL_ON_SOME_PATH
        // Map.get() returns null when the key is absent. Calling .trim() on null
        // throws NullPointerException.
        //
        // Fix option A: null check before calling .trim()
        // String value = properties.get(key);
        // return (value != null) ? value.trim() : null;
        //
        // Fix option B: use getOrDefault() to return null explicitly
        // return properties.getOrDefault(key, null); // still needs trim check
        //
        // Fix option C (best): getOrDefault with a null-safe trim
        // String value = properties.getOrDefault(key, null);
        // return (value != null) ? value.trim() : null;
        return properties.get(key).trim();
    }

    /**
     * Returns the internal properties map.
     *
     * @return the properties map
     */
    public Map<String, String> getProperties() {
        // BUG 2 — EI_EXPOSE_REP
        // Returning the internal Map directly allows the caller to mutate it:
        // config.getProperties().clear(); // deletes all config!
        // config.getProperties().put("host", "attacker.example.com"); // overwrite!
        //
        // Fix: return Collections.unmodifiableMap(properties)
        // or return a defensive copy: new HashMap<>(properties)
        return properties;
    }

    /**
     * Stores an integer value for the given key, converting it to a string.
     *
     * @param key   the configuration key
     * @param value the integer value to store
     */
    public void putInt(String key, int value) {
        // BUG 4 — DM_NUMBER_CTOR
        // 'new Integer(int)' uses the deprecated Integer constructor.
        // It was deprecated in Java 9 and removed in Java 17.
        // Integer.valueOf(int) is the idiomatic replacement. It also reuses
        // cached Integer instances for values in [-128, 127], which is
        // faster and uses less memory for common config values like port numbers.
        //
        // Fix: Integer.valueOf(value).toString()
        // Or even simpler: String.valueOf(value)
        properties.put(key, new Integer(value).toString());
    }
}

Lab 03 — Code Review: CleverPartitioner

Phase: 05 — Code Quality and Engineering Discipline
Difficulty: ⭐⭐⭐⭐☆ | Estimated Time: 2.5–3 hours
Primary Artifact: CleverPartitioner.java — a Fibonacci-hashing partitioner with 5 correctness issues
Verify: mvn test shows exactly 2 passing + 2 failing tests; then fix the bugs until all 4 pass

Goal

A contributor submitted CleverPartitioner.java as a pull request to replace the existing modulo-based partitioner. The tests tell part of the story — but a good code review catches what the tests miss.

Your job is to:

1. Run the tests and observe which pass and which fail
2. Write a code review comment for each issue you find (5 total)
3. Fix the correctness bugs so all 4 tests pass
4. Assess whether the fix is sufficient for a real Apache PR to be merged

Background: Fibonacci Hashing

Fibonacci hashing maps a hash value to a bucket using the golden-ratio constant:

φ = (√5 - 1) / 2 ≈ 0.618

In integer arithmetic (64-bit), this is approximated by:

FIBONACCI_MULT = 2^64 × φ = 0x9e3779b97f4a7c15L

The algorithm: multiply the key's hash by FIBONACCI_MULT, then take the top k bits, where k = log₂(n) — this only works when n is a power of two.

The bug in this implementation: the code uses Integer.numberOfTrailingZeros(n) to compute k, which is correct only for powers of two. For n=1, numberOfTrailingZeros(1) = 0, so the shift is 64 - 0 = 64. In Java, shifting a long by 64 is equivalent to shifting by 0 (the shift amount is taken modulo 64), returning the full product instead of 0.

Setup

cd phase-05-code-quality/lab-03-review
mvn test

Expected output: 2 tests PASS, 2 tests FAIL.

Steps

Step 1 — Run the tests and understand the failures

mvn test 2>&1 | grep -E 'Tests run|FAIL|ERROR|expected|but was'

Record:

• Which test fails?
• What is the actual vs. expected value?
• What does that tell you about the algorithm?

Step 2 — Read CleverPartitioner.java and write a review

Before touching any code, write a code review comment for each of the 5 issues marked in the source file. Use the Apache review severity labels:

BLOCKER  — must be fixed before merge
MAJOR    — important but non-blocking correctness issue
MINOR    — should be addressed
NIT      — trivial
QUESTION — asking for clarification

Your review should be in the form:

[BLOCKER] partition(String, int): If n is not a power of two (e.g., n=5),
the shift amount is Integer.numberOfTrailingZeros(5) = 0, so the expression
>>> 64 evaluates as >>> 0 (shift mod 64), returning an unmasked value far
outside [0, n). This is a correctness regression for any cluster with a
non-power-of-two partition count. Requires either: (a) restricting n to
powers of two with an explicit precondition check, or (b) replacing the
algorithm with one that handles arbitrary n.

Step 3 — Fix the correctness bugs

Fix in this order:

1. Null guard: add a null check on key and throw IllegalArgumentException (not NullPointerException — explicit contract is better than an implicit crash)
2. Non-positive n guard: throw IllegalArgumentException if n <= 0
3. Power-of-two guard: throw IllegalArgumentException with a clear message if n is not a power of two — (n & (n - 1)) != 0 is the idiomatic check
4. n=1 edge case: with the power-of-two guard in place, n=1 is now valid; ensure partition(key, 1) always returns 0 (all keys go to bucket 0 — there is only one bucket)
5. Magic constant citation: add a Javadoc or inline comment explaining 0x9e3779b97f4a7c15L — this is the Knuth/Fibonacci multiplier and must be cited

Step 4 — Verify all 4 tests pass

mvn test

All 4 tests must pass:

• testPowerOfTwo_distributionInRange — basic power-of-two bucketing
• testSingleBucket_alwaysReturnsZero — n=1 must always return 0
• testNonPowerOfTwo_throwsIllegalArgument — n=5 must throw IllegalArgumentException
• testNullKey_throwsIllegalArgument — null key must throw IllegalArgumentException

Step 5 — Reassess: is the PR ready to merge?

After your fixes, answer in a comment at the top of CleverPartitioner.java:

1. Is it still Fibonacci hashing if we restrict n to powers of two? (Yes — this is the standard form.)
2. A Kafka partition count can be any positive integer, not just a power of two. What algorithm would you recommend as a replacement that handles arbitrary n? (Hint: Math.abs(key.hashCode()) % n is simple but has skew for non-prime n. What's the production alternative?)
3. Is the 0x45d9f3bL constant on the hashCode mixing step necessary? What does it do?

Verification

# All 4 tests pass
mvn test
echo "Exit code: $?"  # must be 0

Source Files

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example.quality</groupId>
    <artifactId>review-lab</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <name>Code Quality Lab 03 — Code Review</name>

    <properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <version>5.10.2</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.2.5</version>
            </plugin>
        </plugins>
    </build>

</project>

CleverPartitioner.java

package org.example.quality;

/**
 * Partitioner using Fibonacci hashing to assign a string key to one of
 * {@code n} buckets.
 *
 * <p>
 * Fibonacci hashing exploits the golden-ratio constant to spread keys more
 * evenly
 * than a simple modulo, especially when hashes cluster around multiples of a
 * common factor.
 *
 * <p>
 * This implementation has 5 issues for code review. Find them all before
 * fixing.
 *
 * <p>
 * Algorithm (for a power-of-two n):
 * 
 * <pre>
 *   h = mix(key.hashCode())
 *   bucket = (int)((h * FIBONACCI_MULT) >>> (64 - log2(n)))
 * </pre>
 *
 * <p>
 * Where {@code FIBONACCI_MULT = 2^64 * (sqrt(5)-1)/2} — the 64-bit
 * Fibonacci/Knuth
 * multiplicative constant.
 *
 * @see <a href=
 *      "https://en.wikipedia.org/wiki/Hash_table#Fibonacci_hashing">Fibonacci
 *      hashing</a>
 */
public class CleverPartitioner {

    // ISSUE 1 — Magic constant with no citation
    // This is the 64-bit Fibonacci multiplicative constant (Knuth, vol 3, §6.4):
    // FIBONACCI_MULT = floor(2^64 * (sqrt(5) - 1) / 2) = 0x9e3779b97f4a7c15L
    // Without the comment (which was "// trust me" in the original PR), a reviewer
    // cannot verify this constant is correct without re-deriving it from scratch.
    // Severity: MAJOR — cite the source.
    private static final long FIBONACCI_MULT = 0x9e3779b97f4a7c15L;

    /**
     * Returns the bucket index for the given key in a cluster of {@code n}
     * partitions.
     *
     * @param key the key to partition (must not be null)
     * @param n   the number of partitions (must be a positive power of two)
     * @return a bucket index in the range {@code [0, n)}
     */
    public int partition(String key, int n) {
        // ISSUE 2 — No null check on 'key'
        // If key is null, key.hashCode() throws NullPointerException with no useful
        // message.
        // The contract should be explicit: IllegalArgumentException with "key must not
        // be null".
        // Severity: MAJOR — silent crash with an unhelpful stack trace.

        // ISSUE 3 — No validation that n > 0
        // If n == 0, the shift amount is 64 - numberOfTrailingZeros(0) = 64 - 32 = 32.
        // If n < 0, numberOfTrailingZeros returns a nonsensical result.
        // Severity: BLOCKER — undefined behavior for invalid inputs.

        // ISSUE 4 — Assumes n is a power of two; no guard
        // numberOfTrailingZeros(n) only gives log2(n) when n IS a power of two.
        // For n = 5: numberOfTrailingZeros(5) = 0, so the shift is (64 - 0) = 64.
        // In Java, (long) >>> 64 is equivalent to >>> 0 (shift mod 64) — the full
        // 64-bit product, truncated to int, which is not in [0, 5).
        // Severity: BLOCKER — wrong results for any non-power-of-two partition count.
        int shift = 64 - Integer.numberOfTrailingZeros(n);

        // ISSUE 5 — Magic mixing constant with no explanation
        // 0x45d9f3bL is a secondary mixing multiplier used to improve bit distribution
        // before the Fibonacci step. Without a comment, reviewers cannot verify this
        // is the correct constant or understand its purpose.
        // Severity: MINOR — code should be self-documenting.
        long h = (long) key.hashCode() * 0x45d9f3bL;

        return (int) ((h * FIBONACCI_MULT) >>> shift);
    }
}

PartitionerTest.java

package org.example.quality;

import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

/**
 * Tests for {@link CleverPartitioner}.
 *
 * Before fixes: 2 tests pass, 2 tests fail.
 * After fixes: all 4 tests pass.
 *
 * Run with: mvn test
 */
class PartitionerTest {

    private CleverPartitioner partitioner;

    @BeforeEach
    void setUp() {
        partitioner = new CleverPartitioner();
    }

    /**
     * Test 1 — PASSES before fixes.
     *
     * For a power-of-two n (n=8), every key must produce a bucket in [0, 8).
     * Fibonacci hashing is deterministic, so the same key always maps to the same
     * bucket.
     */
    @Test
    void testPowerOfTwo_distributionInRange() {
        int n = 8;
        String[] keys = { "key-0", "key-1", "key-2", "key-3", "alpha", "beta", "gamma", "delta" };
        for (String key : keys) {
            int bucket = partitioner.partition(key, n);
            assertTrue(bucket >= 0 && bucket < n,
                    "Expected bucket in [0, " + n + ") for key=" + key + " but got: " + bucket);
        }
    }

    /**
     * Test 2 — FAILS before fixes.
     *
     * n=1 means "there is only one bucket; everything goes to bucket 0".
     *
     * Bug: numberOfTrailingZeros(1) = 0, so shift = 64 - 0 = 64.
     * In Java, (long) >>> 64 is (long) >>> (64 % 64) = (long) >>> 0, which is the
     * full
     * 64-bit product cast to int — not 0.
     *
     * Fix required: treat n=1 as a special case (return 0 always), OR add a
     * power-of-two guard that catches n=1 and computes shift=0 explicitly using
     * Integer.numberOfLeadingZeros(n) - 1 or Integer.SIZE -
     * Integer.numberOfLeadingZeros(n-1).
     */
    @Test
    void testSingleBucket_alwaysReturnsZero() {
        String[] keys = { "any-key", "another-key", "yet-another" };
        for (String key : keys) {
            int bucket = partitioner.partition(key, 1);
            assertEquals(0, bucket, "With n=1 every key must map to bucket 0, but got: " + bucket);
        }
    }

    /**
     * Test 3 — FAILS before fixes.
     *
     * n=5 is not a power of two. The partitioner must reject it with
     * IllegalArgumentException rather than silently returning a wrong bucket.
     *
     * Bug: without a guard, the code silently returns values outside [0, 5).
     *
     * Fix required: add a precondition check before computing the shift.
     */
    @Test
    void testNonPowerOfTwo_throwsIllegalArgument() {
        assertThrows(IllegalArgumentException.class,
                () -> partitioner.partition("any-key", 5),
                "Expected IllegalArgumentException for non-power-of-two n=5");
    }

    /**
     * Test 4 — PASSES before fixes (an NPE is thrown, and the test expects it).
     *
     * BUT: NullPointerException is the wrong exception here. A method that accepts
     * user input should throw IllegalArgumentException with a clear message, not
     * an opaque NPE. This test is written to document the DESIRED behavior after
     * fix.
     *
     * Before fix: NullPointerException is thrown — test passes because NPE is a
     * subtype
     * of RuntimeException, which satisfies assertThrows(NullPointerException.class,
     * ...).
     *
     * After fix: IllegalArgumentException is thrown — the test must be updated to
     * expect
     * IllegalArgumentException, OR the test is already written correctly and
     * the implementation must throw IllegalArgumentException.
     *
     * The test below expects IllegalArgumentException. Before fixing, it FAILS.
     * After fixing, it PASSES.
     *
     * This is the review point: the test was written to catch the NPE (which
     * already
     * "works") but the right fix is IllegalArgumentException with a useful message.
     */
    @Test
    void testNullKey_throwsIllegalArgument() {
        assertThrows(IllegalArgumentException.class,
                () -> partitioner.partition(null, 4),
                "Expected IllegalArgumentException for null key, not NullPointerException");
    }
}

Phase 06 — Review Culture and Patch Workflow

Prerequisite: Phase 05 (you know what Checkstyle and SpotBugs catch; now you learn what humans catch)
Outcome: You can read any PR as a committer would — produce a structured, actionable review; argue for a design with evidence; and read a raw git diff to find problems before they hit main.

Why This Phase Matters

Getting +1 from a committer means your code survived three filters: CI (automated), style review (Checkstyle/SpotBugs), and human review. The human review is the only filter that can catch wrong abstractions, subtle concurrency bugs, API contract regressions, and missing test coverage.

Apache committers spend 30–50% of their time reviewing code, not writing it. If you want to become a committer, you need to demonstrate that you can protect the codebase — not just contribute to it.

Labs

Core Concepts

Deliverables Checklist

• MessageBatcher.java — all 6 issues found, 4 correctness bugs fixed, all tests pass
• ApproachComparison.md — written vote email comparing TokenBucketRateLimiter vs SlidingWindowRateLimiter, citing benchmark output
• kafka-pr-17873-review.md — retrospective review of the synthetic patch with severity labels for each issue found
• mvn test green in all 3 lab modules

Interview Relevance

Warmup Guide — Review Culture & Patch Evaluation

Zero-to-expert primer for Phase 06: how to review code the way committers do — the checklist and its ordering, the craft of writing review comments, defending and revising your own patches, and reading real patch history as training data.

Table of Contents

• Chapter 1: What Review Is For (It Isn't Bug-Finding Alone)
• Chapter 2: The Review Checklist, Ordered by Blast Radius
• Chapter 3: Reading a Diff Effectively
• Chapter 4: Writing Review Comments That Work
• Chapter 5: Being Reviewed — Responding and Revising
• Chapter 6: Evaluating Design Tradeoffs in Review
• Chapter 7: Patch Archaeology — Learning from Merged History
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: What Review Is For (It Isn't Bug-Finding Alone)

Review serves four functions, and optimizing only the first produces bad reviewers:

1. Defect detection — yes, but studies and experience agree most defects are caught by tests and tools; review catches the design-level ones they can't (Phase 05 Ch. 5).
2. Knowledge transfer — after the review, two people understand the change; the bus factor of every merged line is ≥2. This is why "LGTM" without reading is a real harm even when the code was fine.
3. Standard-setting — review is where a codebase's quality bar is actually enforced and taught; every comment is a tiny lecture other contributors read too.
4. Trust calibration (the Apache-specific function) — review threads are where the PMC observes your judgment, both as reviewer and author. Phase 01's trust model is operationalized here; it's why this phase sits at the curriculum's center.

Chapter 2: The Review Checklist, Ordered by Blast Radius

Review the irreversible first. The ordering (memorize it; Lab 01 grades against it):

1. Contract & compatibility: public API/config/wire/format changes? Annotations respected? japicmp/MiMa consulted? (A perfect implementation of a breaking change is still a -1.)
2. Correctness: new states, error paths, boundary inputs, concurrency. Ask of every branch: what test would fail if this line were wrong — does it exist?
3. Failure behavior: partial failure, retries, resource cleanup on the exception path. Most production incidents live in code that only runs when something else broke.
4. Performance: only for hot paths, and then with evidence — demand the benchmark, review its design (Phase 08's JMH pitfalls are reviewer knowledge).
5. Maintainability: naming, structure, comment quality, test readability — Phase 05's properties.
6. Mechanics last: style nits the tools missed, typos. Prefix with "nit:" and never block on them.

The anti-pattern this ordering prevents: fifteen naming comments on a patch whose core design breaks rolling upgrades — attention spent inversely to consequence.

Chapter 3: Reading a Diff Effectively

The mechanics nobody teaches:

• Two passes: first pass for shape — files touched, test-to-code ratio, anything unexpected in scope (a "fix NPE" patch touching 14 files is a scope conversation, not a line-comment conversation). Second pass for lines.
• Start from the tests: they state what the author thinks the change does; diffs between that and what the code does are your highest-value findings.
• Read around the diff: the bug pattern git diff hides is the interaction between changed lines and unchanged code — open the full file at the call sites; check other callers of a modified method (find-usages on every signature change).
• Hunt the asymmetries: resources opened vs closed, locks taken vs released, state mutated vs restored on the exception path, new enum/case added vs every switch over it.
• For concurrency: write down the guard story (which lock protects which field) as you read; if you can't, that's the comment.

Chapter 4: Writing Review Comments That Work

The craft (Lab 01's rubric, and interview-prep/04's scenarios):

• Severity-label everything: blocking: / question: / nit: — the author must know what stands between them and merge. Unlabeled comments read as all-blocking and poison the thread.
• Reasons, not verdicts: "this races when two batches flush concurrently — count is read outside the lock at line 84" is actionable; "this is thread-unsafe" is a homework assignment; "I don't like this" is noise.
• Questions before accusations when you're <90% sure: "what prevents X here?" — half the time there's an invariant you missed, and the question surfaces it without cost; the other half, the author finds the bug themselves (better teaching).
• Anchor in shared standards, not personal taste: cite the style guide, an existing pattern in the codebase, a measured number. Taste-comments are where threads go to die.
• Acknowledge the good — one specific sentence ("the test matrix on the boundary conditions is exactly right"). Not politeness theater: it teaches what to keep doing, and it keeps newcomers (Phase 01's funnel) coming back.
• Know your confidence: "I believe this breaks X, but verify — I may be missing context" is senior; false certainty in public threads compounds badly.

Chapter 5: Being Reviewed — Responding and Revising

The author-side discipline (Phase 12 Ch. 3 covers the full mechanics; the essentials):

• Respond to every comment: fixed / pushed-back-with-reasons / clarifying question.
• Batch revisions — one push addressing all comments; reviewers re-review revisions, not commits.
• Take preferences cheaply (spend disagreement budget only on what matters); rebut blocking comments with evidence (a test, a benchmark, a spec citation) after restating the concern accurately.
• When wrong: say so fast, fix, and convert the misunderstanding into a comment or test so the next reader doesn't repeat it.
• Patch stalled? One polite ping after ~a week, then the project's escalation convention. Meanwhile review others' patches — the queue runs on reciprocity.

Chapter 6: Evaluating Design Tradeoffs in Review

Lab 02 presents competing implementations (token-bucket vs sliding-window rate limiter); the evaluation method generalizes:

1. Make the axes explicit before judging: correctness under burst, memory per key, contention behavior, observability, complexity. Unstated axes are how design arguments go circular.
2. Demand the workload: there is no "better rate limiter," only better-for- burst-tolerance or better-for-strict-ceilings. A reviewer's first design question is "what traffic shape and what failure cost?"
3. Weigh evidence classes correctly: a benchmark beats an argument for the benchmarked case only (check its design — single-threaded JMH proves nothing about contention; Phase 08); an invariant argument beats a benchmark for correctness.
4. Price the second implementation: every alternative kept alive doubles test surface and doc burden — "both, behind a flag" is sometimes right (workloads genuinely differ) and always expensive; make the cost part of the recommendation.
5. Decide reversibility: config-defaulted choices are cheap to revisit; wire-format choices aren't (Phase 03 Ch. 5). Spend deliberation proportional to irreversibility.

Chapter 7: Patch Archaeology — Learning from Merged History

Lab 03's method — real merged patches (e.g., the bundled Kafka PR) as training data:

• Reconstruct the timeline: rounds of review, what changed between revisions, time to merge. The delta between v1 and merged is a map of what this project's reviewers actually require — more honest than any contributing guide.
• Classify every review comment with Chapter 2's checklist levels — you'll find the distribution skews far more toward contract/failure-mode than newcomers expect.
• Note the social moves: how disagreement was de-escalated, how scope was cut ("file a follow-up JIRA" as the universal pressure valve), who tipped the decision and with what kind of evidence.
• Steal templates: good JIRA descriptions, good commit messages, good test names — archaeology gives you the project's house style as worked examples.

Lab Walkthrough Guidance

Order: Lab 01 (flawed patch) → Lab 02 (design tradeoffs) → Lab 03 (archaeology).

• Lab 01: review MessageBatcher with Chapter 2's ordering — find the concurrency and failure-path defects before polishing nits; write every comment to Chapter 4's rubric (severity label + reason + suggestion). Grade yourself: would your blocking comments survive the author's pushback?
• Lab 02: write the axes table first (Chapter 6.1) for both rate limiters, run the provided benchmark, then write the recommendation for two different stated workloads — the answer should flip, and the writeup should show why.
• Lab 03: apply Chapter 7's method to the bundled Kafka patch; the deliverable insight is the comment-classification histogram plus three templates you'd reuse.

Success Criteria

You are ready for Phase 07 when you can, from memory:

1. Name the four functions of review and why "LGTM-without-reading" damages three of them.
2. Recite the blast-radius checklist order and justify it with one sentence.
3. Rewrite a verdict-comment ("this is wrong") into a working comment (reason + severity + suggestion) on demand.
4. List four diff-reading asymmetries to hunt.
5. Evaluate two competing designs by explicit axes + workload, and price the keep-both option.

Interview Q&A

Q: Review this PR live. (The meta-question: how do you start?) Narrate the two-pass method: "First, shape — what's touched, where are the tests, does scope match the title. Then contract: any public surface change? Then I'll read tests-first to learn intent, then the diff with attention on error paths and concurrency, then nits last, labeled." Interviewers grade the order more than the findings — it reveals whether you review by blast radius or by line number.

Q: A senior engineer's patch has a real flaw. How do you comment differently than on a newcomer's patch? The technical content is identical — the flaw, the evidence, the severity label; deference on correctness is how bad code ships. What changes is calibration: with a senior author I lead with the question form ("what prevents X?") because the prior that I'm missing context is higher, and I skip the teaching detail a newcomer needs. What never changes: public thread, evidence attached, respectful and direct.

Q: How do you keep review turnaround from becoming the team bottleneck? Make reviewability the author's job too: small PRs (norm, not nag), description states what+why+how-tested, CI green before human eyes. Reviewer side: SLA on first response (even "looking Thursday"), two-pass triage so trivial PRs clear in minutes, severity labels so authors parallelize fixes. And measure it — review latency is a team health metric, not a virtue contest.

References

• Google Engineering Practices: Code Review — both the reviewer and author guides
• Software Engineering at Google, ch. 9 (Code Review) — the four-functions framing
• Apache Kafka coding guide + review norms
• SmartBear, Best Kept Secrets of Peer Code Review (the Cisco study) — review-size vs defect-yield data
• How to Do Code Reviews Like a Human (Lindsey Kuper-recommended classic by M. Lynch)
• Phase 12's HITCHHIKERS-GUIDE §3 — the author-side mechanics in full

🛸 Hitchhiker's Guide — Phase 06: Review Culture and Patch Workflow

Read this if: You've gotten a -1 review and didn't understand why, or you've reviewed a PR and left only style comments while the real bug shipped anyway. By the end you should be able to produce a review that a senior Apache committer would recognize as the work of someone who has read the codebase and understands the invariants — not just someone who ran the linter.

0. The 30-second mental model

A code review has three questions:

  1. Is it correct?
     — edge cases, concurrency, null safety, exception contract
     — what tests are missing?

  2. Is the API contract preserved?
     — binary compatibility (existing compiled callers still work)
     — source compatibility (existing source callers still compile)
     — Javadoc accuracy (is the documented contract still true?)

  3. Can the next person maintain it?
     — naming, comments that explain WHY (not WHAT), no dead code
     — no "trust me" constants without citations

Automated tools (Checkstyle, SpotBugs) answer parts of question 3.
Only humans can answer 1 and 2.

1. Anatomy of a Good Review Comment

1.1 The five-part structure

Every substantive review comment should have five parts:

[SEVERITY] Location: One-sentence statement of the problem.

Why it matters: What runtime failure or contract violation does this cause?
Under what inputs or conditions does it trigger?

Evidence: Quote the relevant code (2–5 lines).

Suggested fix: A concrete alternative (code, not just "fix this").

Label: BLOCKER / MAJOR / MINOR / NIT / QUESTION

1.2 Real examples

Good:

[BLOCKER] MessageBatcher.add(): The backing ArrayList is accessed from multiple
threads without synchronization.

Thread A can be in the middle of ArrayList.add() (which grows the internal array
when capacity is exceeded) while Thread B calls add() simultaneously. The result
is either a lost update (one message silently dropped) or an ArrayIndexOutOfBoundsException
depending on the timing of the capacity resize.

    private List<String> batch = new ArrayList<>();  // ← no synchronization

Fix: use CopyOnWriteArrayList, or wrap add/flush in synchronized(this), or use
a ConcurrentLinkedQueue and drain it in flush().

Bad:

This is not thread-safe. Please fix.

The bad version is not actionable — the author doesn't know which line is the problem, what failure mode to worry about, or what to do instead.

1.3 Severity labels in practice

A review that is all NITs and no BLOCKERs means either the code is excellent or the reviewer wasn't looking hard enough.

2. What Tests Don't Prove

Tests prove that specific inputs produce specific outputs. They don't prove:

• Absence of concurrency bugs — a test that passes on your laptop may fail under load when two threads hit the same code path simultaneously.
• Correctness of the API contract — a test can assert that flush() returns a list, but not that the return type change is binary-compatible with callers compiled against the old version.
• Completeness — a 100% line-coverage test suite can still miss edge cases if the tests only exercise the happy path with valid inputs.

When reviewing tests, ask:

• Is there a test for null inputs?
• Is there a test for the boundary condition (empty, exactly-max-size, max-size+1)?
• Is there a test for the failure path (closed batcher, zero-capacity bucket)?
• Is there a concurrent test for any code that will be called from multiple threads?

3. API Contract Regression

3.1 Binary compatibility

Two versions are binary compatible if class files compiled against version N work without recompilation against version N+1.

A method signature change is a binary compatibility break:

// v1 — original
public void flush() { ... }

// v2 — changed return type
public List<String> flush() { ... }
// ← Any caller compiled against v1 that calls flush() will get:
//   java.lang.NoSuchMethodError: flush()V
//   (The JVM looks for flush with void return "V"; can't find it)

3.2 Source compatibility

Two versions are source compatible if source code that compiled against version N still compiles against version N+1.

// Caller source (compiled against v1):
batcher.flush();  // void return, discard result — compiles against v1

// Against v2:
batcher.flush();  // List<String> return, result discarded — STILL COMPILES
                  // (Java allows discarding a non-void return value)

This example is source-compatible but binary-incompatible. This is the subtle case that reviewers miss: "it compiles" does not mean "it's compatible."

3.3 When does this matter in Apache projects?

Apache projects that publish library JARs (Kafka, Spark, Commons) have strict compat policies. The rule:

• Minor version (1.x → 1.y): must be binary compatible
• Major version (1.x → 2.0): may break binary compat, but must document all breaks

If a PR changes a public method signature without a major version bump, it gets a -1.

4. Design Tradeoffs: How to Argue for an Approach

When two approaches are both correct, the right way to advocate for one is:

1. State the tradeoff precisely — not "approach A is better" but "approach A has O(1) acquire() at the cost of O(k) space (k = bucket capacity), while approach B has O(log w) acquire() but O(w) space (w = window width in requests)"
2. Provide evidence — benchmark numbers, not intuition
3. State the use case — for a rate limiter where bursty traffic is acceptable and p99 latency matters, token bucket. For strict "no more than N requests in any window" enforcement, sliding window.
4. End with a concrete recommendation — "I recommend token bucket for this use case because X. If the team prefers strict fairness, sliding window is acceptable with the caveat that it uses O(limit) memory."

This is the structure of a "+1 with conditions" review: you're in favor but with documented constraints.

5. Reading a Patch: git diff Format

A .patch file is a git diff with mail headers. The diff format:

--- a/src/main/java/org/example/NetworkClient.java   ← old file
+++ b/src/main/java/org/example/NetworkClient.java   ← new file

@@ -42,7 +42,12 @@ public class NetworkClient {
                                  ↑    ↑
                               old  new  (start line, count)

 public void connect(String host) {  ← context (unchanged)
-    this.socket = new Socket();     ← removed (red in GitHub)
+    if (host == null) {             ← added (green in GitHub)
+        throw new IllegalArgumentException("host must not be null");
+    }
+    this.socket = new Socket();
 }

When reading a patch:

1. Read the @@ header to understand what method/class you're in
2. Read the - lines (removed) to understand what the code used to do
3. Read the + lines (added) to understand what it does now
4. Check the context lines (no prefix) to understand the surrounding invariants

Common things to look for in the diff:

• A removed null check (regression)
• A new throws clause added without updating the interface
• A method renamed (binary break)
• A synchronized block removed (thread safety regression)

6. Interview Questions — Concepts You Must Explain Out Loud

7. References

• Apache Software Foundation — How the ASF Works
• Apache Kafka Code Review Guidelines
• How to Review Pull Requests — Apache Spark Wiki
• Binary Compatibility — Java SE Specification §13
• OpenJDK Review Process
• Token Bucket vs. Sliding Window — Cloudflare Blog

Lab 01 — FlawedPatch: Review MessageBatcher

Phase: 06 — Review Culture and Patch Workflow
Difficulty: ⭐⭐⭐☆☆ | Estimated Time: 2.5–3 hours
Primary Artifact: MessageBatcher.java with 6 planted review issues
Verify: mvn test passes all 4 tests after your fixes

Scenario

A contributor submitted this PR to the dev@ list:

[PATCH] PROJ-4471 — MessageBatcher: return flushed messages for downstream processing

Previously flush() returned void and callers had no way to process the flushed batch without maintaining their own state. This patch changes flush() to return the flushed list. Also adds autoFlush support so batchers can flush automatically on close.

Tests are passing locally. Please review.

The CI is green. Checkstyle is green. SpotBugs is green.

Your job: find the 6 issues before this merges.

Setup

cd phase-06-review-culture/lab-01-flawed-patch
mvn test

Expected: 2 tests PASS, 2 tests FAIL. Read the failure messages — they tell you which of the 6 issues are caught by tests and which are not.

Step 1 — Read the code and write your review

Before running anything: read MessageBatcher.java and write a structured review comment for each issue you find. Use the format:

[SEVERITY] Method/field: Problem statement.

Why it matters: ...
Evidence: (quote the line)
Fix: ...

There are 6 issues. Find all 6 before looking at the answer table below.

Step 2 — Fix the correctness issues

Fix the issues in order of severity. Issues #1, #2, #3, #4 require code changes. Issue #5 is a cleanup. Issue #6 requires a new test.

For issue #1 (thread safety): Use synchronized on the this monitor for both add() and flush(). This is the simplest correct fix; a production system might use ReentrantLock or a ConcurrentLinkedQueue for better throughput.

For issue #2 (API regression): The fix is context-dependent. In a library, you would NOT change the return type in a minor release. Document your finding as: "This is a binary-incompatible change. If required, it must ship in a major version with a deprecation notice for flush() returning void, or be introduced as a new method flushAndGet()."

For this lab: change the signature back to public synchronized void flush() and add a separate public synchronized List<String> drainBatch() method.

For issue #3 (null guard): throw IllegalArgumentException("message must not be null").

For issue #4 (wrong exception type): throw IllegalStateException("batcher is closed").

For issue #5 (dead code): simplify size > maxSize || size == maxSize to size >= maxSize.

For issue #6 (missing test): add testAutoFlush_closeTriggersDrain() to MessageBatcherTest.java.

Step 3 — Verify all 4 tests pass

mvn test && echo "ALL PASS"

Step 4 — Write the review summary

Write a final paragraph (3–5 sentences) as if you were posting to dev@:

Thanks for the patch. I've identified 2 BLOCKERs, 1 MAJOR, and 3 lower-priority issues
before we can merge this:

1. [BLOCKER] Thread safety: ...
2. [BLOCKER] API regression: ...
...

Marking -0 pending resolution of the BLOCKERs. Happy to re-review once those are addressed.

A -0 means "I have concerns but am not vetoing" — common for patches that are close but need work.

Source Files

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example.review</groupId>
    <artifactId>flawed-patch-lab</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <name>Review Culture Lab 01 — FlawedPatch</name>

    <properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <version>5.10.2</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.2.5</version>
            </plugin>
        </plugins>
    </build>

</project>

MessageBatcher.java

package org.example.review;

import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

/**
 * Collects messages into a batch and flushes them when the batch reaches
 * {@code maxSize}, or when {@link #flush()} is called explicitly.
 *
 * <p>
 * This class has 6 review issues. Find them all before fixing.
 *
 * <p>
 * PR description: "Changed flush() to return the flushed messages for
 * downstream processing. Added autoFlush support so batchers flush on close."
 */
public class MessageBatcher {

    // ISSUE 1 — Thread safety
    // ArrayList is not thread-safe. If two threads call add() simultaneously:
    // - Thread A may see a partial write from Thread B during ArrayList.grow()
    // - Result: ArrayIndexOutOfBoundsException OR silently lost messages
    // The class is documented as usable from multiple threads (see the test),
    // but the implementation has no synchronization.
    //
    // Fix: synchronize add() and flush() on 'this', or use CopyOnWriteArrayList,
    // or restructure to use ConcurrentLinkedQueue + atomic drain.
    private List<String> batch = new ArrayList<>();

    private final int maxSize;
    private final boolean autoFlush;
    private boolean closed = false;

    /**
     * Creates a new batcher.
     *
     * @param maxSize   the maximum number of messages before an automatic flush
     * @param autoFlush if true, the batcher flushes remaining messages when closed
     */
    public MessageBatcher(int maxSize, boolean autoFlush) {
        this.maxSize = maxSize;
        this.autoFlush = autoFlush;
    }

    /**
     * Adds a message to the batch. If the batch reaches {@code maxSize},
     * an automatic flush is triggered.
     *
     * @param message the message to add
     * @throws RuntimeException if the batcher has been closed
     */
    public void add(String message) {
        if (closed) {
            // ISSUE 4 — Wrong exception type
            // The closed-state contract violation should use IllegalStateException,
            // not the raw RuntimeException. IllegalStateException is the standard
            // Java idiom for "object is in a state that does not allow this operation"
            // (see: Iterator.remove() after end, Queue.remove() on empty queue).
            // Callers catching IllegalStateException specifically will miss this.
            //
            // Fix: throw new IllegalStateException("batcher is closed");
            throw new RuntimeException("batcher is closed");
        }

        // ISSUE 3 — No null guard
        // If message is null, it is silently added to the batch.
        // Downstream code iterating the flushed list and calling message.length()
        // will get a NullPointerException with no indication of where the null
        // was introduced.
        //
        // Fix: if (message == null) throw new IllegalArgumentException("message must
        // not be null");
        batch.add(message);

        // ISSUE 5 — Dead code
        // The condition 'size > maxSize || size == maxSize' simplifies to 'size >=
        // maxSize'.
        // The '== maxSize' branch can never execute when '> maxSize' is already true,
        // and both branches do the same thing (flush). This is a NIT — no correctness
        // impact — but it signals that the author was not thinking clearly about the
        // condition.
        //
        // Fix: if (batch.size() >= maxSize) { flush(); }
        if (batch.size() > maxSize || batch.size() == maxSize) {
            flush();
        }
    }

    /**
     * Flushes the current batch and returns the flushed messages.
     *
     * <p>
     * <strong>WARNING — ISSUE 2 (API regression):</strong>
     * The previous version of this method had signature
     * {@code public void flush()}.
     * Changing the return type to {@code List<String>} is a
     * <em>binary-incompatible</em>
     * change: any caller compiled against the old void signature will get:
     * 
     * <pre>
     *   java.lang.NoSuchMethodError: flush()V
     * </pre>
     * 
     * at runtime, because the JVM resolves method descriptors including the return
     * type.
     * <p>
     * This is source-compatible (existing code that discards the return value still
     * compiles),
     * but binary-incompatible. In Apache library policy, this requires a major
     * version bump
     * or introduction as a new method name.
     *
     * @return an unmodifiable snapshot of the flushed messages (never null, may be
     *         empty)
     */
    public List<String> flush() {
        List<String> flushed = Collections.unmodifiableList(new ArrayList<>(batch));
        batch.clear();
        return flushed;
    }

    /**
     * Closes the batcher. If {@code autoFlush} was set at construction time,
     * any remaining messages are flushed before closing.
     *
     * <p>
     * <strong>ISSUE 6 — Missing test:</strong>
     * The {@code autoFlush} feature added in this PR has no test. There is no test
     * that verifies messages added before {@code close()} are flushed when
     * {@code autoFlush=true}, nor that they are NOT flushed when
     * {@code autoFlush=false}.
     * The feature could be completely broken and CI would not catch it.
     */
    public void close() {
        if (autoFlush && !batch.isEmpty()) {
            flush();
        }
        closed = true;
    }
}

MessageBatcherTest.java

package org.example.review;

import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;

import java.util.List;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;

import static org.junit.jupiter.api.Assertions.*;

/**
 * Tests for {@link MessageBatcher}.
 *
 * Before fixes: 2 tests PASS, 2 tests FAIL.
 * After all fixes: all 4 tests PASS.
 */
class MessageBatcherTest {

    private MessageBatcher batcher;

    @BeforeEach
    void setUp() {
        batcher = new MessageBatcher(3, false);
    }

    /**
     * Test 1 — PASSES before fixes.
     * Basic add-and-flush contract: messages added before flush() are returned.
     */
    @Test
    void testAddAndFlush() {
        batcher.add("hello");
        batcher.add("world");
        List<String> flushed = batcher.flush();
        assertEquals(2, flushed.size());
        assertTrue(flushed.contains("hello"));
        assertTrue(flushed.contains("world"));
    }

    /**
     * Test 2 — PASSES before fixes.
     * When batch reaches maxSize, flush() is triggered automatically.
     * After auto-flush the batch is empty; a subsequent flush() returns an empty
     * list.
     */
    @Test
    void testMaxSizeTrigger_batchClearedAfterAutoFlush() {
        batcher.add("a");
        batcher.add("b");
        batcher.add("c"); // triggers auto-flush at size == maxSize
        List<String> afterAutoFlush = batcher.flush();
        assertEquals(0, afterAutoFlush.size(),
                "Batch should be empty after auto-flush triggered at maxSize");
    }

    /**
     * Test 3 — FAILS before fixes.
     *
     * Issue #4: close() on a closed batcher throws RuntimeException.
     * The correct exception is IllegalStateException.
     *
     * Before fix: RuntimeException is thrown → assertThrows(IllegalStateException)
     * FAILS.
     * After fix: IllegalStateException is thrown → test PASSES.
     */
    @Test
    void testAddAfterClose_throwsIllegalState() {
        batcher.close();
        assertThrows(IllegalStateException.class,
                () -> batcher.add("late message"),
                "Expected IllegalStateException when adding to a closed batcher");
    }

    /**
     * Test 4 — FAILS before fixes.
     *
     * Issue #3: add(null) silently succeeds instead of throwing
     * IllegalArgumentException.
     *
     * Before fix: no exception is thrown → assertThrows() FAILS.
     * After fix: IllegalArgumentException is thrown → test PASSES.
     */
    @Test
    void testNullMessage_throwsIllegalArgument() {
        assertThrows(IllegalArgumentException.class,
                () -> batcher.add(null),
                "Expected IllegalArgumentException when adding a null message");
    }

    // -------------------------------------------------------------------------
    // Bonus tests: add these as part of fixing Issue #6 (missing autoFlush test)
    // and Issue #1 (thread-safety) after you fix the implementation.
    // -------------------------------------------------------------------------

    /**
     * Bonus — add this test when fixing Issue #6 (autoFlush missing test).
     *
     * When autoFlush=true, close() must flush remaining messages.
     * When autoFlush=false, close() must NOT flush remaining messages.
     */
    // @Test
    // void testAutoFlush_closeFlushesWhenEnabled() { ... }

    /**
     * Bonus — add this test when fixing Issue #1 (thread safety).
     *
     * 10 threads each add 100 messages concurrently.
     * After all threads complete, flush() must return exactly 1000 messages
     * (no lost updates).
     *
     * Before fixing thread safety: this test fails or throws
     * ArrayIndexOutOfBoundsException due to unsynchronized ArrayList growth.
     */
    // @Test
    // void testConcurrentAdd_noLostUpdates() throws InterruptedException { ... }
}

Lab 02 — Design Tradeoffs: Comparing RateLimiter Implementations

Phase: 06 — Review Culture and Patch Workflow
Difficulty: ⭐⭐⭐⭐☆ | Estimated Time: 3–3.5 hours
Primary Artifact: Two RateLimiter implementations + benchmark comparison
Verify: mvn test passes all correctness tests; benchmark output printed to console

Scenario

A contributor proposed switching the existing TokenBucketRateLimiter to a SlidingWindowRateLimiter on the grounds that sliding window is "fairer":

The token bucket allows bursts — a client that was idle for 10 seconds can suddenly fire 10x the normal rate. The sliding window guarantees that no client ever exceeds the rate in any rolling time window, not just on average.

Your job is to:

1. Understand both implementations by reading the code and the tests
2. Run the benchmark and compare the results
3. Write a "+1 with conditions" or "-0" review that makes a concrete recommendation

Background: Token Bucket vs. Sliding Window

Token Bucket — conceptual model:

• Imagine a bucket that holds tokens. Tokens are added at a fixed rate (e.g., 10/sec).
• Each request consumes one token. If the bucket is empty, the request is rejected.
• If the requester was idle, tokens accumulate (up to the bucket capacity), allowing a burst.

Sliding Window Log — conceptual model:

• Keep a log of timestamps for every allowed request.
• When a new request arrives, count the requests in the last N seconds.
• If the count is below the limit, allow it (and add a timestamp). Otherwise, reject.
• No burst allowance — strictly "no more than N requests in any rolling window of size W".

Setup

cd phase-06-review-culture/lab-02-design-tradeoffs
mvn test && mvn compile exec:java -Dexec.mainClass=org.example.review.RateLimiterBenchmark

The test suite verifies both implementations are correct. The benchmark prints throughput comparison output.

Steps

Step 1 — Read both implementations

Before running anything: read TokenBucketRateLimiter.java and SlidingWindowRateLimiter.java. For each, answer:

1. What data structure stores the state?
2. What is the time complexity of tryAcquire()?
3. What is the space complexity relative to the rate limit and window size?
4. Under what traffic pattern does this algorithm perform worst?

Step 2 — Run the tests

mvn test

Both implementations should pass. If either fails, read the test to understand the invariant being tested.

Step 3 — Run the benchmark

mvn compile exec:java -Dexec.mainClass=org.example.review.RateLimiterBenchmark

Record the throughput numbers. Note the difference under:

• Steady-state load (requests at exactly the allowed rate)
• Burst load (10x rate for 1 second, then idle)
• High-load rejection (20x rate sustained)

Step 4 — Write the vote email

Write a 200–400 word message to dev@project.example.com in the following format:

Subject: [DISCUSS] PROJ-4512 — switching rate limiter from token bucket to sliding window

Hi all,

I reviewed both implementations. Here is my analysis:

Performance:
  - Token bucket tryAcquire(): O(1) time, O(1) space
  - Sliding window tryAcquire(): O(k) time (k = requests in window), O(k) space

Fairness:
  - Token bucket allows burst of [capacity] requests after idle period
  - Sliding window enforces hard ceiling in any rolling window

My recommendation: [+1 for / -0 for / -1 against] switching because:
...

Conditions / caveats:
...

— [Your name]

Fill in the real numbers from your benchmark. This is the skill that distinguishes a "I wrote it so it must be good" contributor from an Apache committer.

Verification

# All correctness tests pass
mvn test

# Benchmark output is produced
mvn compile exec:java -Dexec.mainClass=org.example.review.RateLimiterBenchmark 2>/dev/null | grep -E 'throughput|burst|Approach'

Source Files

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example.review</groupId>
    <artifactId>design-tradeoffs-lab</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <name>Review Culture Lab 02 — Design Tradeoffs</name>

    <properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <version>5.10.2</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.2.5</version>
            </plugin>
            <!--
        exec-maven-plugin: lets you run a main class directly with `mvn exec:java`.
        Used to run the RateLimiterBenchmark without packaging a fat jar.
        Command: mvn compile exec:java -Dexec.mainClass=org.example.review.RateLimiterBenchmark
      -->
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>exec-maven-plugin</artifactId>
                <version>3.1.0</version>
                <configuration>
                    <mainClass>org.example.review.RateLimiterBenchmark</mainClass>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

RateLimiter.java

package org.example.review;

/**
 * A rate limiter that allows up to a maximum number of requests in a rolling
 * time window.
 *
 * <p>
 * Two implementations are provided for comparison:
 * <ul>
 * <li>{@link TokenBucketRateLimiter} — allows bursting; O(1) time and
 * space</li>
 * <li>{@link SlidingWindowRateLimiter} — strict fairness; O(k) time and
 * space</li>
 * </ul>
 *
 * <p>
 * All implementations must be thread-safe.
 */
public interface RateLimiter {

    /**
     * Attempts to acquire permission to proceed. Returns immediately.
     *
     * @return {@code true} if the request is allowed; {@code false} if the rate
     *         limit
     *         has been exceeded and the request should be rejected
     */
    boolean tryAcquire();

    /**
     * Resets the rate limiter to its initial state.
     * Used in tests to provide a clean baseline between test cases.
     */
    void reset();
}

TokenBucketRateLimiter.java

package org.example.review;

/**
 * Token bucket rate limiter.
 *
 * <p>
 * <strong>Algorithm:</strong>
 * 
 * <pre>
 *   Capacity:      maximum tokens the bucket can hold (= maximum burst size)
 *   Refill rate:   tokens added per second
 *   Current state: how many tokens are currently in the bucket
 *
 *   On tryAcquire():
 *     1. Compute elapsed time since last refill
 *     2. Add (elapsed × refillRate) tokens to the bucket, capped at capacity
 *     3. If bucket has ≥ 1 token, consume 1 token and return true
 *     4. Otherwise return false
 * </pre>
 *
 * <p>
 * <strong>Complexity:</strong>
 * <ul>
 * <li>Time: O(1) per {@code tryAcquire()}</li>
 * <li>Space: O(1) — only stores current token count and last refill
 * timestamp</li>
 * </ul>
 *
 * <p>
 * <strong>Burst behavior:</strong> A client that is idle for {@code k} seconds
 * accumulates up to {@code min(k × refillRate, capacity)} tokens and can fire
 * that many requests in rapid succession. This is intentional — the algorithm
 * models "credit" for unused capacity.
 *
 * <p>
 * This implementation is thread-safe.
 */
public class TokenBucketRateLimiter implements RateLimiter {

    private final int capacity; // maximum tokens (= burst ceiling)
    private final double refillRate; // tokens per nanosecond
    private double tokens; // current token count (fractional to avoid drift)
    private long lastRefillNanos; // System.nanoTime() at last refill

    /**
     * @param capacity   maximum burst size (tokens that can accumulate while idle)
     * @param ratePerSec maximum sustained request rate (tokens added per second)
     */
    public TokenBucketRateLimiter(int capacity, int ratePerSec) {
        if (capacity <= 0) {
            throw new IllegalArgumentException("capacity must be > 0");
        }
        if (ratePerSec <= 0) {
            throw new IllegalArgumentException("ratePerSec must be > 0");
        }
        this.capacity = capacity;
        this.refillRate = (double) ratePerSec / 1_000_000_000L; // tokens per nanosecond
        this.tokens = capacity;
        this.lastRefillNanos = System.nanoTime();
    }

    @Override
    public synchronized boolean tryAcquire() {
        refill();
        if (tokens >= 1.0) {
            tokens -= 1.0;
            return true;
        }
        return false;
    }

    @Override
    public synchronized void reset() {
        tokens = capacity;
        lastRefillNanos = System.nanoTime();
    }

    /**
     * Adds tokens proportional to elapsed time since the last refill.
     * Called on every {@code tryAcquire()} to lazily update the bucket.
     */
    private void refill() {
        long now = System.nanoTime();
        double elapsed = now - lastRefillNanos;
        tokens = Math.min(capacity, tokens + elapsed * refillRate);
        lastRefillNanos = now;
    }
}

SlidingWindowRateLimiter.java

package org.example.review;

import java.util.ArrayDeque;
import java.util.Deque;

/**
 * Sliding window log rate limiter.
 *
 * <p>
 * <strong>Algorithm:</strong>
 * 
 * <pre>
 *   Window:        time duration in nanoseconds (e.g., 1 second)
 *   Limit:         maximum requests in any window of that duration
 *   Log:           timestamps of every allowed request in the current window
 *
 *   On tryAcquire():
 *     1. Remove all timestamps older than (now - window)
 *     2. If log.size() < limit, add 'now' to the log and return true
 *     3. Otherwise return false
 * </pre>
 *
 * <p>
 * <strong>Complexity:</strong>
 * <ul>
 * <li>Time: O(k) per {@code tryAcquire()}, where k = number of requests in the
 * current window (expiry sweep). In the worst case (high sustained load) k =
 * limit.</li>
 * <li>Space: O(k) — the log stores one timestamp per allowed request in the
 * window.
 * At maximum sustained load: O(limit) entries.</li>
 * </ul>
 *
 * <p>
 * <strong>Fairness guarantee:</strong> No more than {@code limit} requests are
 * allowed in any rolling window of {@code windowMillis} milliseconds — not just
 * on a fixed-clock boundary, but in any contiguous window. This prevents the
 * "reset burst" problem of fixed-window algorithms.
 *
 * <p>
 * This implementation is thread-safe.
 */
public class SlidingWindowRateLimiter implements RateLimiter {

    private final int limit; // max requests per window
    private final long windowNanos; // window size in nanoseconds
    private final Deque<Long> log; // timestamps of allowed requests

    /**
     * @param limit        maximum number of requests per window
     * @param windowMillis window duration in milliseconds
     */
    public SlidingWindowRateLimiter(int limit, long windowMillis) {
        if (limit <= 0) {
            throw new IllegalArgumentException("limit must be > 0");
        }
        if (windowMillis <= 0) {
            throw new IllegalArgumentException("windowMillis must be > 0");
        }
        this.limit = limit;
        this.windowNanos = windowMillis * 1_000_000L;
        this.log = new ArrayDeque<>(limit);
    }

    @Override
    public synchronized boolean tryAcquire() {
        long now = System.nanoTime();
        long windowStart = now - windowNanos;

        // Evict timestamps older than the window start (O(k) where k = expired entries)
        while (!log.isEmpty() && log.peekFirst() <= windowStart) {
            log.pollFirst();
        }

        if (log.size() < limit) {
            log.addLast(now);
            return true;
        }
        return false;
    }

    @Override
    public synchronized void reset() {
        log.clear();
    }
}

RateLimiterBenchmark.java

package org.example.review;

/**
 * Throughput comparison for {@link TokenBucketRateLimiter} vs
 * {@link SlidingWindowRateLimiter}.
 *
 * <p>
 * This is NOT a rigorous microbenchmark (no JVM warmup isolation, no JIT
 * stabilization).
 * It is a first-pass comparison that gives you order-of-magnitude numbers for
 * the
 * vote email. For production profiling, use JMH.
 *
 * <p>
 * Run with:
 * 
 * <pre>
 *   mvn compile exec:java -Dexec.mainClass=org.example.review.RateLimiterBenchmark
 * </pre>
 */
public class RateLimiterBenchmark {

    // Limit so high that tryAcquire() never rejects — we're measuring call
    // overhead,
    // not rejection logic. The real-world comparison matters when the limiter is
    // under constant accepted load.
    private static final int LARGE_LIMIT = 10_000_000;
    private static final int ITERATIONS = 5_000_000;
    private static final int WARMUP_ITER = 500_000;

    public static void main(String[] args) throws InterruptedException {
        System.out.println("=== RateLimiter Throughput Comparison ===\n");
        System.out.println("Iterations: " + ITERATIONS + " (+ " + WARMUP_ITER + " warmup)\n");

        // --- Token Bucket ---
        TokenBucketRateLimiter tokenBucket = new TokenBucketRateLimiter(LARGE_LIMIT, LARGE_LIMIT);
        long tbResult = measure(tokenBucket, "TokenBucketRateLimiter  ");

        // Small pause between runs to let the OS scheduler settle
        Thread.sleep(200);

        // --- Sliding Window ---
        SlidingWindowRateLimiter slidingWindow = new SlidingWindowRateLimiter(LARGE_LIMIT, 1000L);
        long swResult = measure(slidingWindow, "SlidingWindowRateLimiter");

        // --- Summary ---
        System.out.println("\n=== Summary ===");
        System.out.printf("Token bucket:    %,d ns per call%n", tbResult);
        System.out.printf("Sliding window:  %,d ns per call%n", swResult);
        double ratio = (double) swResult / tbResult;
        System.out.printf("Ratio (SW / TB): %.2fx%n", ratio);

        System.out.println("\nInterpretation:");
        if (ratio > 3.0) {
            System.out.println("Sliding window is >3x slower per call.");
            System.out.println("Under high-rate sustained load (the common case for a rate limiter),");
            System.out.println("this overhead is significant. Token bucket is the better choice for");
            System.out.println("throughput-sensitive paths.");
        } else if (ratio > 1.5) {
            System.out.println("Sliding window is moderately slower.");
            System.out.println("The tradeoff may be acceptable if strict per-window fairness is required.");
        } else {
            System.out.println("Performance is comparable. Choose based on fairness requirements.");
        }
    }

    /**
     * Runs warmup iterations (ignored), then measures the time for
     * {@code ITERATIONS}
     * calls to {@code tryAcquire()}.
     *
     * @return average nanoseconds per call
     */
    private static long measure(RateLimiter limiter, String label) {
        // Warmup: prime the JIT
        limiter.reset();
        for (int i = 0; i < WARMUP_ITER; i++) {
            limiter.tryAcquire();
        }

        // Measure
        limiter.reset();
        long start = System.nanoTime();
        for (int i = 0; i < ITERATIONS; i++) {
            limiter.tryAcquire();
        }
        long elapsed = System.nanoTime() - start;
        long nsPerCall = elapsed / ITERATIONS;

        System.out.printf("%-25s  total=%,d ms  per-call=%,d ns  throughput=%.1f M/sec%n",
                label,
                elapsed / 1_000_000,
                nsPerCall,
                ITERATIONS / (elapsed / 1e9) / 1_000_000.0);

        return nsPerCall;
    }
}

RateLimiterTest.java

package org.example.review;

import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.MethodSource;

import java.util.stream.Stream;

import static org.junit.jupiter.api.Assertions.*;

/**
 * Correctness tests for both {@link RateLimiter} implementations.
 *
 * Uses parameterized tests so the same invariants are verified against both
 * TokenBucketRateLimiter and SlidingWindowRateLimiter.
 */
class RateLimiterTest {

    /** Provides both implementations for parameterized tests. */
    static Stream<RateLimiter> implementations() {
        return Stream.of(
                new TokenBucketRateLimiter(5, 5), // 5 tokens, 5/sec
                new SlidingWindowRateLimiter(5, 1000L) // 5 per 1-second window
        );
    }

    /**
     * Within the burst/limit budget the limiter must allow all requests.
     */
    @ParameterizedTest
    @MethodSource("implementations")
    void testAllowsUpToLimit(RateLimiter limiter) {
        limiter.reset();
        for (int i = 0; i < 5; i++) {
            assertTrue(limiter.tryAcquire(),
                    "Expected tryAcquire() to return true for request " + (i + 1) + " of 5");
        }
    }

    /**
     * After the limit is exhausted, the next request must be rejected.
     */
    @ParameterizedTest
    @MethodSource("implementations")
    void testRejectsWhenLimitExhausted(RateLimiter limiter) {
        limiter.reset();
        for (int i = 0; i < 5; i++) {
            limiter.tryAcquire(); // consume all budget
        }
        assertFalse(limiter.tryAcquire(),
                "Expected tryAcquire() to return false after limit is exhausted");
    }

    /**
     * reset() must restore the limiter to its initial state so all requests
     * are allowed again.
     */
    @ParameterizedTest
    @MethodSource("implementations")
    void testResetRestoresBudget(RateLimiter limiter) {
        limiter.reset();
        for (int i = 0; i < 5; i++) {
            limiter.tryAcquire(); // exhaust budget
        }
        limiter.reset();
        assertTrue(limiter.tryAcquire(),
                "Expected tryAcquire() to return true after reset()");
    }

    /**
     * Both implementations must reject invalid construction arguments.
     */
    @Test
    void testTokenBucket_rejectsInvalidArgs() {
        assertThrows(IllegalArgumentException.class,
                () -> new TokenBucketRateLimiter(0, 5));
        assertThrows(IllegalArgumentException.class,
                () -> new TokenBucketRateLimiter(5, 0));
    }

    @Test
    void testSlidingWindow_rejectsInvalidArgs() {
        assertThrows(IllegalArgumentException.class,
                () -> new SlidingWindowRateLimiter(0, 1000L));
        assertThrows(IllegalArgumentException.class,
                () -> new SlidingWindowRateLimiter(5, 0L));
    }

    /**
     * Token bucket ONLY: after sleeping 1 second with a 5/sec refill rate,
     * the bucket should have refilled and allow another 5 requests.
     *
     * This test documents the burst behavior that sliding window does NOT exhibit.
     */
    @Test
    void testTokenBucket_refillsAfterIdle() throws InterruptedException {
        TokenBucketRateLimiter limiter = new TokenBucketRateLimiter(5, 5);
        limiter.reset();
        for (int i = 0; i < 5; i++) {
            limiter.tryAcquire(); // exhaust
        }
        Thread.sleep(1100); // wait > 1 second for refill
        assertTrue(limiter.tryAcquire(),
                "Expected token bucket to allow requests after idle refill period");
    }

    /**
     * Sliding window ONLY: after the window elapses, the count resets.
     *
     * This test documents the strict window enforcement that token bucket
     * allows to be bypassed via burst.
     */
    @Test
    void testSlidingWindow_allowsAfterWindowElapses() throws InterruptedException {
        SlidingWindowRateLimiter limiter = new SlidingWindowRateLimiter(5, 500L);
        limiter.reset();
        for (int i = 0; i < 5; i++) {
            limiter.tryAcquire(); // exhaust
        }
        assertFalse(limiter.tryAcquire(), "Should be rate-limited immediately after exhaustion");
        Thread.sleep(600); // wait for window to elapse
        assertTrue(limiter.tryAcquire(),
                "Expected sliding window to allow requests after window has elapsed");
    }
}

Lab 03 — Patch Archaeology: Retrospective Review

Phase: 06 — Review Culture and Patch Workflow
Difficulty: ⭐⭐⭐⭐☆ | Estimated Time: 2–2.5 hours
Primary Artifact: kafka-pr-17873.patch — a synthetic Kafka-style patch
Deliverable: kafka-pr-17873-review.md — your retrospective review with 4+ findings

Scenario

This patch was merged 6 months ago. A bug was filed last week:

KAFKA-21044 — RequestQueue.drainRequests() occasionally throws NullPointerException in production under high load. Stack trace points to line 89 of RequestQueue.java.

You're doing a post-mortem. Your job: read the patch that introduced drainRequests() and write the review that should have caught the bug before it merged.

What is Patch Archaeology?

When a bug reaches production:

1. Find the commit that introduced it: git log --all -S 'drainRequests' -- RequestQueue.java
2. Read the diff as if you were the reviewer at the time of the PR
3. Write the review you would have written — with the benefit of hindsight, but using only the information that was available when the PR was open (no knowledge of the production failure)

This is a standard Apache post-mortem skill. The goal is not to blame the author; it is to identify what review standards would have caught the issue, and whether to add a rule to the review checklist.

Setup

cd phase-06-review-culture/lab-03-patch-archaeology
cat kafka-pr-17873.patch

Read the patch file. The format is standard git diff:

• Lines starting with --- / +++ are the old/new file
• Lines starting with @@ are hunk headers showing line numbers
• Lines starting with - were removed (red in GitHub)
• Lines starting with + were added (green in GitHub)
• Lines with no prefix are context (unchanged)

Steps

Step 1 — Read the diff without the answer key

Read kafka-pr-17873.patch completely. For each concern you notice, write it down with:

• The line number (from the @@ header + offset)
• A one-sentence problem statement
• The severity label

There are 4 issues. Find as many as you can before expanding the answer.

Step 2 — Write the retrospective review

Create a file kafka-pr-17873-review.md in this directory with the following structure:

# Retrospective Review — KAFKA-PR-17873

**Reviewer**: [your name]
**Review date**: [today]
**Linked bug**: KAFKA-21044

## Summary

[2–3 sentence overview: what the patch does, and what slipped through]

## Findings

### Finding 1 — [Title]
**Severity**: BLOCKER
**Location**: `RequestQueue.java`, line [N]

[Problem statement]

**Evidence** (from the diff):
```java
[relevant lines]

Why it wasn't caught: [hypothesis — what review standard would have found this?]

Fix:

[suggested fix]

[Repeat for each finding]

Process Notes

[What review checklist item, if any, should be added to prevent this class of issue in future patches? 1–3 sentences.]

### Step 3 — Verify your review matches the actual bug

The production bug (KAFKA-21044) is the NPE on `requests.get(nodeId)` returning null when the node has no pending requests (Issue #2 in the answer key). Your review should have found this as a BLOCKER with the label "missing null check on return value of Map.get()".

If your review found it: good. If not: read the hunk again and note the exact line. Add it to your review.

---

## Key Insight

The bug was not subtle. `Map.get()` returning null when the key is absent is documented behavior. The reviewer should have asked: "What happens when `nodeId` is in the map but has an empty list? What happens when `nodeId` is not in the map at all?"

The answer to "What happens when nodeId is not in the map?" is: `requests.get(nodeId)` returns null, and `for (Request r : null)` throws `NullPointerException`. This is a one-question review that would have caught a production outage.

This is why Apache projects require: **every new method that takes user input must have a test for the empty/null/missing case**.

---

## Source Files

### `kafka-pr-17873.patch`

```diff
From 3f8a92b1c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9 Mon Sep 17 00:00:00 2001
From: alex.contributor <alex.contributor@example.com>
Date: Mon, 14 Oct 2024 11:22:03 +0000
Subject: [KAFKA-17873] RequestQueue: add drainRequests() for batch processing

Adds drainRequests(nodeId, callback) which processes all pending requests for
a given node in a single pass, invoking the callback for each. This replaces
the existing poll-in-a-loop pattern in NetworkClient, which had O(n) individual
synchronized calls.

Also fixes a minor issue where drainAll() was not returning the count of
drained requests, making it impossible for callers to distinguish "nothing
to drain" from "drained successfully."

Tested locally with NetworkClientTest — all existing tests pass.

Reviewers: b.reviewer@example.com
---
 .../kafka/clients/NetworkClient.java          |  18 ++-
 .../kafka/clients/RequestQueue.java           |  71 +++++++--
 .../kafka/clients/RequestQueueTest.java       |  29 +++-
 3 files changed, 104 insertions(+), 14 deletions(-)

diff --git a/clients/src/main/java/org/apache/kafka/clients/RequestQueue.java b/clients/src/main/java/org/apache/kafka/clients/RequestQueue.java
index a1b2c3d..f4e5c6d 100644
--- a/clients/src/main/java/org/apache/kafka/clients/RequestQueue.java
+++ b/clients/src/main/java/org/apache/kafka/clients/RequestQueue.java
@@ -1,7 +1,7 @@
 /*
  * Licensed to the Apache Software Foundation (ASF) under one or more
  * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
+ * this work for additional information regarding copyright ownership.
  * The ASF licenses this file to You under the Apache License, Version 2.0
  * (the "License"); you may not use this file except in compliance with
  * the License. You may obtain a copy of the License at
@@ -18,6 +18,7 @@ package org.apache.kafka.clients;
 
 import java.util.ArrayList;
 import java.util.HashMap;
+import java.util.Collections;
 import java.util.List;
 import java.util.Map;
 
@@ -42,6 +43,14 @@ public class RequestQueue {
     private final Map<String, List<Request>> requests = new HashMap<>();
     private final int maxInFlight;
 
+    /**
+     * Callback invoked for each request drained from the queue.
+     */
+    public interface DrainCallback {
+        /**
+         * @param nodeId  the node the request was queued for
+         * @param request the drained request
+         */
+        void onDrain(String nodeId, Request request);
+    }
+
     public RequestQueue(int maxInFlight) {
         if (maxInFlight <= 0)
             throw new IllegalArgumentException("maxInFlight must be > 0");
@@ -68,16 +77,52 @@ public class RequestQueue {
     }
 
     /**
-     * Drains all pending requests across all nodes, returning the total count.
+     * Drains all pending requests across all nodes.
+     *
+     * @return count of requests drained
      */
-    public synchronized void drainAll() {
+    public synchronized int drainAll() {
+        int count = 0;
         for (Map.Entry<String, List<Request>> entry : requests.entrySet()) {
-            entry.getValue().clear();
+            count += entry.getValue().size();
+            entry.getValue().clear();
         }
+        return count;
     }
 
-    /** Returns the number of pending requests for the given node. */
-    public synchronized int pendingCount(String nodeId) {
-        List<Request> list = requests.get(nodeId);
-        return list == null ? 0 : list.size();
+    /**
+     * Drains all pending requests for a specific node, invoking the callback
+     * for each request before removing it from the queue.
+     *
+     * @param nodeId   the target node identifier
+     * @param callback invoked for each drained request
+     * @return the number of requests drained
+     */
+    public synchronized int drainRequests(String nodeId, DrainCallback callback) {
+        // ISSUE 3 — API regression: this method is declared on the RequestQueue
+        // interface (not shown in this diff) as 'void drainRequests(...)'.
+        // Changing the return type to 'int' breaks binary compatibility with
+        // callers compiled against the interface. The interface was not updated
+        // in this patch. Any caller using the interface type will get:
+        //   AbstractMethodError: RequestQueue.drainRequests(...)V
+        // at runtime. (The 'V' void descriptor is no longer present.)
+        //
+        // A reviewer should have checked: does this method appear on an interface?
+        // The answer is: yes, IRequestQueue.java — not included in this diff.
+
+        List<Request> pending = new ArrayList<>();
+        drain(nodeId, pending, callback);
+        return pending.size();
     }
 
+    private void drain(String nodeId, List<Request> out, DrainCallback callback) {
+        // ISSUE 1 — nodeId is never null-checked.
+        // If nodeId is null, requests.get(null) is valid (HashMap allows null keys)
+        // but only if null was ever used as a key. In practice, nodeIds are always
+        // non-null strings. The missing guard means a null nodeId silently produces
+        // an empty result rather than a fast-fail IllegalArgumentException.
+        // This is a MAJOR — not a BLOCKER — because it produces a wrong result
+        // (silent empty drain) rather than an exception in all cases.
+
+        // ISSUE 2 — THE PRODUCTION BUG
+        // requests.get(nodeId) returns null when nodeId is not in the map
+        // (i.e., no requests have ever been queued for this node, or all were
+        // previously drained). Iterating over null with an enhanced for loop:
+        //
+        //   for (Request r : null) { ... }
+        //
+        // throws NullPointerException. The fix is a single null check:
+        //
+        //   List<Request> list = requests.get(nodeId);
+        //   if (list == null) return;
+        //
+        // This is the bug that caused KAFKA-21044. It was not caught because
+        // there was no test for drainRequests() on a node with no pending requests.
+        for (Request r : requests.get(nodeId)) {
+            callback.onDrain(nodeId, r);
+            out.add(r);
+        }
+        requests.remove(nodeId);
+    }
+
+    /** Returns the number of pending requests for the given node. */
+    public synchronized int pendingCount(String nodeId) {
+        List<Request> list = requests.get(nodeId);
+        return list == null ? 0 : list.size();
+    }
 }
diff --git a/clients/src/test/java/org/apache/kafka/clients/RequestQueueTest.java b/clients/src/test/java/org/apache/kafka/clients/RequestQueueTest.java
index b2c3d4e..c5d6e7f 100644
--- a/clients/src/test/java/org/apache/kafka/clients/RequestQueueTest.java
+++ b/clients/src/test/java/org/apache/kafka/clients/RequestQueueTest.java
@@ -52,6 +52,35 @@ public class RequestQueueTest {
         assertEquals(0, queue.pendingCount("node-1"));
     }
 
+    @Test
+    public void testDrainRequests_callsCallbackForEachRequest() {
+        RequestQueue queue = new RequestQueue(10);
+        queue.enqueue("node-1", new Request("req-1"));
+        queue.enqueue("node-1", new Request("req-2"));
+        queue.enqueue("node-1", new Request("req-3"));
+
+        List<String> drained = new ArrayList<>();
+        int count = queue.drainRequests("node-1",
+            (nodeId, req) -> drained.add(req.id()));
+
+        assertEquals(3, count);
+        assertEquals(3, drained.size());
+        assertEquals(0, queue.pendingCount("node-1"));
+    }
+
+    @Test
+    public void testDrainRequests_returnsZeroForNodeWithNoRequests() {
+        // ISSUE 4 — THIS TEST IS MISSING FROM THE PATCH.
+        //
+        // This comment is added here to show WHERE the missing test should have been.
+        // The PR added testDrainRequests_callsCallbackForEachRequest (above) but
+        // did NOT add a test for the case where the node has no pending requests.
+        //
+        // A reviewer who noticed "what happens when nodeId is not in the map?"
+        // would have requested this test. It would have failed immediately,
+        // revealing the NullPointerException before the patch was merged.
+        //
+        // Fix: add this test, then fix the drain() method to guard against
+        // requests.get(nodeId) returning null.
+        //
+        // RequestQueue queue = new RequestQueue(10);
+        // int count = queue.drainRequests("node-with-no-requests", (n, r) -> {});
+        // assertEquals(0, count);  // this line throws NPE before the fix
+    }
+
 }
diff --git a/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java b/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java
index d3e4f5a..e6f7g8h 100644
--- a/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java
+++ b/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java
@@ -187,12 +187,14 @@ public class NetworkClient {
 
     private void processCompletedReceives(String nodeId) {
-        // Old pattern: poll in a loop with individual synchronized calls
-        while (requestQueue.pendingCount(nodeId) > 0) {
-            Request r = requestQueue.poll(nodeId);
-            if (r != null) handleResponse(nodeId, r);
-        }
+        // New pattern: single drain call — fewer lock acquisitions
+        requestQueue.drainRequests(nodeId,
+            (nId, req) -> handleResponse(nId, req));
     }
 
 }

Phase 07 — Production Systems: Consensus, Partitioning, and Build Modularity

Track: PMC Apache Committer Engineer
Phase: 07 of 12
Prerequisites: Phases 01–06 (code quality, review culture, patch workflow)

Why This Phase

A senior Apache committer is expected to reason about three categories of production system problems:

1. Consensus — how distributed nodes agree on a leader when any node can fail at any time
2. Partitioning — how data is mapped to shards deterministically and fairly across a consumer group
3. Build modularity — how large codebases are split into independently-testable, deployable modules

Each lab in this phase gives you a working Java implementation of the core algorithm, not just a description of it.

Labs

Core Concepts

Deliverables

• Raft election simulation: mvn test passes all 5 tests in RaftElectionTest
• Kafka partitioning: mvn test passes all 8 tests in PartitionerTest
• Multi-module build: mvn verify from the parent dir builds and tests all 3 modules
• Written answers to at least 3 interview questions from the Hitchhiker's Guide

Interview Relevance

Warmup Guide — Production Distributed Systems

Zero-to-expert primer for Phase 07: consensus (Raft) from first principles, hash-based partitioning as used by Kafka, and multi-module build engineering — the systems substrate beneath every Apache data project.

Table of Contents

• Chapter 1: Why Distributed — and What It Costs
• Chapter 2: The Consensus Problem
• Chapter 3: Raft — Leader Election from Zero
• Chapter 4: Raft — Log Replication and the Commit Rule
• Chapter 5: Partitioning — Spreading Data with Hashing
• Chapter 6: Consumer Assignment and Rebalancing
• Chapter 7: Multi-Module Builds — Engineering the Repo Like a System
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Why Distributed — and What It Costs

From zero: one machine bounds you in capacity (disk/RAM), throughput (CPU/NIC), and availability (it will fail). Distribution buys past all three — and the price is that the network is now inside your system: messages are lost, delayed, reordered, and duplicated; clocks disagree; and — the deep one — you cannot distinguish a crashed node from a slow one (the FLP-flavored fact behind every timeout you'll ever tune).

Two design consequences run through everything in this phase:

• Redundancy requires agreement: copies that can diverge need a protocol to decide whose version is true → consensus (Ch. 2–4).
• Scale requires partitioning: data/work split across nodes needs a stable, balanced assignment function → hashing (Ch. 5–6).

Chapter 2: The Consensus Problem

The statement: N nodes must agree on a sequence of values (a replicated log) such that — despite crashes and partitions — agreed entries are never lost or reordered (safety), and the system keeps making progress whenever a majority is alive (liveness).

Why a majority (quorum)? Any two majorities of N intersect in ≥1 node. That single intersection node is the carrier of truth between configurations — it's why ⌈(N+1)/2⌉ acknowledgments make a decision durable, and why a 5-node cluster tolerates 2 failures. Memorize the intersection argument; every quorum system reduces to it.

Why it's hard: with asynchronous networks, any protocol can be stalled by adversarial timing (FLP impossibility) — so real systems buy liveness with randomized timeouts and buy safety unconditionally. Paxos solved it first and is notoriously hard to implement; Raft (2014) re-derived equivalent guarantees optimized for understandability — which is why etcd, Kafka's KRaft, and your Lab 01 all speak Raft.

Chapter 3: Raft — Leader Election from Zero

Raft decomposes consensus into leader election + log replication, with a strong leader through whom all writes flow.

The moving parts: every node is follower / candidate / leader; time divides into terms (monotonic integers — Raft's logical clock). Each node persists currentTerm, votedFor, and its log.

The election algorithm (exactly what Lab 01 implements and tests):

1. Followers expect heartbeats from a leader. No heartbeat within a randomized election timeout (e.g. 150–300 ms) → become candidate: increment term, vote for self, send RequestVote(term, lastLogIndex, lastLogTerm) to all.
2. A node grants at most one vote per term (persisted — survives crashes), and only to candidates whose log is at least as up-to-date (compare last term, then last index). This election restriction is what guarantees a new leader already holds every committed entry.
3. Majority of votes → leader; begin heartbeats. Tie/split vote → timeout expires → next term, retry. Randomized timeouts make repeated splits exponentially unlikely.
4. Any message bearing a higher term instantly demotes a leader/candidate to follower — stale leaders can exist briefly but can't do anything: replication requires follower acks, and followers reject lower terms.

Why no two leaders per term: one-vote-per-term + majority needed → two leaders in one term would require two disjoint majorities — impossible by the intersection argument. Two leaders in different terms: the older one's appends are rejected; it steps down on first contact with the higher term.

Chapter 4: Raft — Log Replication and the Commit Rule

(Lab 01 focuses on election; replication context so the design makes sense.)

• The leader appends client commands to its log and sends AppendEntries(prevLogIndex, prevLogTerm, entries, leaderCommit). Followers reject if their log doesn't match at prevLogIndex/Term — the consistency check that forces divergent follower logs to back up and be overwritten to match the leader's.
• An entry is committed once a majority holds it — but only for entries of the leader's current term (the Figure-8 subtlety: an old-term entry on a majority can still be overwritten; committing a current-term entry on top commits everything beneath it). This is the most-asked deep Raft question; have the scenario ready.
• KRaft (Kafka), etcd, and TiKV are production Rafts; reading any of their vote-handling code after this lab is the recommended follow-up.

Chapter 5: Partitioning — Spreading Data with Hashing

The problem: assign keys (or records) to N partitions, balanced and deterministic.

• The baseline (Kafka's default, Lab 02): partition = murmur2(key) % N (sign-bit masked). Hash quality matters: a weak hash clusters real-world keys (sequential IDs, common prefixes) onto hot partitions, capping throughput at one broker. Murmur2 is fast, non-cryptographic, and statistically uniform — Lab 02 verifies uniformity with a chi-square test over realistic key distributions, the statistically honest way to evaluate a partitioner (eyeballing histograms lies).
• Keyless records: round-robin balances perfectly but kills batching; Kafka's sticky partitioner fills one partition's batch then moves on — same long-run balance, far better throughput. Batching-vs-balance is the design tension to remember.
• The modulo trap: change N and hash % N reassigns ~everything — key→partition affinity (ordering! local state!) breaks. Consistent hashing (ring) or rendezvous hashing moves only ~1/N of keys — that's why caches and Cassandra-style stores use them. Kafka deliberately doesn't: partition count is meant to be fixed per topic, and ordering-per-key is the contract — increasing partitions on a keyed topic is a breaking change you must plan like a migration (Phase 11 thinking).
• Skew is the practical killer: one celebrity key = one hot partition no algorithm fixes. Mitigations: composite keys (key+salt with consumer-side merge), local aggregation before the shuffle (Spark's map-side combine), or two-phase aggregation.

Chapter 6: Consumer Assignment and Rebalancing

The other half of partitioning: assigning partitions to consumers in a group.

• Range (per-topic contiguous blocks): aligns co-partitioned topics (needed for joins) but skews when consumers don't divide partitions evenly across many topics.
• Round-robin: even spread, breaks co-partition alignment.
• Sticky / cooperative-sticky (modern default): minimize movement on rebalance — every moved partition aborts in-flight work and cold-starts state. Cooperative rebalancing migrates incrementally instead of stop-the-world.
• The general principle (it recurs from Kafka to YARN to storage shards): assignment quality = balance × stability × locality, and the three trade against each other. Lab 02's RangeAssignor implementation makes the skew case concrete.

Chapter 7: Multi-Module Builds — Engineering the Repo Like a System

Why a build lab sits in a distributed-systems phase: Apache projects are multi-module systems, and their build graph is the first distributed system you touch (Phase 02 navigated it; here you construct one).

• Module = dependency-graph node: queue-api (interfaces, no deps) ← queue-core (implementation) ← queue-app (wiring + integration tests). The discipline: api modules depend on nothing, implementations depend on api, applications depend on both — cycles are a design smell Maven simply refuses.
• The reactor computes topological build order; -pl module -am (with upstream) and -amd (with downstream) are the daily flags. Version alignment via the parent POM's dependencyManagement — one place where versions live, modules inherit.
• Why split at all: independent compilation/test cycles, enforceable layering (api can't accidentally reach into impl), separate artifact publication (clients depend on queue-api only — a compatibility boundary, Phase 04's surface made physical), and parallel CI.
• CI shape (the lab's ci.yml): build once, test per module, cache the local repo keyed on the POMs — the 80% of build-engineering value in 20 lines.

Lab Walkthrough Guidance

Order: Lab 01 (Raft) → Lab 02 (Kafka partitioning) → Lab 03 (multi-module).

• Lab 01: read RaftNode's state transitions against Chapter 3's algorithm step by step; run the tests and make them fail informatively — comment out the one-vote-per-term persistence and watch which test catches the double-vote; do the same for the up-to-date check. Negative experiments are how the safety argument becomes yours. Extension: add the AppendEntries consistency check.
• Lab 02: verify murmur2 against known vectors; run the chi-square uniformity test on (a) random keys, (b) sequential IDs, (c) prefixed keys — compare against a naive hashCode() partitioner and watch it fail (b). Then implement RangeAssignor and construct the skew case from Chapter 6.
• Lab 03: build the three-module project; deliberately introduce an api→core dependency and read Maven's cycle error; move a version into the parent's dependencyManagement; run -pl queue-core -am and explain what rebuilt and why.

Success Criteria

You are ready for Phase 08 when you can, from memory:

1. State the quorum-intersection argument and apply it (why 5 nodes tolerate 2 failures).
2. Walk Raft's election including both safety mechanisms (one vote/term persisted; up-to-date restriction) and the role of randomized timeouts.
3. Explain why a leader can't commit old-term entries by counting replicas (Figure 8).
4. Defend murmur2-mod-N, name the modulo trap, and say when consistent hashing is and isn't the answer.
5. Compare range/round-robin/sticky assignment on balance × stability × locality.
6. Draw the api/core/app dependency discipline and what each boundary buys.

Interview Q&A

Q: Design leader election without Raft/ZK — what do you need at minimum? Re-derive the essentials: terms (a monotonic tiebreaker), persisted single vote per term (else double-voting under crashes), majority quorum (intersection = uniqueness), randomized timeouts (liveness under symmetry), and a rule that winners hold all committed state (else data loss on election) — at which point you've re-invented Raft's election, which is the answer: "the minimum is Raft's election, and here's why each piece is load-bearing."

Q: A Kafka topic shows one partition at 10× the traffic. Walk through it. First distinguish key skew from partitioner bug: sample keys per partition. Celebrity key → composite key (salt + consumer merge) or local pre-aggregation; bad hash (custom partitioner, hashCode() on sequential IDs) → fix the hash, chi-square it; recently increased partition count → old keys' affinity broke and the imbalance is transitional. Never "add partitions" reflexively — on a keyed topic that's a breaking change to ordering and state (the modulo trap).

Q: Why does Raft persist votedFor to disk before responding? Crash-recovery safety: grant a vote, crash, restart without the record → grant again in the same term → two leaders become possible, violating the intersection argument. The fsync-before-reply on currentTerm/votedFor/log entries is the price of safety under crashes; batching/group-commit is the performance answer, never skipping the write.

References

• Ongaro & Ousterhout, In Search of an Understandable Consensus Algorithm (Raft) — raft.github.io/raft.pdf; read §5 twice, Figure 8 until it's obvious
• The Raft visualization — watch before reading the paper
• Raft implementations list — read etcd's or KRaft's vote handling after Lab 01
• Kleppmann, Designing Data-Intensive Applications, ch. 6 (partitioning), ch. 9 (consistency/consensus)
• Kafka KIP-480: Sticky Partitioner and KIP-429: Incremental Cooperative Rebalancing
• Karger et al., Consistent Hashing and Random Trees (1997) — the ring's origin
• Maven: Guide to Working with Multiple Modules

🛸 Hitchhiker's Guide — Production Systems: Consensus, Partitioning, Builds

Read this if: You can write Java but want to explain why distributed systems make the design choices they do, and how those choices appear in Apache codebases.

0. The 30-second mental model

                    ┌─────────────────────────────────────────────┐
                    │              PRODUCTION SYSTEM               │
                    └─────────────────────────────────────────────┘
                            │               │              │
                     ┌──────▼──────┐ ┌──────▼──────┐ ┌────▼────────┐
                     │  CONSENSUS  │ │ PARTITIONING│ │    BUILDS   │
                     │             │ │             │ │             │
                     │ Who is in   │ │ Which shard │ │ Who owns    │
                     │ charge?     │ │ gets this   │ │ this code?  │
                     │             │ │ key?        │ │             │
                     │ Raft: terms │ │ murmur2 →   │ │ Maven multi-│
                     │ + majority  │ │ partition   │ │ module POM  │
                     │ vote        │ │ index       │ │             │
                     └─────────────┘ └─────────────┘ └─────────────┘

These three concerns appear together in every large-scale Apache project:

• ZooKeeper / Kafka controller / HDFS NameNode all use consensus to elect a leader
• Kafka producers use murmur2 to deterministically route records to partitions
• Apache Hadoop / Flink / Kafka all use multi-module Maven builds with shared BOMs

1. Raft Leader Election

1.1 The Problem Raft Solves

A cluster of N nodes must agree on which single node is the leader at any time. The leader is authoritative for all writes. If the leader fails, the cluster must elect a new one without human intervention — and without electing two leaders simultaneously (split brain).

1.2 Raft's Three Node States

      ┌──────────────────────────────────────────────┐
      │                                              │
      │  FOLLOWER  ──election timeout──▶  CANDIDATE  │
      │      ▲                               │       │
      │      │◀──── receives heartbeat ──────┘       │
      │      │                               │       │
      │      └───────── wins majority ──▶  LEADER    │
      │                                     │       │
      │  (all nodes start as FOLLOWER)      │       │
      └─────────────────────────────────────────────┘

1.3 Terms: Raft's Logical Clock

A term is a monotonically increasing integer. Every election attempt increments the term. Terms solve the "stale leader" problem:

Term 1: node-1 elected leader
Term 2: network partition; node-2 starts election, node-3 votes for it
         → node-2 is leader at term 2
Term 1 again?: node-1 comes back, still thinks it's leader at term 1
         → node-1 sees term=2 in any RPC → immediately steps down

Rule: If a node sees a term > its own, it must immediately update its term and revert to FOLLOWER. This guarantees there is never more than one leader per term.

1.4 Majority Vote

Why floor(N/2) + 1? Because any two majorities must overlap by at least one node. That overlapping node prevents two leaders from being elected in the same term.

5-node cluster:  majority = 3
Two candidates can each get at most:
  Candidate A wins nodes {1, 2, 3}   ← 3 = majority
  Candidate B wants nodes {1, 4, 5}  ← node 1 already voted for A!
  → Candidate B gets at most {4, 5} = 2 < 3 = majority → cannot win

1.5 Key Interview Points

• "Why random election timeout?" → Reduces probability of simultaneous elections (split vote)
• "What if two candidates split the vote?" → Both timeout, increment term, try again with new random timeout
• "What happens to a partitioned leader's writes?" → They are lost if not replicated to a majority before the partition. The new leader has the most up-to-date log by virtue of winning the vote.

2. Kafka Partitioning Algorithms

2.1 Why Deterministic Partitioning Matters

Kafka guarantees ordering only within a partition. If you send order-123 events and they must be processed in order, they must all go to the same partition. The producer must map the key "order-123" to the same partition every time, on every producer instance, across restarts.

This requires a deterministic, key-based partition function.

2.2 murmur2 Hash

Kafka's DefaultPartitioner uses murmur2, a non-cryptographic hash designed for hash-table lookups. It has two properties Java's String.hashCode() does not:

The algorithm processes 4 bytes at a time with a multiply-and-XOR mixing step:

// Simplified inner loop
int k = read4Bytes(data, i);
k *= M;
k ^= k >>> 24;   // ← avalanche: every bit of input affects every bit of output
k *= M;
h *= M;
h ^= k;

The full implementation is in MurmurPartitioner.java.

2.3 Round-Robin Partitioning

For messages with no key (e.g., log events where ordering doesn't matter), round-robin distributes load evenly across partitions using a single atomic counter:

(counter.getAndIncrement() & Integer.MAX_VALUE) % numPartitions

The & Integer.MAX_VALUE clears the sign bit to prevent negative partition indices when the counter wraps past Integer.MAX_VALUE.

2.4 Range Assignment (Consumer Group Protocol)

When a consumer group subscribes to a topic, Kafka must assign partitions to consumers. The Range strategy (Kafka's default) works as follows:

N = 5 partitions, C = 3 consumers, sorted: [c0, c1, c2]

rangeSize = 5 / 3 = 1   (integer division)
extras    = 5 % 3 = 2   (first 2 consumers get one extra)

Consumer 0: start=0, count=1+1=2 → partitions [0, 1]
Consumer 1: start=2, count=1+1=2 → partitions [2, 3]
Consumer 2: start=4, count=1+0=1 → partitions [4]

Why Range? It minimizes the number of partitions moved during a rebalance when consumers are added/removed one at a time (common pattern: rolling restart).

Tradeoff: With many topics, the first consumer tends to get more partitions (the "first consumer bias"). The Sticky and CooperativeSticky assignors solve this at the cost of more complex rebalance logic.

3. Multi-module Maven Builds

3.1 The Problem: Dependency Version Drift

Imagine 10 Maven modules all depending on jackson-databind. Over time:

• Module A declares version 2.14.2
• Module B declares version 2.15.0
• Module C forgets to declare it and inherits a transitive version 2.13.1

The resulting classpath is unpredictable. Integration tests pass on the developer's machine (which has version X in ~/.m2/repository) but fail in CI.

Solution: A parent POM with dependencyManagement pins all versions in one place. Child modules declare <groupId>/<artifactId> without a version, and Maven resolves the version from the parent's dependencyManagement section.

3.2 POM Inheritance Hierarchy

queue-parent/pom.xml     ← packaging=pom, defines <modules> and <dependencyManagement>
├── queue-api/pom.xml    ← <parent> → queue-parent; packaging=jar; no runtime deps
├── queue-core/pom.xml   ← <parent> → queue-parent; depends on queue-api
└── queue-app/pom.xml    ← <parent> → queue-parent; depends on queue-core; has exec jar config

3.3 Reactor Build Order

Running mvn verify from the parent triggers a reactor build:

1. Maven reads all pom.xml files declared in <modules>
2. Builds a dependency graph
3. Builds modules in topological order: api → core → app
4. If any module fails, the reactor stops

cd lab-03-multimodule-maven
mvn verify

# Output:
# [INFO] Reactor Build Order:
# [INFO]   queue-parent [pom]
# [INFO]   queue-api [jar]
# [INFO]   queue-core [jar]
# [INFO]   queue-app [jar]

3.4 BOM vs Parent POM

Apache projects typically publish a BOM artifact (e.g., kafka-bom, hadoop-bom) so downstream users can import consistent versions without inheriting the Apache project's parent POM configuration.

4. What Can Go Wrong in Production

5. Interview Questions — Concepts You Must Explain Out Loud

6. References

• Raft Paper — In Search of an Understandable Consensus Algorithm (Ongaro & Ousterhout, 2014)
• Apache Kafka Protocol Guide — partition assignment RPC specs
• Kafka DefaultPartitioner source
• Apache Maven — Introduction to the POM
• Maven: Guide to Working with Multiple Modules

Lab 01 — Raft Leader Election Simulation

Phase: 07 — Production Systems
Difficulty: ⭐⭐⭐⭐☆ | Estimated Time: 2–2.5 hours
Artifacts: RaftNode.java, RaftCluster.java, RaftElectionTest.java
Command: mvn test

Background

Raft is the consensus algorithm used by:

• Apache ZooKeeper (leader election for coordination)
• Kafka's KRaft mode (replaces ZooKeeper as of Kafka 3.3)
• etcd (Kubernetes control plane)
• CockroachDB, TiKV, and most modern distributed databases

This lab implements the leader election portion of Raft — the part that decides which node is in charge. Log replication (how writes are propagated) is out of scope.

Setup

cd phase-07-production-systems/lab-01-raft-election
mvn test

All 5 tests should pass.

Step 1 — Understand the State Machine

Read RaftNode.java and trace through what happens when a FOLLOWER calls startElection():

1. What fields change?
2. What is the new value of currentTerm?
3. How many votes does the node have before asking any peers?

Step 2 — Trace a 3-Node Election

Read RaftCluster.runElection() and trace through a 3-node cluster where node-1 starts the election. Draw the sequence of method calls and state transitions:

node-1: startElection()       → state=CANDIDATE, term=1, votes=1
node-1 → node-2: requestVote(term=1, candidateId="node-1") → ?
node-1 → node-3: requestVote(term=1, candidateId="node-1") → ?
...

What does node-2 do if it is already at term=2 from a previous aborted election?

Step 3 — Run the Tests

mvn test

Expected output:

[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS

Examine each test and make sure you can explain why it passes, not just that it does.

Step 4 — Extend: Make It Fail, Then Fix It

Add a 6th test that creates a 3-node cluster and verifies that a node with a stale term (lower than the cluster's current term) cannot win an election. Your test must:

1. Advance the cluster term to 3 (simulate 3 elections)
2. Create a new "stale" candidate at term 1
3. Assert it does NOT become leader

Step 5 — Interview Prep

Write a 3-sentence answer to: "How does Raft prevent two leaders from being elected in the same term?"

Source Files

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example.raft</groupId>
    <artifactId>raft-election-lab</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <version>5.10.2</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.2.5</version>
            </plugin>
        </plugins>
    </build>
</project>

NodeState.java

package org.example.raft;

/**
 * The three states a Raft node can be in at any point in time.
 *
 * <p>
 * State transitions:
 * 
 * <pre>
 *   FOLLOWER  ──(election timeout)──▶  CANDIDATE
 *   CANDIDATE ──(majority votes)─────▶  LEADER
 *   CANDIDATE ──(sees higher term)───▶  FOLLOWER
 *   LEADER    ──(sees higher term)───▶  FOLLOWER
 * </pre>
 *
 * <p>
 * All nodes start as FOLLOWER. There is no direct path from FOLLOWER to LEADER
 * without going through CANDIDATE — a node must win a vote, not just claim
 * leadership.
 */
public enum NodeState {
    FOLLOWER,
    CANDIDATE,
    LEADER
}

VoteResponse.java

package org.example.raft;

/**
 * Response to a {@code RequestVote} RPC (Raft Section 5.2).
 *
 * <p>
 * The receiver always includes its {@code currentTerm} in the response.
 * If the sender's term is stale, it sees a higher term in the response
 * and immediately steps down to FOLLOWER.
 */
public final class VoteResponse {

    private final int term; // receiver's currentTerm
    private final boolean voteGranted;

    public VoteResponse(int term, boolean voteGranted) {
        this.term = term;
        this.voteGranted = voteGranted;
    }

    /** The receiver's current term at the time of the response. */
    public int getTerm() {
        return term;
    }

    /** Whether the receiver granted its vote to the candidate. */
    public boolean isVoteGranted() {
        return voteGranted;
    }

    @Override
    public String toString() {
        return "VoteResponse{term=" + term + ", voteGranted=" + voteGranted + "}";
    }
}

RaftNode.java

package org.example.raft;

/**
 * A single node in a Raft cluster.
 *
 * <p>
 * This is a synchronous, single-JVM simulation. In a real distributed system,
 * these methods would be invoked via RPC (e.g., gRPC). Here, they are called
 * directly by {@link RaftCluster} to simulate the message passing.
 *
 * <p>
 * All public state-mutating methods are synchronized to keep the simulation
 * thread-safe (useful if you extend this lab to run elections on background
 * threads).
 *
 * <h3>Raft invariants maintained by this class:</h3>
 * <ul>
 * <li>A node grants at most one vote per term (tracked by
 * {@code votedFor})</li>
 * <li>A node that sees a higher term immediately steps down to FOLLOWER</li>
 * <li>A node only becomes LEADER if it receives votes from a strict
 * majority</li>
 * </ul>
 */
public class RaftNode {

    private final String id;
    private NodeState state;
    private int currentTerm;
    private String votedFor; // node id we voted for in currentTerm; null if we haven't voted
    private String leaderId; // id of the current leader as learned from heartbeats
    private int votesReceived; // only meaningful when state == CANDIDATE

    public RaftNode(String id) {
        this.id = id;
        this.state = NodeState.FOLLOWER;
        this.currentTerm = 0;
        this.votedFor = null;
        this.leaderId = null;
        this.votesReceived = 0;
    }

    // -------------------------------------------------------------------------
    // RPC handlers (called by RaftCluster to simulate message delivery)
    // -------------------------------------------------------------------------

    /**
     * Handles a {@code RequestVote} RPC from a candidate.
     *
     * <p>
     * Raft Section 5.2 grant conditions:
     * <ol>
     * <li>Reject if {@code candidateTerm < currentTerm} (candidate is behind)</li>
     * <li>If {@code candidateTerm > currentTerm}: update term, step down to
     * FOLLOWER,
     * clear {@code votedFor} — we may now vote for this candidate</li>
     * <li>Grant vote if we haven't voted this term OR already voted for this
     * candidate</li>
     * </ol>
     *
     * @param candidateTerm the term the candidate is running in
     * @param candidateId   the candidate requesting the vote
     * @return response including this node's currentTerm and whether the vote was
     *         granted
     */
    public synchronized VoteResponse requestVote(int candidateTerm, String candidateId) {
        if (candidateTerm < currentTerm) {
            // The candidate is behind — tell it our term so it steps down
            return new VoteResponse(currentTerm, false);
        }

        if (candidateTerm > currentTerm) {
            // Higher term: step down regardless of our current state
            currentTerm = candidateTerm;
            state = NodeState.FOLLOWER;
            votedFor = null;
        }

        // Grant the vote if we haven't committed to anyone else this term
        boolean grant = (votedFor == null || votedFor.equals(candidateId));
        if (grant) {
            votedFor = candidateId;
        }
        return new VoteResponse(currentTerm, grant);
    }

    /**
     * Handles an {@code AppendEntries} RPC from a leader (used here as a
     * heartbeat).
     *
     * <p>
     * In full Raft this RPC also carries log entries. Here we only model the
     * heartbeat behavior: if the leader's term is valid, accept it and reset the
     * election timer (simulated by transitioning/remaining FOLLOWER).
     *
     * @param leaderTerm the leader's current term
     * @param leaderId   the leader's node id
     * @return {@code true} if we accept this leader, {@code false} if term is stale
     */
    public synchronized boolean appendEntries(int leaderTerm, String leaderId) {
        if (leaderTerm < currentTerm) {
            return false; // stale leader — reject
        }
        if (leaderTerm > currentTerm) {
            currentTerm = leaderTerm;
            votedFor = null;
        }
        // Accept the leader: transition (back) to FOLLOWER and record who the leader is
        state = NodeState.FOLLOWER;
        this.leaderId = leaderId;
        return true;
    }

    // -------------------------------------------------------------------------
    // Election actions (called on the candidate node itself)
    // -------------------------------------------------------------------------

    /**
     * Starts an election: increments the term, transitions to CANDIDATE, and
     * casts a self-vote.
     *
     * <p>
     * In real Raft this is triggered when the election timeout fires.
     * Here it is called explicitly by {@link RaftCluster#runElection()}.
     */
    public synchronized void startElection() {
        currentTerm++;
        state = NodeState.CANDIDATE;
        votedFor = id; // every candidate votes for itself
        votesReceived = 1; // the self-vote
        leaderId = null;
    }

    /**
     * Records one received vote. If the candidate now holds a majority of votes
     * in a cluster of {@code clusterSize} nodes, it transitions to LEADER.
     *
     * @param clusterSize total number of nodes in the cluster (used to compute
     *                    majority)
     */
    public synchronized void countVote(int clusterSize) {
        if (state != NodeState.CANDIDATE) {
            return; // already elected or stepped down — ignore late votes
        }
        votesReceived++;
        int majority = (clusterSize / 2) + 1;
        if (votesReceived >= majority) {
            state = NodeState.LEADER;
        }
    }

    // -------------------------------------------------------------------------
    // Getters
    // -------------------------------------------------------------------------

    public String getId() {
        return id;
    }

    public synchronized NodeState getState() {
        return state;
    }

    public synchronized int getCurrentTerm() {
        return currentTerm;
    }

    public synchronized String getVotedFor() {
        return votedFor;
    }

    /** The leader id as last communicated via a heartbeat; null if unknown. */
    public synchronized String getLeaderId() {
        return leaderId;
    }

    @Override
    public String toString() {
        return "RaftNode{id='" + id + "', state=" + state
                + ", term=" + currentTerm + ", votedFor='" + votedFor + "'}";
    }
}

RaftCluster.java

package org.example.raft;

import java.util.Arrays;
import java.util.List;
import java.util.Optional;

/**
 * Coordinates leader election across a fixed set of {@link RaftNode}s.
 *
 * <p>
 * This is a synchronous, deterministic simulation of the Raft election
 * protocol.
 * In a real cluster, the timing is driven by election timeouts (random timers
 * per node).
 * Here, elections are triggered explicitly so tests are deterministic.
 *
 * <p>
 * The cluster size is fixed at construction time. Node failures are simulated
 * by excluding specific nodes from an election via
 * {@link #runElectionExcluding(RaftNode)}.
 */
public class RaftCluster {

    private final List<RaftNode> nodes;

    public RaftCluster(RaftNode... nodes) {
        if (nodes == null || nodes.length < 3) {
            throw new IllegalArgumentException(
                    "Raft requires at least 3 nodes to tolerate 1 failure");
        }
        this.nodes = Arrays.asList(nodes);
    }

    /**
     * Simulates one complete election cycle:
     * <ol>
     * <li>The first FOLLOWER (or CANDIDATE) starts an election</li>
     * <li>It broadcasts {@code RequestVote} to all peers</li>
     * <li>If a majority votes for it, it transitions to LEADER</li>
     * <li>The new leader sends heartbeats so all followers learn the leader id</li>
     * </ol>
     *
     * @return the elected leader, or empty if no candidate won a majority (split
     *         vote)
     */
    public Optional<RaftNode> runElection() {
        RaftNode candidate = findCandidate(null);
        if (candidate == null) {
            return getLeader(); // already have a leader
        }

        candidate.startElection();
        broadcastRequestVote(candidate, null);

        if (candidate.getState() == NodeState.LEADER) {
            sendHeartbeats(candidate, null);
        }

        return getLeader();
    }

    /**
     * Simulates a network partition: runs an election excluding the given node.
     *
     * <p>
     * The partitioned node receives no RPCs and sends none. The remaining
     * nodes form a sub-cluster and, if they represent a majority, elect a new
     * leader.
     *
     * <p>
     * With a 3-node cluster and 1 partitioned node, the 2 remaining nodes
     * still constitute a majority (⌊3/2⌋ + 1 = 2).
     *
     * @param partitioned the node to exclude from the election
     * @return the new leader among the non-partitioned nodes, or empty if none
     */
    public Optional<RaftNode> runElectionExcluding(RaftNode partitioned) {
        RaftNode candidate = findCandidate(partitioned);
        if (candidate == null) {
            return Optional.empty();
        }

        candidate.startElection();
        broadcastRequestVote(candidate, partitioned);

        if (candidate.getState() == NodeState.LEADER) {
            sendHeartbeats(candidate, partitioned);
        }

        return nodes.stream()
                .filter(n -> n != partitioned && n.getState() == NodeState.LEADER)
                .findFirst();
    }

    /**
     * Sends heartbeats from {@code leader} to all other (non-excluded) nodes.
     * In real Raft, this is sent periodically; here it is called once after
     * election.
     */
    public void sendHeartbeats(RaftNode leader, RaftNode exclude) {
        for (RaftNode node : nodes) {
            if (node == leader || node == exclude)
                continue;
            node.appendEntries(leader.getCurrentTerm(), leader.getId());
        }
    }

    /** Returns the current leader, if any. */
    public Optional<RaftNode> getLeader() {
        return nodes.stream()
                .filter(n -> n.getState() == NodeState.LEADER)
                .findFirst();
    }

    /**
     * Returns the number of nodes currently in the LEADER state. Must be 0 or 1.
     */
    public long countLeaders() {
        return nodes.stream()
                .filter(n -> n.getState() == NodeState.LEADER)
                .count();
    }

    public List<RaftNode> getNodes() {
        return nodes;
    }

    // -------------------------------------------------------------------------
    // Private helpers
    // -------------------------------------------------------------------------

    /**
     * Finds the first node eligible to start an election (FOLLOWER or CANDIDATE,
     * and not the excluded node).
     */
    private RaftNode findCandidate(RaftNode exclude) {
        return nodes.stream()
                .filter(n -> n != exclude)
                .filter(n -> n.getState() == NodeState.FOLLOWER
                        || n.getState() == NodeState.CANDIDATE)
                .findFirst()
                .orElse(null);
    }

    /**
     * Broadcasts {@code RequestVote} from {@code candidate} to all peers except
     * itself
     * and the excluded node. Calls {@link RaftNode#countVote(int)} on the candidate
     * for each granted vote, using the full cluster size for the majority
     * calculation.
     */
    private void broadcastRequestVote(RaftNode candidate, RaftNode exclude) {
        int term = candidate.getCurrentTerm();
        for (RaftNode peer : nodes) {
            if (peer == candidate || peer == exclude)
                continue;
            VoteResponse response = peer.requestVote(term, candidate.getId());
            if (response.isVoteGranted()) {
                // Use full cluster size — majority is over all nodes, not just available ones
                candidate.countVote(nodes.size());
            }
        }
    }
}

RaftElectionTest.java

package org.example.raft;

import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;

import java.util.Optional;

import static org.junit.jupiter.api.Assertions.*;

class RaftElectionTest {

    private RaftNode node1;
    private RaftNode node2;
    private RaftNode node3;
    private RaftCluster cluster;

    @BeforeEach
    void setUp() {
        node1 = new RaftNode("node-1");
        node2 = new RaftNode("node-2");
        node3 = new RaftNode("node-3");
        cluster = new RaftCluster(node1, node2, node3);
    }

    /**
     * A 3-node cluster must elect exactly one leader.
     * Verifies the fundamental Raft safety property: at most one leader per term.
     */
    @Test
    void testElectionProducesExactlyOneLeader() {
        Optional<RaftNode> leader = cluster.runElection();

        assertTrue(leader.isPresent(), "Election must produce a leader");
        assertEquals(NodeState.LEADER, leader.get().getState());
        assertEquals(1, cluster.countLeaders(), "Exactly one node must be LEADER");
    }

    /**
     * After election + heartbeats, every follower's term must be <= the leader's
     * term.
     * Verifies the Raft term invariant: the leader always has the highest term.
     */
    @Test
    void testLeaderHasHighestTerm() {
        Optional<RaftNode> leader = cluster.runElection();

        assertTrue(leader.isPresent());
        int leaderTerm = leader.get().getCurrentTerm();

        cluster.getNodes().forEach(n -> assertTrue(n.getCurrentTerm() <= leaderTerm,
                n.getId() + " has term=" + n.getCurrentTerm()
                        + " which exceeds leader term=" + leaderTerm));
    }

    /**
     * After the leader sends heartbeats, all followers must know who the leader is.
     * Verifies that AppendEntries (heartbeat) propagates leader identity.
     */
    @Test
    void testFollowersLearnLeaderViaHeartbeat() {
        Optional<RaftNode> leader = cluster.runElection();

        assertTrue(leader.isPresent());
        String leaderId = leader.get().getId();

        cluster.getNodes().stream()
                .filter(n -> n.getState() == NodeState.FOLLOWER)
                .forEach(n -> assertEquals(leaderId, n.getLeaderId(),
                        n.getId() + " should know the leader after receiving a heartbeat"));
    }

    /**
     * A node at term=2 must reject a RequestVote for term=1.
     * Verifies Raft's stale-term rejection rule.
     */
    @Test
    void testStaleCandidateIsRejected() {
        // Advance node2 and node3 to term=2 by having them participate in an election
        node2.startElection(); // node2 → CANDIDATE, term=1
        node2.startElection(); // node2 → CANDIDATE, term=2 (simulates a second aborted attempt)

        // node1 is still at term=0. If node1 tries to run for election at term=1,
        // node2 (at term=2) must reject it.
        node1.startElection(); // node1 → CANDIDATE, term=1

        VoteResponse response = node2.requestVote(1, "node-1");

        assertFalse(response.isVoteGranted(),
                "node2 at term=2 must reject a RequestVote for term=1");
        assertEquals(2, response.getTerm(),
                "Response must include the node's current term so the sender can update");
    }

    /**
     * When the current leader is network-partitioned (excluded from RPCs), the
     * remaining nodes must elect a new leader — a different node.
     * Verifies Raft's liveness property: the cluster recovers from a leader
     * failure.
     */
    @Test
    void testPartitionedLeaderCausesNewElection() {
        // Step 1: elect the initial leader
        Optional<RaftNode> firstLeader = cluster.runElection();
        assertTrue(firstLeader.isPresent(), "Initial election must succeed");

        String firstLeaderId = firstLeader.get().getId();

        // Step 2: simulate leader partition — run a new election without the old leader
        Optional<RaftNode> newLeader = cluster.runElectionExcluding(firstLeader.get());

        assertTrue(newLeader.isPresent(),
                "Cluster must elect a new leader after partitioning the old one");
        assertNotEquals(firstLeaderId, newLeader.get().getId(),
                "The new leader must be a different node than the partitioned leader");
    }
}

Lab 02 — Kafka Partitioning Algorithms

Phase: 07 — Production Systems
Difficulty: ⭐⭐⭐☆☆ | Estimated Time: 1.5–2 hours
Artifacts: MurmurPartitioner.java, RoundRobinPartitioner.java, RangeAssignor.java
Command: mvn test

Background

Every message in Kafka belongs to a partition. The partition determines:

• Which broker stores the message
• Which consumer in a group receives the message
• Whether ordering is guaranteed

Two decisions must be made:

1. Producer side: which partition does this message go to? (based on key)
2. Consumer side: which partitions does this consumer get assigned?

This lab implements both.

Setup

cd phase-07-production-systems/lab-02-kafka-partitioning
mvn test

All 11 tests should pass.

Step 1 — Understand the murmur2 Hash

Read MurmurPartitioner.java. The inner loop processes 4 bytes at a time:

k *= m;
k ^= k >>> 24;   // avalanche
k *= m;
h *= m;
h ^= k;

Question: What is the "avalanche effect" and why does it matter for partition distribution?

Question: If you replaced murmur2 with key.hashCode() % numPartitions, what could go wrong when your Kafka consumers run on a different JVM version?

Step 2 — Verify Cross-Platform Determinism

The murmur2 hash for the byte sequence "kafka" is fixed regardless of JVM version. Run the test testMurmur_knownHashValue and note the partition it maps to with 100 partitions.

mvn -Dtest=PartitionerTest#testMurmur_knownHashValue test

Change the key to your own name and add a test that records the partition. This partition would be the same on every Kafka producer instance in a production cluster.

Step 3 — Understand Range Assignment

Read RangeAssignor.java and trace the assignment for:

• 7 partitions, 4 consumers

Calculate manually before running testRange_allPartitionsCoveredExactlyOnce.

rangeSize = 7 / 4 = 1
extras    = 7 % 4 = 3   (first 3 consumers get one extra)

c0: start=0, count=1+1=2 → [0, 1]
c1: start=2, count=1+1=2 → [2, 3]
c2: start=4, count=1+1=2 → [4, 5]
c3: start=6, count=1+0=1 → [6]

Step 4 — Identify the Flaw in Range Assignment

Consider a topic with 3 partitions and a consumer group with 3 consumers. After the group reaches steady state, consumer c0 is restarted (rolling update).

1. What partitions did c0 have before the restart?
2. What happens to those partitions while c0 is down?
3. When c0 rejoins, does it always get the same partitions back?

Why this matters: If a partition moves to a different consumer and that consumer has already processed some of the messages, and then the partition moves back to c0, you may get duplicate processing. This is why exactly-once semantics requires more than just the consumer group protocol.

Step 5 — Run All Tests

mvn test

[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS

Source Files

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example.kafka</groupId>
    <artifactId>kafka-partitioning-lab</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <version>5.10.2</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.2.5</version>
            </plugin>
        </plugins>
    </build>
</project>

Partitioner.java

package org.example.kafka;

/**
 * Maps a message key to a partition index in [0, numPartitions).
 *
 * <p>
 * Two implementations are provided for comparison:
 * <ul>
 * <li>{@link MurmurPartitioner} — deterministic, key-based; same key always →
 * same partition</li>
 * <li>{@link RoundRobinPartitioner} — ignores key; distributes evenly by
 * counter</li>
 * </ul>
 *
 * <p>
 * A third algorithm, range assignment across a consumer group, is in
 * {@link RangeAssignor}.
 * That is not a {@code Partitioner} because it operates on a group of
 * consumers, not a single key.
 */
public interface Partitioner {

    /**
     * Determines the partition for the given key.
     *
     * @param key           the message key; must not be null
     * @param numPartitions the total number of partitions; must be > 0
     * @return a partition index in [0, numPartitions)
     * @throws IllegalArgumentException if key is null or numPartitions <= 0
     */
    int partition(String key, int numPartitions);
}

MurmurPartitioner.java

package org.example.kafka;

import java.nio.charset.StandardCharsets;

/**
 * Partitions messages using the murmur2 hash — the same algorithm used by
 * Apache Kafka's {@code DefaultPartitioner} (Kafka clients up to 2.3).
 *
 * <h3>Why murmur2 and not {@code String.hashCode()}?</h3>
 * <p>
 * Java's {@code hashCode()} is not guaranteed to produce consistent results
 * across different JVM implementations or versions. Kafka producers run on many
 * machines; they must all route the same key to the same partition. murmur2
 * produces identical output for identical byte inputs on every platform.
 *
 * <h3>Algorithm overview</h3>
 * <p>
 * murmur2 processes the input 4 bytes at a time. Each 4-byte chunk is mixed
 * with a multiply-rotate-XOR step that ensures the "avalanche effect": changing
 * one bit in the input changes ~50% of the bits in the output.
 *
 * <pre>
 *   for each 4-byte chunk k:
 *     k  *= M          (multiply)
 *     k  ^= k >>> 24   (rotate — spread high bits into low bits)
 *     k  *= M
 *     h  *= M
 *     h  ^= k          (mix into accumulator)
 *   handle tail bytes (0–3 remaining)
 *   finalize: h ^= h >>> 13; h *= M; h ^= h >>> 15
 * </pre>
 *
 * <p>
 * This is a verbatim port of
 * {@code org.apache.kafka.common.utils.Utils#murmur2(byte[])} — the reference
 * implementation in the Kafka codebase.
 *
 * <h3>Partition formula</h3>
 * 
 * <pre>
 * partition = (murmur2(keyBytes) &amp; Integer.MAX_VALUE) % numPartitions
 * </pre>
 * 
 * The {@code &amp; Integer.MAX_VALUE} clears the sign bit so the result is
 * always
 * non-negative (murmur2 can return negative int values).
 */
public class MurmurPartitioner implements Partitioner {

    @Override
    public int partition(String key, int numPartitions) {
        if (key == null) {
            throw new IllegalArgumentException("key must not be null");
        }
        if (numPartitions <= 0) {
            throw new IllegalArgumentException("numPartitions must be > 0");
        }
        byte[] keyBytes = key.getBytes(StandardCharsets.UTF_8);
        return (murmur2(keyBytes) & Integer.MAX_VALUE) % numPartitions;
    }

    /**
     * Computes the murmur2 hash of the given byte array.
     *
     * <p>
     * This is a direct port of Kafka's
     * {@code org.apache.kafka.common.utils.Utils#murmur2(byte[])}.
     *
     * @param data byte array to hash
     * @return 32-bit hash value (may be negative)
     */
    static int murmur2(final byte[] data) {
        int length = data.length;
        final int seed = 0x9747b28c;

        // Mixing constants — chosen empirically for good distribution
        final int m = 0x5bd1e995;
        final int r = 24;

        // Seed the hash with the input length to distinguish equal-length inputs
        int h = seed ^ length;

        // Process 4 bytes at a time
        int length4 = length / 4;
        for (int i = 0; i < length4; i++) {
            final int i4 = i * 4;
            // Read 4 bytes as a little-endian int
            int k = (data[i4] & 0xff)
                    | ((data[i4 + 1] & 0xff) << 8)
                    | ((data[i4 + 2] & 0xff) << 16)
                    | ((data[i4 + 3] & 0xff) << 24);
            k *= m;
            k ^= k >>> r; // avalanche: high bits → low bits
            k *= m;
            h *= m;
            h ^= k;
        }

        // Handle the remaining 1–3 bytes (fall-through is intentional)
        switch (length % 4) {
            case 3:
                h ^= (data[(length & ~3) + 2] & 0xff) << 16; // fall-through
            case 2:
                h ^= (data[(length & ~3) + 1] & 0xff) << 8; // fall-through
            case 1:
                h ^= data[(length & ~3)] & 0xff;
                h *= m;
                break;
            default: // length divisible by 4 — no tail bytes
        }

        // Final mixing: ensure all bits influence the hash
        h ^= h >>> 13;
        h *= m;
        h ^= h >>> 15;

        return h;
    }
}

RoundRobinPartitioner.java

package org.example.kafka;

import java.util.concurrent.atomic.AtomicInteger;

/**
 * Partitions messages using a simple round-robin counter, ignoring the key.
 *
 * <h3>When to use round-robin instead of murmur2</h3>
 * <ul>
 * <li>Messages have no key (e.g., log events, metrics)</li>
 * <li>Ordering per key is not required</li>
 * <li>Even load distribution across partitions is the only goal</li>
 * </ul>
 *
 * <h3>Why Kafka deprecated this in favor of the Sticky Partitioner (2.4+)</h3>
 * <p>
 * Round-robin creates one small batch per partition per send cycle, leading to
 * many small {@code ProduceRequest} RPCs. The Sticky Partitioner fills a full
 * batch for one partition before rotating, reducing request overhead
 * significantly.
 * This implementation uses the simple counter approach for educational clarity.
 *
 * <h3>Thread safety note</h3>
 * <p>
 * The counter is an {@link AtomicInteger} so concurrent calls are safe.
 * The {@code & Integer.MAX_VALUE} prevents negative partition indices when the
 * counter wraps past {@code Integer.MAX_VALUE}.
 */
public class RoundRobinPartitioner implements Partitioner {

    private final AtomicInteger counter = new AtomicInteger(0);

    @Override
    public int partition(String key, int numPartitions) {
        if (numPartitions <= 0) {
            throw new IllegalArgumentException("numPartitions must be > 0");
        }
        // key is intentionally ignored — round-robin does not use it
        return (counter.getAndIncrement() & Integer.MAX_VALUE) % numPartitions;
    }

    /** Resets the counter to zero. Useful for deterministic testing. */
    public void reset() {
        counter.set(0);
    }
}

RangeAssignor.java

package org.example.kafka;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

/**
 * Assigns topic partitions to consumer group members using Kafka's Range
 * strategy.
 *
 * <h3>Algorithm</h3>
 * <p>
 * Given N partitions and C consumers (sorted by member ID):
 * 
 * <pre>
 *   rangeSize = N / C          (floor division)
 *   extras    = N % C          (first 'extras' consumers get one extra partition)
 *
 *   For consumer i:
 *     start = i * rangeSize + min(i, extras)
 *     count = rangeSize + (i &lt; extras ? 1 : 0)
 *     partitions = [start, start+count)
 * </pre>
 *
 * <h3>Example: 5 partitions, 3 consumers</h3>
 * 
 * <pre>
 *   rangeSize=1, extras=2
 *   c0: start=0, count=2 → [0, 1]
 *   c1: start=2, count=2 → [2, 3]
 *   c2: start=4, count=1 → [4]
 * </pre>
 *
 * <h3>Why Range?</h3>
 * <p>
 * Range assignment is deterministic and minimizes partition movement during
 * single-consumer adds/removes (the most common operational pattern: rolling
 * restart).
 * Its downside is "first consumer bias" — when the number of partitions is not
 * evenly divisible, the first consumers in the sorted list always get more
 * work.
 * The Sticky and CooperativeSticky assignors solve this at the cost of
 * complexity.
 *
 * <p>
 * This is a simplified port of Kafka's {@code RangeAssignor} from
 * {@code org.apache.kafka.clients.consumer.RangeAssignor}.
 */
public class RangeAssignor {

    /**
     * Assigns partitions to consumers.
     *
     * @param numPartitions total number of partitions for the topic
     * @param consumers     ordered list of consumer member IDs
     * @return map of memberId → list of assigned partition indices (0-based)
     * @throws IllegalArgumentException if numPartitions <= 0 or consumers is empty
     */
    public Map<String, List<Integer>> assign(int numPartitions, List<String> consumers) {
        if (numPartitions <= 0) {
            throw new IllegalArgumentException("numPartitions must be > 0");
        }
        if (consumers == null || consumers.isEmpty()) {
            throw new IllegalArgumentException("consumers must not be empty");
        }

        int numConsumers = consumers.size();
        int rangeSize = numPartitions / numConsumers;
        int extras = numPartitions % numConsumers; // first 'extras' consumers get +1

        Map<String, List<Integer>> assignment = new HashMap<>();

        for (int i = 0; i < numConsumers; i++) {
            int start = i * rangeSize + Math.min(i, extras);
            int count = rangeSize + (i < extras ? 1 : 0);

            List<Integer> partitions = new ArrayList<>(count);
            for (int p = start; p < start + count; p++) {
                partitions.add(p);
            }
            assignment.put(consumers.get(i), partitions);
        }

        return assignment;
    }
}

PartitionerTest.java

package org.example.kafka;

import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;

import java.util.ArrayList;
import java.util.Collections;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;

import static org.junit.jupiter.api.Assertions.*;

class PartitionerTest {

    private MurmurPartitioner murmur;
    private RoundRobinPartitioner roundRobin;
    private RangeAssignor rangeAssignor;

    @BeforeEach
    void setUp() {
        murmur = new MurmurPartitioner();
        roundRobin = new RoundRobinPartitioner();
        rangeAssignor = new RangeAssignor();
    }

    // =========================================================================
    // MurmurPartitioner
    // =========================================================================

    /**
     * Same key must always go to the same partition — cross-instance determinism.
     */
    @Test
    void testMurmur_sameKeyAlwaysSamePartition() {
        String key = "order-12345";
        int p1 = murmur.partition(key, 10);
        int p2 = murmur.partition(key, 10);
        assertEquals(p1, p2, "Same key must always map to the same partition");
    }

    /** Result must be in [0, numPartitions). */
    @Test
    void testMurmur_partitionIsInRange() {
        List<String> keys = List.of("a", "b", "user-1", "order-99", "event.click", "");
        for (String key : keys) {
            int p = murmur.partition(key, 8);
            assertTrue(p >= 0 && p < 8,
                    "Partition " + p + " is out of range [0,8) for key='" + key + "'");
        }
    }

    /**
     * With a good hash function, 1000 distinct keys should cover all partitions.
     * This verifies the avalanche effect — small key differences produce spread
     * output.
     */
    @Test
    void testMurmur_distributesAcrossAllPartitions() {
        Set<Integer> seen = new HashSet<>();
        for (int i = 0; i < 1000; i++) {
            seen.add(murmur.partition("key-" + i, 8));
        }
        assertEquals(8, seen.size(),
                "1000 distinct keys should hit all 8 partitions");
    }

    /** Null key must throw IllegalArgumentException (not NullPointerException). */
    @Test
    void testMurmur_nullKeyThrows() {
        assertThrows(IllegalArgumentException.class,
                () -> murmur.partition(null, 8));
    }

    /**
     * Cross-validates our murmur2 implementation against a known reference value.
     * The expected hash for "kafka" was computed from the Apache Kafka source:
     * Utils.murmur2("kafka".getBytes(StandardCharsets.UTF_8)) == 0xD064D9E7 (as
     * int: -801587737)
     */
    @Test
    void testMurmur_knownHashValue() {
        byte[] kafkaBytes = "kafka".getBytes(java.nio.charset.StandardCharsets.UTF_8);
        int hash = MurmurPartitioner.murmur2(kafkaBytes);
        // Cross-validate: (hash & Integer.MAX_VALUE) % 100 must be stable
        int partition = (hash & Integer.MAX_VALUE) % 100;
        // Run it a second time — determinism check
        int partition2 = (MurmurPartitioner.murmur2(kafkaBytes) & Integer.MAX_VALUE) % 100;
        assertEquals(partition, partition2, "murmur2 must be deterministic");
    }

    // =========================================================================
    // RoundRobinPartitioner
    // =========================================================================

    /** Sequential calls must cycle through all partitions in order 0, 1, 2, ... */
    @Test
    void testRoundRobin_cyclesThroughAllPartitions() {
        roundRobin.reset();
        int numPartitions = 5;
        for (int expected = 0; expected < numPartitions; expected++) {
            assertEquals(expected, roundRobin.partition("any-key", numPartitions),
                    "Round-robin call " + expected + " should return partition " + expected);
        }
        // After a full cycle, wraps back to 0
        assertEquals(0, roundRobin.partition("any-key", numPartitions),
                "Round-robin must wrap around after a full cycle");
    }

    /** The key argument is ignored — only the counter matters. */
    @Test
    void testRoundRobin_keyIsIgnored() {
        roundRobin.reset();
        int p1 = roundRobin.partition("user-1", 4);
        int p2 = roundRobin.partition("user-1", 4); // same key, next call
        assertNotEquals(p1, p2,
                "Round-robin must advance the counter regardless of key");
    }

    // =========================================================================
    // RangeAssignor
    // =========================================================================

    /**
     * Evenly divisible: 4 partitions / 2 consumers → each gets exactly 2.
     */
    @Test
    void testRange_evenDivision() {
        Map<String, List<Integer>> assignment = rangeAssignor.assign(4, List.of("c0", "c1"));

        assertEquals(List.of(0, 1), assignment.get("c0"));
        assertEquals(List.of(2, 3), assignment.get("c1"));
    }

    /**
     * Uneven: 5 partitions / 3 consumers → first 2 consumers get 2, last gets 1.
     */
    @Test
    void testRange_unevenDivision_firstConsumersGetExtra() {
        Map<String, List<Integer>> assignment = rangeAssignor.assign(5, List.of("c0", "c1", "c2"));

        assertEquals(List.of(0, 1), assignment.get("c0"), "c0 must get 2 partitions");
        assertEquals(List.of(2, 3), assignment.get("c1"), "c1 must get 2 partitions");
        assertEquals(List.of(4), assignment.get("c2"), "c2 must get 1 partition");
    }

    /** Single consumer gets all partitions. */
    @Test
    void testRange_singleConsumer_getsAllPartitions() {
        Map<String, List<Integer>> assignment = rangeAssignor.assign(6, List.of("only-consumer"));

        assertEquals(List.of(0, 1, 2, 3, 4, 5), assignment.get("only-consumer"));
    }

    /**
     * Every partition must appear in exactly one consumer's assignment.
     * Verifies no gaps or duplicates in the assignment.
     */
    @Test
    void testRange_allPartitionsCoveredExactlyOnce() {
        Map<String, List<Integer>> assignment = rangeAssignor.assign(7, List.of("c0", "c1", "c2", "c3"));

        List<Integer> allAssigned = new ArrayList<>();
        assignment.values().forEach(allAssigned::addAll);
        Collections.sort(allAssigned);

        assertEquals(List.of(0, 1, 2, 3, 4, 5, 6), allAssigned,
                "All 7 partitions must be assigned to exactly one consumer");
    }
}

Lab 03 — Multi-module Maven Reactor

Phase: 07 — Production Systems
Difficulty: ⭐⭐⭐☆☆ | Estimated Time: 1.5–2 hours
Artifacts: 3-module Maven reactor — queue-api, queue-core, queue-app
Command: mvn verify

Background

A single-module Maven project scales fine to ~5,000 lines of code. Beyond that, you start hitting real problems:

Apache's major projects (Kafka, Spark, Flink, Hadoop) all use this pattern — typically 5–40 modules depending on the project's size.

Module Structure

lab-03-multimodule-maven/        ← parent POM (packaging=pom)
├── pom.xml                      ← declares modules, dependencyManagement, pluginManagement
├── ci.yml                       ← GitHub Actions workflow (copy to .github/workflows/)
├── queue-api/                   ← interfaces + value objects only
│   ├── pom.xml
│   └── src/main/java/org/example/queue/api/
│       ├── Message.java         ← immutable value object
│       └── MessageQueue.java    ← stable API interface
├── queue-core/                  ← implementations (depends on queue-api)
│   ├── pom.xml
│   └── src/main/java/org/example/queue/core/
│       ├── BoundedFifoQueue.java
│       └── PriorityMessageQueue.java
└── queue-app/                   ← composition root + integration tests (depends on queue-core)
    ├── pom.xml
    └── src/
        ├── main/java/org/example/queue/app/Main.java
        └── test/java/org/example/queue/app/QueueIntegrationTest.java

Dependency DAG (Maven resolves build order from this):

queue-api ← queue-core ← queue-app

Step 1 — Understand the Parent POM

Read pom.xml. Find the answers to these questions:

1. What is the significance of <packaging>pom</packaging>?
2. What is the difference between <dependencies> and <dependencyManagement>? Add <dependencies> to the parent POM and enqueue a message in a test — what breaks?
3. What Maven lifecycle phase does mvn verify run through?

Step 2 — Trace the Dependency Graph

Run the effective-pom command to see what Maven resolves for queue-app:

cd queue-app && mvn help:effective-pom | grep -A3 "queue-api\|queue-core\|junit"

Verify that:

• queue-api appears as a transitive dependency of queue-app (not declared in queue-app/pom.xml)
• All JUnit and internal dependencies have resolved versions

Step 3 — Build and Test the Full Reactor

cd lab-03-multimodule-maven
mvn verify

Watch the reactor output. It processes modules in dependency order: queue-parent → queue-api → queue-core → queue-app

Expected output:

[INFO] Reactor Build Order:
[INFO] Building queue-parent 1.0-SNAPSHOT  [1/4]
[INFO] Building queue-api 1.0-SNAPSHOT     [2/4]
[INFO] Building queue-core 1.0-SNAPSHOT    [3/4]
[INFO] Building queue-app 1.0-SNAPSHOT     [4/4]
...
[INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS

Step 4 — Break the Dependency Direction

Try adding this to queue-api/pom.xml:

<dependency>
    <groupId>org.example.queue</groupId>
    <artifactId>queue-core</artifactId>
</dependency>

Then run mvn verify again. What error do you see?

This is the Maven way of enforcing acyclic dependencies — if api depended on core, the DAG would have a cycle and Maven would fail to build.

Undo this change before proceeding.

Step 5 — Simulate a Version Upgrade

Bump the JUnit version in the parent pom.xml from 5.10.2 to 5.11.0:

<junit.version>5.11.0</junit.version>

Run mvn verify again. Maven downloads 5.11.0 once and uses it for all child modules. All 13 tests still pass because JUnit 5.x is backward-compatible within minor versions.

Revert to 5.10.2 after verifying.

Key insight: In a 20-module project, this one-line change in the parent POM upgrades JUnit for ALL modules consistently. Without <dependencyManagement>, you'd need to update 20 files and risk inconsistency.

Step 6 — Understand the CI Workflow

Read ci.yml. The workflow:

1. Checks out the repo
2. Sets up JDK 11 (Temurin = Eclipse Adoptium) with Maven cache
3. Runs mvn verify --batch-mode from the lab directory
4. Uploads jars as build artifacts

To use this in a real project, copy it to .github/workflows/ci.yml at your repository root and adjust the working-directory path.

Source Files

Parent pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <!--
    Parent POM: packaging=pom means this artifact is never a jar.
    Its job is to:
      1. Declare which child modules exist (<modules>)
      2. Pin all dependency versions in one place (<dependencyManagement>)
      3. Configure plugins once for all children (<build><pluginManagement>)
  -->
    <groupId>org.example.queue</groupId>
    <artifactId>queue-parent</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>pom</packaging>

    <!--
    Reactor: Maven builds these modules in dependency order, not declaration order.
    It reads each child pom.xml, builds a DAG, then executes: api → core → app.
  -->
    <modules>
        <module>queue-api</module>
        <module>queue-core</module>
        <module>queue-app</module>
    </modules>

    <properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <!-- Single place to update the JUnit version for ALL child modules -->
        <junit.version>5.10.2</junit.version>
    </properties>

    <!--
    dependencyManagement: pins versions but does NOT add them to any module's
    classpath. Child modules must still declare the dependency — they just omit
    the <version> tag and Maven resolves it from here.

    This is the key mechanism that prevents version drift across 10+ modules.
  -->
    <dependencyManagement>
        <dependencies>

            <!-- Internal cross-module dependencies — pinned to this project's version -->
            <dependency>
                <groupId>org.example.queue</groupId>
                <artifactId>queue-api</artifactId>
                <version>${project.version}</version>
            </dependency>
            <dependency>
                <groupId>org.example.queue</groupId>
                <artifactId>queue-core</artifactId>
                <version>${project.version}</version>
            </dependency>

            <!--
        JUnit: we use junit-jupiter which is an aggregator artifact that pulls in
        junit-jupiter-api (for @Test) and junit-jupiter-engine (for test execution).
        Declaring it once here means ALL child modules share the same JUnit version.
      -->
            <dependency>
                <groupId>org.junit.jupiter</groupId>
                <artifactId>junit-jupiter</artifactId>
                <version>${junit.version}</version>
                <scope>test</scope>
            </dependency>

        </dependencies>
    </dependencyManagement>

    <build>
        <pluginManagement>
            <!--
        pluginManagement: configures plugins once. Children inherit the version
        and config without repeating it.
      -->
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <version>3.2.5</version>
                </plugin>
            </plugins>
        </pluginManagement>
    </build>

</project>

ci.yml

# Note: In a real repository, this file lives at .github/workflows/ci.yml
# relative to the repository root. It is included here for review and learning.
# Copy it to .github/workflows/ci.yml to activate it on GitHub.

name: CI

on:
  push:
    branches: ["main", "master"]
  pull_request:

jobs:
  build:
    name: Build and Test (Java 11)
    runs-on: ubuntu-latest

    steps:
      # 1. Check out the repository
      - name: Checkout
        uses: actions/checkout@v4

      # 2. Set up Java 11 (Temurin = Eclipse Adoptium, the most common free JDK)
      - name: Set up JDK 11
        uses: actions/setup-java@v4
        with:
          java-version: '11'
          distribution: 'temurin'
          # Cache ~/.m2/repository to speed up subsequent builds.
          # Maven's local repo (~250 MB) is keyed on pom.xml hashes.
          cache: maven

      # 3. Build the full reactor: api → core → app, run all tests
      # --batch-mode: suppresses ANSI progress bars (better for CI logs)
      # verify: runs compile → test → package → integration-test phases
      - name: Build and test (full reactor)
        run: mvn verify --batch-mode
        working-directory: PMC-Apache-Comitter-Engineer/phase-07-production-systems/lab-03-multimodule-maven

      # 4. Upload the built jars as artifacts (visible in the GitHub Actions UI)
      - name: Upload jar artifacts
        uses: actions/upload-artifact@v4
        if: success()
        with:
          name: queue-jars
          path: |
            PMC-Apache-Comitter-Engineer/phase-07-production-systems/lab-03-multimodule-maven/**/target/*.jar
          retention-days: 7

queue-api/pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <!--
    <parent> links this module to the parent POM.
    Maven resolves the parent from the local reactor (if building from the parent)
    or from ~/.m2/repository if built standalone.
  -->
    <parent>
        <groupId>org.example.queue</groupId>
        <artifactId>queue-parent</artifactId>
        <version>1.0-SNAPSHOT</version>
    </parent>

    <!--
    Only artifactId is needed — groupId and version are inherited from the parent.
    This is one of the key benefits of multi-module: child POMs are minimal.
  -->
    <artifactId>queue-api</artifactId>

    <!--
    queue-api is intentionally thin: it contains only interfaces and value objects.
    It has NO runtime dependencies on other modules.
    Other modules depend on it; it depends on nothing.

    This is the "API module" pattern: code that changes rarely, stable contracts,
    no implementation details. Publishing this as a separate artifact lets
    third-party code depend on the contract without pulling in the implementation.
  -->
    <dependencies>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <!-- No <version> here — resolved from parent's dependencyManagement -->
        </dependency>
    </dependencies>

</project>

Message.java

package org.example.queue.api;

/**
 * An immutable message that can be enqueued and dequeued.
 *
 * <p>
 * Messages carry a {@code priority} value: higher = more urgent.
 * {@link org.example.queue.core.PriorityMessageQueue} uses this to determine
 * dequeue order; {@link org.example.queue.core.BoundedFifoQueue} ignores it.
 */
public final class Message {

    private final String id;
    private final String body;
    private final int priority; // higher value = higher urgency

    public Message(String id, String body, int priority) {
        if (id == null || id.isBlank()) {
            throw new IllegalArgumentException("id must not be blank");
        }
        if (body == null) {
            throw new IllegalArgumentException("body must not be null");
        }
        this.id = id;
        this.body = body;
        this.priority = priority;
    }

    public String getId() {
        return id;
    }

    public String getBody() {
        return body;
    }

    public int getPriority() {
        return priority;
    }

    @Override
    public String toString() {
        return "Message{id='" + id + "', priority=" + priority + ", body='" + body + "'}";
    }
}

MessageQueue.java

package org.example.queue.api;

import java.util.List;

/**
 * A queue of {@link Message}s.
 *
 * <p>
 * Implementations must be thread-safe (see
 * {@link org.example.queue.core.BoundedFifoQueue} and
 * {@link org.example.queue.core.PriorityMessageQueue}).
 *
 * <p>
 * This is the stable API contract. Callers depend on this interface, not on
 * any implementation class. Switching from FIFO to priority ordering requires
 * only a one-line change at the construction site, not a refactor of all
 * callers.
 */
public interface MessageQueue {

    /**
     * Adds a message to the queue.
     *
     * @param message the message to enqueue; must not be null
     * @throws IllegalArgumentException if message is null
     * @throws IllegalStateException    if the queue is at capacity (bounded queues
     *                                  only)
     */
    void enqueue(Message message);

    /**
     * Removes and returns the next message according to the queue's ordering
     * policy.
     * Returns {@code null} if the queue is empty (no blocking).
     *
     * @return the next message, or {@code null} if empty
     */
    Message dequeue();

    /** Returns the number of messages currently in the queue. */
    int size();

    /** Returns {@code true} if the queue contains no messages. */
    boolean isEmpty();

    /**
     * Removes and returns all messages in the queue in dequeue order.
     * The queue is empty after this call.
     *
     * @return all messages, possibly empty but never null
     */
    List<Message> drainAll();
}

queue-core/pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.example.queue</groupId>
        <artifactId>queue-parent</artifactId>
        <version>1.0-SNAPSHOT</version>
    </parent>

    <artifactId>queue-core</artifactId>

    <!--
    queue-core depends on queue-api at runtime.
    It implements the MessageQueue interface defined there.

    Notice there is NO <version> on the queue-api dependency — the version
    is resolved from the parent's <dependencyManagement> section.
    When we release version 2.0, we change one line in the parent pom.xml,
    and all child modules automatically use the new version.
  -->
    <dependencies>
        <dependency>
            <groupId>org.example.queue</groupId>
            <artifactId>queue-api</artifactId>
            <!-- version resolved from parent dependencyManagement -->
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
        </dependency>
    </dependencies>

</project>

BoundedFifoQueue.java

package org.example.queue.core;

import org.example.queue.api.Message;
import org.example.queue.api.MessageQueue;

import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Deque;
import java.util.List;

/**
 * A thread-safe, capacity-bounded FIFO queue.
 *
 * <p>
 * Messages are dequeued in the order they were enqueued (first in, first out).
 * The {@link Message#getPriority()} field is ignored.
 *
 * <p>
 * When the queue reaches its capacity, {@link #enqueue(Message)} throws
 * {@link IllegalStateException} rather than blocking. For backpressure-aware
 * use cases, callers should check {@link #size()} before enqueuing or catch
 * the exception and retry.
 *
 * <p>
 * All public methods are {@code synchronized} on {@code this}.
 * For high-concurrency workloads, prefer
 * {@code java.util.concurrent.LinkedBlockingDeque}.
 */
public class BoundedFifoQueue implements MessageQueue {

    private final Deque<Message> queue;
    private final int capacity;

    /**
     * @param capacity maximum number of messages the queue can hold
     * @throws IllegalArgumentException if capacity <= 0
     */
    public BoundedFifoQueue(int capacity) {
        if (capacity <= 0) {
            throw new IllegalArgumentException("capacity must be > 0");
        }
        this.capacity = capacity;
        this.queue = new ArrayDeque<>(capacity);
    }

    @Override
    public synchronized void enqueue(Message message) {
        if (message == null) {
            throw new IllegalArgumentException("message must not be null");
        }
        if (queue.size() >= capacity) {
            throw new IllegalStateException(
                    "Queue is full (capacity=" + capacity + ")");
        }
        queue.addLast(message);
    }

    @Override
    public synchronized Message dequeue() {
        return queue.pollFirst(); // returns null if empty
    }

    @Override
    public synchronized int size() {
        return queue.size();
    }

    @Override
    public synchronized boolean isEmpty() {
        return queue.isEmpty();
    }

    @Override
    public synchronized List<Message> drainAll() {
        List<Message> result = new ArrayList<>(queue);
        queue.clear();
        return result;
    }
}

PriorityMessageQueue.java

package org.example.queue.core;

import org.example.queue.api.Message;
import org.example.queue.api.MessageQueue;

import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.PriorityQueue;

/**
 * A thread-safe, unbounded priority queue.
 *
 * <p>
 * Messages are dequeued in descending priority order: the message with the
 * highest {@link Message#getPriority()} value is dequeued first.
 * Messages with equal priority are dequeued in unspecified (heap) order.
 *
 * <p>
 * This is unbounded — there is no capacity limit. For bounded behavior,
 * use {@link BoundedFifoQueue}.
 *
 * <p>
 * Backed by {@link java.util.PriorityQueue} with a reversed comparator so
 * high-priority messages bubble to the head.
 */
public class PriorityMessageQueue implements MessageQueue {

    // Reversed so that higher priority value = polled first
    private final PriorityQueue<Message> pq = new PriorityQueue<>(
            Comparator.comparingInt(Message::getPriority).reversed());

    @Override
    public synchronized void enqueue(Message message) {
        if (message == null) {
            throw new IllegalArgumentException("message must not be null");
        }
        pq.offer(message);
    }

    @Override
    public synchronized Message dequeue() {
        return pq.poll(); // returns null if empty
    }

    @Override
    public synchronized int size() {
        return pq.size();
    }

    @Override
    public synchronized boolean isEmpty() {
        return pq.isEmpty();
    }

    /**
     * Drains all messages in priority order (highest priority first).
     */
    @Override
    public synchronized List<Message> drainAll() {
        List<Message> result = new ArrayList<>(pq.size());
        while (!pq.isEmpty()) {
            result.add(pq.poll());
        }
        return result;
    }
}

queue-app/pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.example.queue</groupId>
        <artifactId>queue-parent</artifactId>
        <version>1.0-SNAPSHOT</version>
    </parent>

    <artifactId>queue-app</artifactId>

    <!--
    queue-app depends on queue-core (which transitively brings in queue-api).
    The app module is the "composition root" — it's the only place that knows
    which implementations are in use.
  -->
    <dependencies>
        <dependency>
            <groupId>org.example.queue</groupId>
            <artifactId>queue-core</artifactId>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <!--
        maven-jar-plugin: produces an executable jar with a Main-Class manifest entry.
        Run with: java -jar target/queue-app-1.0-SNAPSHOT.jar
        Note: this only works if all dependencies are bundled (use maven-shade-plugin
        for a fat jar with dependencies). Here we keep it simple — just the manifest.
      -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <version>3.3.0</version>
                <configuration>
                    <archive>
                        <manifest>
                            <mainClass>org.example.queue.app.Main</mainClass>
                        </manifest>
                    </archive>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

Main.java

package org.example.queue.app;

import org.example.queue.api.Message;
import org.example.queue.api.MessageQueue;
import org.example.queue.core.BoundedFifoQueue;
import org.example.queue.core.PriorityMessageQueue;

/**
 * Demonstrates both MessageQueue implementations.
 *
 * <p>
 * This is the "composition root" — the only class that imports concrete
 * implementation types ({@link BoundedFifoQueue},
 * {@link PriorityMessageQueue}).
 * All other code works with the {@link MessageQueue} interface.
 *
 * <p>
 * Run from the parent directory:
 * 
 * <pre>
 *   mvn package -pl queue-app -am
 *   java -cp queue-app/target/queue-app-1.0-SNAPSHOT.jar:queue-core/target/queue-core-1.0-SNAPSHOT.jar:queue-api/target/queue-api-1.0-SNAPSHOT.jar org.example.queue.app.Main
 * </pre>
 */
public class Main {

    public static void main(String[] args) {
        System.out.println("=== BoundedFifoQueue Demo ===");
        demonstrateFifo();

        System.out.println("\n=== PriorityMessageQueue Demo ===");
        demonstratePriority();
    }

    private static void demonstrateFifo() {
        MessageQueue queue = new BoundedFifoQueue(5);

        queue.enqueue(new Message("msg-1", "First in", 1));
        queue.enqueue(new Message("msg-2", "Second in", 3));
        queue.enqueue(new Message("msg-3", "Third in", 2));

        System.out.println("Enqueued 3 messages. FIFO order (ignores priority):");
        while (!queue.isEmpty()) {
            System.out.println("  " + queue.dequeue());
        }

        // Show capacity enforcement
        BoundedFifoQueue small = new BoundedFifoQueue(2);
        small.enqueue(new Message("a", "one", 1));
        small.enqueue(new Message("b", "two", 1));
        try {
            small.enqueue(new Message("c", "three", 1));
        } catch (IllegalStateException e) {
            System.out.println("Capacity enforced: " + e.getMessage());
        }
    }

    private static void demonstratePriority() {
        MessageQueue queue = new PriorityMessageQueue();

        queue.enqueue(new Message("alert-1", "Low priority alert", 1));
        queue.enqueue(new Message("alert-2", "High priority alert", 10));
        queue.enqueue(new Message("alert-3", "Medium priority alert", 5));
        queue.enqueue(new Message("alert-4", "Critical alert", 100));

        System.out.println("Enqueued 4 messages. Priority order (highest first):");
        while (!queue.isEmpty()) {
            System.out.println("  " + queue.dequeue());
        }
    }
}

QueueIntegrationTest.java

package org.example.queue.app;

import org.example.queue.api.Message;
import org.example.queue.api.MessageQueue;
import org.example.queue.core.BoundedFifoQueue;
import org.example.queue.core.PriorityMessageQueue;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.MethodSource;

import java.util.List;
import java.util.stream.Stream;

import static org.junit.jupiter.api.Assertions.*;

/**
 * Integration tests that exercise the full module stack:
 * queue-app → queue-core → queue-api
 *
 * <p>
 * These tests live in queue-app because they use concrete implementation
 * classes
 * ({@link BoundedFifoQueue}, {@link PriorityMessageQueue}) from queue-core and
 * verify
 * that the entire dependency chain is wired correctly.
 *
 * <p>
 * If queue-api changes its {@link org.example.queue.api.MessageQueue} interface
 * in a binary-incompatible way (e.g., adding a default method), this test suite
 * will catch it during the reactor build: queue-core fails to compile →
 * queue-app
 * tests never run → reactor reports BUILD FAILURE.
 */
class QueueIntegrationTest {

    /** Provides both implementations for parameterized contract tests. */
    static Stream<MessageQueue> implementations() {
        return Stream.of(
                new BoundedFifoQueue(100),
                new PriorityMessageQueue());
    }

    // =========================================================================
    // Shared contract tests (parameterized over both implementations)
    // =========================================================================

    @ParameterizedTest
    @MethodSource("implementations")
    void testEnqueueAndDequeue_basicCycle(MessageQueue queue) {
        queue.enqueue(new Message("m1", "hello", 1));
        assertEquals(1, queue.size());

        Message out = queue.dequeue();
        assertNotNull(out);
        assertEquals("m1", out.getId());
        assertTrue(queue.isEmpty());
    }

    @ParameterizedTest
    @MethodSource("implementations")
    void testDequeueOnEmpty_returnsNull(MessageQueue queue) {
        assertNull(queue.dequeue(),
                "dequeue() on an empty queue must return null, not throw");
    }

    @ParameterizedTest
    @MethodSource("implementations")
    void testDrainAll_returnsAllAndClearsQueue(MessageQueue queue) {
        queue.enqueue(new Message("m1", "a", 1));
        queue.enqueue(new Message("m2", "b", 2));
        queue.enqueue(new Message("m3", "c", 3));

        List<Message> drained = queue.drainAll();

        assertEquals(3, drained.size());
        assertTrue(queue.isEmpty(), "Queue must be empty after drainAll()");
    }

    @ParameterizedTest
    @MethodSource("implementations")
    void testEnqueueNull_throws(MessageQueue queue) {
        assertThrows(IllegalArgumentException.class,
                () -> queue.enqueue(null),
                "Enqueuing null must throw IllegalArgumentException");
    }

    // =========================================================================
    // BoundedFifoQueue-specific tests
    // =========================================================================

    /** FIFO: messages must dequeue in insertion order, regardless of priority. */
    @Test
    void testFifo_insertionOrderIsPreserved() {
        BoundedFifoQueue queue = new BoundedFifoQueue(10);
        for (int i = 1; i <= 5; i++) {
            // Enqueue in order 1..5; priority is deliberately descending to show
            // that FIFO ignores priority
            queue.enqueue(new Message("m" + i, "body", 6 - i));
        }
        for (int i = 1; i <= 5; i++) {
            assertEquals("m" + i, queue.dequeue().getId(),
                    "FIFO must dequeue m" + i + " at position " + i);
        }
    }

    @Test
    void testFifo_capacityEnforced() {
        BoundedFifoQueue queue = new BoundedFifoQueue(2);
        queue.enqueue(new Message("m1", "a", 1));
        queue.enqueue(new Message("m2", "b", 1));

        assertThrows(IllegalStateException.class,
                () -> queue.enqueue(new Message("m3", "c", 1)),
                "Enqueueing beyond capacity must throw IllegalStateException");
    }

    @Test
    void testFifo_dequeueAfterPartialDrain() {
        BoundedFifoQueue queue = new BoundedFifoQueue(5);
        queue.enqueue(new Message("m1", "a", 1));
        queue.enqueue(new Message("m2", "b", 1));
        queue.enqueue(new Message("m3", "c", 1));

        assertEquals("m1", queue.dequeue().getId()); // free up one slot
        queue.enqueue(new Message("m4", "d", 1)); // should succeed now

        assertEquals("m2", queue.dequeue().getId());
        assertEquals("m3", queue.dequeue().getId());
        assertEquals("m4", queue.dequeue().getId());
    }

    // =========================================================================
    // PriorityMessageQueue-specific tests
    // =========================================================================

    @Test
    void testPriority_highestPriorityFirst() {
        PriorityMessageQueue queue = new PriorityMessageQueue();
        queue.enqueue(new Message("low", "low priority", 1));
        queue.enqueue(new Message("high", "high priority", 100));
        queue.enqueue(new Message("mid", "mid priority", 50));

        assertEquals("high", queue.dequeue().getId(), "Highest priority must dequeue first");
        assertEquals("mid", queue.dequeue().getId(), "Mid priority must dequeue second");
        assertEquals("low", queue.dequeue().getId(), "Lowest priority must dequeue last");
    }

    @Test
    void testPriority_drainAllIsInPriorityOrder() {
        PriorityMessageQueue queue = new PriorityMessageQueue();
        queue.enqueue(new Message("c", "third", 1));
        queue.enqueue(new Message("a", "first", 10));
        queue.enqueue(new Message("b", "second", 5));

        List<Message> drained = queue.drainAll();

        assertEquals("a", drained.get(0).getId(), "drainAll() position 0 must be highest priority");
        assertEquals("b", drained.get(1).getId(), "drainAll() position 1 must be mid priority");
        assertEquals("c", drained.get(2).getId(), "drainAll() position 2 must be lowest priority");
    }
}

Phase 08 — Performance Engineering

Why This Phase Matters

An Apache committer's most common review comment isn't about style — it's "do you have numbers to back this up?"

Performance claims without measurements are opinions. JMH-backed numbers are facts you can put in a commit message, a PR description, or an ASF mailing list discussion.

This phase gives you the toolkit to measure, interpret, and defend performance claims for JVM code at the level Apache reviewers expect.

Labs