Lab 01 — JMH Microbenchmarks

Goal: Write three benchmark classes with JMH, understand how the annotation processor generates harness code, and verify that each benchmark method returns a meaningful result (preventing Dead Code Elimination).

Prerequisites

• JDK 11+ installed (java -version)
• Maven 3.8+ (mvn -version)

Five-Step Lab

Step 1 — Understand why naive benchmarks lie

Read the HITCHHIKERS-GUIDE section "(1) Why naive benchmarks lie". Pay attention to three pitfalls:

Step 2 — Read the JMH annotations

Open StringConcatBenchmark.java and locate these annotations:

@BenchmarkMode(Mode.AverageTime)   // report mean time per operation
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@State(Scope.Thread)               // one benchmark instance per thread
@Warmup(iterations = 3, time = 1)
@Measurement(iterations = 5, time = 1)
@Fork(1)
public class StringConcatBenchmark {

    @Param({"10", "100"})          // JMH runs the benchmark for each value
    int n;

@State(Scope.Thread) means each thread gets its own benchmark object — safe for single-threaded benchmarks. @State(Scope.Benchmark) shares the object across threads, which is what you want for contention benchmarks.

Step 3 — Run the smoke tests

The smoke tests run a minimal JMH harness (no forks, 1 warm-up iteration, 1 measurement iteration) so they complete in seconds. They verify:

1. The JMH annotation processor generated META-INF/BenchmarkList (required for Runner to find benchmark methods).
2. Each benchmark method executes without throwing.
3. Benchmark outputs are semantically correct (all string-concat variants produce the same string, all lookup variants agree on membership, etc.).

mvn test
# Expected: Tests run: 6, Failures: 0, Errors: 0

Step 4 — Run the real benchmarks

For actual performance numbers use the JMH runner directly. The exec-maven-plugin is configured so you do not need to build a fat jar:

# String concatenation — compare +=, StringBuilder, pre-sized StringBuilder
mvn compile exec:java \
  -Dexec.mainClass="org.openjdk.jmh.Main" \
  -Dexec.args="StringConcatBenchmark -wi 3 -i 5 -f 1"

# Collection lookup — ArrayList.contains vs HashSet.contains vs Arrays.binarySearch
mvn compile exec:java \
  -Dexec.mainClass="org.openjdk.jmh.Main" \
  -Dexec.args="CollectionLookupBenchmark -wi 3 -i 5 -f 1"

# Map access patterns — double-check vs getOrDefault vs computeIfAbsent
mvn compile exec:java \
  -Dexec.mainClass="org.openjdk.jmh.Main" \
  -Dexec.args="MapAccessBenchmark -wi 3 -i 5 -f 1"

# Run everything at once
mvn compile exec:java \
  -Dexec.mainClass="org.openjdk.jmh.Main" \
  -Dexec.args="org.example.perf.* -wi 3 -i 5 -f 1"

Step 5 — Interpret the results

A typical StringConcatBenchmark output:

Benchmark                                    (n)  Mode  Cnt     Score     Error  Units
StringConcatBenchmark.concatenation           10  avgt    5   218.432 ±   4.211  ns/op
StringConcatBenchmark.concatenation          100  avgt    5  9834.711 ± 107.302  ns/op
StringConcatBenchmark.stringBuilder           10  avgt    5    82.104 ±   1.873  ns/op
StringConcatBenchmark.stringBuilder          100  avgt    5   614.332 ±   9.541  ns/op
StringConcatBenchmark.stringBuilderPreSized   10  avgt    5    74.221 ±   1.602  ns/op
StringConcatBenchmark.stringBuilderPreSized  100  avgt    5   512.117 ±   7.208  ns/op

Key observations:

• concatenation degrades quadratically (O(n²) copies) while stringBuilder is O(n).
• Pre-sizing the StringBuilder saves one internal array resize per doubling, a small but consistent win.
• The Error column (± value) is the 99.9% confidence interval. A large error relative to the score means noisy measurement — increase iteration count.

Source Files

src/main/java/org/example/perf/
├── StringConcatBenchmark.java       ← @Param n; three concat strategies
├── CollectionLookupBenchmark.java   ← @Param size; ArrayList vs HashSet vs binarySearch
└── MapAccessBenchmark.java          ← fixed MAP_SIZE=1000; three Map access patterns

src/test/java/org/example/perf/
└── BenchmarkSmokeTest.java          ← 3 JMH smoke tests + 3 correctness assertions

StringConcatBenchmark.java

package org.example.perf;

import org.openjdk.jmh.annotations.*;

import java.util.concurrent.TimeUnit;

/**
 * Benchmarks for string concatenation — one of the most common Java
 * anti-patterns
 * caught in Apache code reviews.
 *
 * <p>
 * The += operator in a loop creates O(n) intermediate String objects because
 * String is immutable. Each iteration allocates a new char[] and copies all
 * previous characters. For n=100, that's 100+99+98+...+1 = ~5,000 char copies.
 *
 * <p>
 * StringBuilder avoids this: it maintains a single resizable char[] buffer.
 * Pre-sizing the buffer (when the final length is estimable) eliminates even
 * the
 * internal array copies when the buffer grows.
 *
 * <p>
 * Expected result (approximate, JVM-dependent):
 * 
 * <pre>
 * Method           n=10   n=100
 * concatenation    ~200ns  ~2,000ns  (O(n²) copies)
 * stringBuilder     ~40ns   ~250ns   (O(n) copies)
 * builderPreSized   ~35ns   ~210ns   (zero resize copies)
 * </pre>
 */
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@State(Scope.Thread)
@Warmup(iterations = 3, time = 1)
@Measurement(iterations = 5, time = 1)
@Fork(1)
public class StringConcatBenchmark {

    /**
     * Number of integers to concatenate.
     * Use small values for smoke tests, larger for accurate benchmarks.
     */
    @Param({ "10", "100" })
    public int n;

    /**
     * Naive +=: creates n intermediate String objects.
     * The JVM CANNOT optimize this across loop iterations because the loop
     * bound is not a compile-time constant and the result of each iteration
     * feeds into the next.
     */
    @Benchmark
    public String concatenation() {
        String s = "";
        for (int i = 0; i < n; i++) {
            s += i;
        }
        return s; // return value prevents dead-code elimination
    }

    /**
     * StringBuilder: single char[] buffer, amortized O(1) append.
     * This is what the JVM compiles a += to INSIDE a single statement
     * (e.g., "a" + b + c), but NOT across loop iterations.
     */
    @Benchmark
    public String stringBuilder() {
        StringBuilder sb = new StringBuilder();
        for (int i = 0; i < n; i++) {
            sb.append(i);
        }
        return sb.toString();
    }

    /**
     * Pre-sized StringBuilder: starts large enough to hold all digits.
     * An average int has ~4 characters (0–9999 for small n), so n*4 is a
     * reasonable initial capacity. This eliminates the internal array doubling
     * that occurs when the StringBuilder's buffer fills up.
     *
     * <p>
     * In practice, the difference vs stringBuilder() is small for moderate n,
     * but matters for n > 10,000 where multiple doublings occur.
     */
    @Benchmark
    public String stringBuilderPreSized() {
        StringBuilder sb = new StringBuilder(n * 4);
        for (int i = 0; i < n; i++) {
            sb.append(i);
        }
        return sb.toString();
    }
}

CollectionLookupBenchmark.java

package org.example.perf;

import org.openjdk.jmh.annotations.*;

import java.util.*;
import java.util.concurrent.TimeUnit;

/**
 * Benchmarks for collection membership testing — a classic O(n) vs O(1)
 * comparison.
 *
 * <p>
 * Apache codebases often contain patterns like:
 * 
 * <pre>
 *   List&lt;String&gt; allowed = Arrays.asList("GET", "POST", "PUT");
 *   if (allowed.contains(method)) { ... }
 * </pre>
 * 
 * For small lists, this is fine. But when the list grows (e.g., a deny-list of
 * 1,000
 * known-bad IPs), the O(n) scan becomes a hot path bottleneck.
 *
 * <p>
 * This benchmark measures three strategies:
 * <ol>
 * <li>ArrayList.contains — linear scan, O(n) per lookup</li>
 * <li>HashSet.contains — hash lookup, O(1) per lookup</li>
 * <li>Arrays.binarySearch — O(log n) on a sorted primitive array</li>
 * </ol>
 *
 * <p>
 * Expected result:
 * 
 * <pre>
 * Method             size=100  size=10000
 * arrayListContains   ~200ns    ~15,000ns  (n/2 comparisons on average)
 * hashSetContains       ~15ns       ~15ns  (hash + single equals)
 * binarySearch          ~10ns       ~25ns  (log₂ comparisons)
 * </pre>
 */
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@State(Scope.Thread)
@Warmup(iterations = 3, time = 1)
@Measurement(iterations = 5, time = 1)
@Fork(1)
public class CollectionLookupBenchmark {

    @Param({ "100", "10000" })
    public int size;

    private List<Integer> arrayList;
    private Set<Integer> hashSet;
    private int[] sortedArray;
    private Integer target; // target is always in the middle (worst avg case for ArrayList)

    @Setup
    public void setup() {
        arrayList = new ArrayList<>(size);
        hashSet = new HashSet<>(size * 2); // 0.5 load factor avoids rehashing
        sortedArray = new int[size];
        for (int i = 0; i < size; i++) {
            arrayList.add(i);
            hashSet.add(i);
            sortedArray[i] = i;
        }
        target = size / 2;
    }

    /**
     * O(n) linear scan — ArrayList.contains calls equals() on each element.
     * With target at index size/2, this scans ~half the list on every call.
     */
    @Benchmark
    public boolean arrayListContains() {
        return arrayList.contains(target);
    }

    /**
     * O(1) hash lookup — computes hashCode, finds bucket, checks equals.
     * Performance is largely independent of size (assuming a good hash function
     * and low load factor).
     */
    @Benchmark
    public boolean hashSetContains() {
        return hashSet.contains(target);
    }

    /**
     * O(log n) binary search on a sorted primitive array.
     * Faster than ArrayList.contains for large n, and avoids Integer boxing
     * (compares int primitives directly).
     * Returns a non-negative index on hit (our "contains" equivalent).
     */
    @Benchmark
    public int binarySearch() {
        return Arrays.binarySearch(sortedArray, target);
    }
}

MapAccessBenchmark.java

package org.example.perf;

import org.openjdk.jmh.annotations.*;

import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.TimeUnit;

/**
 * Benchmarks for common Map access patterns.
 *
 * <p>
 * A frequent Apache review comment: "you're doing two map lookups where one
 * would suffice." This benchmark measures the cost difference.
 *
 * <p>
 * Three patterns compared:
 * <ol>
 * <li>{@code containsKey} then {@code get} — two hash lookups per access</li>
 * <li>{@code getOrDefault} — one hash lookup, returns default on miss</li>
 * <li>{@code computeIfAbsent} — one hash lookup, creates value on miss</li>
 * </ol>
 *
 * <p>
 * Expected result (all hits, map size 1000):
 * 
 * <pre>
 * Method              Score
 * containsThenGet     ~30ns  (two bucket scans + two equals)
 * getOrDefault        ~15ns  (one bucket scan + one equals)
 * computeIfAbsent     ~18ns  (one bucket scan + lambda overhead on hit)
 * </pre>
 *
 * <p>
 * The lesson: prefer {@code getOrDefault} for simple default values.
 * Use {@code computeIfAbsent} for lazy initialization (the lambda is only
 * invoked on miss, so it has minimal overhead on the hit path).
 */
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@State(Scope.Thread)
@Warmup(iterations = 3, time = 1)
@Measurement(iterations = 5, time = 1)
@Fork(1)
public class MapAccessBenchmark {

    private static final int MAP_SIZE = 1_000;
    private static final int DEFAULT_VAL = -1;

    private Map<String, Integer> map;
    private String[] keys;
    private int idx;

    @Setup
    public void setup() {
        map = new HashMap<>(MAP_SIZE * 2);
        keys = new String[MAP_SIZE];
        for (int i = 0; i < MAP_SIZE; i++) {
            keys[i] = "key-" + i;
            map.put(keys[i], i);
        }
        idx = 0;
    }

    /**
     * Two-call pattern: containsKey + get.
     * Both calls compute the hash and scan the bucket chain.
     * On a hit, this is exactly twice the work of a single get.
     */
    @Benchmark
    public int containsThenGet() {
        String key = keys[idx++ % MAP_SIZE];
        if (map.containsKey(key)) {
            return map.get(key);
        }
        return DEFAULT_VAL;
    }

    /**
     * One-call pattern: getOrDefault.
     * Single hash lookup. On a hit, returns the value; on a miss, returns default.
     * Preferred over containsKey+get when a default value is acceptable.
     */
    @Benchmark
    public int getOrDefault() {
        String key = keys[idx++ % MAP_SIZE];
        return map.getOrDefault(key, DEFAULT_VAL);
    }

    /**
     * computeIfAbsent: one lookup, creates the value only on a miss.
     * On a hit (99% of the time in this benchmark), the lambda is NOT called —
     * the overhead is just one extra null-check vs getOrDefault.
     *
     * <p>
     * Use this for lazy initialization of expensive objects (connections,
     * compiled patterns, etc.), NOT for simple int defaults.
     */
    @Benchmark
    public int computeIfAbsent() {
        String key = keys[idx++ % MAP_SIZE];
        return map.computeIfAbsent(key, k -> DEFAULT_VAL);
    }
}

BenchmarkSmokeTest.java

package org.example.perf;

import org.junit.jupiter.api.Test;
import org.openjdk.jmh.results.RunResult;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.ChainedOptionsBuilder;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;
import org.openjdk.jmh.runner.options.TimeValue;

import java.util.Collection;

import static org.junit.jupiter.api.Assertions.*;

/**
 * Smoke tests that verify each benchmark class:
 * 1. Compiles (the annotation processor wired the JMH infrastructure correctly)
 * 2. Runs without throwing exceptions
 * 3. Produces at least one result
 *
 * <p>
 * These are NOT performance tests — the numbers at 200ms iterations are
 * meaningless.
 * Their purpose is to catch compilation errors and framework misconfiguration
 * in CI.
 *
 * <p>
 * For real benchmarks, run:
 * 
 * <pre>
 *   mvn compile exec:java \
 *     -Dexec.mainClass="org.openjdk.jmh.Main" \
 *     -Dexec.args="StringConcatBenchmark -wi 3 -i 5 -f 1"
 * </pre>
 */
class BenchmarkSmokeTest {

    /**
     * Builds minimal JMH options for smoke testing:
     * - 1 warm-up iteration (200ms) — enough for the JIT to stabilize
     * - 1 measurement iteration (200ms) — enough to produce a result
     * - forks(0) — run in the same JVM as JUnit (avoids spawning a child process)
     *
     * <p>
     * forks(0) means the JUnit JVM IS the benchmark JVM. This is less accurate
     * (JUnit's own code affects JIT state) but is fine for smoke testing.
     */
    private static ChainedOptionsBuilder minimal(String benchmarkClass) {
        return new OptionsBuilder()
                .include(benchmarkClass)
                .warmupIterations(1)
                .warmupTime(TimeValue.milliseconds(200))
                .measurementIterations(1)
                .measurementTime(TimeValue.milliseconds(200))
                .forks(0)
                .shouldFailOnError(true);
    }

    @Test
    void stringConcatBenchmarkRuns() throws RunnerException {
        Options opts = minimal(StringConcatBenchmark.class.getSimpleName())
                .param("n", "10") // only the smallest param — keeps test fast
                .build();

        Collection<RunResult> results = new Runner(opts).run();

        assertFalse(results.isEmpty(), "StringConcatBenchmark must produce results");
        // Each benchmark method × 1 param value = 3 results
        assertEquals(3, results.size(),
                "Expected concatenation + stringBuilder + stringBuilderPreSized");
    }

    @Test
    void collectionLookupBenchmarkRuns() throws RunnerException {
        Options opts = minimal(CollectionLookupBenchmark.class.getSimpleName())
                .param("size", "100")
                .build();

        Collection<RunResult> results = new Runner(opts).run();

        assertFalse(results.isEmpty(), "CollectionLookupBenchmark must produce results");
        assertEquals(3, results.size(),
                "Expected arrayListContains + hashSetContains + binarySearch");
    }

    @Test
    void mapAccessBenchmarkRuns() throws RunnerException {
        Options opts = minimal(MapAccessBenchmark.class.getSimpleName())
                .build();

        Collection<RunResult> results = new Runner(opts).run();

        assertFalse(results.isEmpty(), "MapAccessBenchmark must produce results");
        assertEquals(3, results.size(),
                "Expected containsThenGet + getOrDefault + computeIfAbsent");
    }

    /**
     * Verifies that all three string-concat variants produce identical output.
     * This confirms the "fast" implementations are not computing different things.
     *
     * <p>
     * This is a correctness test, not a performance test.
     */
    @Test
    void stringConcatVariantsProduceSameResult() {
        StringConcatBenchmark b = new StringConcatBenchmark();
        b.n = 50;

        String naive = b.concatenation();
        String builder = b.stringBuilder();
        String presized = b.stringBuilderPreSized();

        assertEquals(naive, builder, "StringBuilder must produce the same string as +=");
        assertEquals(naive, presized, "Pre-sized StringBuilder must produce the same string as +=");
    }

    /**
     * Verifies HashSet and sorted-array binary search agree with ArrayList on
     * membership.
     */
    @Test
    void collectionLookupVariantsAgreeOnMembership() {
        CollectionLookupBenchmark b = new CollectionLookupBenchmark();
        b.size = 200;
        b.setup();

        // target is always present (size/2 = 100, which is in 0..199)
        assertTrue(b.arrayListContains(), "ArrayList must find target");
        assertTrue(b.hashSetContains(), "HashSet must find target");
        assertTrue(b.binarySearch() >= 0, "Binary search must find target");
    }

    /**
     * Verifies map access variants return the same value for a present key.
     */
    @Test
    void mapAccessVariantsReturnSameValueForPresentKey() {
        MapAccessBenchmark b = new MapAccessBenchmark();
        b.setup();

        int r1 = b.containsThenGet();
        b.setup(); // reset idx
        int r2 = b.getOrDefault();
        b.setup();
        int r3 = b.computeIfAbsent();

        assertEquals(r1, r2, "getOrDefault must return same value as containsKey+get");
        assertEquals(r1, r3, "computeIfAbsent must return same value as containsKey+get");
        assertTrue(r1 >= 0, "Result must be a valid map value (not the -1 default)");
    }
}

How JMH Works Under the Hood

When you run mvn compile, the jmh-generator-annprocess annotation processor reads every @Benchmark method and generates:

1. target/classes/META-INF/BenchmarkList — a text file listing every benchmark method. The Runner reads this at startup to discover what to run.
2. target/generated-sources/annotations/org/example/perf/jmh_generated/ — one *_jmhTest.java per benchmark method, containing the actual timing loop, blackhole sinks, and thread setup.

This is why mvn test must compile before testing: a stale target/ from a previous run (without annotation processing) will be missing BenchmarkList. mvn clean test guarantees a fresh run.

Interview Talking Points

• Why return a value from @Benchmark? — forces the JIT to keep the computation; otherwise Dead Code Elimination can eliminate the entire method.
• What is @State? — controls object lifecycle and visibility; Scope.Thread gives each thread its own object (no sharing, no contention), Scope.Benchmark shares one object across all threads.
• When would you use @Fork(0) vs @Fork(1)? — @Fork(0) runs in the test JVM (fast, debuggable, but JIT may be polluted by test framework); @Fork(1) spawns a fresh JVM for a clean JIT slate (required for production measurements).
• What does the Error column mean? — it is the half-width of the 99.9% confidence interval. A large error means high variance — check for OS noise, GC pauses, or thermal throttling.