race-condition-test-author

Overview

Two complementary tools: a deterministic test that forces the bad interleaving with a start gate (or a hook inside the critical window), and a detector or stress harness (TSan, go test -race, jcstress) that explores interleavings you did not think of. A race fix is done when a test that failed before the fix passes after it, repeatedly.

When to use

• A defect reproduces only under load on shared in-process state (cache, counter, pool, lazy singleton).
• A race incident needs a regression test before the fix lands.
• Adding a race detector to CI, or finding out why the existing one never fails.

Step 1 - Find the shared mutable state

Check each of: module-level / static mutable variables; singletons with mutable fields; get-then-set on caches or maps without a lock or CAS; per-request data stored on shared objects; pools; lazy initialisation ("if not initialised, initialise"). For each, write down the two operations that must not interleave.

Step 2 - Force the interleaving

A start gate makes both threads enter together; a hook inside the window makes the collision certain instead of likely:

import threading

def test_lazy_init_creates_one_instance(monkeypatch):
    in_window = threading.Barrier(2, timeout=0.5)
    real_build = LazyService._build

    def slow_build(self):
        try:
            in_window.wait()      # unfixed: both threads meet here, inside the window
        except threading.BrokenBarrierError:
            pass                  # fixed: only one thread gets here; carry on
        return real_build(self)

    monkeypatch.setattr(LazyService, "_build", slow_build)
    target = LazyService()
    results, errors = [None, None], []

    def worker(i):
        try:
            results[i] = target.lazy_get()
        except Exception as e:    # surface worker failures to the test thread
            errors.append(e)

    threads = [threading.Thread(target=worker, args=(i,)) for i in (0, 1)]
    for t in threads: t.start()
    for t in threads: t.join(timeout=10)
    assert not errors
    assert results[0] is results[1], "lazy init built two instances"

Verified with a check-then-build LazyService: the unfixed code fails 3/3 runs (both threads meet inside the window and build twice); with a lock around check-and-build it passes 3/3, because only one thread reaches _build and its barrier wait times out after 0.5 s. The same test serves before and after the fix.

Assertions belong on the test thread. An assertion or t.Fatal inside a worker thread, goroutine or executor task can fail silently; collect results and errors and assert after join().

Step 3 - Detectors: TSan and Go -race

C / C++ (ThreadSanitizer (opens in new window)):

clang -fsanitize=thread -g -O1 program.c -o program
TSAN_OPTIONS="halt_on_error=0" ./program; echo "exit=$?"

TSan typically costs 5-15x time and 5-10x memory (ThreadSanitizer (opens in new window)). A report changes the exit status to exitcode (default 66); setting exitcode=0 in CI makes a run that found races pass (TSan flags (opens in new window)). Never suppress a finding to get green: a suppression is for third-party code you cannot change, with a ticket. TSAN_OPTIONS=enable_adaptive_delay=1 injects delays at synchronisation points to expose more interleavings (ThreadSanitizer (opens in new window)).

Go: go test -race ./.... The race detector needs cgo, and typically costs 5-10x memory and 2-20x time (Go race detector (opens in new window)). The full Go workflow (GORACE, -count / -cpu stress, start gates with channels, goroutine-leak gating with goleak) is in references/go.md; goleak filter choices are in references/goleak-filter-options.md. The race detector finds unsynchronised memory access only; it says nothing about goroutines that never exit.

Step 4 - jcstress for JVM publication and reordering

jcstress runs @Actor methods concurrently on a fresh @State object per trial and classifies results with @Outcome (jcstress (opens in new window)). Put the shared field on the state object so every trial publishes it anew, and mark the illegal observation FORBIDDEN so one occurrence fails the run:

import org.openjdk.jcstress.annotations.*;
import org.openjdk.jcstress.infra.results.II_Result;
import static org.openjdk.jcstress.annotations.Expect.*;

@JCStressTest
@Outcome(id = "0, 0", expect = ACCEPTABLE, desc = "Nothing seen yet")
@Outcome(id = "1, 1", expect = ACCEPTABLE, desc = "Both writes seen")
@Outcome(id = "0, 1", expect = ACCEPTABLE, desc = "Only x seen")
@Outcome(id = "1, 0", expect = FORBIDDEN,  desc = "y seen without x")
@State
public class PublicationTest {
    volatile int x, y;           // remove volatile to watch 1, 0 become possible

    @Actor public void writer() { x = 1; y = 1; }
    @Actor public void reader(II_Result r) { r.r1 = y; r.r2 = x; }
}

java -jar target/jcstress.jar -t PublicationTest

A clean jcstress run means the forbidden outcome was not observed in the interleavings explored, not that it cannot happen; say so when closing a ticket.

Step 5 - Volume: virtual threads and property tests

Virtual threads (Java 21+) make thousands of concurrent tasks cheap:

@Test
void placesTenThousandOrdersWithoutLoss() throws Exception {
    var service = new OrderService();
    try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
        var futures = IntStream.range(0, 10_000)
            .mapToObj(i -> executor.submit(() -> service.place(i)))
            .toList();
        for (var f : futures) f.get();   // rethrows task failures here
    }
    assertEquals(10_000, service.totalProcessed());
}

Property-based concurrency (Hypothesis) checks invariants over random operation mixes:

import threading
from hypothesis import assume, given, strategies as st

@given(st.lists(st.tuples(st.sampled_from(["read", "write"]), st.integers()), min_size=10, max_size=100))
def test_counter_ends_on_a_written_value(ops):
    writes = [v for op, v in ops if op == "write"]
    assume(writes)
    counter = ThreadSafeCounter()
    threads = [threading.Thread(target=(lambda v=v: counter.set(v)) if op == "write" else counter.get)
               for op, v in ops]
    for t in threads: t.start()
    for t in threads: t.join()
    assert counter.get() in writes

Volume tests raise the odds; they never prove absence. Pair them with Step 2 for the specific bug.

Step 6 - Checkpoint: red before the fix, green after

1. Run the new test against the unfixed code: it must fail. Record the observed numbers (for example "expected 10,000, got 9,412 over 50 rounds") in the ticket.
2. Apply the fix; run the test repeatedly (-count=50, pytest --count=50 with pytest-repeat, or rounds inside the test): it must pass every time.
3. If step 1 passed, the test does not reach the race. Widen the window (Step 2 hook), move the gate closer to the critical section, or add -cpu 1,2,4 / more rounds, and repeat step 1.

Step 7 - CI

- name: Go race detector
  env: { CGO_ENABLED: "1" }
  run: go test -race -count=3 -cpu 1,4 -timeout 20m ./...

- name: TSan build + test
  run: |
    cmake -B build-tsan -DCMAKE_C_FLAGS="-fsanitize=thread -g -O1" -DCMAKE_CXX_FLAGS="-fsanitize=thread -g -O1"
    cmake --build build-tsan
    ./build-tsan/test_runner        # non-zero (66) when TSan reports

- name: jcstress
  run: java -jar target/jcstress.jar -t PublicationTest

Keep at least one instrumented matrix leg; never mark it continue-on-error, and never set exitcode=0.

Anti-patterns

Limitations

• TSan and -race find data races, not logical races in correctly synchronised multi-step operations; Step 2 tests cover those.
• jcstress is JVM-only.
• Database-level races between transactions belong to mvcc-isolation-tests.

References

• tsan (opens in new window) - clang -fsanitize=thread, runtime options, adaptive delay
• jcstress (opens in new window) - @JCStressTest, @Actor, @Outcome outcomes
• TSan flags (opens in new window) - exitcode (default 66), halt_on_error, suppressions
• Go race detector (opens in new window) - overhead, cgo requirement, GORACE
• ThreadSanitizer C++ Manual - github.com/google/sanitizers/wiki/ThreadSanitizerCppManual
• references/go.md - Go race detector + goleak workflow (GORACE, stress, CI matrix)
• references/goleak-filter-options.md - goleak ignore options
• deadlock-detection-harness - sister skill for deadlock-specific patterns
• async-ordering-tests - sister skill for async/await ordering
• mvcc-isolation-tests - database transaction races