Testland
Browse all skills & agents

race-condition-test-author

Build deterministic race-condition tests - identify shared mutable state, drive interleavings via barriers / latches / manual scheduling; use ThreadSanitizer (clang `-fsanitize=thread`) for C/C++ data race detection; run the Go race detector end-to-end (`go test -race`, GORACE tuning, `-count`/`-cpu` stress, goroutine-leak gating with go.uber.org/goleak - references/go.md); use jcstress (`@JCStressTest` + `@Actor` + `@Outcome`) for JVM stress; use Loom virtual-thread interleavings for parallel testing. Use when a defect only reproduces under load on shared in-process state (cache, counter, connection pool, lazy-init singleton), when writing the regression test for a race-condition incident before the fix lands, or when adding `-race` to a Go CI matrix.

Install with skills.sh (any agent)

npx skills add testland/qa --skill race-condition-test-author
View source

race-condition-test-author

Race conditions are the canonical "works on my machine, breaks under load" bug. Tests must drive shared-state access deterministically (barriers, latches) AND non-deterministically (sanitizers, stress) to expose them.

When to use

  • Service handles concurrent requests on shared in-process state (caches, counters, connection pools).
  • Race-condition incident retro: build a regression test before fixing.
  • Pre-release smoke: TSan-instrumented binary catches data races that compile clean.

Step 1 - Identify shared mutable state

Code-review checklist:

  • Static / global / module-level mutable variables
  • Singleton instances with mutable fields
  • Cache structures with read-and-modify ("get-then-set" without CAS)
  • Per-request state stored on shared objects (mutable request context)
  • Connection pools / object pools without internal lock
  • Lazy-init patterns ("if not initialized, initialize")

For each: ask "what if two threads / goroutines / async tasks hit this concurrently?"

Step 2 - Deterministic interleaving via barriers

import threading

def test_lazy_init_thread_safe():
    target = LazyService()  # has private _instance + lazy_get()
    barrier = threading.Barrier(parties=2)
    results = [None, None]

    def worker(idx):
        barrier.wait()  # both threads stop here, then race
        results[idx] = target.lazy_get()

    t1 = threading.Thread(target=worker, args=(0,))
    t2 = threading.Thread(target=worker, args=(1,))
    t1.start(); t2.start()
    t1.join(); t2.join()

    assert results[0] is results[1], "Lazy init created two instances under race"

Barrier ensures both threads start the contended section at the same time - much higher probability of triggering the race than naive threading.Thread().

Step 3 - ThreadSanitizer for C/C++/Go

Per the ThreadSanitizer docs (opens in new window), TSan detects data races at runtime with ~5-15× overhead. For C/C++:

clang -fsanitize=thread -g -O1 program.c -o program
./program

For Go, the native data race detector:

go test -race ./...
go run -race main.go

The full Go workflow - GORACE options (log_path, halt_on_error, history_size), stress amplification via -count/-cpu, go vet loop-capture checks, goroutine-leak gating with goleak, and the CI matrix - is in references/go.md.

Output for a detected race:

WARNING: DATA RACE
Read at 0x... by goroutine 7:
  main.read+0x...
    main.go:42

Previous write at 0x... by goroutine 6:
  main.write+0x...
    main.go:38

Per the ThreadSanitizer docs (opens in new window), adaptive delay injection (TSAN_OPTIONS=enable_adaptive_delay=1) helps surface races at synchronization points.

Step 4 - jcstress for JVM

Per the jcstress docs (opens in new window), jcstress is "the experimental harness ... for the correctness of concurrency support in the JVM."

@JCStressTest
@Outcome(id = "0, 0", expect = ACCEPTABLE, desc = "Initial values")
@Outcome(id = "1, 1", expect = ACCEPTABLE, desc = "Both writes seen")
@Outcome(id = "0, 1", expect = ACCEPTABLE, desc = "Saw partial")
@Outcome(id = "1, 0", expect = FORBIDDEN, desc = "Reordered - bug")
@State
public class CounterTest {
    int x, y;

    @Actor
    public void writer() { x = 1; y = 1; }

    @Actor
    public void reader(II_Result r) {
        r.r1 = y;
        r.r2 = x;
    }
}

@Actor methods run concurrently on different threads; @Outcome classifies observed (r1, r2) pairs. FORBIDDEN outcomes indicate a memory-model violation (in this case, reordering allowed under JMM unless volatile or final).

Run:

java -jar jcstress.jar -m quick CounterTest

Per the jcstress docs (opens in new window): tests are probabilistic; longer runs find more reorderings.

Step 5 - Loom virtual-thread interleavings (Java 21+)

Java 21+ Project Loom enables cheap virtual threads. Use to test many-concurrent-task interleavings without OS thread cost:

@Test
void test_handles_10000_concurrent_orders() throws Exception {
    var orderService = new OrderService();
    try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
        var futures = IntStream.range(0, 10_000)
            .mapToObj(i -> executor.submit(() -> orderService.place(i)))
            .toList();
        for (var f : futures) f.get();
    }
    assertEquals(10_000, orderService.totalProcessed());
}

Virtual threads run M:N on OS threads - the JVM scheduler interleaves them aggressively, exposing schedule-dependent races.

Step 6 - Property-based + concurrency

Combine Hypothesis (Python) / fast-check (JS) with concurrency:

from hypothesis import given, strategies as st
import threading

@given(operations=st.lists(st.tuples(st.sampled_from(["read", "write"]), st.integers()), min_size=10, max_size=100))
def test_counter_property_under_concurrency(operations):
    counter = ThreadSafeCounter()
    threads = []
    for op, val in operations:
        if op == "write":
            threads.append(threading.Thread(target=lambda v=val: counter.set(v)))
        else:
            threads.append(threading.Thread(target=counter.get))

    for t in threads: t.start()
    for t in threads: t.join()

    # Property: final value is one of the written values
    assert counter.get() in [v for op, v in operations if op == "write"]

Cross-ref qa-property-based plugin for property-based test authoring patterns.

Step 7 - CI integration

# Go
- name: Run race detector
  run: go test -race ./...

# C/C++
- name: TSan build + test
  run: |
    cmake -B build -DCMAKE_C_FLAGS="-fsanitize=thread -g -O1"
    cmake --build build
    ./build/test_runner

# Java
- name: jcstress quick run
  run: java -jar jcstress.jar -m quick -r results/

Note the latency cost: go test -race ~3× slower; jcstress quick mode ~minutes per test class.

Anti-patterns

Anti-patternWhy it failsFix
time.sleep(0.001) to "force" interleavingNon-deterministic; flakeUse barriers (Step 2)
Run race tests once and assume greenProbabilistic; some races take hoursMultiple runs OR longer runs OR sanitizers
Skip TSan in CI for "release" buildsRace in release; CI passed without -raceTSan in CI for at least one matrix dimension (Step 7)
Depend on assertion in worker threadThread death silent; main thread sees passUse futures + assert from main
Test only the bug-causing race, not similarOther shared state has same pattern; bugs shipCode-review checklist (Step 1)

Limitations

  • TSan finds only data races (unsynchronized concurrent access), not higher-level race conditions (correct-but-non-atomic multi-step operations).
  • jcstress is JVM-only; no equivalent for non-JVM stacks.
  • Go's -race overhead is real (~5-10× memory); CI matrix budget required.
  • Loom virtual threads (Java 21+) require recent JDK; older versions need OS threads.

References

  • ThreadSanitizer docs (opens in new window) - clang -fsanitize=thread, runtime options, adaptive delay
  • jcstress docs (opens in new window) - @JCStressTest, @Actor, @Outcome outcomes
  • ThreadSanitizer C++ Manual - github.com/google/sanitizers/wiki/ThreadSanitizerCppManual
  • references/go.md - Go race detector + goleak workflow (GORACE, stress, CI matrix)
  • deadlock-detection-harness - sister skill for deadlock-specific patterns
  • async-ordering-tests - sister skill for async/await ordering

Go - race detector workflow (GORACE, stress, goleak)

View source (opens in new window)

Go - race detector workflow (GORACE, stress, goleak)

The Go race detector (ThreadSanitizer compiled into the binary via -race) and goroutine leaks are two separate failure classes: the detector finds concurrent unsynchronized access, goleak finds goroutines that never stop. Run both.

Enable the race detector

Per go.dev/doc/articles/race_detector (opens in new window):

go test -race ./...
go run  -race main.go
go build -race ./cmd/server

Requires cgo and a C compiler on Linux/FreeBSD/Windows (mingw-w64 v8+ on Windows; Darwin ships its own). Expected overhead per the same doc: 2-20x execution time, 5-10x memory, plus 8 bytes per defer/recover accumulating until the goroutine exits - budget CI timeouts accordingly.

A detected race prints the conflicting read/write stacks and goroutine creation sites to stderr (WARNING: DATA RACE); fix by protecting every access to the address with the same primitive (mutex, atomic, or channel hand-off).

GORACE options

GORACE="log_path=/tmp/race/report halt_on_error=1 history_size=2" go test -race ./...
OptionDefaultWhen to change
log_pathstderrFile path so CI can archive race reports as artifacts
halt_on_error01 stops on the first race; local debugging
history_size1Raise to 2-7 when report stacks look truncated (memory cost)
strip_path_prefix""Make report lines repo-relative
exitcode66Override for CI exit-code conventions

Stress with -count and -cpu

The detector only fires on races that actually execute; a single run can miss a real race. Amplify interleaving diversity:

go test -race -count=10 ./...          # 10 runs per package
go test -race -cpu=1,2,4,8 ./...       # re-run per GOMAXPROCS value
go test -race -count=5 -cpu=1,2,4 ./...

GOMAXPROCS=1 surfaces sequencing bugs; higher values surface true parallel races.

go vet - loop-variable capture

Per pkg.go.dev/cmd/vet (opens in new window) (loopclosure), go vet flags goroutines closing over a range variable - the classic pre-Go-1.22 capture race (for _, v := range items { go func() { process(v) }() }). Run go vet ./... before -race to filter this class early; Go 1.22+ makes range variables per-iteration, but audit code that may build on older toolchains.

goleak - goroutine-leak detection

Per github.com/uber-go/goleak (opens in new window) (go get -u go.uber.org/goleak):

func TestWorkerPool(t *testing.T) {
    defer goleak.VerifyNone(t)     // fails if any goroutine outlives the test
    pool := NewWorkerPool(4)
    pool.Submit(func() { /* work */ })
    pool.Shutdown()
}

VerifyNone is incompatible with t.Parallel() - goleak cannot attribute goroutines to parallel sub-tests. For parallel packages wrap the runner:

func TestMain(m *testing.M) { goleak.VerifyTestMain(m) }

Silence expected library goroutines by top-of-stack function:

goleak.VerifyNone(t, goleak.IgnoreTopFunction("database/sql.(*DB).connectionOpener"))

Full filter catalog (IgnoreAnyFunction, IgnoreCurrent, Cleanup) in goleak-filter-options.md (opens in new window).

CI matrix

Gate at least one matrix dimension with -race (per the race-detector doc: "It is recommended to always run race-enabled tests"):

jobs:
  test:
    strategy:
      matrix:
        go-version: ["1.22", "1.23"]
        race: ["", "-race"]
    steps:
      - uses: actions/setup-go@v5
        with: { go-version: "${{ matrix.go-version }}" }
      - env: { GORACE: "log_path=/tmp/race/report halt_on_error=0" }
        run: |
          go vet ./...
          go test ${{ matrix.race }} -count=3 -cpu=1,4 -timeout=10m ./...
      - if: failure()
        uses: actions/upload-artifact@v4
        with: { name: race-reports, path: /tmp/race/report* }

Set -timeout to 5-10x the non-race run time; upload log_path files on failure.

Anti-patterns

Anti-patternWhy it failsFix
Run -race once, see no output, shipDetector only finds races that executed-count / -cpu matrix
Skip -race for "release" builds in CIRace appears in prod, not CIGate one matrix dimension with -race
VerifyNone with t.Parallel()goleak can't attribute goroutinesVerifyTestMain
IgnoreCurrent() at package initSnapshot masks leaks added before each testCall inside each test function
Trust -race to catch goroutine leaksDifferent failure classAdd goleak; the gates are complementary
history_size=7 always128K history per goroutine can OOM CIStart at 1; raise only on truncated stacks

Limitations

  • Cross-compiled binaries (GOOS/GOARCH differing from host) won't run -race unless the target toolchain supports TSan; CGO_ENABLED=0 builds exclude it entirely.
  • goleak supports only the two most recent Go minor versions.
  • goleak has a brief internal retry, but slow-starting background goroutines can false-positive; use IgnoreTopFunction.

References

goleak filter options

Filter options for goleak.VerifyNone / goleak.VerifyTestMain, per pkg.go.dev/go.uber.org/goleak (opens in new window). Pass one or more as trailing arguments to suppress goroutines that are expected rather than leaked.

IgnoreTopFunction

Ignores any goroutine whose top-of-stack frame is the named function. Prefer this when the library goroutine is identifiable by name.

goleak.VerifyNone(t,
    goleak.IgnoreTopFunction("database/sql.(*DB).connectionOpener"),
)

IgnoreAnyFunction (v1.3.0+)

Ignores any goroutine whose stack contains the named function at any depth, not just the top frame. Use when the identifying frame is not at the top.

goleak.VerifyNone(t,
    goleak.IgnoreAnyFunction("google.golang.org/grpc.(*ccBalancerWrapper).watcher"),
)

IgnoreCurrent

Snapshots the goroutines already running at call time and ignores exactly those at verification.

opt := goleak.IgnoreCurrent()
// ... test logic ...
goleak.VerifyNone(t, opt)

Prefer IgnoreTopFunction over IgnoreCurrent when the library goroutine is identifiable by name: IgnoreCurrent silences goroutines that were already running at snapshot time, which can mask leaks introduced before the snapshot.

Cleanup

goleak.Cleanup(func(int)) registers a function goleak calls with the exit code after the leak check, e.g. to log instead of failing the process. Used with VerifyTestMain when the default exit behavior needs to change.

References

Related skills

async-ordering-tests

Test async ordering - event-loop / queue / channel ordering assertions, JS Promise microtask vs macrotask ordering, Python `asyncio.gather` vs `asyncio.wait_for` semantics, Go goroutine + channel happens-before relationships, async/await re-entrancy. Use deterministic schedulers (sinon fake timers, asyncio test mode) to remove run-to-run variance. Use when a callback fires twice, a later response overwrites an earlier one, or a cancelled parent task leaves a child still running - bugs where completion order, not shared memory, is the defect.

deadlock-detection-harness

Build deadlock-detection harnesses - extract lock-acquire-order graph via instrumentation, run cycle detection (DFS) to spot inconsistent ordering, use lock-acquire timeouts to surface rather than hang, JVM `jstack` / `gdb thread apply all bt` for postmortem analysis. Pair with ThreadSanitizer's `detect_deadlocks=1` for runtime detection. Use when a service that holds two or more locks hangs in production with no crash or error, or before release when lock acquisition order across code paths has never been proven consistent.

mvcc-isolation-tests

Build per-database MVCC isolation-level tests - Read Uncommitted vs Read Committed vs Repeatable Read vs Serializable; verify which anomalies are prevented at each level (dirty read, non-repeatable read, phantom read, serialization anomaly, write skew). Per PostgreSQL transaction isolation docs; analogous patterns for MySQL InnoDB, SQL Server, and DynamoDB. Use when two concurrent transactions can touch the same rows (balance debit, seat booking, stock decrement), or before changing a service's default isolation level.