race-condition-test-author
Build deterministic race-condition tests - identify shared mutable state, drive interleavings via barriers / latches / manual scheduling; use ThreadSanitizer (clang `-fsanitize=thread`) for C/C++ data race detection; run the Go race detector end-to-end (`go test -race`, GORACE tuning, `-count`/`-cpu` stress, goroutine-leak gating with go.uber.org/goleak - references/go.md); use jcstress (`@JCStressTest` + `@Actor` + `@Outcome`) for JVM stress; use Loom virtual-thread interleavings for parallel testing. Use when a defect only reproduces under load on shared in-process state (cache, counter, connection pool, lazy-init singleton), when writing the regression test for a race-condition incident before the fix lands, or when adding `-race` to a Go CI matrix.
Install with skills.sh (any agent)
npx skills add testland/qa --skill race-condition-test-authorrace-condition-test-author
Race conditions are the canonical "works on my machine, breaks under load" bug. Tests must drive shared-state access deterministically (barriers, latches) AND non-deterministically (sanitizers, stress) to expose them.
When to use
Step 1 - Identify shared mutable state
Code-review checklist:
For each: ask "what if two threads / goroutines / async tasks hit this concurrently?"
Step 2 - Deterministic interleaving via barriers
import threading
def test_lazy_init_thread_safe():
target = LazyService() # has private _instance + lazy_get()
barrier = threading.Barrier(parties=2)
results = [None, None]
def worker(idx):
barrier.wait() # both threads stop here, then race
results[idx] = target.lazy_get()
t1 = threading.Thread(target=worker, args=(0,))
t2 = threading.Thread(target=worker, args=(1,))
t1.start(); t2.start()
t1.join(); t2.join()
assert results[0] is results[1], "Lazy init created two instances under race"Barrier ensures both threads start the contended section at the same time - much higher probability of triggering the race than naive threading.Thread().
Step 3 - ThreadSanitizer for C/C++/Go
Per the ThreadSanitizer docs (opens in new window), TSan detects data races at runtime with ~5-15× overhead. For C/C++:
clang -fsanitize=thread -g -O1 program.c -o program
./programFor Go, the native data race detector:
go test -race ./...
go run -race main.goThe full Go workflow - GORACE options (log_path, halt_on_error, history_size), stress amplification via -count/-cpu, go vet loop-capture checks, goroutine-leak gating with goleak, and the CI matrix - is in references/go.md.
Output for a detected race:
WARNING: DATA RACE
Read at 0x... by goroutine 7:
main.read+0x...
main.go:42
Previous write at 0x... by goroutine 6:
main.write+0x...
main.go:38Per the ThreadSanitizer docs (opens in new window), adaptive delay injection (TSAN_OPTIONS=enable_adaptive_delay=1) helps surface races at synchronization points.
Step 4 - jcstress for JVM
Per the jcstress docs (opens in new window), jcstress is "the experimental harness ... for the correctness of concurrency support in the JVM."
@JCStressTest
@Outcome(id = "0, 0", expect = ACCEPTABLE, desc = "Initial values")
@Outcome(id = "1, 1", expect = ACCEPTABLE, desc = "Both writes seen")
@Outcome(id = "0, 1", expect = ACCEPTABLE, desc = "Saw partial")
@Outcome(id = "1, 0", expect = FORBIDDEN, desc = "Reordered - bug")
@State
public class CounterTest {
int x, y;
@Actor
public void writer() { x = 1; y = 1; }
@Actor
public void reader(II_Result r) {
r.r1 = y;
r.r2 = x;
}
}@Actor methods run concurrently on different threads; @Outcome classifies observed (r1, r2) pairs. FORBIDDEN outcomes indicate a memory-model violation (in this case, reordering allowed under JMM unless volatile or final).
Run:
java -jar jcstress.jar -m quick CounterTestPer the jcstress docs (opens in new window): tests are probabilistic; longer runs find more reorderings.
Step 5 - Loom virtual-thread interleavings (Java 21+)
Java 21+ Project Loom enables cheap virtual threads. Use to test many-concurrent-task interleavings without OS thread cost:
@Test
void test_handles_10000_concurrent_orders() throws Exception {
var orderService = new OrderService();
try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
var futures = IntStream.range(0, 10_000)
.mapToObj(i -> executor.submit(() -> orderService.place(i)))
.toList();
for (var f : futures) f.get();
}
assertEquals(10_000, orderService.totalProcessed());
}Virtual threads run M:N on OS threads - the JVM scheduler interleaves them aggressively, exposing schedule-dependent races.
Step 6 - Property-based + concurrency
Combine Hypothesis (Python) / fast-check (JS) with concurrency:
from hypothesis import given, strategies as st
import threading
@given(operations=st.lists(st.tuples(st.sampled_from(["read", "write"]), st.integers()), min_size=10, max_size=100))
def test_counter_property_under_concurrency(operations):
counter = ThreadSafeCounter()
threads = []
for op, val in operations:
if op == "write":
threads.append(threading.Thread(target=lambda v=val: counter.set(v)))
else:
threads.append(threading.Thread(target=counter.get))
for t in threads: t.start()
for t in threads: t.join()
# Property: final value is one of the written values
assert counter.get() in [v for op, v in operations if op == "write"]Cross-ref qa-property-based plugin for property-based test authoring patterns.
Step 7 - CI integration
# Go
- name: Run race detector
run: go test -race ./...
# C/C++
- name: TSan build + test
run: |
cmake -B build -DCMAKE_C_FLAGS="-fsanitize=thread -g -O1"
cmake --build build
./build/test_runner
# Java
- name: jcstress quick run
run: java -jar jcstress.jar -m quick -r results/Note the latency cost: go test -race ~3× slower; jcstress quick mode ~minutes per test class.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
time.sleep(0.001) to "force" interleaving | Non-deterministic; flake | Use barriers (Step 2) |
| Run race tests once and assume green | Probabilistic; some races take hours | Multiple runs OR longer runs OR sanitizers |
| Skip TSan in CI for "release" builds | Race in release; CI passed without -race | TSan in CI for at least one matrix dimension (Step 7) |
| Depend on assertion in worker thread | Thread death silent; main thread sees pass | Use futures + assert from main |
| Test only the bug-causing race, not similar | Other shared state has same pattern; bugs ship | Code-review checklist (Step 1) |
Limitations
References
Go - race detector workflow (GORACE, stress, goleak)
View source (opens in new window)Go - race detector workflow (GORACE, stress, goleak)
The Go race detector (ThreadSanitizer compiled into the binary via -race) and goroutine leaks are two separate failure classes: the detector finds concurrent unsynchronized access, goleak finds goroutines that never stop. Run both.
Enable the race detector
Per go.dev/doc/articles/race_detector (opens in new window):
go test -race ./...
go run -race main.go
go build -race ./cmd/serverRequires cgo and a C compiler on Linux/FreeBSD/Windows (mingw-w64 v8+ on Windows; Darwin ships its own). Expected overhead per the same doc: 2-20x execution time, 5-10x memory, plus 8 bytes per defer/recover accumulating until the goroutine exits - budget CI timeouts accordingly.
A detected race prints the conflicting read/write stacks and goroutine creation sites to stderr (WARNING: DATA RACE); fix by protecting every access to the address with the same primitive (mutex, atomic, or channel hand-off).
GORACE options
GORACE="log_path=/tmp/race/report halt_on_error=1 history_size=2" go test -race ./...| Option | Default | When to change |
|---|---|---|
log_path | stderr | File path so CI can archive race reports as artifacts |
halt_on_error | 0 | 1 stops on the first race; local debugging |
history_size | 1 | Raise to 2-7 when report stacks look truncated (memory cost) |
strip_path_prefix | "" | Make report lines repo-relative |
exitcode | 66 | Override for CI exit-code conventions |
Stress with -count and -cpu
The detector only fires on races that actually execute; a single run can miss a real race. Amplify interleaving diversity:
go test -race -count=10 ./... # 10 runs per package
go test -race -cpu=1,2,4,8 ./... # re-run per GOMAXPROCS value
go test -race -count=5 -cpu=1,2,4 ./...GOMAXPROCS=1 surfaces sequencing bugs; higher values surface true parallel races.
go vet - loop-variable capture
Per pkg.go.dev/cmd/vet (opens in new window) (loopclosure), go vet flags goroutines closing over a range variable - the classic pre-Go-1.22 capture race (for _, v := range items { go func() { process(v) }() }). Run go vet ./... before -race to filter this class early; Go 1.22+ makes range variables per-iteration, but audit code that may build on older toolchains.
goleak - goroutine-leak detection
Per github.com/uber-go/goleak (opens in new window) (go get -u go.uber.org/goleak):
func TestWorkerPool(t *testing.T) {
defer goleak.VerifyNone(t) // fails if any goroutine outlives the test
pool := NewWorkerPool(4)
pool.Submit(func() { /* work */ })
pool.Shutdown()
}VerifyNone is incompatible with t.Parallel() - goleak cannot attribute goroutines to parallel sub-tests. For parallel packages wrap the runner:
func TestMain(m *testing.M) { goleak.VerifyTestMain(m) }Silence expected library goroutines by top-of-stack function:
goleak.VerifyNone(t, goleak.IgnoreTopFunction("database/sql.(*DB).connectionOpener"))Full filter catalog (IgnoreAnyFunction, IgnoreCurrent, Cleanup) in goleak-filter-options.md (opens in new window).
CI matrix
Gate at least one matrix dimension with -race (per the race-detector doc: "It is recommended to always run race-enabled tests"):
jobs:
test:
strategy:
matrix:
go-version: ["1.22", "1.23"]
race: ["", "-race"]
steps:
- uses: actions/setup-go@v5
with: { go-version: "${{ matrix.go-version }}" }
- env: { GORACE: "log_path=/tmp/race/report halt_on_error=0" }
run: |
go vet ./...
go test ${{ matrix.race }} -count=3 -cpu=1,4 -timeout=10m ./...
- if: failure()
uses: actions/upload-artifact@v4
with: { name: race-reports, path: /tmp/race/report* }Set -timeout to 5-10x the non-race run time; upload log_path files on failure.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
Run -race once, see no output, ship | Detector only finds races that executed | -count / -cpu matrix |
Skip -race for "release" builds in CI | Race appears in prod, not CI | Gate one matrix dimension with -race |
VerifyNone with t.Parallel() | goleak can't attribute goroutines | VerifyTestMain |
IgnoreCurrent() at package init | Snapshot masks leaks added before each test | Call inside each test function |
Trust -race to catch goroutine leaks | Different failure class | Add goleak; the gates are complementary |
history_size=7 always | 128K history per goroutine can OOM CI | Start at 1; raise only on truncated stacks |
Limitations
References
goleak filter options
View source (opens in new window)goleak filter options
Filter options for goleak.VerifyNone / goleak.VerifyTestMain, per pkg.go.dev/go.uber.org/goleak (opens in new window). Pass one or more as trailing arguments to suppress goroutines that are expected rather than leaked.
IgnoreTopFunction
Ignores any goroutine whose top-of-stack frame is the named function. Prefer this when the library goroutine is identifiable by name.
goleak.VerifyNone(t,
goleak.IgnoreTopFunction("database/sql.(*DB).connectionOpener"),
)IgnoreAnyFunction (v1.3.0+)
Ignores any goroutine whose stack contains the named function at any depth, not just the top frame. Use when the identifying frame is not at the top.
goleak.VerifyNone(t,
goleak.IgnoreAnyFunction("google.golang.org/grpc.(*ccBalancerWrapper).watcher"),
)IgnoreCurrent
Snapshots the goroutines already running at call time and ignores exactly those at verification.
opt := goleak.IgnoreCurrent()
// ... test logic ...
goleak.VerifyNone(t, opt)Prefer IgnoreTopFunction over IgnoreCurrent when the library goroutine is identifiable by name: IgnoreCurrent silences goroutines that were already running at snapshot time, which can mask leaks introduced before the snapshot.
Cleanup
goleak.Cleanup(func(int)) registers a function goleak calls with the exit code after the leak check, e.g. to log instead of failing the process. Used with VerifyTestMain when the default exit behavior needs to change.
References
Related skills
async-ordering-tests
Test async ordering - event-loop / queue / channel ordering assertions, JS Promise microtask vs macrotask ordering, Python `asyncio.gather` vs `asyncio.wait_for` semantics, Go goroutine + channel happens-before relationships, async/await re-entrancy. Use deterministic schedulers (sinon fake timers, asyncio test mode) to remove run-to-run variance. Use when a callback fires twice, a later response overwrites an earlier one, or a cancelled parent task leaves a child still running - bugs where completion order, not shared memory, is the defect.
deadlock-detection-harness
Build deadlock-detection harnesses - extract lock-acquire-order graph via instrumentation, run cycle detection (DFS) to spot inconsistent ordering, use lock-acquire timeouts to surface rather than hang, JVM `jstack` / `gdb thread apply all bt` for postmortem analysis. Pair with ThreadSanitizer's `detect_deadlocks=1` for runtime detection. Use when a service that holds two or more locks hangs in production with no crash or error, or before release when lock acquisition order across code paths has never been proven consistent.
mvcc-isolation-tests
Build per-database MVCC isolation-level tests - Read Uncommitted vs Read Committed vs Repeatable Read vs Serializable; verify which anomalies are prevented at each level (dirty read, non-repeatable read, phantom read, serialization anomaly, write skew). Per PostgreSQL transaction isolation docs; analogous patterns for MySQL InnoDB, SQL Server, and DynamoDB. Use when two concurrent transactions can touch the same rows (balance debit, seat booking, stock decrement), or before changing a service's default isolation level.