jepsen-patterns
Reference for Jepsen-style distributed-systems testing - consistency models hierarchy (linearizability vs sequential vs causal vs monotonic-reads vs eventual), nemesis primitives (network partitions, clock skew, kill nodes), workload generators, Knossos + Elle linearizability checkers. Reference-only because Jepsen tests are typically Clojure-bespoke per system; use this skill to evaluate vendor claims and structure your own test. Use when a datastore vendor advertises a consistency guarantee that has to be checked before adoption, when reading a published Jepsen report for gaps, or when a custom replicated store needs its own consistency test scoped.
Install with skills.sh (any agent)
npx skills add testland/qa --skill jepsen-patternsjepsen-patterns
Per the Jepsen consistency docs (opens in new window), "A consistency model is a safety property which declares what a system can do." Jepsen tests distributed databases by injecting faults (the nemesis) and checking the operation history against a consistency model.
This skill is reference-only - Jepsen itself is a Clojure library
When to use
Step 1 - Map the consistency model your system claims
Per the Jepsen consistency docs (opens in new window), models are organized by their guarantees + the phenomena they prohibit:
| Model | Allowed phenomena | Forbidden phenomena |
|---|---|---|
| Linearizability | None - operations totally ordered respecting real time | Stale read, lost update, write skew |
| Sequential consistency | Per-process order respected; cross-process not real-time | Real-time-ordering violations |
| Causal consistency | Cause-before-effect respected | Causally unrelated operations may appear out of order |
| Monotonic reads | Once a read sees value v, no later read sees an older value | Cross-client divergence allowed |
| Eventual consistency | Convergence eventually | Stale reads, inconsistent windows |
Your system claims one of these (or a hybrid: snapshot isolation, read-your-writes, etc.). The test must match the claim.
Step 2 - Pick a nemesis
Nemesis primitives Jepsen ships:
| Nemesis | What it does |
|---|---|
| Partition | Splits the cluster into N groups; intra-group communication blocked |
| Crash | Hard-kills a process |
| Pause | SIGSTOPs a process (hangs without disconnecting) |
| Clock skew | jiggles gettimeofday() per-node |
| Slow disk | adds I/O latency |
| Bitflip | corrupts disk contents |
Combine nemeses (partition + crash + clock skew) to find compound bugs.
Step 3 - Generator: construct the workload
A Jepsen workload is per-client operations: invoke read / write / cas / append, observe outcome (ok / fail / info).
Pseudocode shape (Jepsen DSL is Clojure):
(generator/mix
[{:f :read, :value nil}
{:f :write, :value (rand-int 100)}
{:f :cas, :value [old new]}])Concurrent N clients hit the system; outcomes recorded as a history (an ordered list of invocations + completions).
Step 4 - Check the history with Knossos / Elle
| Checker | Use |
|---|---|
| Knossos | Linearizability checker for register-style ops (read/write/cas) |
| Elle | Transactional anomaly checker (G0/G1a/G1b/G1c, G-nonadjacent, G-single, G2-item, G2) - finds dirty/non-monotonic/non-repeatable read violations |
Both surface counterexamples (specific operation sequences) that violate the claimed model. Counterexamples are the value: vendor claim says "linearizable"; checker says "here's an op sequence that isn't" → you have evidence.
Step 5 - Workload patterns
Common workload shapes per consistency claim:
| Workload | Tests |
|---|---|
| Register (single-key R/W/CAS) | Linearizability of single-key |
| Append (per-key list, append + read) | Per-key history monotonicity |
| Set (insert + read all) | No lost insert; eventual visibility window |
| Bank transfer (txn-level read + write) | Transactional invariants (sum stays constant) |
Pick the workload closest to your system's user-facing invariants.
Step 6 - Reading vendor Jepsen reports
Check for these red flags:
Per the Jepsen consistency docs (opens in new window), Jepsen's value is that "consistency models and phenomena are often defined in terms of dependencies" - gaps in the test = gaps in confidence.
Step 7 - In-house test scoping
For your own system (custom KV / custom replication):
Out-of-the-box Jepsen test rigs exist for many systems (jepsen-io/jepsen GitHub); fork rather than start from scratch.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Test under stable network only | Real production has partitions; bugs hide | Always include partition nemesis (Step 2) |
| Trust "we did our own consistency tests" without checker | Manual reasoning misses subtle violations | Use Knossos / Elle (Step 4) |
| Single-client workload | Concurrency bugs need concurrency | Multi-client generator (Step 3) |
| Skip clock skew if using NTP | NTP can step backward; bugs trigger | Include clock skew (Step 2) |
| Run for 60s | Bugs may take hours to surface | Run hours; bisect to specific operation in history |
Limitations
References
Related skills
async-ordering-tests
Test async ordering - event-loop / queue / channel ordering assertions, JS Promise microtask vs macrotask ordering, Python `asyncio.gather` vs `asyncio.wait_for` semantics, Go goroutine + channel happens-before relationships, async/await re-entrancy. Use deterministic schedulers (sinon fake timers, asyncio test mode) to remove run-to-run variance. Use when a callback fires twice, a later response overwrites an earlier one, or a cancelled parent task leaves a child still running - bugs where completion order, not shared memory, is the defect.
deadlock-detection-harness
Build deadlock-detection harnesses - extract lock-acquire-order graph via instrumentation, run cycle detection (DFS) to spot inconsistent ordering, use lock-acquire timeouts to surface rather than hang, JVM `jstack` / `gdb thread apply all bt` for postmortem analysis. Pair with ThreadSanitizer's `detect_deadlocks=1` for runtime detection. Use when a service that holds two or more locks hangs in production with no crash or error, or before release when lock acquisition order across code paths has never been proven consistent.
go-race-detector-workflow
Runs the Go race detector and goroutine-leak checker end-to-end: instrument with `go test -race`, read race reports, configure GORACE options, stress with `-count`/`-cpu`, detect goroutine leaks with go.uber.org/goleak, and gate both checks in CI. Use when a Go service has shared state accessed by concurrent goroutines, when a race-related incident needs a regression harness, or when adding `-race` to a CI matrix for a Go module. Does not cover barrier-based deterministic interleaving or forced goroutine scheduling; use race-condition-test-author for that.
mvcc-isolation-tests
Build per-database MVCC isolation-level tests - Read Uncommitted vs Read Committed vs Repeatable Read vs Serializable; verify which anomalies are prevented at each level (dirty read, non-repeatable read, phantom read, serialization anomaly, write skew). Per PostgreSQL transaction isolation docs; analogous patterns for MySQL InnoDB, SQL Server, and DynamoDB. Use when two concurrent transactions can touch the same rows (balance debit, seat booking, stock decrement), or before changing a service's default isolation level.
race-condition-test-author
Build deterministic race-condition tests - identify shared mutable state, drive interleavings via barriers / latches / manual scheduling; use ThreadSanitizer (clang `-fsanitize=thread`) for C/C++/Go data race detection; use jcstress (`@JCStressTest` + `@Actor` + `@Outcome`) for JVM stress; use Loom virtual-thread interleavings for parallel testing. Use when a defect only reproduces under load on shared in-process state (cache, counter, connection pool, lazy-init singleton), or when writing the regression test for a race-condition incident before the fix lands.