Flaky Tests and Test Reliability

A flaky test is one that passes and fails against the same code. That definition is uncontroversial and almost useless, because it describes a symptom shared by a dozen unrelated causes: a race in the application, a race in the test, shared state between workers, a network dependency, a clock, a fixture that leaks, an assertion that never had a stable value to assert on.

The reason flakiness is worth a topic of its own is that it destroys something automation exists to provide. A suite is a decision-making instrument. Once engineers learn that a red build might mean nothing, they stop reading the build, and the suite's cost stays while its value goes to zero. Every hour spent maintaining a suite nobody trusts is waste.

Testland's position: "re-run it" is not a flake strategy, it is the absence of one. Retries hide the signal that tells you which part of your system is non-deterministic, and the systems that produce flaky tests are usually the same systems that produce intermittent production incidents.

Diagnose before you fix

Most flaky-test advice fails because it prescribes a fix without naming the cause. The useful first move is classification, and the classification is empirical: run the test under varied conditions and see which axis reproduces the failure. Isolation, ordering, parallelism, viewport, network throttling, and clock skew are each a different suspect, and a test that only fails under parallel execution is a different problem from one that only fails when it runs third.

How to fix flaky tests: a four-step triage lays out the four-step detect, categorize, fix, verify loop that this topic assumes as a baseline. Flaky Playwright tests: five root causes and fixes is the framework-specific version, covering the failure modes that auto-wait does not cover: non-retrying assertions, brittle locators, network races, and hard sleeps.

Quarantine is a policy, not a button

Every test runner can skip a test. Almost no team has agreed on what happens next, which is how quarantine directories turn into graveyards holding tests that were disabled two years ago for reasons nobody recorded.

A working quarantine policy answers four questions before the first test enters it: who may quarantine, what evidence is required, what the expiry is, and who owns the fix. Without an expiry, quarantine is deletion with extra steps. How to organize regression testing for web apps covers where quarantine sits inside a tiered suite, and why the fast pull-request tier and the nightly soak tier need different flakiness thresholds.

Measuring flakiness

Pass and fail are the two least informative things a test run produces. A suite that is 99% green and 30% flaky on its slowest ten tests looks identical, in a status report, to one that is genuinely stable.

The metrics that distinguish them are flake rate per test, duration trend, retry frequency, and time-to-green per pull request. All four require storing history rather than reading the latest run, which is the actual infrastructure investment behind flake management. Test observability: metrics beyond red and green covers what to collect and what each number tells you about where the CI bottleneck is.

Retries: when they are legitimate

Automatic retry is defensible in exactly one situation: the flakiness originates outside your control and is genuinely transient, such as a third-party sandbox that rate-limits under load. Even then, a retry that is not counted is a lie. If a test passes on attempt two, the run is not green, it is amber, and the reporting should say so.

Retries applied as a default policy across a whole suite are how teams stop noticing that their application has a race condition. The bug does not go away when the test stops reporting it.

Skills and agents for flake triage

qa-flake-triage packages this topic as installable Claude Code skills and agents. The flake pattern reference is the eight-pattern classification catalogue with the code-level fix for each pattern. flaky-test-quarantine is the quarantine workflow: the skip or fixme annotation records the observed failure rate and a bisect link, carries an expiry date, and feeds a CI report of every quarantined test whose expiry has passed. flake-dashboard-author builds the history view the metrics above depend on.

For diagnosis, flake-axis-bisection is the method: hold the test constant, vary one axis at a time, and check whether the gap between two conditions is larger than sampling noise. The e2e-flake-bisector agent runs that loop against a target test and, when parallel execution is implicated, instruments database rows, environment variables, files, ports, and module state to name the resource two workers are colliding on. A test that now fails on every run is a regression rather than a flake, and regression-bisector drives git bisect to the commit that introduced it. How these install and get invoked in Claude Code is covered under Claude Code for QA.

What this topic does not cover

Genuine product bugs that happen to be intermittent are not flaky tests, they are intermittent bugs, and the fix belongs in the application. The distinction matters: a test that correctly reports a real race condition 40% of the time is doing its job, and quarantining it is the wrong response.