flake-pattern-reference

Terminology note: "flaky test" is a practitioner-emergent term popularized by the Google Testing Blog (google-causes (opens in new window), google-flaky (opens in new window)); ISTQB does not maintain a canonical entry. This catalog reflects industry-engineering consensus, not ISTQB authority.

A flake is rarely random - it almost always falls into one of eight recurring patterns. Identifying the pattern early shrinks the bisect search space dramatically. This catalog is a reference, not a workflow; the matching workflow is in flaky-test-quarantine.

The Google Testing Blog observed a near-linear correlation between test size and flakiness rate across ~4.2M tests (google-causes (opens in new window)) - larger tests touch more of the eight patterns at once.

Pattern 1: async / timing

The most common flake category in UI and integration tests.

Remediation:

• Replace fixed sleeps with deterministic waits - await expect(loc).toBeVisible(), page.waitForLoadState('networkidle'), page.waitForFunction(...), etc.
• For animations, disable them in test setup (animations: 'disabled' in Playwright; Cypress.config('animationDistanceThreshold', 0) in Cypress).
• For absolute clock dependencies, freeze time with sinon.useFakeTimers() / vi.useFakeTimers() / Playwright's page.clock.install().

Pattern 2: test ordering

Tests pass alone, fail when run with siblings.

Remediation:

• Run the suite with explicit randomization in CI to surface ordering deps early (jest --randomize, pytest --random-order, mocha --sort reverse).
• Move ANY shared setup into beforeEach / afterEach, never rely on beforeAll for state that the test mutates.
• Database tests: roll back transactions after each test instead of truncating between describe blocks.

Pattern 3: shared parallel state

Tests pass sequentially, fail when run in parallel workers.

Remediation:

• Find the shared state two workers collide on - DB rows, files, ports, or global module state.
• Per-worker isolation: per-worker DB schemas (PG_SCHEMA=test_${WORKER}), per-worker temp dirs (TMPDIR=/tmp/test-${WORKER}), per-worker port ranges.
• For unique IDs: use UUIDs or a per-worker namespace prefix, not auto-increment integers shared across workers.

Pattern 4: resource leaks

Tests pass on a fresh machine, fail after the test process has run for hours.

Remediation:

• Always await browser.close() / await server.close() in afterAll, with a try/finally so failed tests still clean up.
• Set per-test timeouts and ensure the framework kills the process, not just the test (--testTimeout, test.setTimeout()).
• Run lsof | wc -l and ps aux | wc -l before / after the suite in CI to detect leaks; alert when growth exceeds a threshold.

Pattern 5: network / external service

Tests pass when the upstream is healthy, fail otherwise.

Remediation:

• Mock at the boundary - never let test code reach a real network endpoint. Use Mock Service Worker (MSW), nock, WireMock, or Playwright's page.route().
• For tests that must hit a real service (smoke / contract tests), isolate them in a separate suite that doesn't gate the main CI.
• DNS-level: pin to specific resolvers in CI to avoid resolver variance.

Pattern 6: locator drift

UI tests pass when the page looks one way, fail when it shifts.

Remediation:

• Use role-based selectors first (page.getByRole('button', { name: 'Submit' })), then data-testid, only text= / CSS as a last resort.
• For Playwright: enable strict: true so any ambiguous selector fails immediately rather than silently picking the first match.
• For viewport-specific UIs: snapshot at every breakpoint via responsive-breakpoint-runner; visual signal exposes layout-shift flakes faster than text checks.

Pattern 7: environment variance

Tests pass on Linux CI, fail on macOS dev machines (or vice versa).

Remediation:

• Pin CI to one OS / one timezone (TZ=UTC) for deterministic runs.
• Run snapshot updates only in CI, never from a developer laptop (per playwright-snapshots).
• For path-sensitive code, normalize with path.posix.join() / node:path.

Pattern 8: randomness

Tests use random data without a controlled seed.

Remediation:

• Seed every random source: Math.random via seedrandom, faker via faker.seed(N), property-based testing via fc.assert(prop, { seed }).
• For property-based failures, don't mark them as flake - copy the failing seed into a regression test.
• Persist the seed used in each CI run as a build artifact so a flake can be replayed.

Triage decision tree

Test fails ~50% of runs?
├── Yes → likely "shared parallel state" or "test ordering"
└── No → fails ~5-20% of runs?
    ├── Yes → likely "async/timing" or "network"
    └── No → fails only on specific OS / runner?
        ├── Yes → "environment variance"
        └── No → fails after long suite duration?
            ├── Yes → "resource leaks"
            └── No → fails after unrelated UI change?
                ├── Yes → "locator drift"
                └── No → does the test use random data?
                    ├── Yes → "randomness"
                    └── No → run a structured bisect

For systematic bisection, run a structured bisect that varies one axis at a time per the patterns above.

References

• google-flaky (opens in new window) - Google Testing Blog overview.
• google-causes (opens in new window) - Google's correlation analysis on ~4.2M tests; "test size correlates with flakiness rate."
• flaky-test-quarantine - workflow that uses this catalog during triage.