How to Fix Flaky Tests: A Four-Step Triage

A test that passes sometimes and fails other times is worse than no test at all. It erodes trust in your test suite, wastes time on false alarms, and eventually gets ignored. Or worse, disabled.

The problem isn't rare, even at companies with mature test infrastructure. Google reported in 2016 that "almost 16% of our tests have some level of flakiness associated with them," with "a continual rate of about 1.5% of all test runs reporting a 'flaky' result" (Google Testing Blog). Microsoft's flaky test management system had identified roughly 49,000 flaky tests by February 2022 and helped pass 160,000 sessions that would otherwise have failed on a flaky result.

Fixing flakes one at a time, in whatever order they annoy people, doesn't scale. What works is triage: find the flaky tests, rank them, name the cause of each, apply the fix that matches the cause, and prove the fix held. Here's that process in four steps.

Step 1: Detect, track, and rank flaky tests

You can't fix what you can't see. Before anything else, you need to know which tests are flaky and how often they fail.

Automatic detection methods:

• Rerun failures: Run failed tests 2-3 times. If a test passes on retry, it's likely flaky.
• Historical tracking: Log test results over time. Tests that alternate between pass and fail are your culprits.
• CI annotations: Most CI systems can flag tests with inconsistent results across runs.

Some runners do the classification for you. With retries enabled, Playwright Test sorts every test into passed, failed, or flaky, where flaky means "failed on the first run, but passed when retried" (Playwright docs: retries). Its failOnFlakyTests option makes CI exit with an error when any test lands in that bucket, so a flake can't hide behind a green retry. On the Python side, the pytest-rerunfailures plugin reruns failed tests and marks every rerun in the output, so a test that only passed on a rerun stands out.

Here's a simple pytest approach to track flaky tests:

# conftest.py
import json
from pathlib import Path

FLAKY_LOG = Path("flaky_tests.json")

def pytest_runtest_makereport(item, call):
    if call.when == "call" and call.excinfo is not None:
        # Test failed - log it
        log = json.loads(FLAKY_LOG.read_text()) if FLAKY_LOG.exists() else {}
        test_name = item.nodeid
        log[test_name] = log.get(test_name, 0) + 1
        FLAKY_LOG.write_text(json.dumps(log, indent=2))

Review this log weekly. Tests that fail intermittently but pass on retry need investigation. Doing this properly means storing history rather than reading the latest run: Test Observability: Metrics Beyond Red and Green covers which numbers are worth collecting and what each one tells you.

Then rank. Sort the list by failure rate times how often the test runs, and work from the top. A test that fails 2% of the time in every pull request costs more than one that fails 20% of the time in a weekly job, and the ranking keeps the team on the flakes that actually block merges.

Step 2: Categorize the cause

Once you've identified a flaky test, figure out why it's flaky. Most flaky tests fall into four categories:

Quick diagnosis:

1. Run the test 10 times in isolation (npx playwright test checkout.spec.ts --repeat-each=10, or pytest --count=10 test_checkout.py with pytest-repeat). If it fails, it's likely a timing issue.
2. Run the test with other tests. If it only fails in the full suite, suspect shared state.
3. Run the test in CI vs locally. Different results point to environment issues.

Those three runs vary one condition at a time, and that's the whole trick. When they don't settle it, keep going down the list of axes (worker count, execution order, viewport, network latency) and record a pass/fail count for each. Testland's flake-axis-bisection skill turns that into a measured procedure, including how many runs it takes before a difference between two conditions is more than noise, and the flake pattern reference extends these four categories to eight, with the code-level fix for each.

Step 3: Fix by category

Fixing timing issues

Timing issues are the first suspect for any UI or async test. The fix: stop using fixed waits and start using condition-based waits.

Bad - hardcoded sleep:

# This might work locally but fail in CI
def test_form_submission(page):
    page.click("button#submit")
    time.sleep(3)  # Hoping the server responds in 3 seconds
    assert page.locator("#success").is_visible()

Good - wait for condition:

# Waits until the condition is true, up to timeout
def test_form_submission(page):
    page.click("button#submit")
    expect(page.locator("#success")).to_be_visible(timeout=10000)

For Playwright specifically, take advantage of auto-waiting:

# Playwright auto-waits for element to be actionable
def test_login(page):
    page.get_by_role("button", name="Submit").click()  # Auto-waits
    expect(page).to_have_url("/dashboard")  # Auto-retries assertion

For API tests, add retry logic for transient failures:

from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, max=10))
def call_api(url):
    response = requests.get(url, timeout=5)
    response.raise_for_status()
    return response.json()

Fixing shared state problems

When tests share data (database records, files, global variables), they can step on each other's toes. The fix: isolate each test completely.

Bad - shared test data:

# All tests use the same user - disaster waiting to happen
TEST_USER = {"email": "test@example.com", "id": 1}

def test_update_user():
    api.update_user(TEST_USER["id"], {"name": "New Name"})
    # Another test might be reading this user right now

def test_delete_user():
    api.delete_user(TEST_USER["id"])
    # Now test_update_user fails because user doesn't exist

Good - isolated test data:

import uuid

@pytest.fixture
def test_user(api_client):
    """Create a unique user for this test only."""
    unique_email = f"test_{uuid.uuid4()}@example.com"
    user = api_client.create_user(email=unique_email)
    yield user
    api_client.delete_user(user["id"])  # Cleanup after test

def test_update_user(test_user):
    api.update_user(test_user["id"], {"name": "New Name"})
    # Only this test touches this user

def test_delete_user(test_user):
    api.delete_user(test_user["id"])
    # Different user instance, no conflict

For database tests, use transactions that roll back after each test:

@pytest.fixture
def db_session():
    connection = engine.connect()
    transaction = connection.begin()
    session = Session(bind=connection)
    yield session
    transaction.rollback()  # All changes disappear
    connection.close()

Fixing environment issues

Environment flakiness usually comes from hardcoded paths, timing assumptions, or missing dependencies. The fix: make tests environment-agnostic.

# Bad - hardcoded path
config_path = "/Users/dev/project/config.json"

# Good - relative to test location
config_path = Path(__file__).parent / "fixtures" / "config.json"

For CI specifically:

• Use environment variables for configuration
• Mock external services instead of calling them
• Pin dependency versions exactly

Step 4: Verify the fix, then guard against new flakes

A flaky test that passes once after a fix hasn't proven anything. Rerun it under the condition that reproduced the failure, many times:

# Playwright: 50 repetitions, in parallel, to recreate CI pressure
npx playwright test tests/checkout.spec.ts --repeat-each=50 --workers=4

# pytest with pytest-repeat and pytest-xdist
pytest tests/test_checkout.py --count=50 -n 4

Pick the run count from the failure rate you measured in Step 1. If the test failed about 1 run in 20 before the fix, the chance of 50 clean runs with the bug still present is 0.95^50, a little under 8%. Fifty green runs is good evidence. Five is not.

Once the fix holds, put guardrails in place:

Code review checklist:

• [ ] No time.sleep() or Thread.sleep() calls
• [ ] All waits are condition-based with timeouts
• [ ] Test data is isolated (unique per test)
• [ ] External services are mocked
• [ ] No hardcoded paths or environment-specific values

CI configuration:

• Run flaky test detection on every PR
• Quarantine known flaky tests while fixing them, with an expiry and a named owner (How to Organize Regression Testing for Web Apps covers where quarantine sits in a tiered suite). The flaky-test-quarantine skill writes the annotation with the observed failure rate and an expiry date, and reports every quarantined test whose expiry has passed
• Set up alerts when test flakiness exceeds a threshold

Team practices:

• Fix flaky tests immediately - don't let them pile up
• Track flaky test metrics over time
• Make "no new flaky tests" part of your definition of done

Fixing your flakiest tests first

Start by identifying your top 5 flakiest tests. Categorize each one, apply the appropriate fix, and verify the fix holds across multiple runs.

If you're using Playwright, the framework-specific failure modes are covered in Flaky Playwright Tests: Five Root Causes and Fixes, and Playwright's own best practices guide is worth reading alongside it.

The ranking from Step 1 is what keeps this from turning into whack-a-mole. Work the top five, verify each fix with repeated runs, and re-rank every week: the list should get shorter, and the failure rates at the top should get smaller.

Everything Testland has written on this subject is collected under flaky tests and test reliability, and the diagnosis workflow above is packaged as installable agents in qa-flake-triage.