Free tools

Flaky Test Finder: Compare Test Report Runs

Drop reports from several runs of the same suite on the same code. Tests that pass in one run and fail in another, or pass only on a retry, are flaky.

Runs in your browserNothing is uploadedNo sign-up

Choose or drop two or more report files of the same suite, one per run, in run order. Any format the report viewer reads works, and formats can be mixed.

Or drop files on the box. Nothing is uploaded.

How it decides a test is flaky

A flaky test passes and fails on the same code. The finder looks for that in two places:

  1. Across runs. Load reports from several runs of the same suite, in order. A test that passed in at least one run and failed in at least one other is flaky.
  2. Within a run. A test the runner retried into a pass is flaky even in a single report: Playwright marks it flaky, Maven Surefire records <flakyFailure>, CTRF sets flaky, and a report that lists the same test twice, failed and then passed, counts too.

A test that failed in every run it ran is listed apart as always failing. That is a bug or a broken test, and retrying it wastes CI time. Tests that never ran are left out.

For each flaky test you get the result of every run as a row of dots, the fail rate (failed runs over runs where the test ran), how many runs needed a retry to pass, and the last failure message, which usually names the cause: a timeout, a race, a shared resource.

Get the comparison right

  • Same code, different runs. Compare reruns of one commit, or nightly runs on a branch nobody changed. A test that starts failing after a merge is a regression, not a flake.
  • More runs, better rates. Three runs catch tests that fail often. A test that fails one run in fifty needs many more, which is why teams keep a history instead of comparing by hand.
  • Same names. Tests are matched by suite and name. A test whose name includes a timestamp or a random id looks like a new test every run.

What to do with a flaky test

Fix it if the failure message points somewhere. If it doesn't, quarantine it so it stops blocking merges while someone investigates, and keep running it so you know when it is fixed. The flake pattern reference maps common symptoms (only in CI, only at night, only in parallel) to their usual causes, and flaky test quarantine sets up the quarantine.

Sources

Testland for teams

In development

Automated test reports, manual test runs and regression tracking in one workspace. A separate product, built on the same skills.

One email when it opens, nothing else. Privacy policy