How it decides a test is flaky
A flaky test passes and fails on the same code. The finder looks for that in two places:
A test that failed in every run it ran is listed apart as always failing. That is a bug or a broken test, and retrying it wastes CI time. Tests that never ran are left out.
For each flaky test you get the result of every run as a row of dots, the fail rate (failed runs over runs where the test ran), how many runs needed a retry to pass, and the last failure message, which usually names the cause: a timeout, a race, a shared resource.
Get the comparison right
What to do with a flaky test
Fix it if the failure message points somewhere. If it doesn't, quarantine it so it stops blocking merges while someone investigates, and keep running it so you know when it is fixed. The flake pattern reference maps common symptoms (only in CI, only at night, only in parallel) to their usual causes, and flaky test quarantine sets up the quarantine.
Sources
Testland for teams
In developmentAutomated test reports, manual test runs and regression tracking in one workspace. A separate product, built on the same skills.
One email when it opens, nothing else. Privacy policy