Testland
Skills & AgentsTopicsBlogAbout
Browse all skills & agents

flaky-test-quarantine

Builds a quarantine workflow for flaky tests - marks the test with the framework's skip/fixme/retry annotation, records the failure-rate observation and a bisect link in the annotation body, sets an auto-expiry date, and produces a CI report listing every quarantined test that has expired and needs re-evaluation. Use when a flaky test is blocking the trunk and must be removed from the gating path without losing track of it.

Install with skills.sh (any agent)

npx skills add testland/qa --skill flaky-test-quarantine
View source

flaky-test-quarantine

Overview

A "flaky test" is a test that produces inconsistent pass/fail results across runs without an underlying code change (google-flaky (opens in new window)). Industry consensus from Google Testing Blog and similar practitioner-engineering sources is that flaky tests should be isolated from the gating path rather than left to mask real regressions or be silently ignored (google-flaky (opens in new window)).

Terminology note: "flaky test" is a practitioner-emergent term popularized by the Google Testing Blog. ISTQB does not maintain a canonical entry for it. This skill cites industry-engineering sources, not ISTQB authority.

How to use

  1. Confirm it is a flake, not a regression - a 1% to 50% failure rate, not 100% (see "When to use"). A 100%-failing test is a regression: bisect and fix, do not quarantine.
  2. Mark the test with the framework's skip / fixme / retry annotation (Step 1).
  3. Annotate the body with date, issue link, failure rate, bisect status, and a Re-evaluate by date (Step 2) - this is the load-bearing, machine-parseable part.
  4. Set the TTL that produces the Re-evaluate by date (Step 3).
  5. Wire the expired-quarantine report into a scheduled CI job (Step 4 and references/ci-quarantine-report.md).
  6. Prune on expiry - fix and un-quarantine, renew once (never past two consecutive renewals), or delete (Step 5).

When to use

  • A test is failing on the trunk between 1% and 50% of runs (above 50% it's not flaky, it's broken; under 1% the noise is acceptable for most projects).
  • The team's incident process has been triggered more than once by the same test.
  • A new feature merge is blocked by a known-flaky pre-existing test that's unrelated to the change.

If the test fails 100% of the time after a code change, it's a regression - bisect to the introducing commit and fix, do not quarantine.

Worked example

Test tests/checkout.spec.ts:42 fails about 12% of runs on the tablet-768 project in CI. An unrelated feature PR is blocked by it.

  1. Confirm it is a flake. 12% sits between 1% and 50%, and the test predates the PR - quarantine it, do not treat it as a regression of the PR under review.
  2. Mark and annotate with a single test.fixme() carrying the full record (Steps 1 and 2):
test('checkout flow flaky test', async ({ page }) => {
  test.fixme(
    true,
    'Quarantined 2026-07-20 (#1234) - fails ~12% of runs on tablet-768; bisect inconclusive. Re-evaluate by 2026-08-19. Owner: @web-platform.',
  );
  // ... test body, no longer runs
});
  1. Set the TTL. The default 30-day TTL (Step 3) puts Re-evaluate by 2026-08-19 in the annotation. The PR merges the same day, unblocked.
  2. Let CI track it. The Monday report job (Step 4) greps every test.fixme annotation. On the first Monday after 2026-08-19 it prints EXPIRED for this entry and opens a tracking issue.
  3. Prune. A fix for the tablet-768 layout race has since landed, so remove test.fixme(), re-run the test at depth to confirm it is green, and close the issue. Had it still failed, renew once with an updated annotation - never past two consecutive renewals.

Step 1 - Mark the test

Playwright

test.fixme() is the canonical Playwright primitive for "this test is broken; do not run past this point" (pw-test (opens in new window)):

test('checkout flow flaky test', async ({ page }) => {
  test.fixme(
    true,
    'Quarantined 2026-05-04 (#1234) - fails ~12% of runs on tablet-768; bisect inconclusive. Re-evaluate by 2026-06-04.',
  );
  // ... test body, no longer runs
});

test.fixme(condition, description) skips with the description visible in the report. Unlike test.skip(), fixme carries the explicit "this needs to be fixed" intent (pw-test (opens in new window)).

If the goal is to allow retries before quarantining, use the retries config first (pw-retries (opens in new window)):

// playwright.config.ts
export default defineConfig({
  retries: process.env.CI ? 2 : 0,
});

A test that passes on retry is reported with the flaky status (distinct from passed and failed); track these separately - flaky-but-passing tests are quarantine candidates, not yet quarantined (pw-retries (opens in new window)).

Cypress

Cypress configures retries at the suite level via Cypress.config('retries', { runMode: 2, openMode: 0 }). For quarantining individual specs, use it.skip(...) or the cypress-grep (opens in new window) plugin's tagging convention.

Jest / Vitest

test.skip(...) and test.todo(...) are the canonical primitives. For periodic auto-evaluation, use test.skip.if(condition) patterns or introduce a project-specific tagging convention parsed by your CI.

JUnit / TestNG (JVM)

JUnit 5: @Disabled("Quarantined 2026-05-04 (#1234) - ..."). TestNG: @Test(enabled = false, description = "..."). For per-method retries before quarantine, JUnit 5's @RetryingTest(N) extension and TestNG's @Test(retryAnalyzer = ...).

Step 2 - Annotate with failure rate + bisect link + expiry

The annotation body is the load-bearing part of the workflow. Every quarantine record carries:

FieldRequiredFormat
DateyesYYYY-MM-DD of the quarantine.
Issue linkyes#1234 or full URL - links a tracked ticket.
Failure rateyes~12% of runs - measured, not guessed.
Bisect statusyesbisect inconclusive / bisected to commit abc1234 / not yet bisected.
Re-evaluate byyesYYYY-MM-DD - the auto-expiry date.
Owneroptional@team-handle for routing.

The format is parseable by the re-evaluation report (Step 4):

Quarantined 2026-05-04 (#1234) - fails ~12% of runs on tablet-768;
bisect inconclusive. Re-evaluate by 2026-06-04. Owner: @web-platform.

Step 3 - Auto-expiry

Default TTL: 30 days. Picked because:

  • Long enough to fix non-trivial issues without creating churn.
  • Short enough that quarantined tests don't become a permanent graveyard.

Adjust per project:

  • 90 days for low-traffic suites where the underlying issue is known but un-prioritized.
  • 14 days for high-traffic CI where flakiness is an actively-monitored metric.

Step 4 - Re-evaluation report

A scheduled CI job greps every quarantine annotation, extracts the Re-evaluate by date, and lists the entries that have expired - then opens a tracking issue (or posts to Slack) per expired entry. Because the Step 2 format is machine-parseable, the whole report is a short grep. The Bash report script and the scheduled GitHub Actions workflow are in references/ci-quarantine-report.md.

Step 5 - Pruning rules

When a re-evaluation expires, the team has three options:

OutcomeAction
Underlying issue fixedRemove test.fixme() and re-run; close the issue.
Underlying issue still presentRenew the quarantine for one more TTL with updated annotation; never more than two consecutive renewals - at that point, delete the test or rewrite it.
The test is no longer relevantDelete the test outright; close the issue.

The two-renewal cap is the lever that prevents quarantine from becoming a permanent dead-letter. Past two renewals, the team has either lost interest in the assertion or the test is fundamentally unfixable - both signal "delete."

References

Automating the expired-quarantine report in CI

View source (opens in new window)

Automating the expired-quarantine report in CI

Deep reference for flaky-test-quarantine SKILL.md. The Step 4 re-evaluation report script and its scheduled CI wiring, split out of the main workflow so the mark / annotate / expiry / prune decisions stay in front.

The annotation format from Step 2 is what makes this automatable: every quarantine body carries a Re-evaluate by YYYY-MM-DD field, so a scheduled job can grep the suite, extract the date, and list the entries that have expired.

The report script

A nightly (or weekly) job greps all quarantine annotations, extracts the Re-evaluate by date, and lists expired entries. A minimal Bash version against a Playwright suite:

#!/usr/bin/env bash
# scripts/list-expired-quarantines.sh
set -e
TODAY=$(date -u +%Y-%m-%d)

grep -rn -B1 -A5 "test\.fixme(" tests/ \
  | awk '/Re-evaluate by/ { print FILENAME ":" $0 }' \
  | while IFS= read -r line; do
      EXPIRY=$(echo "$line" | grep -oE 'Re-evaluate by [0-9]{4}-[0-9]{2}-[0-9]{2}' | awk '{print $3}')
      if [[ "$EXPIRY" < "$TODAY" ]]; then
        echo "EXPIRED: $line"
      fi
    done

Run it as a scheduled GitHub Action and post the output to a Slack channel or open a tracking issue per expired entry.

The scheduled workflow

# .github/workflows/quarantine-report.yml
name: quarantine-report

on:
  schedule:
    - cron: '0 9 * * 1'   # Mondays 09:00 UTC
  workflow_dispatch:

jobs:
  list-expired:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5

      - name: List expired quarantines
        run: bash scripts/list-expired-quarantines.sh > expired.txt

      - name: Open tracking issue per expired entry
        if: ${{ hashFiles('expired.txt') != '' }}
        run: |
          while IFS= read -r line; do
            gh issue create --title "Expired quarantine: ${line%%:*}" --body "$line"
          done < expired.txt
        env:
          GH_TOKEN: ${{ github.token }}

Each expired entry becomes a tracking issue that routes back to Step 5 pruning: fix and un-quarantine, renew once more (never past two consecutive renewals), or delete the test.

Related skills

flake-axis-bisection

Locates the condition a known-flaky test actually depends on by holding the test constant and varying one axis at a time (isolation, execution order, worker count, viewport, network latency, repetition depth), recording a pass/fail count per variation, and testing whether the gap between two conditions exceeds sampling noise. Covers choosing the run count N from the failure rate you need to detect, binomial confidence intervals on a measured reproduction rate, a two-proportion comparison rule, what a zero-failure result does and does not prove, and the resource-collision walk (DB row, DB schema, file path, port, env var, module state, inode, cookie jar) used once parallelism is implicated. Use when a specific test is already known to fail intermittently, reading its source has not explained why, and a decision about what to change must rest on measurement rather than on a plausible-sounding guess.

flake-dashboard-author

Builds a persistent flakiness infrastructure dashboard from JUnit XML or JSON CI run history: defines the flake-rate metric (failures per test over a configurable window), authors the data model, generates a Grafana time-series panel JSON or configures a Datadog CI Visibility view, derives the quarantine-candidate query, and wires trend alerts. Use when a team needs a long-lived observability surface for test reliability that outlasts any single weekly report.

flake-pattern-reference

Reference catalog of flake patterns - async/timing, test ordering, shared parallel state, resource leaks, network, locator drift, environment variance, randomness - with detection heuristics and remediation per pattern. Use when triaging an unknown flake to identify the category before bisecting.

flake-remediation-guide

Provides concrete code-level fixes for each of the eight recurring flake patterns in flake-pattern-reference: replacing fixed sleeps with framework auto-waits, isolating state in beforeEach fixtures, stable role-based locators, mocking network + clock, seeding RNG, closing leaked resources, and the per-worker DB-schema fix for shared parallel state. Use when a flake is already classified by pattern and you need the specific code change to apply; to classify the pattern first use flake-pattern-reference or flake-axis-bisection, and to quarantine the test while the fix is in review use flaky-test-quarantine.