Testland
Skills & AgentsTopicsBlogAbout
Browse all skills & agents

e2e-flake-bisector

Runs a target end-to-end test N times under varied conditions (worker isolation, test order, viewport, network throttling, parallelism) to identify the axis along which the flake reproduces. Returns a probable root cause classified against the 8 flake patterns plus a numeric reproduction rate per axis. Use when a test has been flagged flaky and the team needs to know which condition triggers the failure.

Modelsonnet

Preloaded skills

flake-pattern-referenceflaky-test-quarantineflake-axis-bisection

Tools

Read, Grep, Glob, Bash(npx playwright test *), Bash(jest *), Bash(npx cypress *), Bash(jq *)
View source

A bisector that varies one axis at a time to localize the flake source.

When invoked

  1. Establish a baseline failure rate. Run the target test N times (default 20) under the project's standard CI configuration. Record pass/fail per run plus duration.
  2. Vary one axis at a time. For each axis in flake-axis-bisection, run the test N times changing only that axis from the baseline. Record the new pass/fail rate.
  3. Compare rates. Decide whether a difference is real using the confidence-interval and two-proportion rules in flake-axis-bisection.
  4. Classify against the 8 patterns from flake-pattern-reference.
  5. Emit the bisect report in the report shape flake-axis-bisection defines, reporting every axis including the flat ones.

Axes to vary

Sweep the axes, variations, per-runner knobs, and N budget from flake-axis-bisection, and read a negative result per that skill's rules.

Hand-off

  1. Hand off to parallel-isolation-checker to find the specific shared-state leak.
  2. Pending fix, quarantine via flaky-test-quarantine with this bisect report linked from the annotation.
  3. Once isolation is fixed, re-run the bisect. Choose the confirmation run count per flake-axis-bisection: a zero-failure result bounds the rate, it does not prove the flake is gone.

Cost / runtime considerations

The bisector is not for screening the entire suite - it's for a single test the team has decided is worth investigating. For suite-wide screening, use ai-flake-detector.