Testland
testland/qaInstall

e2e-flake-bisector

Runs a target end-to-end test N times under varied conditions (worker isolation, test order, viewport, network throttling, parallelism) to identify the axis along which the flake reproduces, then - when the bisect implicates parallel execution - runs a stage-2 shared-state isolation check that instruments DB rows, env vars, files, ports, and global module state to name the specific resource two workers are colliding on, with file:line evidence. Returns a probable root cause classified against the 8 flake patterns plus a numeric reproduction rate per axis.

Install with Claude Code
/plugin install qa-flake-triage@testland-qa

Agents run in Claude Code. This installs the qa-flake-triage plugin. First time? Add the marketplace with /plugin marketplace add testland/qa

View source (opens in a new tab)
Agent
Plugin
qa-flake-triage
Category
Operations & resilience
Version
v1.5.5
Tools
Read, Grep, Glob, Bash(npx playwright test *), Bash(jest *), Bash(npx cypress *), Bash(lsof *), Bash(ps *), Bash(jq *)
Preloads
flake-pattern-referenceflaky-test-quarantineflake-axis-bisection

A bisector that varies one axis at a time to localize the flake source, then digs into shared state when parallelism is the implicated axis.

Stage 1 - axis bisection

  1. Establish a baseline failure rate. Run the target test N times (default 20) under the project's standard CI configuration. Record pass/fail per run plus duration.
  2. Vary one axis at a time. For each axis in flake-axis-bisection, run the test N times changing only that axis from the baseline. Record the new pass/fail rate.
  3. Compare rates. Decide whether a difference is real using the confidence-interval and two-proportion rules in flake-axis-bisection.
  4. Classify against the 8 patterns from flake-pattern-reference.
  5. Emit the bisect report in the report shape flake-axis-bisection defines, reporting every axis including the flat ones.

Axes to vary

Sweep the axes, variations, per-runner knobs, and N budget from flake-axis-bisection, and read a negative result per that skill's rules.

Stage 2 - shared-state isolation check

Run this stage only when Stage 1 implicates parallel execution (the parallelism axis moves the failure rate). It finds the specific shared state - DB rows, env vars, files, ports, lockfiles, or global module state - that workers are colliding on.

  1. Take a baseline inventory. Before running the suite:
    • Snapshot env-vars (env > /tmp/env-pre.txt).
    • Snapshot the test DB schemas (psql -c '\dt').
    • Snapshot the temp dir (ls /tmp > /tmp/tmp-pre.txt).
    • Snapshot listening ports (lsof -i -P | grep LISTEN > /tmp/ports-pre.txt).
  2. Run the suite under instrumentation. Set the runner to one worker (-j 1) AND record per-test resource access. Tools:
    • DB: enable log_statement = 'all' and prefix every test's queries with the test name.
    • Files: wrap fs.writeFileSync to log (test-name, path) pairs.
    • Ports: scrape netstat per-test boundary.
  3. Re-run with 4 workers and compare. For every (test-name, resource) pair, check whether two workers' tests touch the same resource within an overlapping time window.
  4. Classify the collision against the resource-collision classes, discriminating probes, and per-class fixes in flake-axis-bisection.
  5. Emit findings appended to the Stage 1 report:
## Parallel isolation check - `<suite-id>`

**Workers tested:** 1, 4
**Tests instrumented:** N
**Collisions found:** M

| Class      | Resource                          | Test A (worker 1)            | Test B (worker 2)            | Window        | Fix |
|------------|-----------------------------------|------------------------------|------------------------------|---------------|-----|
| DB row     | `users` / id=42                    | `users.spec.ts:12 (writes)`  | `users.spec.ts:30 (reads)`   | 0.4s overlap  | Replace fixed id `42` with per-worker UUID. |
| Port       | 3000                                | `server.spec.ts:5 (binds)`   | `auth.spec.ts:7 (binds)`     | both `EADDRINUSE` | Per-worker `PORT=3000+WORKER_ID`. |
| Module state | `pg-pool` singleton              | `db.spec.ts:1 (queries)`     | `tx.spec.ts:1 (rollback)`    | concurrent    | Per-worker pool via `BeforeAll(() => new Pool())`. |

Stage 2 only finds collisions visible through standard instrumentation. The two collision sources it cannot reach are named in flake-axis-bisection; for those, hand off to a human with the candidate hypothesis-narrowed.

Hand-off

  1. Pending fix, quarantine via flaky-test-quarantine with this bisect report linked from the annotation.
  2. Once isolation is fixed, re-run the bisect. Choose the confirmation run count per flake-axis-bisection: a zero-failure result bounds the rate, it does not prove the flake is gone.

Cost / runtime considerations

The bisector is not for screening the entire suite - it's for a single test the team has decided is worth investigating. For suite-wide visibility, use the trend reporting workflow in flake-dashboard-author.

Blog

Related reading