qa-flake-triage
Flake triage: 4 skills (flake-dashboard-author, flake-pattern-reference, flake-remediation-guide, flaky-test-quarantine) and 5 agents (ai-flake-detector, e2e-flake-bisector, e2e-test-trend-reporter, parallel-isolation-checker, regression-bisector).
Install this plugin
/plugin install qa-flake-triage@testland-qaPart of role bundles: qa-starter, qa-role-automation-engineer, qa-role-sdet
qa-flake-triage
Flake triage workflow: axis bisector with shared-state isolation checking, regression bisector (pass/fail and perf-measurement modes), flake pattern catalog with per-pattern code fixes, dashboards with periodic trend reports, and the quarantine workflow.
Components
| Type | Name | Description |
|---|---|---|
| Skill | flaky-test-quarantine | Quarantine workflow: mark, annotate (rate + bisect link + expiry), auto-expiry report, two-renewal cap. |
| Skill | flake-pattern-reference | Catalog of 8 flake patterns (timing, ordering, shared state, leaks, network, locator, environment, randomness) with detection signals, remediation, and per-pattern code fixes in references. |
| Skill | flake-dashboard-author | Build a persistent flakiness dashboard from run history (Grafana / Datadog CI Visibility), plus the weekly / monthly trend report with week-over-week deltas. |
| Skill | flake-axis-bisection | Finds which condition a flaky test depends on by varying one axis at a time, with confidence intervals on the measured rate and a rule for when a difference exceeds sampling noise. |
| Agent | e2e-flake-bisector | Vary one axis at a time (worker count, random order, network throttle, viewport, animations, OS, sequential reps) over N runs to localize the flake source; when parallelism is implicated, stage 2 finds the shared state two workers collide on with file:line evidence. |
| Agent | regression-bisector | git bisect run orchestrator: build the test script, mark good/bad, handle exit-125 skips, report the introducing commit; perf-measurement mode bisects on a k6 / Lighthouse budget instead of pass/fail. |
Install
/plugin marketplace add testland/qa
/plugin install qa-flake-triage@testland-qaSkills
flake-axis-bisection
Locates the condition a known-flaky test actually depends on by holding the test constant and varying one axis at a time (isolation, execution order, worker count, viewport, network latency, repetition depth), recording a pass/fail count per variation, and testing whether the gap between two conditions exceeds sampling noise. Covers choosing the run count N from the failure rate you need to detect, binomial confidence intervals on a measured reproduction rate, a two-proportion comparison rule, what a zero-failure result does and does not prove, and the resource-collision walk (DB row, DB schema, file path, port, env var, module state, inode, cookie jar) used once parallelism is implicated. Use when a specific test is already known to fail intermittently, reading its source has not explained why, and a decision about what to change must rest on measurement rather than on a plausible-sounding guess.
flake-dashboard-author
Builds a persistent flakiness infrastructure dashboard from JUnit XML or JSON CI run history: defines the flake-rate metric (failures per test over a configurable window), authors the data model, generates a Grafana time-series panel JSON or configures a Datadog CI Visibility view, derives the quarantine-candidate query, and wires trend alerts. Also generates the periodic (weekly / monthly) test-suite trend report - total runs, suite duration, flakiness rate, top failing tests, time-to-green per PR, week-over-week deltas - as a markdown summary for a team Slack channel or wiki page. Use when a team needs a long-lived observability surface for test reliability, or a scheduled comparable health report on top of it.
flake-pattern-reference
Reference catalog of the eight flake patterns - async/timing, test ordering, shared parallel state, resource leaks, network, locator drift, environment variance, randomness - with detection heuristics, remediation per pattern, and the concrete code-level fixes: replacing fixed sleeps with framework auto-waits, isolating state in beforeEach fixtures, per-worker DB schemas via workerIndex, try/finally teardown, mocking network + clock at the boundary, stable role-based locators, TZ pinning, and RNG seeding. Use when triaging an unknown flake to identify the category before bisecting, or when a classified flake needs the specific code change to apply.
flaky-test-quarantine
Builds a quarantine workflow for flaky tests - marks the test with the framework's skip/fixme/retry annotation, records the failure-rate observation and a bisect link in the annotation body, sets an auto-expiry date, and produces a CI report listing every quarantined test that has expired and needs re-evaluation. Use when a flaky test is blocking the trunk and must be removed from the gating path without losing track of it.
Agents
e2e-flake-bisector
Runs a target end-to-end test N times under varied conditions (worker isolation, test order, viewport, network throttling, parallelism) to identify the axis along which the flake reproduces, then - when the bisect implicates parallel execution - runs a stage-2 shared-state isolation check that instruments DB rows, env vars, files, ports, and global module state to name the specific resource two workers are colliding on, with file:line evidence. Returns a probable root cause classified against the 8 flake patterns plus a numeric reproduction rate per axis. Use when a test has been flagged flaky and the team needs to know which condition triggers the failure.
regression-bisector
Orchestrates `git bisect` against a target test or build script to identify the introducing commit of a regression. Wraps the bad/good marking, the `git bisect run` script, the 125 exit code for unbuildable revisions, and the final culprit report. Includes a perf-measurement mode that bisects on a perf metric threshold instead of pass/fail - a per-commit k6 or Lighthouse CI run with a single budget assertion - and hands the culprit's hot path to flame-graph or query-plan analysis. Use when a test that previously passed has started failing 100% of the time on the trunk, or when load testing / Lighthouse CI shows a perf regression whose introducing commit is unclear.