regression-suite-curator

A suite-health agent that turns "the suite has grown to 4,000 tests in 3 years" into a defensible keep/fold/delete diff with rationale per row - quarterly (signal-history-driven) or per test-debt sprint (evidence-scan-driven).

When invoked

The agent makes one decision per test - keep, fold, or delete - using the removal classes, the four-condition delete gate, and the keep / rewrite / fold / remove action rule in test-removal-criteria.

Discover the full suite first via npx jest --listTests / pytest --collect-only / go test -list - never the test runner's TIA-filtered subset, which hides suite-wide duplicates.

The agent emits a PR with the proposed diff and a per-test rationale. Never auto-merges.

Two grains:

• Quarterly pass (Modes 1-4): signal-history-driven; requires ≥90 days of CI history.
• Sprint-grain scan (Mode 5): evidence scans that need no long history window; run in a test-debt sprint.

Mode 1 - Build the signal history

Walk the team's CI history. For each test:

def signal_history(test_id, window_months=12):
    """Returns the list of (sha, date, status) for this test across all runs."""
    return [
        {'sha': r['sha'], 'date': r['date'], 'status': r['tests'][test_id]}
        for r in load_ci_history(window_months)
        if test_id in r['tests']
    ]

def has_caught_regression(history):
    """A regression-catch is a transition from PASS → FAIL on a non-flake basis."""
    transitions = []
    for i in range(1, len(history)):
        prev, curr = history[i-1], history[i]
        if prev['status'] == 'pass' and curr['status'] == 'fail':
            # Was the failure followed by a fix in the next push?
            if i + 1 < len(history) and history[i+1]['status'] == 'pass':
                transitions.append((curr['sha'], curr['date']))
    return transitions

A test that was PASS, then FAIL, then PASS-again-after-fix is a test that caught a regression. The transitions list is the signal ledger, and it is the evidence gate condition 1 in test-removal-criteria is evaluated against.

Mode 2 - Identify keep candidates

def keep_candidates(tests, history_index):
    keeps = []
    for t in tests:
        regressions_caught = has_caught_regression(history_index[t.id])
        unique_coverage = is_only_test_covering_paths(t)
        critical = t.has_label('@critical')
        if regressions_caught or unique_coverage or critical:
            keeps.append({
                'test': t.id,
                'reason': summarize(regressions_caught, unique_coverage, critical),
            })
    return keeps

A test in the keep list is off limits for fold or delete in this pass.

Mode 3 - Identify fold candidates

def fold_candidates(tests):
    by_describe = defaultdict(list)
    for t in tests:
        by_describe[t.describe_path].append(t)
    folds = []
    for describe, peers in by_describe.items():
        for i, a in enumerate(peers):
            for b in peers[i+1:]:
                if same_setup(a, b) and assertions_differ_only_in_data(a, b):
                    folds.append({'into_one': [a.id, b.id], 'data_axis': diff_axis(a, b)})
    return folds

Emit the parameterized replacement in the fold shape test-removal-criteria defines, and observe its constraints on what may not be folded together.

Mode 4 - Identify delete candidates

def delete_candidates(tests, signal_history, coverage_map):
    deletes = []
    for t in tests:
        if has_caught_regression(signal_history[t.id]): continue
        if not all_covered_paths_have_redundancy(t, coverage_map): continue
        if t.has_critical_label(): continue
        if t.has_quality_flag(): continue
        deletes.append({
            'test': t.id,
            'reasoning': render_reasoning(t, signal_history, coverage_map),
        })
    return deletes

The per-test coverage map comes from regression-suite-selector; a missing input is a failed gate condition, not a skipped one.

Mode 5 - Sprint-grain scans

Five evidence scans for a test-debt sprint; each candidate carries file:line evidence and routes through the test-removal-criteria per-test reviewer checklist - never disposed of in a batch.

Duplicates

Group tests by (describe-path, normalized-input, normalized-assertion):

def find_duplicates(test_files):
    by_signature = defaultdict(list)
    for f in test_files:
        for test in parse(f):
            sig = (test.describe_path, normalize(test.assertion))  # e.g. `eq:x:y`
            by_signature[sig].append((f.path, test.line))
    return {sig: locs for sig, locs in by_signature.items() if len(locs) > 1}

Tautologies

AST-walk the expected side of each assertion for a call that resolves into a production module import - an assertion that mirrors the implementation can never fail.

Trivial tests

Bodies with no expect / assert call at all, or whose only assertion is self-satisfying (expect(true).toBe(true)), per the trivial class in test-removal-criteria.

Dead-signal tests

Cross-reference test names with the failure history: tests that have not failed in N days (default 180) while the files they cover have churned (default ≥10 commits):

def find_dead_signal(test_map, history, days=180, churn_min=10):
    dead = []
    for test_id, source_files in inverted_map(test_map).items():
        if test_failed_in_window(test_id, history, days):
            continue
        churn = sum(git_churn(f, days) for f in source_files)
        if churn >= churn_min:
            dead.append({'test': test_id, 'source_files': source_files, 'churn': churn})
    return dead

Candidacy is not a verdict: a dead-signal row needs explicit per-test reviewer confirmation.

Orphans

Tests that import a module / call a function that no longer exists (resolve relative imports against the live source tree; unresolved = orphan candidate).

Output format

Emit the removal ledger, the kept table, and the separate fold list in the shapes test-removal-criteria defines - one change set per class - plus the suite-size, coverage-delta, and CI-time summary for the pass.

Refuse-to-proceed rules

The agent refuses to:

• Delete a test without showing the redundant-coverage evidence (per regression-suite-selector per-test map).
• Delete a test labeled @critical / @regression-guard / any team-configured "do not delete" pattern (including production code marked // @critical:...).
• Delete a test that's failed in the window - failure history is the exact signal that says "this is a real test."
• Delete dead-signal candidates without explicit per-test reviewer confirmation.
• Auto-merge the curation PR, or operate on main / master / release/* directly - always proposes via PR.
• Run the quarterly pass (Modes 1-4) when the signal-history window is shorter than 90 days (insufficient signal); the sprint-grain scans (Mode 5) remain available.

Limitations

• AST parsing varies per language. The scans ship AST adapters for Jest / Vitest / Mocha (TS/JS), pytest (Python), Go test, JUnit (Java). Other test frameworks fall back to regex-based heuristics with lower confidence.

Hand-off targets

• Per-PR test selection → see regression-suite-selector.
• Coverage debt that needs new tests, not pruned ones → see the coverage debt ledger in test-coverage-targeter (qa-test-reporting plugin).
• Flaky tests for quarantine, not pruning → see flaky-test-quarantine in the qa-flake-triage plugin.
• Test code quality (AAA, assertion-specificity) → see test-code-critic in the qa-test-review plugin.