regression-suite-selector
Builds a CI workflow that runs only the subset of tests impacted by a PR's changes - combines a per-test → source-file dependency map (built from coverage profiles or, in build-graph projects, queried from the build system itself like Bazel `rdeps`) with the PR's `git diff --name-only`, then selects the union of (impacted by changed files + previously failing + newly added). Always pairs with a periodic full-suite run so a misconfigured map can't silently shrink coverage. Use when the regression suite is large enough that PR-time CI is the bottleneck and a full run is reserved for nightly / pre-release.
Install with skills.sh (any agent)
npx skills add testland/qa --skill regression-suite-selectorregression-suite-selector
Overview
Test Impact Analysis (TIA) runs only the tests a change can affect: map production sources to the tests that exercise them, then intersect that map with the PR diff (tia-fowler (opens in new window)). The map is bidirectional - one test exercises a subset of sources; one source is exercised by a subset of tests. Azure Pipelines builds it from per-test dynamic dependencies; Bazel derives it statically from the build graph (tia-azure (opens in new window)).
This skill builds a TIA-style selector for any team - stitching coverage data, git diff, and a safe fallback policy - without vendor tooling.
When to use
If the build is a Bazel / Pants / Buck monorepo, the selection already comes from the build graph (Step 5) and this skill is mostly orchestration around it.
Step 1 - Decide the selection policy
A robust selector runs impacted + previously-failing + newly-added tests, and falls back to the full suite for any change it can't map (tia-azure (opens in new window)).
The selection set per PR:
impacted ∪ previously_failing ∪ newly_added ∪ (FALLBACK if any change is unmappable)Hard-coded fallback triggers (run everything):
Azure's TIA reasons only about managed code on a single machine; it can't reason about HTML / CSS changes and falls back to running all tests (tia-azure (opens in new window)).
Step 2 - Build the per-test → source map
Emit per-test coverage (not merged), then invert it into a file → tests map. The core inversion:
# scripts/build_test_map.py
def build_map(per_test_coverage):
"""returns {file_path: [test_id, ...]}"""
inverted = defaultdict(list)
for test_id, coverage in per_test_coverage.items():
for file_path, hits in coverage.items():
if any(h > 0 for h in hits):
inverted[file_path].append(test_id)
return dict(inverted)Persist as test-map.json, checked into the repo or stored as a CI artifact updated on every main run. Per-framework recipes for emitting per-test coverage (Jest, pytest + coverage.py, JaCoCo) and the Bazel / Pants / Buck build-graph path (where rdeps gives the map directly) live in references/instrumentation-and-ci.md.
Step 3 - Compute the changed-file set
git diff --name-only origin/${{ github.base_ref }}...HEADImportant: ... (three dots), not ... Three-dot diff is "what changed on this branch since it diverged from main", which matches PR semantics. Two-dot diff is "differences vs current main HEAD" which can show changes the PR didn't make if main moved forward.
Step 4 - Combine
def select_tests(map, changed_files, previously_failing, newly_added):
impacted = set()
for f in changed_files:
if f in map:
impacted.update(map[f])
else:
return ('FALLBACK', f) # unknown file type
return ('SELECTED', impacted | previously_failing | newly_added)previously_failing comes from the most recent full-suite run on main (CI artifact). newly_added comes from git diff --diff-filter=A --name-only filtered to test files.
Step 5 - Add safety: periodic full run + drift detection
Pair selection with a full-suite run so a stale map can't silently shrink coverage. Azure's pattern: run selected tests (T1) and all tests (T2) in sequence and confirm T2 reports the same result as T1 (tia-azure (opens in new window)).
CI workflow (selected run + shadow full run): references/instrumentation-and-ci.md.
Step 6 - Surface the selection
PR-comment summary so reviewers know what ran:
## Test Impact Analysis - `<sha>`
**Selected:** 47 tests of 1,283 total (3.7%)
**Strategy:** impacted ∪ previously_failing ∪ newly_added
**Reason for selection:**
| Source | Tests added |
|---------------------|------------:|
| Impacted by changes | 39 |
| Previously failing | 5 |
| Newly added | 3 |
**Files driving impacted set:**
- `src/checkout/cart.ts` → 12 tests
- `src/checkout/promo.ts` → 18 tests
- `src/api/orders.ts` → 9 tests
**Last full-suite run:** 2026-05-04 22:00 UTC (12 hours ago) - passed.Step 7 - Configurable overrides
The team should be able to opt out for a specific build, matching Azure's DisableTestImpactAnalysis build-variable escape hatch (tia-azure (opens in new window)):
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Selection without periodic full-suite safety net | Map staleness causes missed coverage; bugs ship. | Pattern A or B (Step 5). |
git diff origin/main..HEAD (two dots) | Picks up commits that landed on main after the PR diverged; selection is wrong. | Use three dots (Step 3). |
| Treating an empty map result as "no impacted tests" → run nothing | A new file type (e.g. *.proto) isn't in the map → selector returns nothing → bugs ship. | Fallback to full suite (Step 1). |
Skipping previously_failing from the union | Flaky / known-broken tests don't run; the broken state is invisible. | Always include the previously-failing set (Step 4). |
| Map updated only on full-suite runs that succeed | A failing full-suite run doesn't update the map → next PR uses stale data. | Update the map on every full run regardless of pass/fail (the data is still valid). |
| One global map for a multi-language repo | Per-language coverage tools emit different test IDs; the map merges incorrectly. | Per-language maps + per-language selectors; combine selections, not maps. |
| Selecting only "impacted" without "newly_added" | A new test file with no map entry never runs in PR. | Detect new test files via git diff --diff-filter=A (Step 4). |
| Hard-coded 5-PR full-run cadence with no opt-out | A user with 50-PR streak runs full suite 10× even if they're trivial. | Optional [full-suite] PR title override (Step 7). |
Limitations
References
Per-test instrumentation and CI
View source (opens in new window)Per-test instrumentation and CI
Supporting detail for regression-suite-selector: per-framework recipes for emitting per-test coverage, the Bazel build-graph selection path, and the CI workflow. The core selector (build_map, git diff, select_tests) stays inline in SKILL.md.
Per-test coverage instrumentation (Path A)
Modify the test runner to emit per-test coverage instead of merged coverage, then feed it to build_map (SKILL.md Step 2):
Persist the built map as test-map.json, checked into the repo or stored as a CI artifact updated on every main run.
Path B - From the build graph (Bazel / Pants / Buck)
In Bazel projects, the dependency graph IS the test-source map:
# What tests depend on changed files?
bazel query 'kind("_test", rdeps(//..., set(<changed-files>)))'A Bazel target is actually dependent on target Y if Y must be present, built, and up-to-date for X to build correctly; rdeps(<scope>, <target>) reverses the edge and finds targets that depend on <target> (https://bazel.build/concepts/dependencies).
CHANGED=$(git diff --name-only origin/main...HEAD | sed 's|^|//|')
bazel query "kind('_test', rdeps(//..., set(${CHANGED})))" \
| xargs bazel testDeclared dependencies must comprehensively cover actual dependencies for correct incremental rebuilds, so the build-graph approach is only as good as BUILD-file discipline. Lint via buildozer / gazelle to catch missing declarations.
CI workflow (selected run + shadow full run)
# .github/workflows/regression.yml
jobs:
selected:
runs-on: ubuntu-latest
outputs:
verdict: ${{ steps.run.outcome }}
steps:
- uses: actions/checkout@v5
with: { fetch-depth: 0 } # full history for diff
- name: Compute selection
id: pick
run: |
CHANGED=$(git diff --name-only origin/${{ github.base_ref }}...HEAD)
python scripts/select_tests.py --changed "$CHANGED" --map test-map.json > selection.txt
echo "count=$(wc -l < selection.txt)" >> "$GITHUB_OUTPUT"
- name: Run selected
id: run
run: xargs -a selection.txt npm test --
shadow-full:
if: github.run_attempt == 1 && (github.event.pull_request.number % 5 == 0)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- run: npm test
- name: Compare with selected
run: python scripts/compare_results.py selected.xml shadow.xmlRelated skills
coverage-debt-tracker
Builds a per-file coverage-debt ledger by walking N runs of historical coverage data - flags files whose line% / branch% has slid more than M pp over the period (`falling`), files whose coverage hasn't moved while their churn has (`stale`), and files that lost their last covering test (`orphan`). Emits a sorted backlog the team can ratchet down: each PR fixes one or two debt items, the rest stays visible. Use when whole-repo coverage is "fine" but specific modules are eroding silently and the team needs a stack-ranked list to fix.
test-removal-criteria
Decides which existing tests should stop existing, using five removal classes (duplicate, tautology, trivial, dead-signal, orphan) and a four-condition delete gate where every condition must hold or the verdict reverts to keep. Puts the burden of proof on removal: each deletion carries a written reason, redundant-coverage evidence from a per-test source map, and a named reviewer; a test that has merely never failed is treated as possibly load-bearing, not worthless. Use when a suite has outgrown its signal value and someone is opening a pruning change set that deletes test files; for choosing which tests to RUN per change (not delete) use regression-suite-selector, and to track accumulating low-value coverage over time use coverage-debt-tracker.