test-suite-health-audit

What this owns, and what it does not

This audit owns the measurement layer: how each test file is assigned a tier, how the observed ratio is compared to a target, how flake rate and return per tier are banded into severities, and how those severities reduce to a single categorical verdict.

It deliberately does not own:

Read the boundary this way: if the question is what the ratio should be, that is the balancing decision and this audit does not answer it. If the question is what the ratio is, and how far that is from the number the team already agreed to, that is this audit.

The audit needs a target as input. If the team documented one, use it verbatim. If it did not, say plainly in the output which target you applied and that the team never committed to it, because the severity numbers below are meaningless without naming the baseline they are measured against.

Step 1 - Classify each test file by tier

Apply the rules in order. First match wins, so a file under e2e/ that also imports a database driver is E2E, not integration.

An explicit team convention outranks all three rules. If the suite tags tiers through a runner marker, a config-level project split, or a directory contract, read the tag and use it: the heuristic exists only for suites that never declared one.

The E2E and integration rows lean on the same distinction the pyramid draws: unit tests have "the narrowest scope of all the tests in your test suite", while work that crosses a serialization boundary, including "reading from and writing to databases" and calls to another application's API, is integration territory (Fowler and Vocke, The Practical Test Pyramid).

The classification is a heuristic, and it will be wrong sometimes

Path-based tiering labels location, not behavior. A test sitting under a unit/ directory that opens a real database connection is an integration test, because reading from and writing to a database is exactly what the cited source puts in the integration bucket (Fowler and Vocke, The Practical Test Pyramid). The reverse also happens: a file under integration/ that stubs every collaborator is a unit test wearing the wrong directory.

The import-signal rules in the table above catch the common cases and miss the clever ones, such as a test that imports nothing suspicious but starts an in-process HTTP server in a fixture. Two consequences follow, and both belong in the output:

1. Report a confidence figure, not just a count: the share of files that matched a rule cleanly, excluding files that matched contradictory signals or matched nothing at all.
2. Never present the ratio as exact. It is an estimate with a stated confidence, and every downstream severity inherits that uncertainty.

A team that needs exact tiering should add explicit tier markers. Say so in the output rather than pretending the heuristic is a measurement.

Step 2 - Compare the ratio to the target

Compute the per-layer share of total test files (or of total test cases, if the count is available: state which one you used, because the two diverge when one E2E file holds thirty scenarios). Then compute the signed delta in percentage points against the target for each layer.

What is grounded and what is convention here. The inversion rule is grounded: the cited source warns to "watch out that you don't end up with a test ice-cream cone that will be a nightmare to maintain and takes way too long to run" (Fowler and Vocke, The Practical Test Pyramid), and the pyramid's own premise is that "you should have many more low-level UnitTests than high level BroadStackTests running through a GUI" (Fowler, TestPyramid). A suite with more E2E files than unit files has inverted that premise outright, so it earns the top severity regardless of arithmetic.

The 10 and 25 percentage-point cuts are conventions of this audit, not values published by any source. Their reasoning: the pyramid is explicitly a heuristic whose proportions vary by codebase and team, so a small deviation carries no information and should not generate a finding at all. Ten points is roughly the width of the disagreement between reasonable published targets for the same codebase, which makes it the smallest gap worth reporting. Twenty-five points is the width at which a layer's share has changed category rather than drifted: a layer targeted at 20 percent sitting at 45 percent is not a tuning problem, it is a different suite. Substitute your own cuts if you have historical data; keep them fixed across audits so consecutive reports compare.

Do not convert this into a recommendation for a new target. Report the delta and the severity, and hand the rebalancing decision to whatever owns it.

Step 3 - Per-layer flake rate

For each tier, compute the share of runs that failed without an accompanying code change, over a fixed window of recent runs. A run counts as flaky under the cited definition: "A test is non-deterministic when it passes sometimes and fails sometimes, without any noticeable change in the code, tests, or environment" (Fowler, Eradicating Non-Determinism in Tests).

These four bands are practitioner conventions. No standard defines them, and no cited source publishes a numeric flake threshold. Each cut has a reason, and the reason is what you should argue with, not the number:

• The 2% floor is set by measurement resolution, not by quality. A 50-run window cannot distinguish anything finer than one failure in fifty, which is 2%. Below that cut you are reporting "no flake observed in the window", which is a different statement from "the flake rate is low". Widen the window if you need finer resolution, and move the floor with it.
• The 5% cut is where the cost becomes routine rather than occasional. At one failure in twenty runs, a developer who opens twenty pull requests in a sprint will hit a false red at least once. That is frequent enough to teach the habit of re-running, and the habit is the damage.
• The 15% cut is where the layer stops functioning as a regression signal. At roughly one failure in six, a red result carries almost no information, and the cited source describes exactly where that leads: once a team starts disregarding a regression test failure, the test has lost its value, because nobody can separate a real defect from noise anymore (Fowler, Eradicating Non-Determinism in Tests). That article also warns that quarantine is a holding action rather than a fix, and recommends bounding it with a hard cap on count or on age so quarantined tests do not become permanent.

If no per-run history exists, this axis emits not assessed and the audit continues. A missing flake number is not a Healthy flake number, and the output must not let a reader confuse the two.

Step 4 - ROI per tier

For each tier, compute:

roi(layer) = defects_caught(layer) / total_run_minutes(layer)

total_run_minutes is the layer's wall-clock cost across the same window used in Step 3. defects_caught has no direct measurement, so use a proxy: the count of defect-fix or revert changes in the window that were accompanied by a change to a test in that layer. State which proxy you used and its window.

Then flag any layer whose ROI is below 10 percent of the best-scoring layer as a prune-or-refactor candidate.

This ratio and its 10 percent cut are practitioner conventions. The reasoning for the cut is the weakness of the proxy: a defect-fix-commit count under-counts badly (it misses defects caught before commit, and misses layers whose tests are edited without a fix marker), so a small ROI gap between two layers is indistinguishable from proxy noise. Setting the flag an order of magnitude below the best layer means only a gap far larger than the proxy's error can trigger a recommendation to delete tests. Deleting tests on a weak signal is the expensive mistake here, so the cut is deliberately conservative.

What is grounded is the ordering the ratio tends to produce, and why: cost and execution speed increase as you move up the layers, unit tests "run very fast" while end-to-end tests "require a lot of maintenance and run pretty slowly" (Fowler and Vocke, The Practical Test Pyramid), and UI-driven tests are "brittle, expensive to write, and time consuming to run" (Fowler, TestPyramid). So the denominator grows sharply toward the top of the pyramid. The numerator does not move the same way: the counter-argument is that "as you move up the pyramid, the confidence quotient of each form of testing increases" and that "integration tests strike a great balance on the trade-offs between confidence and speed/expense" (Dodds, Write tests. Not too many. Mostly integration.). Expect the middle layer to score well and expect a bottom-heavy suite of trivial unit tests to score worse than its file count suggests. Neither expectation is a target: report what the numbers say.

A low-ROI unit layer is a different finding from a low-ROI E2E layer. Cheap tests with low return are usually redundant and can be pruned; expensive tests with low return are usually testing the wrong thing at the wrong level and want rewriting. Say which one you found.

Step 5 - Reduce to one verdict

Evaluate in order. The first matching row is the verdict.

The pruning-versus-refactor split is a convention of this audit, and the line is the shape of the remediation, not its size: pruning means the fix is deletion (there is too much of something that returns little), while refactor means the fix is rewriting or moving tests (the coverage exists at the wrong level, or exists but cannot be trusted). Splitting on remediation shape is what makes the verdict actionable, because the two land on different people in different sprints.

What makes a suite un-assessable

Emit Cannot assess and name which condition fired. Do not emit a softened verdict instead: an audit that guesses is worse than one that abstains, because the guess gets quoted later without its caveat.

When you emit Cannot assess, name the specific input that would unlock it: a tier-marker convention, a directory split, a runner config tag, a documented target ratio, or a larger sample. "Insufficient data" without that sentence is not a useful output.

Output format

Emit one Markdown block, in this order.

## Suite health verdict: <Healthy | Needs pruning | Needs refactor | Cannot assess>

**Suite:** <identifier>   **Window:** <dates, run count>
**Target ratio applied:** <unit/int/e2e> (<documented by the team | applied as a
default because none was documented>)
**Tier classification confidence:** <n>% of <total> files tiered cleanly

| Axis | Severity | Finding | Remediation shape |
|---|---|---|---|
| Tier classification | <severity or n/a> | <% clean, what defeated the rest> | <add tier markers / keep> |
| Pyramid ratio | <severity> | <actual vs target, signed delta per layer> | <prune / add / keep> |
| Flake rate per layer | <severity or not assessed> | <rate per layer, or the missing input> | <quarantine / keep> |
| ROI per tier | <severity or not assessed> | <ratio per layer, which layer is the reference> | <prune / rewrite / keep> |

## Top 3 findings

1. <largest blast radius first, naming the axis and the number behind it>
2. <second>
3. <third>

## Not assessed

- <every axis that emitted `not assessed`, with the input that would fix it>
- <the confidence gap, if classification landed under 100%>

## Not decided here

The target ratio and the rebalancing plan are a separate decision. This report
measures distance from the target named above; it does not propose a new one.

Keep the last section even when it feels redundant. It is what stops a reader from treating a measurement as a mandate.

Worked examples

Example 1 - inverted pyramid, no run history

A suite of 50 unit, 100 integration, and 200 E2E files. No CI history available. Team documented a 70/20/10 target.

• Step 1: 350 files, 331 tiered cleanly (95%). Above the 80% floor, so the audit proceeds.
• Step 2: actual 14/29/57. E2E file count (200) exceeds unit file count (50), so the axis is Critical by the inversion rule, whatever the deltas say.
• Step 3: not assessed, no run history.
• Step 4: not assessed, no defect-fix history in the window.
• Step 5: a Critical exists, so the verdict is Needs refactor. Top finding: the suite is an ice-cream cone, and the remediation shape is moving coverage down a layer, not deleting it.
• "Not assessed" lists both missing inputs so nobody reads the silence on flake as a clean bill of health.

Example 2 - healthy

A suite of 350 unit, 80 integration, and 30 E2E files against a documented 70/20/10 target, with 200 runs of history.

• Step 2: actual 76/17/7. Worst delta is 6 points (unit), inside the 10-point band. Axis Healthy.
• Step 3: 0.5% unit, 1.1% integration, 1.8% E2E, all under the 2% floor. Axis Healthy, reported as "no flake observed in a 200-run window" rather than as a rate.
• Step 4: integration scores highest; unit at 62% of the integration figure and E2E at 31%. Nothing below the 10% flag. Axis Healthy.
• Step 5: no Critical, no Important. Verdict Healthy.

Example 3 - un-assessable

A flat directory of 60 test files with no tier convention, no runner markers, and imports that mix a database client and a component renderer in the same files.

• Step 1: 38 of 60 files tiered cleanly (63%), below the 80% floor.
• Verdict: Cannot assess (tier classification confidence 63%).
• Unlocking input, stated explicitly: split the directory by tier, or add a runner-level tier marker to each file, then re-run. Do not report a ratio in the meantime; a 63%-confidence ratio will be quoted as fact within a week.

Anti-patterns

Limitations

• Static, not runtime. Tiering reads paths and imports. It does not run the suite, so a test that behaves differently from how it reads is misfiled.
• The ratio is over files unless test cases are counted. One E2E file with thirty scenarios and one unit file with one assertion weigh the same in a file-count ratio. Count cases when the runner exposes them, and say which you used.
• The flake window bounds the resolution. Every band in Step 3 is limited by how many runs you have. Do not report a 1% rate from a 50-run window.
• The defect proxy is weak. Teams that squash commits, omit fix markers, or fix defects without touching a test will see an ROI axis that is not assessed or misleading. Prefer not assessed over a proxy you do not trust.
• One repository per run. A monorepo's test directories can be walked together, but a separate test-only repository is a separate audit: merging them averages away the signal that would drive any decision.
• No effort estimate. Findings are ranked by blast radius. What each fix costs depends on team familiarity with the code and is outside this scope.

References

• Fowler and Vocke, The Practical Test Pyramid: unit tests have "the narrowest scope of all the tests in your test suite"; work that reads from and writes to databases or calls another application's API is integration testing; unit tests "run very fast" and end-to-end tests "require a lot of maintenance and run pretty slowly"; "watch out that you don't end up with a test ice-cream cone that will be a nightmare to maintain and takes way too long to run"; the pyramid's proportions are a heuristic to adapt, not a fixed doctrine.
• Fowler, TestPyramid: "you should have many more low-level UnitTests than high level BroadStackTests running through a GUI"; tests that run end-to-end through the UI are "brittle, expensive to write, and time consuming to run"; testing through the UI is slow and more prone to non-determinism.
• Fowler, Eradicating Non-Determinism in Tests: "A test is non-deterministic when it passes sometimes and fails sometimes, without any noticeable change in the code, tests, or environment"; a regression test that the team has started to disregard has lost its value; quarantine is a holding action that needs a hard cap on count or age.
• Dodds, Write tests. Not too many. Mostly integration.: "as you move up the pyramid, the confidence quotient of each form of testing increases"; "Integration tests strike a great balance on the trade-offs between confidence and speed/expense"; "When you mock something you're removing all confidence in the integration between what you're testing and what's being mocked."