test-pyramid-balancer

Step 1 - Inventory current test mix

Per-language adapters classify by path heuristic:

# scripts/test-mix-inventory.py
import re
from pathlib import Path

EXTENSIONS = {'.js', '.ts', '.py', '.kt', '.java', '.rb', '.go'}

def classify(path_str, content):
    if any(s in path_str for s in ['/playwright/', '/cypress/', '/selenium/', '/e2e/']):
        return 'e2e'
    if any(s in content for s in ['playwright', 'cypress', 'selenium-webdriver']):
        return 'e2e'
    if any(s in path_str for s in ['/integration/', '/it/']):
        return 'integration'
    if 'testcontainers' in content or 'WebApplicationFactory' in content:
        return 'integration'
    return 'unit'   # default

mix = {'unit': 0, 'integration': 0, 'e2e': 0}
for path in Path('.').rglob('*'):
    if path.suffix not in EXTENSIONS:
        continue
    if not re.search(r'(test|spec)\.|test_|_test\.', path.name):
        continue
    content = path.read_text(errors='ignore')
    layer = classify(str(path), content)
    mix[layer] += content.count('test(') + content.count('it(') + content.count('def test_')

print(mix)

Validation checkpoint (Step 1): Before proceeding, sample 5 - 10 tests from each classified bucket and confirm the layer assignment looks correct. If a "unit" test hits a real database, reclassify it as integration. Adjust the classify() heuristics for any systematic misclassification before continuing.

Step 2 - Inventory current runtime

# Per-layer runtime (one-off measurement)
time npm test                    # unit + integration via Jest
time npx playwright test         # E2E

# OR via JUnit XML aggregation per junit-xml-analysis

Step 3 - Compare to ideal ratios

Per test-pyramid (opens in new window): the right ratio depends on the codebase. Defaults:

The change-shape input comes from the change-shape classifier bundled with test-effort-estimation (its references/change-shape-classifier.md), which walks a window of git log and classifies each commit by path and content signal. This step consumes that distribution: it does not recompute it.

Step 4 - Detect anti-patterns

Validation checkpoint (Step 4): Present the diagnosed anti-pattern to the user and ask them to confirm the diagnosis before generating migration recommendations. Large-scale test moves are multi-sprint work; confirm the direction is correct first.

Ice-cream cone (E2E-heavy)

Current: 30 unit / 10 integration / 60 E2E  →  Verdict: ICE-CREAM CONE (60% E2E vs target 5%)

Symptoms: E2E count > unit count; total runtime dominated by E2E; per-PR feedback time >15 min.

Fix: identify E2E tests that test pure logic; rewrite at the unit layer. Keep a hero-flow floor of 5 - 15 E2E tests for critical journeys. Tune target ratios per change shape (Step 3) rather than applying a single universal ratio.

Hourglass (integration-thin)

Current: 200 unit / 8 integration / 30 E2E  →  Verdict: HOURGLASS (3% integration vs target 20%)

Symptoms: many unit + many E2E; very few integration; multi-module bugs slip through (units pass; E2E catches but late).

Fix: add integration tests covering the cross-module seams that unit tests can't reach and E2E tests catch too late. Note that path-based classification can mislead - read file content for hint signals (testcontainers, WebApplicationFactory) to catch unit tests that secretly hit a real DB.

Inverted pyramid

Current: 50 unit / 100 integration / 80 E2E  →  Verdict: INVERTED PYRAMID (heaviest at the top; UI tests dominate)

Symptoms: same as ice-cream cone but with integration-heavy variant; CI is slow; flake is high.

Fix: aggressive layer-down - move tests to lower layers where they catch the same bugs faster. Evaluate using layer ratio + runtime + flake rate together, not runtime alone.

Step 5 - Recommend specific changes

Output a stack-ranked list of layer-changes:

## Test pyramid analysis - `<repo>`

**Date:** YYYY-MM-DD   **Last 90 days commits classified:** 142

### Current

| Layer       | Tests | % of total | Avg runtime | Cost factor |
|-------------|------:|-----------:|------------:|------------:|
| Unit         |   840 |       59%  |       12 ms |       1×    |
| Integration   |    98 |        7%  |      1.4 s  |       3×    |
| E2E          |   485 |       34%  |      8.2 s  |      10×    |

**Verdict:** ICE-CREAM CONE - E2E % (34) far exceeds target (5).

### Recommended (per change shape: 70% service-layer, 30% pure-logic)

| Layer        | Target % | Target tests | Δ        |
|--------------|---------:|-------------:|----------|
| Unit          |      75% |        ~1100 | +260     |
| Integration   |      20% |        ~290  | +192     |
| E2E           |       5% |          ~75 | -410 (!) |

### Top recommendations

1. **Identify E2E tests testing pure logic** (`grep -l "expect.*\bcalculate\|format\|parse" e2e/`).
   Likely candidates: 80-120 tests. Move to unit layer.
2. **Identify E2E tests testing service-layer integration**. Move to
   integration layer with testcontainers.
3. **Review the remaining 75 E2E tests** for hero-flow coverage. If they
   cover 5-10 distinct critical journeys, the suite is healthy.

### Estimated impact

- CI time: ~38 min → ~12 min (per the cost-factor math).
- Flake rate: typical reduction 50-70% (E2E dominates flake).
- Per-PR feedback: <5 min for unit + integration (vs current 15 min).

Step 6 - Confirm improvement

After migrations are complete, re-run scripts/test-mix-inventory.py and compare the new ratios against the targets from Step 3. If any layer is still outside its target band, return to Step 4 to diagnose residual anti-patterns before closing the work.

Step 7 - Cadence

Limitations

• Heuristic classification. Some tests legitimately span layers (an integration test exercising a UI fragment). Manual triage needed for ambiguous cases.
• Doesn't measure test value. Two unit tests of equal runtime can have very different bug-catching power. Pair with mutation testing for value signal.

References

• tp (opens in new window) - Mike Cohn's pyramid: unit / service / UI; "many more low-level UnitTests than high level BroadStackTests"; UI tests "brittle, expensive to write, and time consuming to run."
• test-coverage-targeter - risk-weighted "what to add at unit layer" once the team decides to layer-down.
• e2e-suite-budget - sibling skill for capping E2E suite size.