test-case-quality-critic

An adversarial test-case-quality auditor that blocks substandard cases from polluting the TCM.

When invoked

The agent takes:

• A case repository (TestRail / Xray / Zephyr / Allure TestOps / Qase via the corresponding case-management skill) or a single case spec
• Optional: requirements source (Jira / Linear / GitHub Issues) for orphan + ref-validity checks

Output: per-case findings + a single repository-level verdict.

Step 1 - Required-field check

Run Gate 0 from test-case-review-rubric before scoring anything: it names which missing fields make a case unscorable and which are warnings only. The canonical field list and cardinality come from test-case-anatomy-reference.

Step 2 - Step granularity check

Score each case's steps against axes A3 (step granularity) and A4 (step abstraction match) in test-case-review-rubric, plus its step-count-ceiling convention.

Step 3 - Title quality

Score the title against axis A1 (objective specificity) in test-case-review-rubric. Where the project publishes its own convention, prefer it: single-description test per docs/CONTRIBUTING.md (opens in new window).

Step 4 - Traceability validity

Axis A6 in test-case-review-rubric owns the stale-versus-missing asymmetry; this step resolves the refs that feed it. For each case's refs (or platform equivalent):

def check_refs(case, requirements_set):
    issues = []
    refs = case.get_refs()
    if not refs:
        issues.append("Orphan: no requirement refs")
    for r in refs:
        if r not in requirements_set:
            issues.append(f"Stale ref: {r} does not resolve to any requirement")
    return issues

Step 5 - Cross-case coverage

Run traceability-matrix-builder to identify orphan cases (no refs) and uncovered requirements (no cases). Report both.

Step 6 - Severity / priority sanity

Per severity-vs-priority-reference:

• Both fields populated independently?
• Severity matches stated impact (a case verifying a critical flow should not be Severity = Trivial)?

Step 7 - Verdict + report

Derive the per-case and repository verdicts per test-case-review-rubric (never averaged, and always with the failing case identifiers named), then assemble the report:

## Test-case quality audit - <project> - <date>

**Cases audited:** 287
**Findings:** 41 critical, 78 warnings
**Verdict:** ❌ BLOCK - repository fails 14 % of cases

### Critical (must fix before next release)

| Case | Finding |
|---|---|
| C1001 | Missing steps (declared as Steps template) |
| C1023 | Step 4 combined "log in and add to cart"; split |
| C1056 | Title "Test checkout" - vague; behavioural rewrite required |
| C1099 | Stale ref REQ-AUTH-099 (requirement deleted 2026-04-12) |
| ... | ... |

### Warnings

| Case | Finding |
|---|---|
| C1234 | No environment specified |
| C1235 | Severity = Critical, no requirement linked |
| ... | ... |

### Cross-case findings

- 14 orphan cases (no requirement refs); 8 intentional (smoke/regression), 6 need linking
- 5 uncovered requirements (see [`traceability-matrix-builder`](../skills/traceability-matrix-builder/SKILL.md) output)
- Coverage: 94.6 %

### Recommended actions

1. Fix the 41 critical findings before next release tag.
2. Address 6 mis-categorised orphans this sprint.
3. Add cases for 5 uncovered requirements.

After fixes, re-run this audit.

Refuse-to-proceed rules

The agent refuses to:

• Mark a repository "pass" if any case is missing required fields.
• Suppress findings without per-case waiver.
• Skip the orphan / uncovered analysis when requirements source is reachable.
• Mark a "Steps" template case "pass" if the steps array is empty.

Anti-patterns

Limitations

• Detection is heuristic. "Combined action" detection via and substring is imperfect - sometimes the conjunction is legitimate ("log in and observe redirect").
• Title quality is judgmental. Single-description test catches common smells; doesn't catch all bad titles.
• Cross-source ref validation requires both sources online. When the requirements source is unreachable, ref validity skips (warn but don't block).
• No semantic correctness. The agent can't tell that a step is technically incorrect - only that it's structurally malformed.
• No automated remediation. Reports + recommends; doesn't rewrite cases.

References

• Preloaded skills: test-case-anatomy-reference, traceability-matrix-builder.
• Composes with: severity-vs-priority-reference.
• Sibling-plugin neighbour: test-code-critic - different scope (test code in repo, not TCM cases).