ai-spec-coverage-mapper

Build-an-X workflow that uses an LLM to map existing tests to spec sections - given a spec doc + the test suite, the LLM identifies which tests cover which sections, surfaces uncovered sections (gap), and recommends specific tests to add. Output is a coverage matrix per spec ID. Scope is mapping tests that already exist and naming the gaps, not authoring tests for new acceptance criteria. Use when a spec doc and a test suite both exist but nobody can say which requirements are actually covered - before a release sign-off, an audit, or a decision about where to spend the next round of test effort.

Install with skills.sh (any agent)

npx skills add testland/qa --skill ai-spec-coverage-mapper

View source

ai-spec-coverage-mapper

Overview

Coverage tools (per lcov-analysis in the qa-test-reporting plugin) report which lines are tested. They don't report which spec sections are tested.

A spec section like "AC-1.4: Already-applied promo shows 'Already applied'" might be:

Covered by a test (good).
Partially covered (the AC says "Already applied"; the test asserts "Already used" - semantic drift).
Not covered.

This skill uses an LLM to map tests to spec sections semantically - the mapping human coverage tools cannot produce.

When to use

The team has a spec doc + a test suite and wants spec-coverage visibility.
Before a release sign-off - to answer "which requirements are actually covered?"
An audit / compliance review needs spec-test traceability.
After AC drift - tests written months ago for ACs that have since changed.

Scope is mapping tests that already exist and naming the gaps - not authoring tests for the gaps (that is ai-test-generator).

How to use

Point the mapper at the spec doc + the test globs, and set the AC-ID extraction pattern.
Run the LLM mapper - it reads the spec and the tests and emits, per AC ID, a coverage tier (full / partial / none) plus the covering test names.
Read the coverage matrix - every partial and none row is a gap.
File one action item per gap (fix the drift, or add the missing test).
Schedule it weekly and verify the LLM's claims - see references/continuous-coverage-and-verification.md.

Worked example - checkout promo spec

Inputs. Point the mapper at the spec and the tests, and declare how AC IDs are written in the spec:

spec_path: "docs/specs/checkout.md"
test_globs:
  - "tests/checkout/**/*.spec.ts"
  - "features/checkout/*.feature"
ac_extraction:
  pattern: "AC-(\\d+\\.\\d+):"   # AC IDs in the spec

Run the mapper. The LLM reads both the spec and the tests and classifies each AC:

# scripts/ai-coverage.py
import openai

spec_text = read(spec_path)
ac_list = extract_acs(spec_text)
test_files = read_all(test_globs)

system_prompt = """
You map AC IDs to tests. For each AC ID, identify:
- which test files cover it
- which test names within those files cover it
- coverage tier: full | partial | none

If partial, explain what aspect is missing.
"""

response = openai.chat.completions.create(
    model='gpt-4',
    messages=[
        {'role': 'system', 'content': system_prompt},
        {'role': 'user', 'content': f"Spec:\n{spec_text}\n\nTests:\n{test_files}"},
    ],
)
print(response.choices[0].message.content)

Output. A coverage matrix per AC, the action items, and a trend vs. the prior run:

## Spec -> test coverage map

**Spec:** `docs/specs/checkout.md`
**ACs:** 12
**Tests inventoried:** 47

### Coverage matrix

| AC ID    | Description                              | Coverage | Tests                                               |
|----------|------------------------------------------|----------|-----------------------------------------------------|
| AC-1.1    | Valid promo applies discount               |   ✅ full | `promo.spec.ts > "applies WELCOME10"`              |
| AC-1.2    | Expired promo shows error                   |   ✅ full | `promo.spec.ts > "shows error for EXPIRED50"`      |
| AC-1.3    | Invalid format shows "Code not found"        |   ✅ full | `promo.spec.ts > "rejects NOTREAL"`                  |
| AC-1.4    | Already-applied promo shows "Already applied" |   ⚠ partial | `promo.spec.ts > "rejects duplicate"`. ⚠ Test asserts "Already used" - message drift from AC. |
| AC-1.5    | Promo applies before tax                    |   ❌ none |                                                       |
| AC-2.1    | Stripe webhook delivery retried              |   ✅ full | `webhook.spec.ts > "retries on 500"`               |
| AC-2.2    | Stripe webhook delivery DLQ after 3 fails    |   ❌ none |                                                       |
| ...

### Action items

| AC ID   | Action                                          |
|---------|-------------------------------------------------|
| AC-1.4   | Update test message assertion to match AC ("Already applied"). |
| AC-1.5   | Add test asserting promo applies before tax.    |
| AC-2.2   | Add test for DLQ-after-3-fails behavior.        |

### Coverage trend

(Compare with prior run)
- AC count: 12 (was 10)
- Full coverage: 9/12 = 75% (was 8/10 = 80%)
- 2 new ACs added in this PR; both uncovered.

Each partial / none row becomes one action item; the two new uncovered ACs are the highest-priority gaps to close before sign-off.

Anti-patterns

Anti-pattern	Why it fails	Fix
Trusting LLM's "full coverage" claim without verification	Hallucination; tests don't actually cover the AC.	Spot-check + cross-reference every high-priority AC (see references).
Running on the entire codebase repeatedly	Cost + slow.	Filter to changed ACs / tests since last run.
One-shot mapping; never updated	Drift; mapping stale.	Schedule weekly (see references).
No action items per gap	Coverage gaps surface but nothing happens.	One action item per gap (see the worked example).

Limitations

LLM quality varies. Same caveats as ai-test-generator.
Spec format matters. Well-structured ACs (numbered, clearly scoped) map cleanly; vague specs produce vague mappings.
No code-execution verification. The LLM reads code; doesn't run it. A test that imports correctly but throws at runtime may be classified "covered."
Test naming influences mapping quality. Per-test names like "test_1" don't help the LLM identify what's covered.

References

references/continuous-coverage-and-verification.md - weekly CI scheduling and how to verify the LLM's coverage claims.
ai-test-generator - sister skill: generates tests for the gaps this skill identifies.
acceptance-test-from-criteria (in the qa-bdd plugin) - for tag-based AC traceability without LLM.
coverage-debt-tracker (in the qa-test-impact-analysis plugin) - line-coverage debt; complementary to spec coverage.

Continuous coverage mapping and LLM-claim verification

View source

Continuous coverage mapping and LLM-claim verification

Deep reference for the ai-spec-coverage-mapper SKILL.md. Consult when scheduling the mapper on a cadence, or when hardening the report against LLM-hallucinated coverage claims.

Run it on a weekly cadence

A single mapping goes stale the moment ACs or tests change. Schedule it weekly and open an issue with the report:

on:
  schedule:
    - cron: '0 4 * * MON'

jobs:
  spec-coverage:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - run: python scripts/ai-coverage.py
      - uses: peter-evans/create-issue-from-file@v5
        with:
          title: 'Spec coverage report - week of ${{ github.event.repository.updated_at }}'
          content-filepath: spec-coverage-report.md

Filter to the ACs and tests changed since the last run to keep cost and runtime down - re-mapping the whole codebase every week is slow and expensive for no new signal.

Verify the LLM's coverage claims

LLMs may claim a test "covers" an AC when it doesn't. Before trusting a full row:

Spot-check the highest-priority ACs manually - read the named test and confirm it asserts what the AC requires.
Cross-reference with acceptance-test-from-criteria (in the qa-bdd plugin) if the team uses @AC-X.Y tags - those tags are the ground truth the LLM's semantic guess can be checked against.
Compare the LLM's claim against the test code in human review - the report is a starting point for triage, not an authority.

The mapper reads code; it does not run it. A test that imports cleanly but throws at runtime can still be classified "covered," so a green matrix is a prompt to verify, not a sign-off on its own.

Related skills

ai-test-generator

Generates tests from natural-language specs (acceptance criteria, user stories, requirements) using an LLM, with confidence scoring per test case (LLM self-assessment plus heuristics: assertion quality, naming, completeness), batching uncertain cases for human review, and integration with the team's existing test framework. Use when the user asks to generate unit tests from acceptance criteria, convert user stories to test cases, automate test creation from requirements, or augment a spec-driven test suite with AI-generated stubs that are then curated before merge.

input-domain-coverage-audit

Audits a test file's input-domain coverage per entry point across three axes: equivalence partitioning (clustering the literal values the tests actually pass, to infer which partitions are exercised), boundary value analysis (recorded n/a when the entry point declares no bound), and error/negative-path coverage (classifying every matcher as positive or negative and computing the negative-assertion ratio). Emits a PASS / SHALLOW / N/A verdict per axis per entry point, with the evidence that produced it. Owns whether the test data spans the input space, not whether an individual assertion is specific enough: matcher specificity belongs to `test-code-conventions`. Use when a test file's cases all look alike - every argument the same shape, every response a success, no thrown-error case - and the suite needs a defensible answer on whether it exercises more than one equivalence class before it is approved.

model-based-test-graph-author

Build-an-X workflow for model-based testing (MBT) per the canonical definition - authors a state-machine model of the SUT (states + transitions + guards + actions), validates the model is connected and complete, and feeds the model to a test generator (manual / AI / dedicated MBT tool) that produces test paths covering each transition. Per Wikipedia (en.wikipedia.org/wiki/Model-based_testing): MBT "leverages model-based design for designing and possibly executing tests." Use when a complex stateful flow (checkout, onboarding, multi-step wizard) needs systematic coverage that ad-hoc tests miss.