Testland
Browse all skills & agents

coverage-diff-reporter

Builds a per-PR coverage delta report from any pair of LCOV / Cobertura / JSON coverage outputs (current run + baseline from the merge target) - emits a per-file table with line% / branch% deltas, called-out new files, hidden drops (overall +0.1pp but one file -8pp), and a single-line PR-comment summary. Use when the team has coverage in CI but needs human-readable PR feedback that points at the specific file the reviewer should focus on, not just an aggregate number.

Install with skills.sh (any agent)

npx skills add testland/qa --skill coverage-diff-reporter
View source

coverage-diff-reporter

Overview

A whole-repo coverage gate is necessary but not sufficient. A drop of 0.4pp overall might hide a 12pp drop in one critical file. A new file at 35% line coverage might pass an aggregate gate but leave a regression-risk hot spot.

This skill builds a coverage diff report that solves the "reviewer can see what to look at" problem:

  1. Parse the current run's coverage (LCOV / Cobertura).
  2. Parse a baseline from the merge target.
  3. Emit a per-file delta table sorted by absolute drop.
  4. Highlight new files below threshold, files with disproportionate drops, and files that gained coverage (positive feedback).
  5. Post a single-line summary as the PR top-of-comment, with click-through to the full table.

When to use

  • A PR is missing reviewer signal about which file's coverage changed.
  • The repo has coverage but no PR-time visualization (the data is in the artifacts but nobody opens them).
  • A coverage SaaS isn't an option (compliance, cost) and the team wants a self-hosted equivalent.

This skill does not decide pass/fail - that's the gate's job (see lcov-analysis or cobertura-analysis Step 5). This skill just makes the diff legible.

Step 1 - Pick the parser

Match the existing CI's reporter:

Existing reporterUse
LCOV .infolcov-analysis parser
Cobertura XMLcobertura-analysis parser
Jest JSON / V8 coverageConvert to LCOV first (jest --coverageReporters=lcov)
JaCoCo XMLUse jacoco-analysis, or convert to Cobertura
coverage.pycoverage xml → Cobertura, OR py2lcov → LCOV

The reporter writes to current.json (parsed). The same parser runs against the baseline → baseline.json.

Step 2 - Get the baseline

Two patterns:

Pattern A - cached artifact (recommended)

The main branch's last successful CI run uploaded its coverage as an artifact. PR jobs download it.

- name: Restore baseline
  uses: dawidd6/action-download-artifact@v3
  with:
    workflow: coverage.yml
    branch: main
    name: coverage-baseline
    path: baseline/

- name: Parse current
  run: python scripts/parse_lcov.py coverage/lcov.info > current.json

- name: Parse baseline
  run: python scripts/parse_lcov.py baseline/lcov.info > baseline.json

- name: Generate diff
  run: python scripts/coverage_diff.py current.json baseline.json > diff.md

Pattern B - recompute the baseline in the PR job

The PR job checks out main, runs tests + coverage, then checks out the PR head. Slower (~2x runtime) but always-fresh.

- name: Checkout main
  run: git fetch origin main && git checkout origin/main

- name: Run tests on main
  run: npm test -- --coverage && cp coverage/lcov.info baseline.lcov

- name: Checkout PR head
  run: git checkout ${{ github.event.pull_request.head.sha }}

- name: Run tests on PR head
  run: npm test -- --coverage

Pattern A is the default. Pattern B is the fallback when artifact retention has expired or main coverage is non-deterministic.

Step 3 - Compute the per-file delta

# scripts/coverage_diff.py
def compute_diff(current, baseline):
    base_idx = {f['path']: f for f in baseline}
    rows = []
    for f in current:
        b = base_idx.get(f['path'])
        line_now    = pct(f.get('lh', 0), f.get('lf', 0))
        branch_now  = pct(f.get('brh', 0), f.get('brf', 0))
        line_then   = pct(b.get('lh', 0), b.get('lf', 0)) if b else None
        branch_then = pct(b.get('brh', 0), b.get('brf', 0)) if b else None
        rows.append({
            'path': f['path'],
            'is_new': b is None,
            'line_now': line_now,    'line_delta':   delta(line_now, line_then),
            'branch_now': branch_now,'branch_delta': delta(branch_now, branch_then),
        })
    # Also catch deletions - files in baseline but not current.
    for path, b in base_idx.items():
        if path not in {f['path'] for f in current}:
            rows.append({'path': path, 'is_deleted': True, 'line_now': None, 'line_then': pct(b.get('lh', 0), b.get('lf', 0))})
    return rows

Step 4 - Sort and classify

Reviewers care most about big drops. Sort by line_delta ascending (most-negative first), with new sub-threshold files at the top:

def classify(row):
    if row.get('is_deleted'):                      return 'deleted'
    if row.get('is_new') and row['line_now'] < 80: return 'new_below_threshold'
    if row.get('is_new'):                          return 'new_ok'
    if row['line_delta'] is not None and row['line_delta'] <= -5:   return 'regressed'
    if row['line_delta'] is not None and row['line_delta'] <  0:    return 'declined'
    if row['line_delta'] is not None and row['line_delta'] >  0:    return 'improved'
    return 'unchanged'

The thresholds (80% for new files, -5pp for regression) are tunable per repo.

Step 5 - Render the report

## Coverage diff - `<sha>` vs `main` `<base-sha>`

**Overall:** line 84.2% (-0.3pp) | branch 71.5% (-0.1pp)
**Files changed:** 7 (3 regressed, 1 new, 2 improved, 1 deleted)

### âš  Regressions (4)

| File                                  | Line%       | Branch%     |
|---------------------------------------|-------------|-------------|
| `src/checkout/cart.ts`                | 65.4 (-12.8 ⬇) | 50.0 (-25.0 ⬇) |
| `src/checkout/promo.ts`               | 78.0 (-8.5 ⬇)  | 60.0 (-15.0 ⬇) |

### 🆕 New files (1)

| File                                  | Line%       | Branch%     |
|---------------------------------------|-------------|-------------|
| `src/checkout/discount-stack.ts`      | 35.0 (NEW, below 80% threshold) | 25.0 |

### ✅ Improvements (2)

| File                                  | Line%       | Branch%     |
|---------------------------------------|-------------|-------------|
| `src/orders/list.ts`                  | 92.0 (+4.5 ⬆) | 85.0 (+10.0 ⬆) |

### 🗑 Deleted (1)

| File                                  | Was line%   |
|---------------------------------------|-------------|
| `src/legacy/old-checkout.ts`          | 22.0        |

The four-section split (Regressions / New / Improvements / Deleted) matches reviewer attention budget. Improvements get airtime - positive feedback prevents the gate from feeling adversarial.

Step 6 - One-line summary for the PR top

PR comment APIs render long markdown by default; the summary line sits at the top so the reviewer doesn't have to scroll:

📉 Coverage 84.2% (-0.3pp) - 3 files regressed, 1 new file below threshold. See full report below.

Or if all-clear:

✅ Coverage 84.5% (+0.2pp) - no regressions, 2 files improved.

Step 7 - Post to the PR

- name: Generate diff report
  run: python scripts/coverage_diff.py current.json baseline.json > diff.md

- name: Post / update PR comment
  uses: marocchino/sticky-pull-request-comment@v2
  with:
    header: coverage-diff
    path: diff.md

sticky-pull-request-comment uses the header to update the same comment across pushes - the reviewer doesn't see N copies of the report as the PR evolves.

Anti-patterns

Anti-patternWhy it failsFix
Posting only the aggregate (overall ± Xpp)Hides which file regressed; reviewer can't act.Per-file table sorted by drop (Step 4 - 5).
One thread per push (new comment per commit)PR conversation drowns in coverage churn; nobody reads.Sticky comment updated in place (Step 7).
Showing every unchanged file500-row tables; the 3 regressions are buried.Filter to only changed files; one summary line for unchanged count.
Adversarial framing ("FAIL: coverage dropped")Reviewer associates coverage tool with friction; team disables.Show improvements too (Step 5). Gate failures are the gate's job; this report is informational.
Using PR's merge-base coverage (re-runs main coverage)Doubles CI cost; flake risk on the main re-run.Cache main coverage as artifact (Step 2 Pattern A).
Hiding new files because they "don't have a baseline"New files are exactly where regressions enter the codebase.Always show new files; flag the sub-threshold ones explicitly (Step 4).
Ignoring deleted filesCoverage went up because high-coverage code was deleted; aggregate misleads.Show deletions (Step 3); explain in summary if they cause aggregate movement.

Limitations

  • Per-line uncovered detail isn't shown. This skill is the file-level summary; for per-line drilldown, generate the language-native HTML report (Allure / genhtml / coverage html).
  • No semantic awareness. A 50% drop because the file got 2x bigger (more code, same number of tests) reads the same as a drop because tests were deleted. The reviewer still has to look.
  • Sub-1pp deltas are noise. Coverage tools have measurement jitter from non-deterministic test ordering; show only deltas above a threshold (typical: ±0.5pp).
  • Baseline staleness. Pattern A assumes main's last coverage artifact is recent. If main is stale, surface the baseline age in the report header.

References

  • lcov-analysis - LCOV parser this skill consumes.
  • cobertura-analysis - Cobertura parser this skill consumes.
  • jest-coverage-analysis, jacoco-analysis, coverage-py-analysis - language-specific parsers; convert to LCOV / Cobertura before feeding this skill.
  • test-coverage-targeter - downstream skill that reads the same data to suggest which uncovered branches to target next.

Related skills

allure-reports

Configures Allure Report (test-runner adapter install, `allure-results` directory wiring, `categories.json` for failure classification, `history-trend.json` retention via the copy-history-between-runs pattern), runs the Allure CLI to convert `allure-results` to a static HTML site, and uploads the report as a CI artifact. Use when the team needs richer test reporting than JUnit XML - step-level attachments, per-test history, retry tracking, and severity / epic / feature labeling across framework-agnostic adapters (pytest, Jest, JUnit, TestNG, NUnit, Mocha). As a rich static HTML report generator, it is the open-source alternative to extentreports (JVM/.NET per-test HTML narrative); for hosted cross-run flakiness analytics rather than a static per-run report use currents-integration.

cobertura-analysis

Parses Cobertura XML coverage reports (the JVM-canonical format originally from the cobertura-cobertura tool, also emitted by JaCoCo `--coverage-xml`, coverage.py `--xml`, Istanbul / Jest `cobertura` reporter, gocover-cobertura, and dotnet's `coverlet`). Walks the coverage-04 DTD structure (coverage → packages → classes → methods → lines + conditions), computes per-file deltas, and emits PR-time gating verdicts. Use when the existing CI emits Cobertura XML - typical for JVM-heavy stacks and tools that ship Cobertura as a default reporter.

coverage-py-analysis

Configures coverage.py for Python projects - wires `coverage run` (replacing `python` for instrumentation), enables branch coverage via the `--branch` flag or `branch = True` config, manages the `.coverage` data file (single-process and `combine` for parallel pytest-xdist runs), authors `.coveragerc` with `source` / `omit` / `fail_under`, and emits the format the downstream tool needs (`coverage report` for terminal, `coverage xml` for Cobertura, `coverage html` for human review, `coverage lcov` for SaaS, `coverage json` for programmatic post-processing). Use for any Python test stack (pytest, unittest, nose) that needs PR-time coverage signal.

currents-integration

Wires Currents.dev cross-run test analytics into a Playwright suite: installs `@currents/playwright`, authors `currents.config.ts` (env-sourced `recordKey` + `projectId`), registers `currentsReporter()`, enables trace/video/screenshot artifacts, and runs via `npx pwc` so per-test traces stream to the Currents dashboard with over-time flakiness, slowest-test, and pass-rate trends. Use when a Playwright suite needs hosted cross-run suite-health analytics; for a static per-run report use extentreports or allure-reports, and to sync results into Jira test management use zephyr-integration or xray-integration.

extentreports

Configures ExtentReports v5 for a JVM (or .NET via `extentreports-dotnet`) test run: wires `ExtentSparkReporter`, `attachReporter`, `createTest`, the `info`/`pass`/`warning`/`skip`/`fail` log chain, screenshots via `MediaEntityBuilder`, hierarchical `createNode` parent/child tests, and category/author/device labels, emitting a static HTML report alongside JUnit XML for CI artifact upload. Use when a suite on the Aventstack ExtentReports stack wants a richer per-test HTML narrative than JUnit XML gives; for code-coverage reporting use jacoco-analysis, and for hosted cross-run flakiness analytics use currents-integration.

jacoco-analysis

Configures JaCoCo for JVM projects (Java / Kotlin / Scala / Groovy) - wires the runtime agent via `jacoco-maven-plugin` `prepare-agent`, generates per-build reports (HTML / XML / CSV) via the `report` goal, gates the build via the `check` goal with element / limit / minimum rules, parses the six native counters (instructions, branches, lines, methods, classes, cyclomatic complexity), and converts JaCoCo XML to LCOV / Cobertura when downstream tools need a different format. Use when the JVM build is Maven / Gradle and the team wants the canonical JVM coverage tool - or to convert JaCoCo output for cross-language coverage aggregation.

jest-coverage-analysis

Configures Jest's built-in coverage (Istanbul-instrumented `babel` provider or V8-native `v8` provider), wires the right `coverageReporters` for downstream consumption (`lcov` for SaaS / cross-tool, `cobertura` for Jenkins, `text-summary` for terminal, `html` for human review), authors per-file `coverageThreshold` rules that focus the gate on critical paths (vs the global-only foot-gun), and parses the per-file JSON output for PR-time deltas. Use when the project tests with Jest (or Vitest, which uses the same Istanbul/V8 provider) and the team needs PR-time coverage signal that's both local-runnable and CI-gateable.

junit-xml-analysis

Parses JUnit-format XML reports (the de-facto interchange format every CI ingests - Jenkins, GitHub Actions, GitLab, Buildkite, CircleCI) into structured, machine-readable per-suite and per-case metrics tables (passed / failed / errored / skipped, time, classname, message, stack), groups failures by classname for trend analysis, and distinguishes "new failures vs flakes" by cross-referencing the `flakyFailure` and `rerunFailure` rerun elements. Use when the downstream consumer is a dashboard, script, or aggregator - not when the goal is a human-readable prose summary (use test-run-summary-author for that). Single-run, in-XML aggregation only; for cross-run cross-environment roll-ups, use a cross-run test-suite aggregator.

lcov-analysis

Parses LCOV `.info` text files (the de-facto coverage interchange format produced by gcov, llvm-cov, Coverage.py via `py2lcov`, JaCoCo via `xml2lcov`, Devel::Cover, Jest via `lcov` reporter, NYC, and most others). Extracts per-file line / function / branch metrics from the canonical record keywords (TN/SF/FN/FNDA/FNF/FNH/BRDA/BRF/BRH/DA/LH/LF), computes the diff vs a baseline, and emits per-file gating verdicts. Use for PR coverage gates that don't depend on a specific language runtime.

test-coverage-targeter

Builds a "what to test next" recommendation by combining a coverage report (LCOV / Cobertura / coverage.py JSON / Jest JSON / JaCoCo XML) with the PR's `git diff`, ranking uncovered branches by risk × cost - risk weighted by McCabe cyclomatic complexity and code-churn frequency, cost weighted by the unit-test pyramid layer (unit tests cheaper than integration than E2E). Emits a prioritized list with concrete file:line targets and the test layer recommended for each. Use when a team has the budget to write 5 - 10 new tests and needs help picking which uncovered code to target first instead of blindly chasing 100% coverage.

test-run-summary-author

Build-an-X workflow that turns a structured test-run artifact (JUnit XML, Allure JSON, TestRail / Xray / Zephyr export) plus optional release context (version, build URL, deploy target) into a narrative markdown summary for release notes, an exec status update, or a stand-up Slack post. Distinct from the per-framework parsers junit-xml-analysis / allure-reports / coverage-diff-reporter, which emit structured tabular reports: this skill takes the same data and writes the human-readable narrative. Use when a manager needs a draft release note or stand-up summary from a single run; for cross-run trend analytics use currents-integration.

testrail-integration

Syncs test runs / results / cases between an automated test suite and TestRail (Gurock / Idera) - opens a Test Run for the build (`add_run`), batches per-case results back via `add_results_for_cases` (preferred over per-test `add_result_for_case` - N+1 API calls vs 1), maps the test framework's pass/fail/skip to TestRail status IDs, and attaches build URL + version + elapsed time. Use when the team's test management is standalone TestRail (not a Jira app) and automated suites must update it without a human copy-paste step; when the TCM is instead a Jira app use xray-integration (Xray) or zephyr-integration (Zephyr Scale), and for hosted cross-run flakiness analytics rather than TCM sync use currents-integration.

xray-integration

Imports CI test results into Xray for Jira - authenticates via the `client_id` + `client_secret` → JWT exchange (Cloud) or PAT / Basic (Server), posts to the format-specific `/api/v2/import/execution/*` endpoint (`/junit` for JUnit XML, `/cucumber` for Cucumber JSON, `/nunit` / `/testng` / `/robot` for the others), and maps automated results to existing Xray Test issues via the `xray-junit-extensions` `@XrayTest(key="...")` annotation. Use when the team uses the Xray Jira app to manage Test, Test Set, and Test Execution issue types and CI must keep those execution issues in sync; for the other Jira TCM app use zephyr-integration (Zephyr Scale), for standalone non-Jira TestRail use testrail-integration, and for hosted cross-run flakiness analytics rather than TCM sync use currents-integration.

zephyr-integration

Syncs automated test results to Zephyr Scale for Jira (formerly TM4J / SmartBear / Adaptavist): picks the product variant (Scale Cloud / Squad / Enterprise), authenticates with a long-lived API token as a Bearer header, opens a Test Cycle per build, posts executions via `POST /testexecutions` (or bulk JUnit via `/automations/executions/junit`), and maps test methods to Zephyr Test Cases via `@TestCaseKey`-style annotations. Use when the team's Jira test management is Zephyr Scale; for the Xray Jira app use xray-integration, for standalone TestRail use testrail-integration, and for over-time flakiness analytics rather than TCM sync use currents-integration.