coverage-diff-reporter
Builds a per-PR coverage delta report from any pair of LCOV / Cobertura / JSON coverage outputs (current run + baseline from the merge target) - emits a per-file table with line% / branch% deltas, called-out new files, hidden drops (overall +0.1pp but one file -8pp), and a single-line PR-comment summary. Use when the team has coverage in CI but needs human-readable PR feedback that points at the specific file the reviewer should focus on, not just an aggregate number.
Install with skills.sh (any agent)
npx skills add testland/qa --skill coverage-diff-reportercoverage-diff-reporter
Overview
A whole-repo coverage gate is necessary but not sufficient. A drop of 0.4pp overall might hide a 12pp drop in one critical file. A new file at 35% line coverage might pass an aggregate gate but leave a regression-risk hot spot.
This skill builds a coverage diff report that solves the "reviewer can see what to look at" problem:
When to use
This skill does not decide pass/fail - that's the gate's job (see lcov-analysis or lcov-analysis references/cobertura.md). This skill just makes the diff legible.
Step 1 - Pick the parser
Match the existing CI's reporter:
| Existing reporter | Use |
|---|---|
LCOV .info | lcov-analysis parser |
| Cobertura XML | lcov-analysis (references/cobertura.md) parser |
| Jest JSON / V8 coverage | Convert to LCOV first (jest --coverageReporters=lcov) |
| JaCoCo XML | Use jacoco-analysis, or convert to Cobertura |
| coverage.py | coverage xml → Cobertura, OR py2lcov → LCOV |
The reporter writes to current.json (parsed). The same parser runs against the baseline → baseline.json.
Step 2 - Get the baseline
Two patterns:
Pattern A - cached artifact (recommended)
The main branch's last successful CI run uploaded its coverage as an artifact. PR jobs download it.
- name: Restore baseline
uses: dawidd6/action-download-artifact@v3
with:
workflow: coverage.yml
branch: main
name: coverage-baseline
path: baseline/
- name: Parse current
run: python scripts/parse_lcov.py coverage/lcov.info > current.json
- name: Parse baseline
run: python scripts/parse_lcov.py baseline/lcov.info > baseline.json
- name: Generate diff
run: python scripts/coverage_diff.py current.json baseline.json > diff.mdPattern B - recompute the baseline in the PR job
The PR job checks out main, runs tests + coverage, then checks out the PR head. Slower (~2x runtime) but always-fresh.
- name: Checkout main
run: git fetch origin main && git checkout origin/main
- name: Run tests on main
run: npm test -- --coverage && cp coverage/lcov.info baseline.lcov
- name: Checkout PR head
run: git checkout ${{ github.event.pull_request.head.sha }}
- name: Run tests on PR head
run: npm test -- --coveragePattern A is the default. Pattern B is the fallback when artifact retention has expired or main coverage is non-deterministic.
Step 3 - Compute the per-file delta
# scripts/coverage_diff.py
def compute_diff(current, baseline):
base_idx = {f['path']: f for f in baseline}
rows = []
for f in current:
b = base_idx.get(f['path'])
line_now = pct(f.get('lh', 0), f.get('lf', 0))
branch_now = pct(f.get('brh', 0), f.get('brf', 0))
line_then = pct(b.get('lh', 0), b.get('lf', 0)) if b else None
branch_then = pct(b.get('brh', 0), b.get('brf', 0)) if b else None
rows.append({
'path': f['path'],
'is_new': b is None,
'line_now': line_now, 'line_delta': delta(line_now, line_then),
'branch_now': branch_now,'branch_delta': delta(branch_now, branch_then),
})
# Also catch deletions - files in baseline but not current.
for path, b in base_idx.items():
if path not in {f['path'] for f in current}:
rows.append({'path': path, 'is_deleted': True, 'line_now': None, 'line_then': pct(b.get('lh', 0), b.get('lf', 0))})
return rowsStep 4 - Sort and classify
Reviewers care most about big drops. Sort by line_delta ascending (most-negative first), with new sub-threshold files at the top:
def classify(row):
if row.get('is_deleted'): return 'deleted'
if row.get('is_new') and row['line_now'] < 80: return 'new_below_threshold'
if row.get('is_new'): return 'new_ok'
if row['line_delta'] is not None and row['line_delta'] <= -5: return 'regressed'
if row['line_delta'] is not None and row['line_delta'] < 0: return 'declined'
if row['line_delta'] is not None and row['line_delta'] > 0: return 'improved'
return 'unchanged'The thresholds (80% for new files, -5pp for regression) are tunable per repo.
Step 5 - Render the report
## Coverage diff - `<sha>` vs `main` `<base-sha>`
**Overall:** line 84.2% (-0.3pp) | branch 71.5% (-0.1pp)
**Files changed:** 7 (3 regressed, 1 new, 2 improved, 1 deleted)
### ⚠ Regressions (4)
| File | Line% | Branch% |
|---------------------------------------|-------------|-------------|
| `src/checkout/cart.ts` | 65.4 (-12.8 ⬇) | 50.0 (-25.0 ⬇) |
| `src/checkout/promo.ts` | 78.0 (-8.5 ⬇) | 60.0 (-15.0 ⬇) |
### 🆕 New files (1)
| File | Line% | Branch% |
|---------------------------------------|-------------|-------------|
| `src/checkout/discount-stack.ts` | 35.0 (NEW, below 80% threshold) | 25.0 |
### ✅ Improvements (2)
| File | Line% | Branch% |
|---------------------------------------|-------------|-------------|
| `src/orders/list.ts` | 92.0 (+4.5 ⬆) | 85.0 (+10.0 ⬆) |
### 🗑 Deleted (1)
| File | Was line% |
|---------------------------------------|-------------|
| `src/legacy/old-checkout.ts` | 22.0 |The four-section split (Regressions / New / Improvements / Deleted) matches reviewer attention budget. Improvements get airtime - positive feedback prevents the gate from feeling adversarial.
Step 6 - One-line summary for the PR top
PR comment APIs render long markdown by default; the summary line sits at the top so the reviewer doesn't have to scroll:
📉 Coverage 84.2% (-0.3pp) - 3 files regressed, 1 new file below threshold. See full report below.Or if all-clear:
✅ Coverage 84.5% (+0.2pp) - no regressions, 2 files improved.Step 7 - Post to the PR
- name: Generate diff report
run: python scripts/coverage_diff.py current.json baseline.json > diff.md
- name: Post / update PR comment
uses: marocchino/sticky-pull-request-comment@v2
with:
header: coverage-diff
path: diff.mdsticky-pull-request-comment uses the header to update the same comment across pushes - the reviewer doesn't see N copies of the report as the PR evolves.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Posting only the aggregate (overall ± Xpp) | Hides which file regressed; reviewer can't act. | Per-file table sorted by drop (Step 4 - 5). |
| One thread per push (new comment per commit) | PR conversation drowns in coverage churn; nobody reads. | Sticky comment updated in place (Step 7). |
| Showing every unchanged file | 500-row tables; the 3 regressions are buried. | Filter to only changed files; one summary line for unchanged count. |
| Adversarial framing ("FAIL: coverage dropped") | Reviewer associates coverage tool with friction; team disables. | Show improvements too (Step 5). Gate failures are the gate's job; this report is informational. |
| Using PR's merge-base coverage (re-runs main coverage) | Doubles CI cost; flake risk on the main re-run. | Cache main coverage as artifact (Step 2 Pattern A). |
| Hiding new files because they "don't have a baseline" | New files are exactly where regressions enter the codebase. | Always show new files; flag the sub-threshold ones explicitly (Step 4). |
| Ignoring deleted files | Coverage went up because high-coverage code was deleted; aggregate misleads. | Show deletions (Step 3); explain in summary if they cause aggregate movement. |
Limitations
References
Related skills
allure-reports
Configures Allure Report (test-runner adapter install, `allure-results` directory wiring, `categories.json` for failure classification, `history-trend.json` retention via the copy-history-between-runs pattern), runs the Allure CLI to convert `allure-results` to a static HTML site, and uploads the report as a CI artifact. Use when the team needs richer test reporting than JUnit XML - step-level attachments, per-test history, retry tracking, and severity / epic / feature labeling across framework-agnostic adapters (pytest, Jest, JUnit, TestNG, NUnit, Mocha). As a rich static HTML report generator, it is the open-source alternative to the sunset ExtentReports (JVM/.NET per-test HTML narrative); for hosted cross-run flakiness analytics rather than a static per-run report use currents-integration.
coverage-py-analysis
Configures coverage.py for Python projects - wires `coverage run` (replacing `python` for instrumentation), enables branch coverage via the `--branch` flag or `branch = True` config, manages the `.coverage` data file (single-process and `combine` for parallel pytest-xdist runs), authors `.coveragerc` with `source` / `omit` / `fail_under`, and emits the format the downstream tool needs (`coverage report` for terminal, `coverage xml` for Cobertura, `coverage html` for human review, `coverage lcov` for SaaS, `coverage json` for programmatic post-processing). Use for any Python test stack (pytest, unittest, nose) that needs PR-time coverage signal.
currents-integration
Wires Currents.dev cross-run test analytics into a Playwright suite: installs `@currents/playwright`, authors `currents.config.ts` (env-sourced `recordKey` + `projectId`), registers `currentsReporter()`, enables trace/video/screenshot artifacts, and runs via `npx pwc` so per-test traces stream to the Currents dashboard with over-time flakiness, slowest-test, and pass-rate trends. Use when a Playwright suite needs hosted cross-run suite-health analytics; for a static per-run report use extentreports or allure-reports, and to sync results into TestRail / Xray / Zephyr test management use test-management-sync.
jacoco-analysis
Configures JaCoCo for JVM projects (Java / Kotlin / Scala / Groovy) - wires the runtime agent via `jacoco-maven-plugin` `prepare-agent`, generates per-build reports (HTML / XML / CSV) via the `report` goal, gates the build via the `check` goal with element / limit / minimum rules, parses the six native counters (instructions, branches, lines, methods, classes, cyclomatic complexity), and converts JaCoCo XML to LCOV / Cobertura when downstream tools need a different format. Use when the JVM build is Maven / Gradle and the team wants the canonical JVM coverage tool - or to convert JaCoCo output for cross-language coverage aggregation.
junit-xml-analysis
Explains CI test numbers that disagree with what the suite actually did - a 'slowest tests' list dominated by the wrong suite, a release gate or dashboard reading only the summary attributes on the suite element and never the cases below them, or a pass rate that quietly counts skipped tests as passes. Parses JUnit-format XML (the interchange format Jenkins, GitHub Actions, GitLab, Buildkite, and CircleCI all ingest) into per-suite and per-case metrics tables - passed / failed / errored / skipped, time, classname, message, stack - groups failures by classname for trend analysis, and separates new failures from flakes by cross-referencing the `flakyFailure` and rerun elements. Use when a report, gate, or metric derived from test results cannot be trusted.
lcov-analysis
Parses both mainstream coverage interchange formats: LCOV `.info` text files (produced by gcov, llvm-cov, Coverage.py via `py2lcov`, JaCoCo via `xml2lcov`, Devel::Cover, Jest via `lcov` reporter, NYC, and most others) and Cobertura XML (coverage-04.dtd - emitted by JaCoCo, coverage.py `--xml`, Jest's `cobertura` reporter, coverlet, gocover-cobertura; full parser in references/cobertura.md). Extracts per-file line / function / branch metrics from the canonical record keywords (TN/SF/FN/FNDA/FNF/FNH/BRDA/BRF/BRH/DA/LH/LF), computes the diff vs a baseline, and emits per-file gating verdicts. Use for PR coverage gates that don't depend on a specific language runtime, whichever of the two formats the CI emits.
test-coverage-targeter
Builds a "what to test next" recommendation by combining a coverage report (LCOV / Cobertura / coverage.py JSON / Jest JSON / JaCoCo XML) with the PR's `git diff`, ranking uncovered branches by risk × cost - risk weighted by McCabe cyclomatic complexity and code-churn frequency, cost weighted by the unit-test pyramid layer (unit tests cheaper than integration than E2E). Also carries the coverage debt ledger: a weekly per-file drift report over N historical main runs flagging `falling` (line% slid >M pp from peak), `stale` (flat coverage + high churn), and `orphan` (lost last covering test) files, whose rows feed the same targeting. Emits a prioritized list with concrete file:line targets and the test layer recommended for each. Use when a team has the budget to write 5 - 10 new tests and needs help picking which uncovered code to target first instead of blindly chasing 100% coverage, or when specific modules are eroding silently while whole-repo coverage looks fine.
test-management-sync
Syncs automated test results into test management tools - TestRail (standalone, `add_run` + batched `add_results_for_cases`), Xray for Jira (JWT auth + `/api/v2/import/execution/*`), and Zephyr Scale (Bearer token + `/testexecutions`) - from CI. The body carries the vendor-independent push-results workflow (map tests to case IDs, open a run / execution / cycle per build, batch results back, close on main only, run as an `if: always()` step) with TestRail as the worked example; full vendor specifics live in references/ (testrail.md, xray.md, zephyr.md). Use when automated suites must keep the team's test management view in sync without a human copy-paste step; for hosted cross-run flakiness analytics rather than TCM sync use currents-integration, and for authoring / migrating test CASES rather than pushing results see qa-test-management's tcm-case-management.
test-run-summary-author
Build-an-X workflow that turns a structured test-run artifact (JUnit XML, Allure JSON, TestRail / Xray / Zephyr export) plus optional release context (version, build URL, deploy target) into a narrative markdown summary for release notes, an exec status update, or a stand-up Slack post. Distinct from the per-framework parsers junit-xml-analysis / allure-reports / coverage-diff-reporter, which emit structured tabular reports: this skill takes the same data and writes the human-readable narrative. Use when a manager needs a draft release note or stand-up summary from a single run; for cross-run trend analytics use currents-integration.