embedded-coverage-strategy-reference

Overview

This skill is a pure reference consumed by the per-tool skills (googletest-embedded-arm, unity-test-framework-c, ceedling-build-runner, qemu-system-test-runner) and by the HIL reference (hardware-in-loop-reference).

When to use

• Choosing a coverage criterion for a new embedded test suite - what does "enough coverage" mean for this project?
• Wiring gcov or llvm-cov into the cross-compile build.
• Reading a coverage report and translating between gcov, LCOV, and llvm-cov formats.
• Negotiating coverage requirements against a safety standard (MISRA-C, DO-178C, ISO 26262, IEC 62304).
• Deciding whether to instrument the host build, the QEMU build, or the on-target build.

Coverage criteria hierarchy

The standard hierarchy, from weakest to strongest. ISTQB glossary terms (cite by stable term ID; the glossary is a JS SPA and not WebFetchable):

The escalation matters because higher criteria find different defect classes - branch coverage finds an unreached else; MC/DC finds a short-circuit-evaluated condition whose change never alters the decision.

gcov toolchain (GCC, ARM-GCC, AVR-GCC)

Per gcc.gnu.org/onlinedocs/gcc/Invoking-Gcov.html:

Compilation

Build the program (or test harness) with --coverage - a convenience alias that "tells the compiler to generate additional information needed by gcov (basically a flow graph of the program) and also includes additional code in the object files for generating the extra profiling information" (per the GCC gcov invocation page).

arm-none-eabi-gcc --coverage -O0 -g \
    main.c parser.c tests.c -o test_binary
# Equivalent to: -fprofile-arcs -ftest-coverage -lgcov

Compilation produces a .gcno file (flow-graph) per source file "at compile time". Running the binary produces a .gcda file per source file with the accumulated execution counts (both per the GCC docs).

Running gcov

./test_binary           # writes parser.gcda etc.
gcov -b -c parser.c     # produces parser.c.gcov text report

Key flags from the same invocation page:

The text .gcov file annotates each source line with an execution count (or - for non-executable, ##### for unexecuted).

LCOV info format

For HTML reports + CI integration, post-process with lcov/genhtml:

lcov --capture --directory . --output-file coverage.info
genhtml coverage.info --output-directory coverage-html/

LCOV's .info is the de-facto interchange format consumed by Codecov, Coveralls, SonarQube. (LCOV is not GCC; it is a separate Linux Test Project tool that wraps gcov.)

LLVM source-based coverage (clang, arm-linux-clang)

Per clang.llvm.org/docs/SourceBasedCodeCoverage.html:

Compilation

clang --target=arm-none-eabi \
      -fprofile-instr-generate -fcoverage-mapping \
      -O0 -g main.c parser.c tests.c -o test_binary

The two flags are independent: -fprofile-instr-generate enables "instrumentation based profiling"; -fcoverage-mapping emits the mapping that "describes the mapping between the source ranges and the profiling instrumentation counters" (per the LLVM Coverage Mapping Format docs at llvm.org/docs/CoverageMappingFormat.html).

For MC/DC, add -fcoverage-mcdc per the Clang docs.

Running + profile merge

Running the binary writes default.profraw in the current directory (or to the path in LLVM_PROFILE_FILE, with pattern strings %p for PID, %h for hostname, %Nm for merge-pool - per the Clang docs).

LLVM_PROFILE_FILE="raw/%p.profraw" ./test_binary
llvm-profdata merge -sparse raw/*.profraw -o test.profdata

-sparse "produces smaller indexed profiles" (per Clang docs).

Reports

llvm-cov show   ./test_binary -instr-profile=test.profdata \
                              -format=html -output-dir=cov-html/
llvm-cov report ./test_binary -instr-profile=test.profdata
llvm-cov export ./test_binary -instr-profile=test.profdata \
                              -format=lcov > coverage.info

show emits per-line annotations, report emits the file-level summary table, export -format=lcov produces a file compatible with the gcov-flavoured LCOV .info consumed by Codecov / SonarQube. (Per the same Clang Source-Based Code Coverage page.)

LLVM coverage mapping regions

The LLVM coverage-mapping format (per llvm.org/docs/CoverageMappingFormat.html) distinguishes:

• Code regions - associate source ranges with counters.
• Skipped regions - preprocessor-excluded code (e.g. #ifdef branches not taken at compile time).
• Expansion regions - macro expansions, so a macro that fires from multiple call sites has separate coverage per site.
• Branch regions - true / false condition paths (added with -fcoverage-mapping).

This region taxonomy is why llvm-cov can show separate counts inside macro expansions - gcov cannot.

On-target vs host vs QEMU instrumentation

A practical trade-off for embedded teams:

For most safety-critical projects, the standard recipe is host build for fast loop, QEMU build for arch sanity, on-target build for the certification artefact.

Safety-standard coverage expectations

These are cited by stable ID - the standards themselves are gated and not WebFetchable.

The number "100% MC/DC" in aviation is famously expensive; the standard accepts "MC/DC of the integrated executable object code" which is interpreted differently by certifiers. The reader should treat these as the floor, not a turnkey recipe.

Coverage report formats

For embedded CI, the rule of thumb is: produce lcov.info as the durable artefact; render HTML for the developer; gate the PR on the .info totals.

Coverage gates that work

A flat "global ≥85% branch" gate is the failure mode - it penalises the team for unreviewed legacy code and rewards removing tests for hard-to-cover error paths.

Anti-patterns

Limitations

• Statement coverage is the weakest meaningful target - a 100%-statement-covered suite can miss every else branch.
• Branch coverage from gcov has known imprecision on short-circuit && / || - these compile to two branches; the count attributes to the source line, not the individual operand. For per-operand visibility, use clang -fcoverage-mcdc per the Clang docs.
• .gcda files accumulate across runs. Re-run without deleting them and counters keep climbing. Use __gcov_reset() (provided by libgcov) to zero between scenarios - per the gcov source.
• Counter overflow. Default counters are 64-bit on modern GCC but were 32-bit historically - on a long-running on-target run, check gcov-tool overlap for saturated counts.
• No path coverage from either toolchain. Path coverage ("path coverage" per ISTQB) is exponential and neither gcov nor llvm-cov measures it. For path-sensitive testing pair with fuzzing or symbolic execution.

References

Cited inline above. Foundational documents:

• GCC gcov manual - gcc.gnu.org/onlinedocs/gcc/Invoking-Gcov.html.
• Clang Source-Based Code Coverage - clang.llvm.org/docs/SourceBasedCodeCoverage.html.
• LLVM Coverage Mapping Format - llvm.org/docs/CoverageMappingFormat.html.
• ARM GNU Toolchain docs (for --specs=rdimon.specs semihosting) - developer.arm.com/Tools and Software/GNU Toolchain.
• ISTQB glossary V4.7.1 terms cited by stable term ID (glossary is a JS SPA at glossary.istqb.org - not WebFetchable).
• MISRA-C:2012 §8 Coverage Guidance (gated standard - cite by stable ID).
• DO-178C §6.4.4 Structural Coverage (gated standard - cite by stable ID).
• ISO 26262-6:2018 Table 12 (gated standard - cite by stable ID).