crash-triage-reference
Pure-reference catalog for manually triaging individual fuzzer crash artifacts - reading ASan, UBSan, and MSan output; classifying findings as LIKELY-EXPLOITABLE, MEDIUM, or BENIGN; deduplicating by stack-hash; and minimizing reproducers with -minimize_crash. Use when you need to understand what a specific crash means, build exploitability intuition, or manually work a small set of findings. For automated bulk triage across a full artifact directory, use fuzz-findings-critic instead.
crash-triage-reference
Overview
Pure-reference catalog for working with fuzzer crash artifacts produced by libFuzzer, AFL++, or cargo-fuzz campaigns using clang sanitisers. Covers reading crash output from ASan, UBSan, and MSan; distinguishing LIKELY-EXPLOITABLE from BENIGN findings; collapsing duplicates by stack-hash; and minimizing reproducers. Consumed by the fuzz-findings-critic agent, which automates these steps across a full artifact directory. For sanitiser build flags and compatibility, see sanitiser-integration-reference.
When to use
Reading ASan crash output
AddressSanitizer (per clang.llvm.org/docs/AddressSanitizer.html) reports all memory errors through a structured output block. The ==ERROR: line always carries the bug class, and the access line carries the direction (READ or WRITE) and size.
Annotated example:
==1234==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7f...
WRITE of size 4 at 0x7f... thread T0 <-- direction + size
#0 0x4015a3 in process_input src/parser.c:42:5
#1 0x4012f0 in LLVMFuzzerTestOneInput fuzz_target.cc:10:3
#2 ...
0x7f... is located 0 bytes to the right of 16-byte region [0x7f..., 0x7f...)
allocated by thread T0 here: <-- allocation site
#0 0x40e7c0 in __interceptor_malloc
#1 0x4015a3 in process_input src/parser.c:39:9
freed by thread T0 here: <-- deallocation site (UAF only)
#0 0x40e8b0 in __interceptor_free
...Key fields (per clang.llvm.org/docs/AddressSanitizer.html):
| Field | Location in output | What to extract |
|---|---|---|
| Bug class | ==ERROR: AddressSanitizer: <class> line | e.g. heap-buffer-overflow, use-after-free |
| Access direction | READ / WRITE of size N | determines exploitability tier |
| Crash site | #0 frame after ==ERROR | source file + line of the fault |
| Allocation site | after allocated by thread T0 here: | where the corrupted memory came from |
| Deallocation site | after freed by thread T0 here: | present only for use-after-free |
ASan detects: out-of-bounds accesses to heap, stack, and globals; use-after-free; double-free; invalid free; memory leaks (per clang.llvm.org/docs/AddressSanitizer.html). It does not produce false positives.
Reading UBSan crash output
UndefinedBehaviorSanitizer (per clang.llvm.org/docs/UndefinedBehaviorSanitizer.html) uses a runtime error: prefix rather than ==ERROR::
src/math.c:17:5: runtime error: signed integer overflow: 2147483647 + 1 cannot be represented in type 'int'The general pattern is: <file>:<line>: runtime error: <check>: <detail>.
Common UBSan check identifiers to recognize:
| Identifier in output | Check |
|---|---|
signed integer overflow: | Signed overflow; per UBSan docs |
division by zero | Integer divide-by-zero |
null pointer dereference | Null pointer use |
misaligned address | Alignment violation |
index N out of bounds | Array subscript OOB (static bounds) |
load of value N is not valid for type | Invalid enum / bool load |
call to function through pointer to incorrect function type | Function-pointer type mismatch |
UBSan's runtime is "not expected to produce false positives" (per clang.llvm.org/docs/UndefinedBehaviorSanitizer.html), but its production use needs care: recovery modes that continue execution instead of aborting can mask bugs from the fuzzer. Always build with -fno-sanitize-recover=all for fuzzing (see sanitiser-integration-reference).
Reading MSan crash output
MemorySanitizer (per clang.llvm.org/docs/MemorySanitizer.html) reports with a WARNING: prefix rather than ==ERROR::
WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x... in check_header src/decode.c:55
...
Uninitialized value was created by a heap allocation
#0 0x... in parse_frame src/decode.c:22When built with -fsanitize-memory-track-origins the report also shows where the uninitialised value was created and the intermediate stores it passed through. Without origins, the report names only the use site, making root cause harder to locate (per clang.llvm.org/docs/MemorySanitizer.html).
MSan findings classify as MEDIUM by default (uninitialized reads rarely give an attacker write primitives), but escalate if the value flows into a branch that controls a WRITE operation.
Exploitability classification
Classify each deduplicated finding by the bug class and access direction. The access direction (READ vs WRITE) is in the ASan line immediately after ==ERROR:.
| Bug class | Direction | Exploitability | Rationale |
|---|---|---|---|
heap-buffer-overflow | WRITE | LIKELY-EXPLOITABLE | Attacker-controlled write to adjacent heap; classic exploitation primitive |
use-after-free | WRITE | LIKELY-EXPLOITABLE | Write to freed memory; allocator-reuse exploitation |
double-free / invalid-free | any | LIKELY-EXPLOITABLE | Corrupts allocator metadata; exploitation is well-documented |
heap-buffer-overflow | READ | MEDIUM | Leaks heap contents; information disclosure |
use-after-free | READ | MEDIUM | Information disclosure; no write primitive directly |
stack-buffer-overflow | WRITE | MEDIUM | Stack corruption; exploitability depends on stack layout |
stack-buffer-overflow | READ | MEDIUM | Stack disclosure |
signed integer overflow | any | MEDIUM | Context-dependent; may widen to a write if used as an array index |
null pointer dereference | any | BENIGN | Crash-only in user-space protected-zero-page model |
memory-leak | any | BENIGN | DoS risk only; no memory corruption |
use-of-uninitialized-value (MSan) | - | MEDIUM | Information disclosure or branch confusion; escalate if controls a WRITE |
division by zero | any | BENIGN | Process termination; no memory corruption |
timeout / OOM artifact | any | BENIGN | Denial-of-service risk only |
Note: LIKELY-EXPLOITABLE is a triage signal, not a CVE severity. A security engineer must confirm before disclosure.
Deduplication by stack-hash
libFuzzer saves one artifact per unique crash input. A single bug can produce dozens of artifacts with slightly different inputs. Deduplicate before counting bugs.
Stack-hash key: take the top 3 non-sanitiser frames from the symbolised #N lines of the crash report. Exclude frames whose function names contain sanitizer, interceptor, or LLVMFuzzerTestOneInput - they are harness frames, not the fault site.
# Extract top 3 meaningful frames and hash them
grep -E '^\s+#[0-9]+ 0x' /tmp/report.txt \
| grep -v 'sanitizer\|interceptor\|LLVMFuzzerTestOneInput' \
| head -3 \
| sha1sum | cut -c1-8Artifacts sharing the same 8-character stack-hash represent the same bug. Keep the smallest artifact per hash - it is the easiest reproducer to attach to a bug report.
If the binary was built without -g (no debug info), the #N lines carry only hex addresses. The hash still works for dedup within a campaign but loses file/line attribution needed for bug tickets. Always build fuzz targets with -g -O1 (per clang.llvm.org/docs/AddressSanitizer.html and the libFuzzer build examples at llvm.org/docs/LibFuzzer.html).
Reproducer minimization with -minimize_crash
A crash artifact produced during fuzzing is often much larger than necessary. Minimizing it reduces review time, makes root-cause analysis easier, and produces a cleaner bug-report attachment.
libFuzzer's -minimize_crash=1 flag reduces the crash input to its smallest form that still reproduces the same crash (per llvm.org/docs/LibFuzzer.html):
# Minimize a single crash artifact
# -minimize_crash=1 requires a time or run budget
./fuzz_target -minimize_crash=1 \
-max_total_time=60 \
-exact_artifact_path=./minimized_crash \
./crash-a3f2c1b0
# Or bound by iteration count instead of time
./fuzz_target -minimize_crash=1 \
-runs=10000 \
-exact_artifact_path=./minimized_crash \
./crash-a3f2c1b0The -exact_artifact_path flag writes the minimized result to a single named file instead of using the default checksum-prefixed naming; -artifact_prefix can be used instead to write to a directory (per llvm.org/docs/LibFuzzer.html).
For AFL++ crashes, use afl-tmin (from the AFL++ toolchain) rather than libFuzzer minimization - they use different transport formats.
After minimization, re-run the minimized artifact to confirm it still triggers the same crash class and the same stack-hash before attaching it to the bug report:
ASAN_OPTIONS=abort_on_error=1:symbolize=1 \
UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=1 \
./fuzz_target -runs=1 ./minimized_crash 2>&1The -runs=1 flag re-runs the file as a single test input without fuzzing, as described in the libFuzzer options at llvm.org/docs/LibFuzzer.html.
Artifact naming quick-reference
libFuzzer saves artifacts with a class prefix followed by a content checksum (per llvm.org/docs/LibFuzzer.html):
| Artifact prefix | Meaning |
|---|---|
crash-<sha1> | Input triggered a crash or sanitiser abort |
leak-<sha1> | Input triggered LSan memory-leak detection |
timeout-<sha1> | Input exceeded -timeout wall-clock limit |
oom-<sha1> | Input exceeded -rss_limit_mb in fork mode |
AFL++ crash artifacts land under output/default/crashes/ with filenames of the form id:N,sig:N,src:N,.... They carry the same information but require symbolization separately via the AFL++ target binary - the class is not encoded in the filename.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Counting raw artifact files as bug count | One bug produces many artifacts with varied inputs | Deduplicate by stack-hash first |
| Classifying BENIGN without checking direction | A heap-buffer-overflow READ is MEDIUM, not BENIGN | Always read the READ/WRITE line before classifying |
| Minimizing before verifying the stack-hash match | Minimized input may trigger a different code path | Confirm stack-hash matches after minimization |
| Treating integer-overflow as always BENIGN | May feed into an index that drives a WRITE | Trace the value's use before downgrading to BENIGN |
Skipping -g in fuzz target builds | Stack traces become raw hex; dedup still works but root cause is unattributable | Always build with -g -O1 |
| Filing bugs on un-minimized artifacts | Large inputs slow review and bisection | Run -minimize_crash=1 before filing |