flame-graph-analyzer

Use when a performance regression is traced to a commit but the hot path inside it is unclear. Reads flame graphs from py-spy, async-profiler, Go pprof or clinic.js, finds the hottest frames, tells CPU-bound from lock contention or allocation pressure and suggests the next step.

/plugin install qa-load-testing@testland-qa

First time in Claude Code? Add the marketplace with /plugin marketplace add testland/qa

View source (opens in a new tab)

Overview

Canonical flame-graph reference: brendan-gregg-flame (opens in new window). The widest leaf (bottom of the stack) is the hot path.

Each runtime produces flame graphs from its own profiler - py-spy (Python), async-profiler (JVM), Go's built-in pprof, clinic.js flame (Node.js), and perf / eBPF (native C/C++/Rust). This skill is language-agnostic: it consumes the profiler output (SVG, JSON, or folded-stacks .txt) and surfaces a hypothesis the engineer can act on. Per-runtime capture commands live in references/capturing-and-interpreting-flame-graphs.md.

When to use

  • A perf regression has been bisected but the introducing commit touches multiple functions; the team needs to know which function is the actual hot path.
  • A load test under k6-load-testing or a sibling shows latency growth, but the API code hasn't visibly changed - the flame graph reveals the runtime cause.
  • A production incident showed CPU saturation; the team has captured a profile and needs to triage before reducing fleet size.
  • An EXPLAIN ANALYZE trace suggests CPU is the bottleneck rather than I/O - the flame graph confirms.

How to use

  1. Capture a profile under realistic load with the runtime's profiler and export folded stacks - per-runtime capture commands in references/capturing-and-interpreting-flame-graphs.md.
  2. Sort the folded stacks by sample count and read the top 5 leaves (the actual working code, not framework wrappers).
  3. Classify each hot leaf's sample-time signature (CPU-bound vs allocator pressure vs lock contention vs I/O-misclassified) using the table below.
  4. Propose the next step for the dominant leaf; the category-to-fix remediation catalog is in references/capturing-and-interpreting-flame-graphs.md.
  5. Re-profile after the change to confirm the delta, then hand off to perf-budget-gate to confirm the regression delta closes.

Reading the hot path

Read the folded stacks (or extract them from the SVG / JSON), sort by sample count, and identify the top leaves - the bottom-of-stack frames that are the actual working code, not framework wrappers:

# top 5 leaves by sample count
sort -k2 -n -r folded.txt | head -5

Each hot leaf's sample-time signature points to one category:

CategorySignature in the flame graph
CPU-bound (hot algo)A wide leaf in user code (a regex, a JSON serializer, a hash function).
Allocator pressureWide GC frames (gc::scavenge, Java GC, gc.collect).
Lock contentionWide synchronization frames (pthread_mutex_lock, Object.wait, parking).
I/O wait misclassifiedOn-CPU profilers don't show I/O blocks; switch to wall-clock profiling.
Reflection / dynamic dispatchWide reflection.invoke, method_missing, getattr chains.
Logging overheadWide log.format, Logger.debug, serialization for log lines.

Worked example

A Node.js endpoint regressed after a bisected commit. Capture under load, then read the hot path end to end.

Capture (Node.js clinic.js flame; other runtimes in the reference):

npx clinic flame -- node app.js
# drive load against the process, then stop it; clinic writes flame.html + folded stacks

Sort the folded stacks and read the top leaves:

sort -k2 -n -r folded.txt | head -5
main;handleRequest;serializeJson;JSON.stringify   4521
main;handleRequest;dbQuery;Array.from              2103
main;handleRequest;dbQuery;parseRows               1832
main;authCheck;jwt.verify;crypto.createHash        904
main;handleRequest;serializeJson;Buffer.from       312

Emit the analysis:

## Flame graph analysis - `flame.html`

**Runtime:** node
**Profile duration:** 30s under load

| Rank | Sample share | Stack (leaf)                          | Category |
|-----:|-------------:|---------------------------------------|----------|
| 1    | 38%          | `JSON.stringify` (in `serializeJson`) | CPU-bound hot algo |
| 2    | 17%          | `Array.from` (in `dbQuery`)           | Allocator pressure |
| 3    | 15%          | `parseRows` (in `dbQuery`)            | CPU-bound hot algo |
| 4    | 8%           | `jwt.verify` (in `authCheck`)         | CPU-bound hot algo (crypto) |
| 5    | 3%           | `Buffer.from` (in `serializeJson`)    | Allocator pressure |

### Hypothesis
The serialization path is load-bearing: rank 1 (`JSON.stringify`, 38%) plus
rank 5 (`Buffer.from`, 3%) account for ~41% of sampled time combined.

### Recommended next step
1. Switch to a streaming JSON serializer (`fast-json-stringify` in Node,
   `orjson` in Python, Jackson's `JsonGenerator` in the JVM) to eliminate the
   intermediate string allocation.
2. Re-profile; expect rank 1 to drop below 10%.
3. Hand off to `perf-budget-gate` to confirm the regression delta closes.

More interpreted cases (GC pressure, lock contention, reflection overhead) and the category-to-fix remediation catalog are in references/capturing-and-interpreting-flame-graphs.md.

Anti-patterns

Anti-patternWhy it failsFix
Reading the SVG visually only, no quantitative dataEasy to mis-judge widths; biases toward dramatic-looking deep stacks.Always work from folded stacks; sort by sample count.
Profiling under load that is too lowOne request per second can't expose contention or allocator pressure.Profile under realistic load - pair with k6-load-testing.
Optimizing rank 5 first because rank 1 looks "structural"Premature optimization; misses the dominant cost.Always start with rank 1; only descend if rank 1 is genuinely framework-bound (e.g. event_loop).
On-CPU profiler for an I/O-bound workloadI/O wait doesn't appear; the flame graph shows what's running, not what's waiting.Use wall-clock / off-CPU profiling for I/O-bound workloads.
Single 30-second capture under highly variable loadThe sample is unrepresentative.Capture multiple samples across the load-test duration; merge.

Limitations

  • Symbolication. Without debug symbols, the flame graph shows hex addresses. Profile with debug info enabled in CI builds intended for analysis.
  • Inlining. Aggressive inlining (especially in the JVM hot-path optimizer) can hide functions; the flame graph shows the post-inlining shape, which may not match the source.
  • Sampling vs. tracing. Sample-based flame graphs give relative weights; for absolute timings of specific operations, use a tracer instead.

References

Reference

Capturing and interpreting flame graphs per runtime

View source (opens in a new tab) (opens in new window)

Deep reference for flame-graph-analyzer SKILL.md. Consult for the full per-runtime capture wiring, the category-to-fix remediation catalog, and worked interpretation cases beyond the single inline example.

Per-runtime capture wiring

Each runtime produces flame-graph data from its own profiler. Export folded stacks (.txt, one line per unique stack + sample count) wherever the profiler supports it - that is the machine-readable form the analyzer reads.

Python - py-spy

py-spy (opens in new window):

py-spy record -o flame.svg -d 30 --pid <pid>
# or run-and-record
py-spy record -o flame.svg -d 30 -- python app.py
# raw folded stacks for analysis
py-spy record -f raw -o folded.txt -d 30 --pid <pid>

JVM - async-profiler

async-profiler (opens in new window):

java -agentpath:/path/to/libasyncProfiler.so=start,event=cpu,duration=30s,file=flame.html ...
# JFR output
java -agentpath:.../libasyncProfiler.so=start,event=cpu,file=profile.jfr ...

Output: HTML flame graph or JFR (Java Flight Recorder) format.

Go - pprof

Go's built-in pprof (opens in new window):

# in-process: import _ "net/http/pprof"; http.ListenAndServe(":6060", nil)
go tool pprof -http=:8080 http://localhost:6060/debug/pprof/profile?seconds=30

Open the served URL, then VIEW -> Flame Graph.

Node.js - clinic.js flame

Clinic.js (opens in new window):

npx clinic flame -- node app.js
# writes flame.html when the process exits

Native (C/C++/Rust) - perf

perf record piped through Brendan Gregg's stackcollapse-perf.pl and flamegraph.pl produces folded stacks and the SVG (bg (opens in new window)).

Folded stacks - the universal format

All major profilers can emit folded stacks: one line per unique stack with its sample count.

main;handleRequest;serializeJson;Buffer.from 4521
main;handleRequest;dbQuery;parseRows 1832
main;handleRequest;authCheck;jwtVerify 904

flamegraph.pl consumes this format directly. Folded stacks are the canonical machine-readable input for this skill's analysis.

Category-to-fix remediation catalog

Once a hot leaf is classified, map the category to a typical fix:

CategoryTypical fix
CPU-bound hot algoCache the result; switch to a faster algorithm; move it out of the hot path.
Allocator pressureReuse buffers / pools; switch to streaming serialization; escape-analysis fixes for the JVM.
Lock contentionReduce critical-section scope; move to lock-free data structures; per-shard locking.
Reflection overheadReplace dynamic dispatch with cached call-sites or codegen.
Logging overheadLazy log-message construction; level-check before format.

Worked interpretation cases

GC pressure

| Rank | Share | Stack (leaf)     | Category |
|-----:|------:|------------------|----------|
| 1    | 32%   | `gc.collect`     | Allocator pressure |
| 2    | 18%   | `dict.update`    | (callsite) |
| 3    | 14%   | `parse_response` | CPU-bound hot algo |

GC at 32% of samples means allocator pressure dominates. The fix is not making any one function faster - it is reducing the rate of allocations from dict.update and parse_response (object pooling, streaming parsing).

Lock contention

| Rank | Share | Stack (leaf)          | Category |
|-----:|------:|-----------------------|----------|
| 1    | 41%   | `pthread_mutex_lock`  | Lock contention |
| 2    | 12%   | `cache.get`           | (callsite) |

41% of samples in lock acquisition. The fix is not "make cache.get faster" - it is "reduce the contention" (per-shard locks, lock-free structures, or a lock-free cache such as Caffeine for the JVM).

Reflection / dynamic-dispatch overhead

| Rank | Share | Stack (leaf)                | Category |
|-----:|------:|-----------------------------|----------|
| 1    | 28%   | `Method.invoke` / `getattr` | Reflection overhead |

A common surprise - an ORM's reflective field access dominates the profile of an otherwise simple endpoint. The fix is the ORM-equivalent of "compile the mapping": cached method handles in the JVM, __slots__ in Python, generated SQL in Go.