flame-graph-analyzer

Overview

Canonical flame-graph reference: brendan-gregg-flame (opens in new window). The widest leaf (bottom of the stack) is the hot path.

Each runtime produces flame graphs from its own profiler - py-spy (Python), async-profiler (JVM), Go's built-in pprof, clinic.js flame (Node.js), and perf / eBPF (native C/C++/Rust). This skill is language-agnostic: it consumes the profiler output (SVG, JSON, or folded-stacks .txt) and surfaces a hypothesis the engineer can act on. Per-runtime capture commands live in references/capturing-and-interpreting-flame-graphs.md.

When to use

• A perf regression has been bisected but the introducing commit touches multiple functions; the team needs to know which function is the actual hot path.
• A load test under k6-load-testing or a sibling shows latency growth, but the API code hasn't visibly changed - the flame graph reveals the runtime cause.
• A production incident showed CPU saturation; the team has captured a profile and needs to triage before reducing fleet size.
• An EXPLAIN ANALYZE trace suggests CPU is the bottleneck rather than I/O - the flame graph confirms.

How to use

1. Capture a profile under realistic load with the runtime's profiler and export folded stacks - per-runtime capture commands in references/capturing-and-interpreting-flame-graphs.md.
2. Sort the folded stacks by sample count and read the top 5 leaves (the actual working code, not framework wrappers).
3. Classify each hot leaf's sample-time signature (CPU-bound vs allocator pressure vs lock contention vs I/O-misclassified) using the table below.
4. Propose the next step for the dominant leaf; the category-to-fix remediation catalog is in references/capturing-and-interpreting-flame-graphs.md.
5. Re-profile after the change to confirm the delta, then hand off to perf-budget-gate to confirm the regression delta closes.

Reading the hot path

Read the folded stacks (or extract them from the SVG / JSON), sort by sample count, and identify the top leaves - the bottom-of-stack frames that are the actual working code, not framework wrappers:

# top 5 leaves by sample count
sort -k2 -n -r folded.txt | head -5

Each hot leaf's sample-time signature points to one category:

Worked example

A Node.js endpoint regressed after a bisected commit. Capture under load, then read the hot path end to end.

Capture (Node.js clinic.js flame; other runtimes in the reference):

npx clinic flame -- node app.js
# drive load against the process, then stop it; clinic writes flame.html + folded stacks

Sort the folded stacks and read the top leaves:

sort -k2 -n -r folded.txt | head -5

main;handleRequest;serializeJson;JSON.stringify   4521
main;handleRequest;dbQuery;Array.from              2103
main;handleRequest;dbQuery;parseRows               1832
main;authCheck;jwt.verify;crypto.createHash        904
main;handleRequest;serializeJson;Buffer.from       312

Emit the analysis:

## Flame graph analysis - `flame.html`

**Runtime:** node
**Profile duration:** 30s under load

| Rank | Sample share | Stack (leaf)                          | Category |
|-----:|-------------:|---------------------------------------|----------|
| 1    | 38%          | `JSON.stringify` (in `serializeJson`) | CPU-bound hot algo |
| 2    | 17%          | `Array.from` (in `dbQuery`)           | Allocator pressure |
| 3    | 15%          | `parseRows` (in `dbQuery`)            | CPU-bound hot algo |
| 4    | 8%           | `jwt.verify` (in `authCheck`)         | CPU-bound hot algo (crypto) |
| 5    | 3%           | `Buffer.from` (in `serializeJson`)    | Allocator pressure |

### Hypothesis
The serialization path is load-bearing: rank 1 (`JSON.stringify`, 38%) plus
rank 5 (`Buffer.from`, 3%) account for ~41% of sampled time combined.

### Recommended next step
1. Switch to a streaming JSON serializer (`fast-json-stringify` in Node,
   `orjson` in Python, Jackson's `JsonGenerator` in the JVM) to eliminate the
   intermediate string allocation.
2. Re-profile; expect rank 1 to drop below 10%.
3. Hand off to `perf-budget-gate` to confirm the regression delta closes.

More interpreted cases (GC pressure, lock contention, reflection overhead) and the category-to-fix remediation catalog are in references/capturing-and-interpreting-flame-graphs.md.

Anti-patterns

Limitations

• Symbolication. Without debug symbols, the flame graph shows hex addresses. Profile with debug info enabled in CI builds intended for analysis.
• Inlining. Aggressive inlining (especially in the JVM hot-path optimizer) can hide functions; the flame graph shows the post-inlining shape, which may not match the source.
• Sampling vs. tracing. Sample-based flame graphs give relative weights; for absolute timings of specific operations, use a tracer instead.

References

• Brendan Gregg's flame graphs page (opens in new window) - canonical reference for the visualization technique.
• Per-runtime capture wiring, the remediation catalog, and extended interpretation cases: references/capturing-and-interpreting-flame-graphs.md.
• py-spy - https://github.com/benfred/py-spy
• async-profiler - https://github.com/async-profiler/async-profiler
• Go pprof - https://pkg.go.dev/net/http/pprof
• Clinic.js - https://clinicjs.org/
• k6-load-testing - runner that produces the load under which the profile is captured.