FlameGraph for Performance Analysis

FlameGraph visualises perf data.

Install

FlameGraphs are visualizations of profile data. The visualization makes hot paths visible at a glance; the team's CPU investigation is dramatically accelerated when FlameGraphs are part of the toolkit.

What installation looks like:

• Brendan Gregg's flamegraph.pl scripts.: The original FlameGraph tooling. Brendan Gregg's Perl scripts produce SVG output from profile data; the scripts are widely available and standalone.
• Standalone Perl scripts.: The scripts have minimal dependencies. Perl is widely available; the scripts work on most systems; the installation is bounded.
• Or use vendor-integrated FlameGraph features.: Many observability vendors include FlameGraph generation. Datadog, Grafana, Honeycomb, others all support FlameGraphs natively; the integration is convenient.
• Multiple input formats.: FlameGraph supports profile data from perf, py-spy, dtrace, async-profiler, and others. The team's profiling tool can feed FlameGraph directly.
• Open-source.: The tooling is open. The team's discipline is not vendor-locked; the FlameGraph format and tooling are portable.

Installation is bounded. The discipline is fluency; the value is the visualization.

Read

Reading FlameGraphs is a learnable skill. Width is time, height is call depth, wide bars at top are bottlenecks; the visualization conveys the data efficiently.

• Width equals time.: The width of each function's bar is proportional to the time spent in it. Wide bars are heavy contributors; the eye finds them quickly.
• Height equals call depth.: The height represents call stack depth. The bottom is the entry point; the top is the deepest call. Each level shows callers and callees.
• Wide bars at top equal bottlenecks.: A function that is wide at the top (deep in the stack) is a bottleneck. The function consumes significant time; optimization targets it.
• Hot functions surface at the top.: The visualization makes hot functions visually prominent. The team's eyes find them quickly; the analysis is fast.
• Compare profiles.: Two FlameGraphs side by side reveal optimization effects. Before and after; the team verifies improvements; the data drives decisions.

Reading is the analytical skill. Once learned, FlameGraphs accelerate every CPU investigation.

Integrate

FlameGraphs are most valuable when integrated into the team's continuous profiling. Production samples produce continuous FlameGraph data; the analysis is data-driven; the discipline scales.

• Auto-generated for production sampling.: The team's sampling pipeline produces FlameGraphs automatically. Continuous data flows; the team's analysis builds on the data; the discipline is sustained.
• Continuous profiling tools.: Tools like Pyroscope and Parca specialize in continuous profiling. They produce FlameGraphs at scale; the team's observability includes performance.
• Per-service FlameGraphs.: Each service can have its own FlameGraph. The team's investigation is per-service; the data is targeted; the analysis is fast.
• Time-comparison.: Continuous profiling supports time-comparison FlameGraphs. Performance regression between two time windows is visible; root cause is identifiable.
• Sampling overhead.: Continuous profiling has overhead. Modern tools keep this small; the team's discipline accepts the bounded overhead in exchange for the visibility.

FlameGraph for perf is one of those performance-investigation skills that pays off across many incidents. Nova AI Ops integrates with profiling telemetry, surfaces FlameGraph data, and supports the team's performance discipline.