Create a gist now

Instantly share code, notes, and snippets.

Quick steps of how to create a flame graph using perf

The prep-script.sh will setup the latest Node and install the latest perf version on your Linux box.

When you want to generate the flame graph, run the following (folder locations taken from install script):

sudo sysctl kernel.kptr_restrict=0
# May also have to do the following:
# (additional reading http://unix.stackexchange.com/questions/14227/do-i-need-root-admin-permissions-to-run-userspace-perf-tool-perf-events-ar )
sudo sysctl kernel.perf_event_paranoid=0

perf record -i -g -e cycles:u -- ~/sources/node/node --perf-basic-prof script.js

perf script | egrep -v "( __libc_start| LazyCompile | v8::internal::| Builtin:| Stub:| LoadIC:|\[unknown\]| LoadPolymorphicIC:)" | sed 's/ LazyCompile:[*~]\?/ /' | ~/sources/FlameGraph/stackcollapse-perf.pl > out.perf-folded

~/sources/FlameGraph/flamegraph.pl out.perf-folded > node-flame.svg

The data munging is to help expose the most important bits. There is still some play that I'm working with, but right now it seems to be working.

Alternatives:

perf script | egrep -v "( __libc_start|node::Start\(| LazyCompile | Builtin:| Stub:| LoadIC:|\[unknown\]| LoadPolymorphicIC:)" | ../FlameGraph/stackcollapse-perf.pl | grep "uv_run" > out.perf-folded

perf script | egrep -v "( __libc_start|node::Start\(| LazyCompile | v8::internal::| Builtin:| Stub:| LoadIC:|\[unknown\]| LoadPolymorphicIC:)" | ../FlameGraph/stackcollapse-perf.pl | grep "uv_run" > out.perf-folded
#!/bin/bash
sh /etc/lsb-release
wget -O - http://llvm.org/apt/llvm-snapshot.gpg.key|sudo apt-key add -
apt-add-repository "deb http://llvm.org/apt/${DISTRIB_CODENAME}/ llvm-toolchain-${DISTRIB_CODENAME} main"
apt-get update
apt-get -y upgrade
apt-get -y install make build-essential linux-tools-common elfutils libelf-dev flex bison libunwind8 libunwind8-dev libaudit-dev libdw-dev binutils-dev libnuma-dev libslang2-dev asciidoc llvm-3.6 clang-3.6 lldb-3.6 libllvm3.6 subversion libc6-dev-i386 libgtk2.0-dev libperl-dev python-dev git tmux nginx gdb apache2-utils
ln -s /usr/bin/clang-3.6 /usr/bin/clang
ln -s /usr/bin/clang++-3.6 /usr/bin/clang++
ln -s /usr/bin/lldb-3.6 /usr/bin/lldb
echo 'export CC=clang' > ~/.bash_aliases
echo 'export CXX=clang++' >> ~/.bash_aliases
echo 'export GYP_DEFINES="clang=1"' >> ~/.bash_aliases
. ~/.bash_aliases
chmod 777 /usr/share/nginx/html
ln -s /usr/share/nginx/html www
mkdir sources
cd sources
wget https://github.com/joyent/node/archive/v0.11.13.tar.gz
tar xvf v0.11.13.tar.gz
rm v0.11.13.tar.gz
mv node-0.11.13 node
cd node
./configure --gdb
make
make install
cd ~/sources
wget https://github.com/brendangregg/FlameGraph/archive/master.tar.gz
tar xvf master.tar.gz
rm master.tar.gz
mv FlameGraph-master FlameGraph
#cd ~/sources
#wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.15.6.tar.gz
#tar xvf linux-3.15.6.tar.gz
#rm linux-3.15.6.tar.gz
#cd linux-3.15.6/tools/perf/
#make -f Makefile.perf install prefix=/usr
@dignifiedquire

I used this to get a html version (which is much more usable in the browser) of the graph using stackvis:

$ npm i -g stackvis
$ perf script | egrep -v "( __libc_start| LazyCompile | v8::internal::| Builtin:| Stub:| LoadIC:|\[unknown\]| LoadPolymorphicIC:)" | sed 's/ LazyCompile:[*~]\?/ /' > intermediate.out
$ stackvis perf < intermediate.out > flamegraph.htm
@arunoda
arunoda commented Aug 2, 2014

What's the use of nginx? to watch the dumped traces?
If so, using python -m SimpleHTTPServer

@bjouhier
bjouhier commented Aug 4, 2014

@trevnorris
Thanks, that's very helpful. I found two glitches on ubuntu:

  • the lines that add the exports to .bash_aliases broke the build. I had to remove these exports.
  • I got the following XML error in some of the generated SVGs:
<g class="func_g" onmouseover="s('RegExp:(^|[\r\n])([^"'\/]*)\/(?:\*[\s\S]*?\*\/|\/.*) (11 samples, 0.12%)')" onmouseout="c()">
-------------------------------------------------------^

I hacked around it that by inserting the following at line 460 of flamegraph.pl:

$info =~ s/\[\^\"\'/[^/g;

I don't know if the problem comes from flamegraph.pl or from one of the preprocessing steps.

@balupton
balupton commented Aug 5, 2014

Would be awesome to get a docker image that does this.

@aseemk
aseemk commented Sep 3, 2014

This is awesome. Any way to achieve this on Mac OS X?

@trevnorris
Owner

@Dignifiedquire Very nice. Thanks for sharing.

@arunoda The reason for nginx is because I found it was the easiest way to view the flamegraphs myself and share them with others (simple as a single URL). Instead of needing to scp/ftp the file down to my box to look at it.

@bjouhier Thanks for the info about the broken XML. Sorry about the .bash_aliases break. I used this script on fresh servers that hadn't been touched yet.

@ryanstevens

@trevnorris I just tried to do run perf after installing all the dependancies via the setup script. I've experienced the following error both in Ubuntu and Debian.

After running this step

$ perf record -i -g -e cycles:u -- ~/sources/node/node --perf-basic-prof script.js

I receive this

$ Perf session creation failed.
@thlorenz

@aseemk you can do that with the flamegraph app I created.
Includes instructions on how to get things working on Linux and Mac.

Also removes the LazyCompile for you when you load a map file.

@jlukanta
jlukanta commented Jan 8, 2015

I encountered the following errors:

E: Unable to locate package llvm-3.6
E: Couldn't find any package by regex 'llvm-3.6'
E: Unable to locate package clang-3.6
E: Couldn't find any package by regex 'clang-3.6'
E: Unable to locate package lldb-3.6
E: Couldn't find any package by regex 'lldb-3.6'
E: Unable to locate package libllvm3.6
E: Couldn't find any package by regex 'libllvm3.6'

It seems that llvm/clang/lldb/libllvm 3.6 packages were removed. To fix this issue, I simply switched to 3.5 packages. Hopefully this will help some people.

@Redsandro
$ perf script > perf.data.script
[kernel.kallsyms] with build id 9a87d34680d3efd31b5d57f56906d8c2fc38fc5e not found, continuing without symbols
Failed to open /tmp/perf-29438.map, continuing without symbols

What did I miss?

@cantremember

probably the node --perf-basic-prof part

@Redsandro

I'm interested in the function calls. There's a lot of module._something, garbage collection and _ZN2v88internal5Scope11LookupLocalEPKNS0_12AstRawStringE stuff in the graph together with function calls. Can I remove all the other stuff and only have the function calls?

@denys
denys commented Nov 30, 2016

^^ +1 for the question above. Thx.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment