perf-guide.md

perf guide

Profiling tools are critical for trying to understand performance bottlenecks in code.

Without real data about what a program is doing while running, detecting bottlenecks is at best a process of trial and error for both users and developers. While thoughtful testing of various program configurations along with measuring time elapsed for decrete runs can often be enough - why not learn faster and funner ways to rapidly collect real data about what a program is doing?

Actual data about program execution like which functions are being called while a program is active helps point to hot parts of the code where most time may be being spent. While users of applications may not easily understand the output of profiling tools, being equipped to generate profiling output can be extremely useful for sharing with developers, since the time to set up robust test cases for developers is can be greater than the time it takes to understand and optimize slow code paths. Therefore it can be invaluable to get a rough snapshot of profiling output from users.

This guide serves to instruct you, the user, to generate some useful profiling output for the program you are running, to share with developers.

Many profiling tools exist on linux - others I've used with success include gprof and oprofile.

But the two most valuable in my opinion are gdb and perf.

GDB

GDB, or the GNU debugger, can be used to temporarily stop a program and print what it is doing (a backtrace) and it can do this for all the potential threads the program is using (which can be critical).

Install gdb like:

sudo apt-get install gdb

Next you need the PID, or process ID of the program you are interested in. Make sure it is running and then find the ID using methods like:

# if the application is python based, for example, do:
ps aux | grep python

# or more direct:
pidof python

Now, run as root and attach to the process using gdb. Be aware that your program will become hung by attaching, but when we exit gdb it will wake up.

sudo gdb --pid `pidof python`

The above command should send you into the gdb interpreter and you'll see a line starting with (gdb).

Now, type this command to generate a backtrace of what all threads are doing in the program:

thread apply all backtrace

Now open a plain text document, create a header called "*** BACKTRACE 1 ***", and paste all the lines gdb output into the doc.

Next, leave gdb and allow your program to continue running again. To do this type:

quit

Then gdb should prompt you to confirm and will detach (leaving the program running as it was before you attached). The output should look something like:

(gdb) quit
The program is running.  Quit anyway (and detach it)? (y or n) y
Detaching from process 55606.

Now, for a program that is running very very slowly a single backtrace can often help pinpoint (when a developer sees it) exactly what is going wrong - e.g. some function in the code is being called that indicates the program was misconfigured and explains why the app is not working well.

But, for programs that are running fast (using lots of CPU) a single GDB backtrace will not be trustable in isolation. So, next you should repeat the above steps to generate several more GDB traces.

You can run this quickly in succession like:

gdb -ex "set pagination 0" -ex "thread apply all bt" --batch --pid `pidof python`

Or in a loop like:

PID=`pidof python`
RUNS=10
SLEEP=1
for x in $(seq 1 ${RUNS})
  do
    echo "*** BACKTRACE $x ***";
    sudo gdb -ex "set pagination 0" -ex "thread apply all bt" --batch --pid $PID;
    sleep ${SLEEP};
  done

For more info on this approach see: http://poormansprofiler.org/

Perf

search for the latest linux-tools-*

apt-cache search linux-tools

On 12.04 at the time of this writing it was linux-tools-3.2.0-24

sudo apt-get install linux-tools-common linux-tools-3.2.0-24

You should now have a command of perf_3.2.0-24:

perf_3.2.0-24 --help

Slick thing to do now is to run, as root, perf top:

sudo perf_3.2.0-24 top

Activity Monitor.app on OSX

On OS X when a program starts running slowly, or appears hung, the first thing to do is run a trace using Activity Monitor. Activity Monitor is a GUI program located at /Applications/Utilities/Activity Monitor.app. It offers several windows but what we want is the main window also called "Activity Monitor" and activated with ⌘1 when the app is open. The main window will display all running processes. Find the process name or id of the process you are interested in and hit "Sample Process" which will run for several seconds and then output call graphs of each thread.