Memory profiling Rust code with heaptrack in 2019
I recently ran into a classic case of "our code is using way more memory than it should". So I took my first dive into memory profiling Rust code. I read several posts about this, including the following
- KodrAus' gist on profiling rust
- This reddit thread
- Brendan D. Gregg's post about memory flamegraphs
- Bradlee Speice's A Case Study in Heaptrack
valgrind massif, jemalloc's
ex_stats_print, flamegraphs and
I tried heaptrack first, because it looked like the simplest solution to me.
The tutorials I found all required changing the allocator, since
heaptrack doesn't play nicely with jemalloc.
But, good news: Since Rust 1.32, jemalloc has been replaced as default allocator by the system default allocator. This means that we no longer need to make changes to our code for memory profiling.
So, lets take a look at memory profiling.
On Ubuntu (18.04) heaptrack can be installed via apt:
sudo apt install heaptrack heaptrack-gui
Running your code
Use heaptrack to run your binary and pass parameters as usual. An example for our code looks like this:
heaptrack target/release/rbt call-consensus-reads 1.fq.gz 2.fq.gz out.1.fq.gz out.2.fq.gz
heaptrack output will be written to "heaptrack.rbt.17073.gz"
starting application, this might take some time...
[some output of rbt]
Heaptrack finished! Now run the following to investigate the data:
heaptrack --analyze "heaptrack.rbt.17073.gz"
As you can see I used the release build (without any debug symbols). Using heaptrack slows down execution of the program, but not in a huge way. In my case, running without heaptrack took ~4 minutes, running with heaptrack took 6-7 minutes.
Running heaptrack like this writes a gzipped result file that we can now take a look at.
heaptrack_gui tool generates a nice collection of graphs and different views on the results.
You might need to wait a little for all graphs to render. Unfinished tabs stay grayed-out until they are done.
For my specific problem, I used the "Consumed" tab to take a look at memory usage over time and was able to identify the culprit:
Other visualizations include a flamegraph, bottom-up and top-down lists, and a nice summary which also identified
SliceConcatExt as the peak contributor to memory load consumption.
Fixing my problem
In our case, with the information that we need to look for a concatenation of slices we quickly found some old code that could be refactored to work without these additional allocations. After fixing our code, I ran heaptrack again and, as you can see, peak memory use dropped from 400MB to <100MB, leveling out at about 60MB instead of 380MB.
Since Rust 1.32 (January 2019),
heaptrack works out of the box for memory profiling Rust code.
You no longer need to change the allocator, making it a nice and quick way to take a look at you memory footprint.