I recently ran into a classic case of "our code is using way more memory than it should". So I took my first dive into memory profiling Rust code. I read several posts about this, including the following
- KodrAus' gist on profiling rust
- This reddit thread
- Brendan D. Gregg's post about memory flamegraphs
- Bradlee Speice's A Case Study in Heaptrack
which mentioned valgrind massif, jemalloc's ex_stats_print, flamegraphs and heaptrack.
I tried heaptrack first, because it looked like the simplest solution to me.
The tutorials I found all required changing the allocator, since heaptrack doesn't play nicely with jemalloc.
But, good news: Since Rust 1.32, jemalloc has been replaced as default allocator by the system default allocator. This means that we no longer need to make changes to our code for memory profiling.
So, lets take a look at memory profiling.
On Ubuntu (18.04) heaptrack can be installed via apt:
sudo apt install heaptrack heaptrack-gui
Alternatively, (and for older ubunutu versions) you can download the source code from the github repo and compile it yourself.
Use heaptrack to run your binary and pass parameters as usual. An example for our code looks like this:
heaptrack target/release/rbt call-consensus-reads 1.fq.gz 2.fq.gz out.1.fq.gz out.2.fq.gz
heaptrack output will be written to "heaptrack.rbt.17073.gz"
starting application, this might take some time...
[some output of rbt]
Heaptrack finished! Now run the following to investigate the data:
heaptrack --analyze "heaptrack.rbt.17073.gz"
As you can see I used the release build (without any debug symbols). Using heaptrack slows down execution of the program, but not in a huge way. In my case, running without heaptrack took ~4 minutes, running with heaptrack took 6-7 minutes.
Running heaptrack like this writes a gzipped result file that we can now take a look at.
The heaptrack_gui tool generates a nice collection of graphs and different views on the results.
You might need to wait a little for all graphs to render. Unfinished tabs stay grayed-out until they are done.
heaptrack_gui heaptrack.rbt.17073.gz
For my specific problem, I used the "Consumed" tab to take a look at memory usage over time and was able to identify the culprit: SliceConcatExt
Other visualizations include a flamegraph, bottom-up and top-down lists, and a nice summary which also identified SliceConcatExt as the peak contributor to memory load consumption.
In our case, with the information that we need to look for a concatenation of slices we quickly found some old code that could be refactored to work without these additional allocations. After fixing our code, I ran heaptrack again and, as you can see, peak memory use dropped from 400MB to <100MB, leveling out at about 60MB instead of 380MB.
Since Rust 1.32 (January 2019), heaptrack works out of the box for memory profiling Rust code.
You no longer need to change the allocator, making it a nice and quick way to take a look at you memory footprint.