HOWTO: heap profiling with jemallocator

README.md

1. Make sure your rust application uses https://github.com/gnzlbg/jemallocator as the global memory allocator (when in doubt, grep jemallocator in your Cargo.lock).
2. Install jemalloc (we'll only need jeprof), dot, ps2pdf and libunwind on your system. Enable jemallocator's profiling feature (if jemallocator is an indirect dependency, one trick to do is to add a dependency jemallocator = { version = "*", features = ["profiling"] } to your app and let cargo select the || of features for you).
3. export _RJEM_MALLOC_CONF=prof:true,lg_prof_interval:32,lg_prof_sample:19. lg_prof_interval sets how often profile dump should be written to disk measured in allocated bytes. The value is passed as a power of two, which is 2^32 in our case, i.e. every 4 GiB of allocations of long-lived objects (see https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling). lg_prof_sample:19 tells jemalloc to take a profiling sample every 2^19 = 512 KiB.
4. Running your binary should produce a bunch of jeprof.*.heap files (depending on your _RJEM_MALLOC_CONF).
5. To produce a PDF output, run jeprof --show_bytes --pdf "/path/to/<binary>" jeprof.*.heap > <binary>.pdf, where <binary> is the binary you are profiling, e.g. jeprof --show_bytes --pdf `which ripgrep` jeprof.2805.204.i204.heap > ripgrep.pdf. You probably want to select the latest jeprof.*.heap file, see http://jemalloc.net/mailman/jemalloc-discuss/2015-November/001205.html.

does not work

For systemd service, I had to use something like

Environment=_RJEM_MALLOC_CONF=prof:true,prof_final:true,prof_leak:true,prof_gdump:true,lg_prof_interval:32,prof_prefix:/tmp/jeprof

It saves the files to like /tmp/jeprof.3051418.60.i60.heap

Make sure to also build your binary with debug symbols, for example adding this to Cargo.toml:

[profile.release]
debug = true

View the graph with jeprof --web target/release/mybin $(ls -S1 /tmp/*.heap | head -n 1).

Note: jeprof.heap stores absolute paths to libraries and the running executable.
Run jeprof on the same system that the binary is built on, else you may get errors like this:

Using local file ~/jeprof.3085546.9823.i24.heap.
/usr/bin/objdump: '/usr/lib64/libpcre.so.1.2.12': No such file
/usr/bin/objdump: '/usr/lib64/libselinux.so.1': No such file
/usr/bin/objdump: '/usr/lib64/librt-2.33.so': No such file
/usr/bin/objdump: '/usr/lib64/libnss_usrfiles.so.2': No such file
/usr/bin/objdump: '/usr/lib64/libnss_files-2.33.so': No such file
/usr/bin/objdump: '/usr/lib64/libc-2.33.so': No such file
/usr/bin/objdump: '/usr/lib64/libm-2.33.so': No such file
/usr/bin/objdump: '/usr/lib64/libpthread-2.33.so': No such file
/usr/bin/objdump: '/usr/lib64/libdl-2.33.so': No such file
/usr/bin/objdump: '/usr/lib64/ld-2.33.so': No such file
/usr/bin/addr2line: '/usr/lib64/libpthread-2.33.so': No such file
/usr/bin/addr2line: '/usr/lib64/libc-2.33.so': No such file

Alternatively, you can copy the shared libraries, and update the paths in the jeprof.heap file to match your local copies.

Actually: run jeprof on the same system that the binary is running on (i.e. where the dump files are generated)