How to debug a try build on macOS with lldb with full debug information and source

debug-try-build-macos.md

I tried this with an x86_64 try build running on an M1 machine, and it worked. It was this build (from Aug 26, 2021) and I'll use it as an example.

TL;DR

You need the XUL.dSYM and a local source checkout of the same revision, and then tell lldb about them like this:

% lldb /Volumes/Firefox\ Nightly/Firefox Nightly.app/Contents/MacOS/firefox
(lldb) target modules add "/Volumes/Firefox Nightly/Firefox Nightly.app/Contents/MacOS/XUL" --symfile ~/Downloads/target.crashreporter-symbols-full/XUL/E13E188188F0368CB252DC67549D134B0/XUL.dSYM/Contents/Resources/DWARF/XUL
(lldb) settings set target.source-map /builds/worker/checkouts/gecko /Users/mstange/code/try-checkout
(lldb) run

Prerequisites

You need the following pieces on your local file system:

1. The target.dmg from the build job.
2. The target.crashreporter-symbols-full.tar.zst artifact from the same build job.
3. A local checkout of mozilla-central, at the try revision. hg pull -r bd821b2ea80f57e9c3dc6595ac32626fb85847f8 try && hg up bd821b2ea80f57e9c3dc6595ac32626fb85847f8

Getting right dSYM

The crashreporter symbols bundle contains symbols for multiple variants of the XUL library. We need to find the ID for the right one. The easiest way to do this is:

1. Launch the Firefox build from the try job.
2. Open the error console with Cmd+Shift+J.
3. If there is no textbox, first go to about:config and set devtools.chrome.enabled to true.
4. In the textbox, paste this and press enter: Services.profiler.sharedLibraries.find(l => l.name == "XUL").breakpadId

Now you have the correct breakpadId for XUL. In my case, this ID is E13E188188F0368CB252DC67549D134B0.

The next step is to extract that dSYM from the crashreporter symbols:

1. brew install zstd if you don't have it.
2. Extract the zstd file to a directory, using zstd -d ~/Downloads/target.crashreporter-symbols-full.tar.zst.
3. In the extracted directory, find XUL/<breakpadId>/XUL.dSYM.tar.
4. Extract XUL.dSYM.tar.

Now you have XUL.dSYM. Great.

Using lldb

Now you need to launch lldb, tell it about the XUL.dSYM before launching or attaching to Firefox, and also tell it about the location of the source checkout.

Importantly, it looks like lldb actually needs the path to the XUL object inside the dSYM bundle, not just the path to the dSYM itself. For example, you'll need to give it the full path to XUL.dSYM/Contents/Resources/DWARF/XUL

And it bears repeating: If this lldb session has already discovered the XUL binary because you launched Firefox with it earlier (or attached to it), then it ignores the dSYM file. You need to tell it about the symbols before it discovers the binary.

The DWARF data in the dSYM file refers to the source files at an absolute path that starts with /builds/worker/checkouts/gecko. To let lldb find the local checkout, you'll need to tell it to redirect those source paths, with the settings set target.source-map command.

% lldb /Volumes/Firefox\ Nightly/Firefox\ Nightly.app/Contents/MacOS/firefox
(lldb) target create "/Volumes/Firefox Nightly/Firefox Nightly.app/Contents/MacOS/firefox"
Current executable set to '/Volumes/Firefox Nightly/Firefox Nightly.app/Contents/MacOS/firefox' (x86_64).
(lldb) target modules add "/Volumes/Firefox Nightly/Firefox Nightly.app/Contents/MacOS/XUL" --symfile ~/Downloads/target.crashreporter-symbols-full/XUL/E13E188188F0368CB252DC67549D134B0/XUL.dSYM/Contents/Resources/DWARF/XUL
(lldb) settings set target.source-map /builds/worker/checkouts/gecko /Users/mstange/code/try-checkout
(lldb) run

Now you should have full source and debug information in your debugging session. If you're debugging code that's not compiled into XUL, you can do similar setup for the library you need (e.g. libmozglue.dylib).