Skip to content

Instantly share code, notes, and snippets.

@RaisinTen

RaisinTen/Debugging native C++ crashes in Electron and Chromium using GDB.md

Last active April 15, 2024 14:16
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save RaisinTen/5c065d28789d7a95a23af2db8e0e6c75 to your computer and use it in GitHub Desktop.
Save RaisinTen/5c065d28789d7a95a23af2db8e0e6c75 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Debugging native C++ crashes in Electron and Chromium using GDB.md

Debugging native C++ crashes in Electron and Chromium using GDB

Need for debugging symbols

Setup

Let’s try to debug the following program.

test.c

#include <stdlib.h>

int main() {
  abort(); // This call crashes the program.
  return 0;
}

If we compile the program and run it, it crashes this way.

$ gcc -o test test.c 
$ ./test
Aborted (core dumped)

We know the source of this crash because we have read the code and it is self-explanatory. However, it is impossible to tell where the crash is coming from just by looking at the output. Now imagine that a relatively large program, like Electron, crashes this way. The source code is public but it’s too huge to pinpoint the source of a crash. What do we do if such a thing happens? Don’t panic just yet! :)

Debugging

Let’s try to investigate with GDB (GNU Debugger) because that’s what it’s supposed to do, help us in debugging!

$ gdb -q test
Reading symbols from test...
(No debugging symbols found in test)
(gdb) run
Starting program: /tmp/test 

Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff7de6859 in __GI_abort () at abort.c:79
#2  0x0000555555555156 in main ()
  • gdb starts the debugger
    • -q silences the big annoying copyright message in the beginning
    • test is the file path of the executable we want to debug
  • run starts the program with no command line arguments
  • bt is supposed to print the call stack after the crash happens

Hmm, the stack trace doesn’t seem to be very useful here. #2 0x0000555555555156 in main () does not tell me where in main() the crash happens from. That’s actually expected because this is a release build, i.e., the binary doesn’t have any debugging symbols as correctly detected by the debugger - (No debugging symbols found in test).

Investigating a debug build crash

Setup

Let’s try to compile the same program with debugging symbols by passing the -g option to the compiler.

$ gcc -g -o test test.c
$ ./test
Aborted (core dumped)

Debugging

Compiling with debugging symbols won’t affect the output here. It’s supposed to help us in detecting the source of the crash while debugging.

$ gdb -q test
Reading symbols from test...
(gdb) run
Starting program: /tmp/test 

Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff7de6859 in __GI_abort () at abort.c:79
#2  0x0000555555555156 in main () at test.c:4

Voila! The crash comes from line number 4 in test.c.

Crash from a release build with debugging symbols in a separate file

In most real-world cases, we’ll be debugging release build binaries because debugging symbols take up a lot of space and those are stripped out in production. Luckily, the debugging symbols are not completely removed. Those are shipped separately to make it possible to debug such builds.

Setup

Let’s try to replicate such a scenario.

First, we’ll create a debug build.

$ gcc -g -o test test.c
$ ls -l
total 24
-rwxrwxr-x 1 parallels parallels 17800 Apr 22 16:26 test
-rw-rw-r-- 1 parallels parallels    59 Apr 22 11:52 test.c
Then we’ll copy only the debugging symbols to another file called test.debug.
$ cp test test.debug
$ ls -l
total 44
-rwxrwxr-x 1 parallels parallels 17800 Apr 22 16:26 test
-rw-rw-r-- 1 parallels parallels    59 Apr 22 11:52 test.c
-rwxrwxr-x 1 parallels parallels 17800 Apr 22 16:26 test.debug
$ strip --only-keep-debug test.debug 
$ ls -l
total 32
-rwxrwxr-x 1 parallels parallels 17800 Apr 22 16:26 test
-rw-rw-r-- 1 parallels parallels    59 Apr 22 11:52 test.c
-rwxrwxr-x 1 parallels parallels  6424 Apr 22 16:27 test.debug

Now we’ll strip out the debugging symbols from test (note that it reduces the size of the binary).

$ strip --strip-debug --strip-unneeded test
$ ls -l
total 28
-rwxrwxr-x 1 parallels parallels 14472 Apr 22 16:29 test
-rw-rw-r-- 1 parallels parallels    59 Apr 22 11:52 test.c
-rwxrwxr-x 1 parallels parallels  6424 Apr 22 16:27 test.debug

That’s it! Our testing setup is ready.

Debugging

Now we have a release build called test which has no debugging symbols and a file called test.debug which contains only the debugging symbols. The presence of the debugging symbols can also be verified using the file command.

$ file test
test: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=196f0d6393b1405a9b8bd07a9797ddef14e1f8c1, for GNU/Linux 3.2.0, stripped
$ file test.debug 
test.debug: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter *empty*, BuildID[sha1]=196f0d6393b1405a9b8bd07a9797ddef14e1f8c1, for GNU/Linux 3.2.0, with debug_info, not stripped

For test, the output says stripped which means that it has been stripped off of debugging symbols and for test.debug, it says with debug_info, not stripped which means that it’s not stripped and it has all the debugging symbols we need.

Now, it’s time to debug the crash in the release build, test, by making use of the debugging symbols from the separate file, test.debug.

First, we need to associate the debugging symbols in test.debug with the executable, test. This creates a .gnu_debuglink section inside the executable containing the required information.

$ objcopy --add-gnu-debuglink=test.debug test
$ ls -l
total 28
-rwxrwxr-x 1 parallels parallels 14568 Apr 22 16:39 test
-rw-rw-r-- 1 parallels parallels    59 Apr 22 11:52 test.c
-rwxrwxr-x 1 parallels parallels  6424 Apr 22 16:27 test.debug

Now, we can run the program in the debugger.

$ gdb -q test
Reading symbols from test...
Reading symbols from /tmp/test.debug...
(gdb) run
Starting program: /tmp/test 

Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff7de6859 in __GI_abort () at abort.c:79
#2  0x0000555555555156 in main () at test.c:4

As we can see, GDB picks up the reference from the release binary, test, and successfully loads the debugging symbols from test.debug and displays the human readable stack trace!

Investigating a crash in Electron

Now, we'll move on to a real-world example where we will debug a crash in Electron!

Setup

We would need to install Electron v18.0.3.

$ npm i electron@18.0.3

The test case tries to call safeStorage.isEncryptionAvailable() - https://www.electronjs.org/docs/latest/api/safe-storage#safestorageisencryptionavailable.

test.js

const { safeStorage } = require('electron');
console.log(safeStorage.isEncryptionAvailable());

The crash can be reproduced by running the following command.

$ node_modules/electron/dist/electron test.js 
Trace/breakpoint trap (core dumped)

Since this is a release build, just running the program inside GDB won’t be of much use.

$ gdb -q node_modules/electron/dist/electron
Reading symbols from node_modules/electron/dist/electron...
(No debugging symbols found in node_modules/electron/dist/electron)
(gdb) run test.js
Starting program: /tmp/node_modules/electron/dist/electron test.js
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff3faf700 (LWP 235726)]
[Detaching after fork from child process 235727]
[Detaching after fork from child process 235728]
[Detaching after fork from child process 235729]
[New Thread 0x7ffff37ae700 (LWP 235732)]
[New Thread 0x7ffff2fad700 (LWP 235733)]
[New Thread 0x7ffff27ac700 (LWP 235734)]
[New Thread 0x7ffff1fab700 (LWP 235735)]
[New Thread 0x7ffff17aa700 (LWP 235736)]
[New Thread 0x7ffff0fa9700 (LWP 235737)]
[New Thread 0x7ffff07a8700 (LWP 235738)]
[New Thread 0x7fffeff0a700 (LWP 235739)]
[New Thread 0x7fffef709700 (LWP 235740)]
[New Thread 0x7fffeef08700 (LWP 235741)]
[New Thread 0x7fffee707700 (LWP 235742)]
[New Thread 0x7fffedf06700 (LWP 235743)]

Thread 1 "electron" received signal SIGTRAP, Trace/breakpoint trap.
0x000055555b8b4faf in ?? ()
(gdb) bt
#0  0x000055555b8b4faf in ?? ()
#1  0x0000000000000000 in ?? ()
  • run test.js runs the program with the JS file path, test.js, as the first argument

We need to substitute the ??s with debugging symbols.

Debugging the crash

The debugging symbols for Electron’s x64 linux builds can be found in https://github.com/electron/electron/releases/download/v18.0.3/electron-v18.0.3-linux-x64-debug.zip. Let’s download it and extract it into a directory called debug.

$ curl -LO https://github.com/electron/electron/releases/download/v18.0.3/electron-v18.0.3-linux-x64-debug.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   671  100   671    0     0   2192      0 --:--:-- --:--:-- --:--:--  2185
100 1146M  100 1146M    0     0  4415k      0  0:04:25  0:04:25 --:--:-- 3133k
$ unzip -d debug electron-v18.0.3-linux-x64-debug.zip 
Archive:  electron-v18.0.3-linux-x64-debug.zip
  inflating: debug/LICENSE           
  inflating: debug/LICENSES.chromium.html  
  inflating: debug/version           
  inflating: debug/debug/chrome_crashpad_handler.debug  
  inflating: debug/debug/electron.debug  
  inflating: debug/debug/libffmpeg.so.debug  
  inflating: debug/debug/libvk_swiftshader.so.debug  
  inflating: debug/debug/libEGL.so.debug  
  inflating: debug/debug/libGLESv2.so.debug

Now, let’s try to run the release binary with the debugging symbols!

$ gdb -q -s debug/debug/electron.debug -e node_modules/electron/dist/electron
Reading symbols from debug/debug/electron.debug...
(gdb) run test.js 
Starting program: /tmp/node_modules/electron/dist/electron test.js
warning: Probes-based dynamic linker interface failed.
Reverting to original interface.
[New LWP 235230]
[Detaching after fork from child process 235231]
[Detaching after fork from child process 235232]
[Detaching after fork from child process 235233]
[New LWP 235239]
[New LWP 235240]
[New LWP 235241]
[New LWP 235242]
[New LWP 235243]
[New LWP 235244]
[New LWP 235245]
[New LWP 235246]
[New LWP 235247]
[New LWP 235248]
[New LWP 235249]
[New LWP 235250]

Thread 1 "electron" received signal SIGTRAP, Trace/breakpoint trap.
0x000055555b8b4faf in (anonymous namespace)::CreateKeyStorage () at ../../components/os_crypt/os_crypt_linux.cc:74
74	../../components/os_crypt/os_crypt_linux.cc: No such file or directory.
(gdb) bt
#0  0x000055555b8b4faf in (anonymous namespace)::CreateKeyStorage () at ../../components/os_crypt/os_crypt_linux.cc:74
#1  0x000055555b8b53e9 in (anonymous namespace)::GetPasswordV11 () at ../../components/os_crypt/os_crypt_linux.cc:117
#2  0x000055555b8b4b79 in OSCrypt::IsEncryptionAvailable () at ../../components/os_crypt/os_crypt_linux.cc:245
#3  0x00005555575c6f11 in base::RepeatingCallback<bool ()>::Run() const & (this=0x7fffffffbc50) at ../../base/callback.h:241
#4  gin_helper::Invoker<gin_helper::IndicesHolder<>>::DispatchToCallback<bool>(base::RepeatingCallback<bool ()>) (callback=..., this=<optimized out>)
    at ../../electron/shell/common/gin_helper/function_template.h:222
#5  gin_helper::Dispatcher<bool ()>::DispatchToCallback(v8::FunctionCallbackInfo<v8::Value> const&) (info=...)
    at ../../electron/shell/common/gin_helper/function_template.h:264
#6  0x00005555587307e1 in v8::internal::FunctionCallbackArguments::Call (this=0x7fffffffbce8, handler=...)
    at ../../v8/src/api/api-arguments-inl.h:152
#7  v8::internal::(anonymous namespace)::HandleApiCallHelper<false> (isolate=0x17ec0060c000, fun_data=..., receiver=..., function=..., 
    new_target=..., args=...) at ../../v8/src/builtins/builtins-api.cc:112
#8  v8::internal::Builtin_Impl_HandleApiCall (args=..., isolate=0x17ec0060c000) at ../../v8/src/builtins/builtins-api.cc:142
#9  v8::internal::Builtin_HandleApiCall (args_length=5, args_object=<optimized out>, isolate=0x17ec0060c000)
    at ../../v8/src/builtins/builtins-api.cc:130
#10 0x00005555572cb5f8 in Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit ()
#11 0x000055555724ae22 in Builtins_InterpreterEntryTrampoline ()
  • -s debug/debug/electron.debug is used to communicate the symbol-file path to GDB
  • -e node_modules/electron/dist/electron is used to pass the executable-file path to GDB
  • run test.js runs the program with the JS file path, test.js, as the first command line argument

There we go! The crash happens from the CreateKeyStorage() function at line 74 of components/os_crypt/os_crypt_linux.cc, which is present here - https://source.chromium.org/chromium/chromium/src/+/main:components/os_crypt/os_crypt_linux.cc;l=74;drc=35be6215ec8f09e50176f36753c68f26c63d1885;bpv=1;bpt=0.

// Create the KeyStorage. Will be null if no service is found. A Config must be
// set before every call to this function.
std::unique_ptr<KeyStorageLinux> CreateKeyStorage() {
  CHECK(g_cache.Get().config);
  std::unique_ptr<KeyStorageLinux> key_storage =
      KeyStorageLinux::CreateService(*g_cache.Get().config);
  g_cache.Get().config.reset();
  return key_storage;
}

The release CHECK() assertion here is failing which means that the config wasn’t set during the current call, dammit! Follow the rest of the debugging in the Electron issue on GitHub - electron/electron#32206!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment