Debugging MongoDB with GDB

debugging_mongodb_gdb.md

Debugging MongoDB with gdb

gdb is a command line tool that you can use to examine the state of a (1) terminated or (2) running C or C++ process.

Background: Debugging a terminated process

A process can terminate (uncleanly) for a few reasons, such as triggering a segmentation fault, failing an invariant, or throwing an uncaught exception.

A binary can be compiled so that when a process terminates (uncleanly), the process produces a core dump, which is a file containing a "frozen" record of the state of all threads that existed in the process at the time it terminated.

You can use gdb as "viewer" of this core dump file to examine the frozen state. For example, you can use gdb on the core dump file to:

• list all threads
• in each thread, show the current backtrace
• for each frame in each thread's backtrace, print the values of the local variables in that frame
• print values of the global variables

Though having a core dump is useful to examine the "end state" of a process that terminated uncleanly, it doesn't tell you how the process reached the "end state."

Examining a core dump with gdb is most useful for understanding why a process terminated unexpectedly in the wild.

Background: Debugging a running process

If you want to trace how a process reaches a particular state, it’s more useful to execute the process through gdb.

This allows you to use breakpoints to specify at what points the process should "freeze."

Once the process has hit a breakpoint and frozen, you can use gdb to examine its current state, just as you would examine a core dump.

The difference is that, once you are done examining the frozen state, you can use gdb to "unfreeze" the process so that the process runs until the next breakpoint is hit.

Stepping through code

If you're interested in stepping through some region of code, it would be rather tedious to specify a breakpoint for each line. For this reason, gdb provides "syntactic sugar" commands that, once a breakpoint is hit, allow you to advance the process by one statement ("step") or one line ("next") at a time.

So, you only need to specify a breakpoint on the first line of the region.

Using gdb to step through a running process is most useful when you can reproduce some behavior locally, but want to trace the code paths and changes in the process’s state that lead to that behavior.

Setting up your system to debug MongoDB with gdb

To debug with gdb, you have to

1. install gdb on your local machine
2. compile MongoDB binaries for use with gdb (that is, pass flags to the compiler to produce binaries that contain debug symbols).

This tutorial assumes you are using a linux-based operating system (not OS X or Windows; the author uses Ubuntu 16.04), and assumes you know (or are comfortable looking up) how to:

• do basic commands on your machine's terminal (edit files, change file permissions, execute a binary)
• compile MongoDB binaries on your machine (directly with scons or through ninja)

Install gdb (and gdbserver)

If you are a MongoDB employee, you should already have installed the MongoDB toolchain, and you should be using either the gcc or clang compiler in the toolchain to compile your MongoDB binaries.

Using the compilers (gcc, clang) and debugger (gdb) from the toolchain ensures that the versions of your compiler and debugger are compatible with each other and support the pretty printers for MongoDB types (more on those below).

How to install the MongoDB toolchain

Follow the directions in this email to kernel@10gen.com to install the MongoDB toolchain for your operating system.

How to compile with the gcc or clang in the toolchain

If you are compiling directly with buildscripts/scons.py, then to compile with the gcc in the toolchain, pass --variables-files=etc/scons/mongodbtoolchain_gcc.vars to your scons invocation, e.g.

$ ./buildscripts/scons.py --variables-files=etc/scons/mongodbtoolchain_gcc.vars mongod mongos mongo

Similarly, to compile with the clang in the toolchain, pass --variables-files=etc/scons/mongodbtoolchain_clang.vars:

$ ./buildscripts/scons.py --variables-files=etc/scons/mongodbtoolchain_clang.vars mongod mongos mongo

If you are compiling through ninja, you will have to rebuild your .ninja executable with the --variables-files argument. See the ninja github repo for directions on recompiling .ninja.

Once you have the toolchain set up, you should see a copy of gdb and gdbserver in your /opt/mongodbtoolchain/gdb/bin directory:

$ ls /opt/mongodbtoolchain/gdb/bin
gcore  gdb  gdbserver

You should now either

1. add /opt/mongodbtoolchain/gdb/bin to your $PATH (see here for a quick refresher on editing your $PATH), or
2. create an alias for "gdb" and "gdbserver" to point to the one in the toolchain (see here for a quick refresher on creating an alias).

If you are not a MongoDB employee, you can install gdb and gdbserver using your system's package manager. For example:

$ sudo apt-get install gdb gdbserver

The gdb and gdbserver executables should automatically be placed in your /usr/bin directory, which should already be in your $PATH.

You will have to ensure on your own that the version of gdb you install is compatible with the version of gcc or clang used to compile the binaries.

Compile MongoDB binaries with debug symbols

Once you have installed gdb, the next step is to compile your binaries with debug symbols. Fortunately, MongoDB has set up our build system (SCons) to automatically compile with debug symbols.

Quick background on debug symbols

When you compile source code, the compiler builds a symbol table that maps variable names, function names, and literals to their type, scope, and other information.

You can use gdb on any binary compiled from C or C++ source code - unit test, dbtest, mongod, mongos, mongo, etc.

How to use gdb

Examining a core dump

Pass gdb the core dump of a process that has terminated, plus the binary that was used to launch the process

Attach gdb to a hung (but still running) process

Similar to debugging a core dump

Execute a binary entirely through gdb

Some basic gdb commands to get you started

Examine frozen state

info threads

backtrace or bt

Step through code

run or r

step or s

next or n

continue or c

What's next?

Become a gdb power user

Install gdb pretty-printers for the C++ STL types, Boost types, and MongoDB types

Pretty printers produce user-friendly string descriptions of types.

These all involve installing Python libraries and registering them with gdb through your ~/.gdbinit.

• C++ Standard Library Data structures (vectors, stacks, etc): https://sourceware.org/gdb/wiki/STLSupport
• Boost objects: https://github.com/ruediger/Boost-Pretty-Printer
• MongoDB: https://github.com/mongodb/mongo/blob/master/buildscripts/gdb/mongo.py

See all gdb commands and command-line options

http://www.yolinux.com/TUTORIALS/GDB-Commands.html

Step through a multithreaded process

Use gdbserver

Compile the binaries with --gdbserver so that they hang rather than terminate

This is mainly useful for replica sets and sharded clusters:

Compile the binaries with --gdbserver, then start the processes as normal. If a breakpoint is hit or the process would have terminated (due to a segmentation fault, invariant failure, or uncaught exception), instead of terminating it will launch a "gdbserver" process and hang. You can then use gdb to attach to the "gdbserver" that is attached to the hung process.

Related topics and tools

Related topics:

• compiling with and without optimizations
• compiling with clang vs. compiling with gcc

Related tools:

• lldb (gdb for OS X)
• C++ demangler
• symbolizing a stack trace
• nm to see symbols in a binary

Thx Esha. Very helpful pointers.

…

On 23. Jan 2022, at 22:04, Esha Maharishi ***@***.***> wrote: @EshaMaharishi commented on this gist. Hi @sebastiangajek <https://github.com/sebastiangajek> , you should be able to use a publicly available gcc or clang. There are some instructions on the mongo GitHub repo wiki: https://github.com/mongodb/mongo/blob/98f6025860618721a2fa414a0345caf4c4d5bd57/docs/building.md <https://github.com/mongodb/mongo/blob/98f6025860618721a2fa414a0345caf4c4d5bd57/docs/building.md> The MongoDB Community Forum is also a great resource if you have trouble: https://www.mongodb.com/community/forums/ <https://www.mongodb.com/community/forums/>. — Reply to this email directly, view it on GitHub <https://gist.github.com/88965c523625e077adf7884474942e68#gistcomment-4038957>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG3M27N2UGZSBKFTBU3V6ZLUXRUMDANCNFSM5MPSKLYA>. You are receiving this because you were mentioned.

It seems that you haven't talked about what kind of aspects we should be concerned when debugging mongo server. There's one I want to mention. That is, when debugging the primary node of a replica set, in order to prevent it from stepping down, we should issue the following command before running in gdb :

cfg=rs.conf(); cfg.settings.heartbeatTimeoutSecs=3600 ; cfg.settings.electionTimeoutMillis=3600000; rs.reconfig(cfg)

Above assuming that we're not debugging the election part of mongo server.

I have another gdb problem that you might be interested in. Any help is appreciated.

https://stackoverflow.com/questions/71193608/problem-internal-to-gdb-has-been-detected-and-debugging-session-dies-in-multi-th

Thanks. That’s a good hint. I don’t know what exactly to debug yet. Looking for a way to turn debug flags on. Apologies for brevity, message sent from my iPhone

…

On 20. Feb 2022, at 11:13, Lewis Chan ***@***.***> wrote: ***@***.*** commented on this gist. It seems that you haven't talked about what kind of aspects we should be concerned when debugging mongo server. There's one I want to mention. That is, when debugging the primary node of a replica set, in order to prevent it from stepping down, we should issue the following command before running in gdb : cfg=rs.conf(); cfg.settings.heartbeatTimeoutSecs=3600 ; cfg.settings.electionTimeoutMillis=3600000; rs.reconfig(cfg) Above assuming that we're not debugging the election part of mongo server. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

@baiwfg2 , that is indeed a good way to prevent a primary from stepping down while debugging. Thank you for mentioning it. Another way to do this is to set the secondaries' replica set priority to 0 (https://docs.mongodb.com/manual/core/replica-set-priority-0-member/).

@sebastiangajek , I confirmed that we actually always compile with debug symbols, so my text was misleading. I have updated the text. This comment also gave some context.