Skip to content

Instantly share code, notes, and snippets.

@masonmark

masonmark/gdb backtrace

Created September 21, 2011 06:49
Show Gist options
  • Save masonmark/1231423 to your computer and use it in GitHub Desktop.
Save masonmark/1231423 to your computer and use it in GitHub Desktop.
Download ZIP
passenger phubar - various debuggery
Raw
gdb backtrace
mason@ubuntu:~$ sudo bash
[sudo] password for mason:
root@ubuntu:~#
root@ubuntu:~#
root@ubuntu:~#
root@ubuntu:~#
root@ubuntu:~#
root@ubuntu:~# sudo gdb
sudo: gdb: command not found
root@ubuntu:~# apt-get install gdb
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
libpython2.7
Suggested packages:
gdb-doc gdbserver
The following NEW packages will be installed:
gdb libpython2.7
0 upgraded, 2 newly installed, 0 to remove and 44 not upgraded.
Need to get 3,187 kB of archives.
After this operation, 9,081 kB of additional disk space will be used.
Do you want to continue [Y/n]?
Get:1 http://us.archive.ubuntu.com/ubuntu/ natty/main libpython2.7 amd64 2.7.1-5ubuntu2 [1,196 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu/ natty/main gdb amd64 7.2-1ubuntu11 [1,992 kB]
Fetched 3,187 kB in 9s (326 kB/s)
Selecting previously deselected package libpython2.7.
(Reading database ... 33024 files and directories currently installed.)
Unpacking libpython2.7 (from .../libpython2.7_2.7.1-5ubuntu2_amd64.deb) ...
Selecting previously deselected package gdb.
Unpacking gdb (from .../gdb_7.2-1ubuntu11_amd64.deb) ...
Processing triggers for man-db ...
Setting up libpython2.7 (2.7.1-5ubuntu2) ...
Setting up gdb (7.2-1ubuntu11) ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
root@ubuntu:~# gdb
GNU gdb (Ubuntu/Linaro 7.2-1ubuntu11) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
(gdb) attach 21116
Attaching to process 21116
Reading symbols from /usr/local/bin/ruby...done.
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x7fe8b86b5700 (LWP 21185)]
Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/librt.so.1
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /lib/x86_64-linux-gnu/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libcrypt.so.1
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/local/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so...done.
Loaded symbols for /usr/local/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
Reading symbols from /usr/local/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so...done.
Loaded symbols for /usr/local/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
Reading symbols from /usr/local/lib/ruby/1.9.1/x86_64-linux/socket.so...done.
Loaded symbols for /usr/local/lib/ruby/1.9.1/x86_64-linux/socket.so
Reading symbols from /usr/local/lib/ruby/1.9.1/x86_64-linux/etc.so...done.
Loaded symbols for /usr/local/lib/ruby/1.9.1/x86_64-linux/etc.so
Reading symbols from /lib/x86_64-linux-gnu/libnss_compat.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_compat.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libnsl.so.1
Reading symbols from /lib/x86_64-linux-gnu/libnss_nis.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_nis.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2
Reading symbols from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/ext/ruby/ruby-1.9.2-x86_64-linux/passenger_native_support.so...done.
Loaded symbols for /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/ext/ruby/ruby-1.9.2-x86_64-linux/passenger_native_support.so
Reading symbols from /usr/local/lib/ruby/1.9.1/x86_64-linux/fcntl.so...done.
Loaded symbols for /usr/local/lib/ruby/1.9.1/x86_64-linux/fcntl.so
Reading symbols from /usr/local/lib/ruby/1.9.1/x86_64-linux/strscan.so...done.
Loaded symbols for /usr/local/lib/ruby/1.9.1/x86_64-linux/strscan.so
0x00007fe8b8b135b7 in sched_yield () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) thread apply bt all
(gdb) thread apply all bt
Thread 2 (Thread 0x7fe8b86b5700 (LWP 21185)):
#0 0x00007fe8b96c4f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005270f4 in native_cond_timedwait (dummy=<value optimized out>) at thread_pthread.c:145
#2 thread_timer (dummy=<value optimized out>) at thread_pthread.c:802
#3 0x00007fe8b96bfd8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007fe8b8b4104d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7fe8b9aeb720 (LWP 21116)):
#0 0x00007fe8b8b135b7 in sched_yield () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00000000005296c6 in rb_threadptr_execute_interrupts_rec (th=0x18e7b40, sched_depth=0) at thread.c:1268
#2 0x000000000052c39b in rb_threadptr_execute_interrupts (th=0x18e7b40, deadlockable=1) at thread.c:1333
#3 sleep_forever (th=0x18e7b40, deadlockable=1) at thread.c:852
#4 0x000000000052c3fd in thread_join_sleep (arg=140733935317296) at thread.c:682
#5 0x000000000041a3db in rb_ensure (b_proc=0x52c3c0 <thread_join_sleep>, data1=140733935317296,
e_proc=0x524dc0 <remove_from_join_list>, data2=140733935317296) at eval.c:743
#6 0x00000000005267dd in thread_join (target_th=0x21a26c0, delay=<value optimized out>) at thread.c:715
#7 0x0000000000526920 in thread_join_m (argc=<value optimized out>, argv=<value optimized out>,
self=<value optimized out>) at thread.c:796
#8 0x000000000052122d in vm_call_cfunc (th=0x18e7b40, cfp=0x7fe8b87b5988, num=<value optimized out>,
blockptr=<value optimized out>, flag=<value optimized out>, id=<value optimized out>, me=0x19f2580,
recv=35225360) at vm_insnhelper.c:402
#9 vm_call_method (th=0x18e7b40, cfp=0x7fe8b87b5988, num=<value optimized out>, blockptr=<value optimized out>,
flag=<value optimized out>, id=<value optimized out>, me=0x19f2580, recv=35225360) at vm_insnhelper.c:524
#10 0x0000000000515b54 in vm_exec_core (th=<value optimized out>, initial=<value optimized out>)
at insns.def:1006
#11 0x000000000051b336 in vm_exec (th=0x18e7b40) at vm.c:1147
#12 0x000000000051c082 in invoke_block_from_c (th=0x18e7b40, block=<value optimized out>, self=26276400, argc=0,
argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0) at vm.c:558
#13 0x0000000000521d24 in vm_yield (val=35225360) at vm.c:588
#14 rb_yield_0 (val=35225360) at vm_eval.c:740
#15 rb_yield (val=35225360) at vm_eval.c:750
#16 0x0000000000535be6 in rb_ary_each (ary=26275800) at array.c:1427
#17 0x000000000052122d in vm_call_cfunc (th=0x18e7b40, cfp=0x7fe8b87b5a90, num=<value optimized out>,
blockptr=<value optimized out>, flag=<value optimized out>, id=<value optimized out>, me=0x199e6b0,
recv=26275800) at vm_insnhelper.c:402
#18 vm_call_method (th=0x18e7b40, cfp=0x7fe8b87b5a90, num=<value optimized out>, blockptr=<value optimized out>,
flag=<value optimized out>, id=<value optimized out>, me=0x199e6b0, recv=26275800) at vm_insnhelper.c:524
#19 0x0000000000515b54 in vm_exec_core (th=<value optimized out>, initial=<value optimized out>)
at insns.def:1006
#20 0x000000000051b336 in vm_exec (th=0x18e7b40) at vm.c:1147
#21 0x000000000051c082 in invoke_block_from_c (th=0x18e7b40, block=<value optimized out>, self=28452240, argc=0,
argv=<value optimized out>, blockptr=<value optimized out>, cref=0x0) at vm.c:558
#22 0x0000000000521d24 in vm_yield (val=28413960) at vm.c:588
#23 rb_yield_0 (val=28413960) at vm_eval.c:740
#24 rb_yield (val=28413960) at vm_eval.c:750
#25 0x0000000000535be6 in rb_ary_each (ary=28410400) at array.c:1427
#26 0x000000000052122d in vm_call_cfunc (th=0x18e7b40, cfp=0x7fe8b87b5ca0, num=<value optimized out>,
blockptr=<value optimized out>, flag=<value optimized out>, id=<value optimized out>, me=0x199e6b0,
recv=28410400) at vm_insnhelper.c:402
#27 vm_call_method (th=0x18e7b40, cfp=0x7fe8b87b5ca0, num=<value optimized out>, blockptr=<value optimized out>,
flag=<value optimized out>, id=<value optimized out>, me=0x199e6b0, recv=28410400) at vm_insnhelper.c:524
#28 0x0000000000515b54 in vm_exec_core (th=<value optimized out>, initial=<value optimized out>)
at insns.def:1006
#29 0x000000000051b336 in vm_exec (th=0x18e7b40) at vm.c:1147
#30 0x0000000000522570 in rb_iseq_eval (iseqval=29651600) at vm.c:1374
#31 0x0000000000570d4e in rb_load_internal (fname=29655400, wrap=<value optimized out>) at load.c:294
#32 0x0000000000570eae in rb_f_load (argc=<value optimized out>, argv=<value optimized out>) at load.c:367
#33 0x000000000052122d in vm_call_cfunc (th=0x18e7b40, cfp=0x7fe8b87b5f08, num=<value optimized out>,
blockptr=<value optimized out>, flag=<value optimized out>, id=<value optimized out>, me=0x19e07f0,
recv=26511760) at vm_insnhelper.c:402
#34 vm_call_method (th=0x18e7b40, cfp=0x7fe8b87b5f08, num=<value optimized out>, blockptr=<value optimized out>,
flag=<value optimized out>, id=<value optimized out>, me=0x19e07f0, recv=26511760) at vm_insnhelper.c:524
#35 0x0000000000515b54 in vm_exec_core (th=<value optimized out>, initial=<value optimized out>)
at insns.def:1006
#36 0x000000000051b336 in vm_exec (th=0x18e7b40) at vm.c:1147
#37 0x000000000052267f in rb_iseq_eval_main (iseqval=26328920) at vm.c:1388
#38 0x000000000041755a in ruby_exec_internal (n=0x191bf58) at eval.c:214
#39 0x0000000000419be6 in ruby_exec_node (n=<value optimized out>) at eval.c:261
#40 ruby_run_node (n=<value optimized out>) at eval.c:254
#41 0x00000000004170e9 in main (argc=3, argv=0x7fff2c390438) at main.c:35
(gdb)
Raw
passenger-status --show=backtraces
mason@ubuntu:~$ passenger-status --show=backtraces
*** ERROR: You are not authorized to query the status for this Phusion Passenger instance. Please try again with 'sudo'.
mason@ubuntu:~$ sudo passenger-status --show=backtraces
[sudo] password for mason:
Thread 'Main thread':
in 'void Server::mainLoop()' (HelperAgent.cpp:1281)
in 'int main(int, char**)' (HelperAgent.cpp:1342)
Thread 'Client thread 2':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'Client thread 1':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'Client thread 3':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'Client thread 4':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'Client thread 5':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'Client thread 6':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'Client thread 7':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'ApplicationPool analytics collector':
in 'void Passenger::ApplicationPool::Pool::analyticsCollectionThreadMainLoop()' (Pool.h:870)
Thread 'Client thread 8':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'Client thread 9':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'Client thread 10':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'Client thread 11':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'Client thread 12':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'Client thread 13':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'Client thread 14':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'Client thread 15':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'Client thread 17':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'Client thread 16':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'Client thread 18':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'Client thread 20':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'Client thread 21':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'Client thread 22':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'Client thread 19':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'Client thread 23':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'Client thread 24':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:427)
in 'void Client::threadMain()' (HelperAgent.cpp:950)
Thread 'MessageServer thread':
in 'void Passenger::MessageServer::mainLoop()' (MessageServer.h:537)
Thread 'ApplicationPool cleaner':
(empty)
Thread 'MessageServer client thread 35':
in 'virtual bool Passenger::BacktracesServer::processMessage(Passenger::MessageServer::CommonClientContext&, Passenger::MessageServer::ClientContextPtr&, const std::vector<std::basic_string<char> >&)' (BacktracesServer.h:47)
in 'void Passenger::MessageServer::clientHandlingMainLoop(Passenger::FileDescriptor&)' (MessageServer.h:470)
mason@ubuntu:~$
Raw
what top looks like
top - 01:08:34 up 7:59, 6 users, load average: 1.20, 1.26, 1.34
Tasks: 96 total, 3 running, 93 sleeping, 0 stopped, 0 zombie
Cpu(s): 12.6%us, 87.4%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1019220k total, 516840k used, 502380k free, 43532k buffers
Swap: 1046524k total, 0k used, 1046524k free, 276712k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21116 rollerba 20 0 56476 11m 2636 R 95.3 1.2 153:54.18 passenger
1 root 20 0 23876 2036 1324 S 0.0 0.2 0:01.58 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:01.78 ksoftirqd/0
5 root 20 0 0 0 0 S 0.0 0.0 0:02.47 kworker/u:0
6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
7 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 cpuset
8 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper
9 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns
10 root 20 0 0 0 0 S 0.0 0.0 0:00.07 sync_supers
11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 bdi-default
12 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd
13 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd
14 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kacpid
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment