mongodb config server logfiles

server 1

Thu Mar 17 19:03:41 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:03:45 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:03:48 [conn47] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:03:48 [conn47] fsync from getlasterror
Thu Mar 17 19:03:48 [conn47] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:03:48 [conn47] fsync from getlasterror
Thu Mar 17 19:03:49 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:03:50 [conn47] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:03:50 [conn47] fsync from getlasterror
Thu Mar 17 19:03:50 [conn49] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:03:50 [conn49] fsync from getlasterror
Thu Mar 17 19:03:51 [DataFileSync] flushing diag log
Thu Mar 17 19:03:53 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:03:57 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:03:58 [conn27] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:03:58 [conn27] fsync from getlasterror
Thu Mar 17 19:03:58 [conn27] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:03:58 [conn27] fsync from getlasterror
Thu Mar 17 19:04:00 [conn47] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:04:00 [conn47] fsync from getlasterror
Thu Mar 17 19:04:00 [conn49] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:04:00 [conn49] fsync from getlasterror
Thu Mar 17 19:04:01 [DataFileSync] flushing diag log
Thu Mar 17 19:04:01 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:04:05 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:04:09 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:04:10 [conn47] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:04:10 [conn47] fsync from getlasterror
Thu Mar 17 19:04:10 [conn49] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:04:10 [conn49] fsync from getlasterror
Thu Mar 17 19:04:11 [DataFileSync] flushing diag log
Thu Mar 17 19:04:13 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:04:14 Got signal: 11 (Segmentation fault).

Thu Mar 17 19:04:14 Backtrace:
0x8a5359 0x7ffe3455c560 0x607db0 0x7fd35b 0x7ff594 0x7dc5f0 0x7ddb21 0x645565 0x64addc 0x757f15 0x75a440 0x8a617e 0x8b92d0 0x7ffe3500565d 0x7ffe345fae1d 
 mongod(_ZN5mongo10abruptQuitEi+0x399) [0x8a5359]
 /lib64/libc.so.6 [0x7ffe3455c560]
 mongod(_ZN5mongo12ScriptEngine14getPooledScopeERKSs+0x250) [0x607db0]
 mongod(_ZN5mongo12GroupCommand5groupESsRKSsRKNS_7BSONObjES3_SsSsPKcS3_SsRSsRNS_14BSONObjBuilderE+0x4b) [0x7fd35b]
 mongod(_ZN5mongo12GroupCommand3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0xe34) [0x7ff594]
 mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x9a0) [0x7dc5f0]
 mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x831) [0x7ddb21]
 mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x35) [0x645565]
 mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x31ac) [0x64addc]
 mongod [0x757f15]
 mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x5b0) [0x75a440]
 mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x21e) [0x8a617e]
 mongod(thread_proxy+0x80) [0x8b92d0]
 /lib64/libpthread.so.0 [0x7ffe3500565d]
 /lib64/libc.so.6(clone+0x6d) [0x7ffe345fae1d]

Thu Mar 17 19:04:14 dbexit: 
Thu Mar 17 19:04:14 [conn47] shutdown: going to close listening sockets...
Thu Mar 17 19:04:14 [conn47] closing listening socket: 6
Thu Mar 17 19:04:14 [conn47] closing listening socket: 7
Thu Mar 17 19:04:14 [conn47] closing listening socket: 8
Thu Mar 17 19:04:14 [conn47] closing listening socket: 9
Thu Mar 17 19:04:14 [conn47] removing socket file: /tmp/mongodb-27217.sock
Thu Mar 17 19:04:14 [conn47] removing socket file: /tmp/mongodb-28217.sock
Thu Mar 17 19:04:14 [conn47] shutdown: going to flush diaglog...
Thu Mar 17 19:04:14 [conn47] flushing diag log
Thu Mar 17 19:04:14 [conn47] shutdown: going to close sockets...
Thu Mar 17 19:04:14 [conn47] shutdown: waiting for fs preallocator...
Thu Mar 17 19:04:14 [conn47] shutdown: closing all files...
Thu Mar 17 19:04:14 closeAllFiles() finished
Thu Mar 17 19:04:14 [conn47] shutdown: removing fs lock...
Thu Mar 17 19:04:14 dbexit: really exiting now
Thu Mar 17 19:04:14 ERROR: Client::~Client _context should be null but is not; client:conn

server 2

Thu Mar 17 19:02:49 [conn12] killcursors: found 0 of 1
Thu Mar 17 19:02:49 [conn12] killcursors: found 0 of 1
Thu Mar 17 19:02:49 [conn12] killcursors: found 0 of 1
Thu Mar 17 19:02:49 [conn12] ClientCursor::find(): cursor not found in map 0 (ok after a drop)
killcursors: found 0 of 1
Thu Mar 17 19:02:49 [conn12] killcursors: found 0 of 1
Thu Mar 17 19:02:49 [conn12] killcursors: found 0 of 1
Thu Mar 17 19:02:49 [conn12] killcursors: found 0 of 1
Thu Mar 17 19:02:49 [conn12] killcursors: found 0 of 1
Thu Mar 17 19:02:49 [conn12] killcursors: found 0 of 1
Thu Mar 17 19:02:49 [conn12] killcursors: found 0 of 1
Thu Mar 17 19:02:49 [conn12] killcursors: found 0 of 1
Thu Mar 17 19:02:49 [conn12] killcursors: found 0 of 1
Thu Mar 17 19:02:49 [conn12] killcursors: found 0 of 1
Thu Mar 17 19:02:50 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:02:50 [conn46] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:02:50 [conn46] fsync from getlasterror
Thu Mar 17 19:02:50 [conn47] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:02:50 [conn32] CMD fsync:  sync:1 lock:0

[…]

Thu Mar 17 19:03:38 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:03:40 [conn46] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:03:40 [conn46] fsync from getlasterror
Thu Mar 17 19:03:40 [conn47] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:03:40 [conn47] fsync from getlasterror
Thu Mar 17 19:03:42 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:03:43 [DataFileSync] flushing diag log
Thu Mar 17 19:03:46 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:03:48 [conn46] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:03:48 [conn46] fsync from getlasterror
Thu Mar 17 19:03:48 [conn46] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:03:48 [conn46] fsync from getlasterror
Thu Mar 17 19:03:50 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:03:50 [conn46] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:03:50 [conn46] fsync from getlasterror
Thu Mar 17 19:03:50 [conn47] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:03:50 [conn47] fsync from getlasterror
Thu Mar 17 19:03:53 [DataFileSync] flushing diag log
Thu Mar 17 19:03:54 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:03:58 [conn31] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:03:58 [conn31] fsync from getlasterror
Thu Mar 17 19:03:58 [conn31] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:03:58 [conn31] fsync from getlasterror
Thu Mar 17 19:03:58 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:04:00 [conn46] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:04:00 [conn46] fsync from getlasterror
Thu Mar 17 19:04:00 [conn47] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:04:00 [conn47] fsync from getlasterror
Thu Mar 17 19:04:02 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:04:03 [DataFileSync] flushing diag log
Thu Mar 17 19:04:04 [conn49] end connection 172.19.117.51:60587
Thu Mar 17 19:04:06 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:04:10 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:04:10 [conn46] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:04:10 [conn46] fsync from getlasterror
Thu Mar 17 19:04:10 [conn47] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:04:10 [conn47] fsync from getlasterror
Thu Mar 17 19:04:13 [DataFileSync] flushing diag log
Thu Mar 17 19:04:14 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:04:14 Got signal: 11 (Segmentation fault).

Thu Mar 17 19:04:14 Backtrace:
0x8a5359 0x7fe33236d560 0x607db0 0x7fd35b 0x7ff594 0x7dc5f0 0x7ddb21 0x645565 0x64addc 0x757f15 0x75a440 0x8a617e 0x8b92d0 0x7fe332e1665d 0x7fe33240be1d 
 mongod(_ZN5mongo10abruptQuitEi+0x399) [0x8a5359]
 /lib64/libc.so.6 [0x7fe33236d560]
 mongod(_ZN5mongo12ScriptEngine14getPooledScopeERKSs+0x250) [0x607db0]
 mongod(_ZN5mongo12GroupCommand5groupESsRKSsRKNS_7BSONObjES3_SsSsPKcS3_SsRSsRNS_14BSONObjBuilderE+0x4b) [0x7fd35b]
 mongod(_ZN5mongo12GroupCommand3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0xe34) [0x7ff594]
 mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x9a0) [0x7dc5f0]
 mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x831) [0x7ddb21]
 mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x35) [0x645565]
 mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x31ac) [0x64addc]
 mongod [0x757f15]
 mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x5b0) [0x75a440]
 mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x21e) [0x8a617e]
 mongod(thread_proxy+0x80) [0x8b92d0]
 /lib64/libpthread.so.0 [0x7fe332e1665d]
 /lib64/libc.so.6(clone+0x6d) [0x7fe33240be1d]

Thu Mar 17 19:04:14 dbexit: 
Thu Mar 17 19:04:14 [conn46] shutdown: going to close listening sockets...
Thu Mar 17 19:04:14 [conn46] closing listening socket: 6
Thu Mar 17 19:04:14 [conn46] closing listening socket: 7
Thu Mar 17 19:04:14 [conn46] closing listening socket: 8
Thu Mar 17 19:04:14 [conn46] closing listening socket: 9
Thu Mar 17 19:04:14 [conn46] removing socket file: /tmp/mongodb-27217.sock
Thu Mar 17 19:04:14 [conn46] removing socket file: /tmp/mongodb-28217.sock
Thu Mar 17 19:04:14 [conn46] shutdown: going to flush diaglog...
Thu Mar 17 19:04:14 [conn46] flushing diag log
Thu Mar 17 19:04:14 [conn46] shutdown: going to close sockets...
Thu Mar 17 19:04:14 [conn46] shutdown: waiting for fs preallocator...
Thu Mar 17 19:04:14 [conn46] shutdown: closing all files...
Thu Mar 17 19:04:14 closeAllFiles() finished
Thu Mar 17 19:04:14 [conn46] shutdown: removing fs lock...
Thu Mar 17 19:04:14 dbexit: really exiting now
Thu Mar 17 19:04:14 ERROR: Client::~Client _context should be null but is not; client:conn

server 3

Thu Mar 17 19:03:40 [conn49] fsync from getlasterror
Thu Mar 17 19:03:40 [conn51] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:03:40 [conn51] fsync from getlasterror
Thu Mar 17 19:03:42 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:03:46 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:03:48 [DataFileSync] flushing diag log
Thu Mar 17 19:03:48 [conn49] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:03:48 [conn49] fsync from getlasterror
Thu Mar 17 19:03:48 [conn49] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:03:48 [conn49] fsync from getlasterror
Thu Mar 17 19:03:50 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:03:50 [conn49] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:03:50 [conn49] fsync from getlasterror
Thu Mar 17 19:03:50 [conn51] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:03:50 [conn51] fsync from getlasterror
Thu Mar 17 19:03:54 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:03:58 [DataFileSync] flushing diag log
Thu Mar 17 19:03:58 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:03:58 [conn29] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:03:58 [conn29] fsync from getlasterror
Thu Mar 17 19:03:58 [conn29] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:03:58 [conn29] fsync from getlasterror
Thu Mar 17 19:04:00 [conn49] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:04:00 [conn49] fsync from getlasterror
Thu Mar 17 19:04:00 [conn51] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:04:00 [conn51] fsync from getlasterror
Thu Mar 17 19:04:02 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:04:06 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:04:08 [DataFileSync] flushing diag log
Thu Mar 17 19:04:10 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:04:10 [conn49] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:04:10 [conn49] fsync from getlasterror
Thu Mar 17 19:04:10 [conn51] CMD fsync:  sync:1 lock:0
Thu Mar 17 19:04:10 [conn51] fsync from getlasterror
Thu Mar 17 19:04:14 [snapshotthread] cpu: elapsed:4000  writelock: 0%
Thu Mar 17 19:04:14 Got signal: 11 (Segmentation fault).

Thu Mar 17 19:04:14 Backtrace:
0x8a5359 0x7f339b2e7560 0x607db0 0x7fd35b 0x7ff594 0x7dc5f0 0x7ddb21 0x645565 0x64addc 0x757f15 0x75a440 0x8a617e 0x8b92d0 0x7f339bd9065d 0x7f339b385e1d 
 mongod(_ZN5mongo10abruptQuitEi+0x399) [0x8a5359]
 /lib64/libc.so.6 [0x7f339b2e7560]
 mongod(_ZN5mongo12ScriptEngine14getPooledScopeERKSs+0x250) [0x607db0]
 mongod(_ZN5mongo12GroupCommand5groupESsRKSsRKNS_7BSONObjES3_SsSsPKcS3_SsRSsRNS_14BSONObjBuilderE+0x4b) [0x7fd35b]
 mongod(_ZN5mongo12GroupCommand3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0xe34) [0x7ff594]
 mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x9a0) [0x7dc5f0]
 mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x831) [0x7ddb21]
 mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x35) [0x645565]
 mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x31ac) [0x64addc]
 mongod [0x757f15]
 mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x5b0) [0x75a440]
 mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x21e) [0x8a617e]
 mongod(thread_proxy+0x80) [0x8b92d0]
 /lib64/libpthread.so.0 [0x7f339bd9065d]
 /lib64/libc.so.6(clone+0x6d) [0x7f339b385e1d]

Thu Mar 17 19:04:14 dbexit: 
Thu Mar 17 19:04:14 [conn49] shutdown: going to close listening sockets...
Thu Mar 17 19:04:14 [conn49] closing listening socket: 6
Thu Mar 17 19:04:14 [conn49] closing listening socket: 7
Thu Mar 17 19:04:14 [conn49] closing listening socket: 8
Thu Mar 17 19:04:14 [conn49] closing listening socket: 9
Thu Mar 17 19:04:14 [conn49] removing socket file: /tmp/mongodb-27217.sock
Thu Mar 17 19:04:14 [conn49] removing socket file: /tmp/mongodb-28217.sock
Thu Mar 17 19:04:14 [conn49] shutdown: going to flush diaglog...
Thu Mar 17 19:04:14 [conn49] flushing diag log
Thu Mar 17 19:04:14 [conn49] shutdown: going to close sockets...
Thu Mar 17 19:04:14 [conn49] shutdown: waiting for fs preallocator...
Thu Mar 17 19:04:14 [conn49] shutdown: closing all files...
Thu Mar 17 19:04:14 closeAllFiles() finished
Thu Mar 17 19:04:14 [conn49] shutdown: removing fs lock...
Thu Mar 17 19:04:14 dbexit: really exiting now
Thu Mar 17 19:04:14 ERROR: Client::~Client _context should be null but is not; client:conn

http://jira.mongodb.org/browse/SERVER-2786