MongoDB database corruption after reboot and journal recovery

gistfile1.txt

Tue Jan 29 10:27:21 [conn91] insert WF_0.Logs keyUpdates:0 locks(micros) w:6536794 6536ms
Tue Jan 29 10:27:21 [conn91] insert WF_0.Logs keyUpdates:0 locks(micros) w:225414 225ms
Tue Jan 29 10:27:30 [conn91] insert WF_0.Logs keyUpdates:0 locks(micros) w:266027 266ms
Tue Jan 29 10:27:30 [conn91] insert WF_0.Logs keyUpdates:0 locks(micros) w:327808 327ms
Tue Jan 29 10:27:31 [conn91] insert WF_0.Logs keyUpdates:0 locks(micros) w:223957 223ms
Tue Jan 29 10:27:31 [conn91] insert WF_0.Logs keyUpdates:0 locks(micros) w:183785 183ms
Tue Jan 29 10:27:33 [conn91] insert WF_0.Logs keyUpdates:0 locks(micros) w:240481 240ms
Tue Jan 29 10:27:37 [conn91] insert WF_0.Logs keyUpdates:0 locks(micros) w:232044 232ms
Tue Jan 29 10:27:44 [conn91] insert WF_0.Logs keyUpdates:0 locks(micros) w:156225 156ms
Tue Jan 29 10:27:46 [DataFileSync] flushing mmaps took 18136ms  for 5 files
Tue Jan 29 10:27:47 [conn91] insert WF_0.Logs keyUpdates:0 locks(micros) w:101432 101ms


***** SERVER RESTARTED *****


Tue Jan 29 10:29:57 [initandlisten] MongoDB starting : pid=2631 port=27017 dbpath=/mail/db 32-bit host=cerberus.redwind.net
Tue Jan 29 10:29:57 [initandlisten]
Tue Jan 29 10:29:57 [initandlisten] ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
Tue Jan 29 10:29:57 [initandlisten] **       see http://blog.mongodb.org/post/137788967/32-bit-limitations
Tue Jan 29 10:29:57 [initandlisten] **       with --journal, the limit is lower
Tue Jan 29 10:29:57 [initandlisten]
Tue Jan 29 10:29:57 [initandlisten] db version v2.2.2, pdfile version 4.5
Tue Jan 29 10:29:57 [initandlisten] git version: d1b43b61a5308c4ad0679d34b262c5af9d664267
Tue Jan 29 10:29:57 [initandlisten] build info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_49
Tue Jan 29 10:29:57 [initandlisten] options: { config: "/home/product/code/config/mongod.conf", dbpath: "/mail/db", fork: "true", journal: "true", logappend: "true", logpath: "/mail/log/mongo/mongod.log", pidfilepath: "/var/run/mongo/mongod.pid" }
Tue Jan 29 10:29:58 [initandlisten] journal dir=/mail/db/journal
Tue Jan 29 10:29:58 [initandlisten] recover begin
Tue Jan 29 10:29:58 [initandlisten] recover lsn: 440388820
Tue Jan 29 10:29:58 [initandlisten] recover /mail/db/journal/j._36
Tue Jan 29 10:29:58 [initandlisten] recover skipping application of section seq:425117054 < lsn:440388820
Tue Jan 29 10:29:58 [initandlisten] recover skipping application of section seq:425176524 < lsn:440388820
Tue Jan 29 10:29:58 [initandlisten] recover skipping application of section seq:425236094 < lsn:440388820
Tue Jan 29 10:29:58 [initandlisten] recover skipping application of section seq:425295604 < lsn:440388820
Tue Jan 29 10:29:58 [initandlisten] recover skipping application of section seq:425355184 < lsn:440388820
Tue Jan 29 10:29:58 [initandlisten] recover skipping application of section seq:425414724 < lsn:440388820
Tue Jan 29 10:29:58 [initandlisten] recover skipping application of section seq:425474274 < lsn:440388820
Tue Jan 29 10:29:58 [initandlisten] recover skipping application of section seq:425533844 < lsn:440388820
Tue Jan 29 10:29:58 [initandlisten] recover skipping application of section seq:425593404 < lsn:440388820
Tue Jan 29 10:29:58 [initandlisten] recover skipping application of section more...
Tue Jan 29 10:30:05 [initandlisten] recover /mail/db/journal/j._37
Tue Jan 29 10:30:10 [initandlisten] recover cleaning up
Tue Jan 29 10:30:10 [initandlisten] removeJournalFiles
Tue Jan 29 10:30:10 [initandlisten] recover done
Tue Jan 29 10:30:10 [initandlisten] preallocateIsFaster=true 5.5
Tue Jan 29 10:30:11 [initandlisten] preallocateIsFaster=true 8.54
Tue Jan 29 10:30:12 [initandlisten] preallocateIsFaster=true 7.86
Tue Jan 29 10:30:12 [initandlisten] preallocating a journal file /mail/db/journal/prealloc.0
Tue Jan 29 10:30:22 [initandlisten] preallocating a journal file /mail/db/journal/prealloc.1
Tue Jan 29 10:30:32 [websvr] admin web console waiting for connections on port 28017
Tue Jan 29 10:30:32 [initandlisten] waiting for connections on port 27017
Tue Jan 29 10:30:39 [initandlisten] connection accepted from 127.0.0.1:51989 #1 (1 connection now open)
Tue Jan 29 10:31:33 [initandlisten] connection accepted from 127.0.0.1:51995 #2 (2 connections now open)
Tue Jan 29 10:31:39 [initandlisten] connection accepted from 127.0.0.1:51996 #3 (3 connections now open)
Tue Jan 29 10:31:44 [initandlisten] connection accepted from 127.0.0.1:51997 #4 (4 connections now open)
Tue Jan 29 10:31:46 [initandlisten] connection accepted from 127.0.0.1:51999 #5 (5 connections now open)
Tue Jan 29 10:31:46 [conn5] query WF_0.Logs query: { orderby: { $natural: -1 }, query: {} } ntoreturn:1 ntoskip:0 nscanned:1 keyUpdates:0 locks(micros) W:83813 r:132806 nreturned:1 reslen:42 132ms
Tue Jan 29 10:31:49 [conn5] assertion 10334 Invalid BSONObj size: 1702065410 (0x02757365) first element: : ?type=114 ns:WF_0.Logs query:{ _id: { $gte: ObjectId('510814a4b8eff17100000806') } }
Tue Jan 29 10:31:49 [conn5] problem detected during query over WF_0.Logs : { $err: "Invalid BSONObj size: 1702065410 (0x02757365) first element: : ?type=114", code: 10334 }
Tue Jan 29 10:31:49 [conn5] query WF_0.Logs query: { _id: { $gte: ObjectId('510814a4b8eff17100000806') } } ntoreturn:0 keyUpdates:0 exception: Invalid BSONObj size: 1702065410 (0x02757365) first element: : ?type=114 code:10334 numYields: 1 locks(micros) r:3533881 reslen:118 3292ms
Tue Jan 29 10:31:49 [conn5] end connection 127.0.0.1:51999 (4 connections now open)
Tue Jan 29 10:31:54 [initandlisten] connection accepted from 127.0.0.1:52001 #6 (5 connections now open)
Tue Jan 29 10:31:55 [conn6] Assertion: 10334:Invalid BSONObj size: 1702065410 (0x02757365) first element: : ?type=114
0x8759eaa 0x8730e15 0x871a980 0x871ab26 0x81940ef 0x843c433 0x843cd05 0x8273226 0x8464f4c 0x846500e 0x846a443 0x846ca50 0x83e5591 0x83e6c15 0x81902b4 0x8746731 0x77db450b 0x77d42f4e
 /home/product/code/firmware/current/mongodb/mongod(_ZN5mongo15printStackTraceERSo+0x2a) [0x8759eaa]
 /home/product/code/firmware/current/mongodb/mongod(_ZN5mongo10logContextEPKc+0xa5) [0x8730e15]
 /home/product/code/firmware/current/mongodb/mongod(_ZN5mongo11msgassertedEiPKc+0xc0) [0x871a980]
 /home/product/code/firmware/current/mongodb/mongod [0x871ab26]
 /home/product/code/firmware/current/mongodb/mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x44f) [0x81940ef]
 /home/product/code/firmware/current/mongodb/mongod(_ZNK5mongo19CoveredIndexMatcher7matchesERKNS_7BSONObjERKNS_7DiskLocEPNS_12MatchDetailsEb+0x183) [0x843c433]
 /home/product/code/firmware/current/mongodb/mongod(_ZNK5mongo19CoveredIndexMatcher14matchesCurrentEPNS_6CursorEPNS_12MatchDetailsE+0xb5) [0x843cd05]
 /home/product/code/firmware/current/mongodb/mongod(_ZN5mongo6Cursor14currentMatchesEPNS_12MatchDetailsE+0x46) [0x8273226]
 /home/product/code/firmware/current/mongodb/mongod(_ZN5mongo20QueryResponseBuilder14currentMatchesERNS_12MatchDetailsE+0x3c) [0x8464f4c]
 /home/product/code/firmware/current/mongodb/mongod(_ZN5mongo20QueryResponseBuilder8addMatchEv+0x4e) [0x846500e]
 /home/product/code/firmware/current/mongodb/mongod(_ZN5mongo23queryWithQueryOptimizerEiRKSsRKNS_7BSONObjERNS_5CurOpES4_S4_RKN5boost10shared_ptrINS_11ParsedQueryEEES4_RKNS_17ShardChunkVersionERNS7_10scoped_ptrINS_25PageFaultRetryableSectionEEERNSG_INS_19NoPageFaultsAllowedEEERNS_7MessageE+0x573) [0x846a443]
 /home/product/code/firmware/current/mongodb/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x14f0) [0x846ca50]
 /home/product/code/firmware/current/mongodb/mongod [0x83e5591]
 /home/product/code/firmware/current/mongodb/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x3d5) [0x83e6c15]
 /home/product/code/firmware/current/mongodb/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x84) [0x81902b4]
 /home/product/code/firmware/current/mongodb/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x421) [0x8746731]
 /lib/libpthread.so.0 [0x77db450b]
 /lib/libc.so.6(__clone+0x5e) [0x77d42f4e]
Tue Jan 29 10:31:55 [conn6] assertion 10334 Invalid BSONObj size: 1702065410 (0x02757365) first element: : ?type=114 ns:WF_0.Logs query:{ _id: { $gte: ObjectId('510814a4b8eff17100000806') } }
Tue Jan 29 10:31:55 [conn6] problem detected during query over WF_0.Logs : { $err: "Invalid BSONObj size: 1702065410 (0x02757365) first element: : ?type=114", code: 10334 }
Tue Jan 29 10:31:55 [conn6] query WF_0.Logs query: { _id: { $gte: ObjectId('510814a4b8eff17100000806') } } ntoreturn:0 keyUpdates:0 exception: Invalid BSONObj size: 1702065410 (0x02757365) first element: : ?type=114 code:10334 numYields: 1 locks(micros) r:837009 reslen:118 488ms