juju-db-startup-error.log

Oct 10 15:16:22 juju-84a348-0 mongod.37017[20713]: [initandlisten] MongoDB starting : pid=20713 port=37017 dbpath=/var/lib/juju/db 64-bit host=juju-84a348-0
Oct 10 15:16:22 juju-84a348-0 mongod.37017[20713]: [initandlisten] db version v3.2.9
Oct 10 15:16:22 juju-84a348-0 mongod.37017[20713]: [initandlisten] git version: 22ec9e93b40c85fc7cae7d56e7d6a02fd811088c
Oct 10 15:16:22 juju-84a348-0 mongod.37017[20713]: [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
Oct 10 15:16:22 juju-84a348-0 mongod.37017[20713]: [initandlisten] allocator: system
Oct 10 15:16:22 juju-84a348-0 mongod.37017[20713]: [initandlisten] modules: none
Oct 10 15:16:22 juju-84a348-0 mongod.37017[20713]: [initandlisten] build environment:
Oct 10 15:16:22 juju-84a348-0 mongod.37017[20713]: [initandlisten]     distarch: s390x
Oct 10 15:16:22 juju-84a348-0 mongod.37017[20713]: [initandlisten]     target_arch: s390x
Oct 10 15:16:22 juju-84a348-0 mongod.37017[20713]: [initandlisten] options: { net: { ipv6: true, port: 37017, ssl: { PEMKeyFile: "/var/lib/juju/server.pem", PEMKeyPassword: "<password>", mode: "requireSSL" } }, replication: { oplogSizeMB: 1024, replSet: "juju" }, security: { authorization: "enabled", keyFile: "/var/lib/juju/shared-secret" }, storage: { dbPath: "/var/lib/juju/db", engine: "wiredTiger", journal: { enabled: true } }, systemLog: { destination: "syslog", quiet: true } }
Oct 10 15:16:22 juju-84a348-0 mongod.37017[20713]: [initandlisten] Detected unclean shutdown - /var/lib/juju/db/mongod.lock is not empty.
Oct 10 15:16:22 juju-84a348-0 mongod.37017[20713]: [initandlisten] Recovering data from the last clean checkpoint.
Oct 10 15:16:22 juju-84a348-0 mongod.37017[20713]: [initandlisten] wiredtiger_open config: create,cache_size=22G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
Oct 10 15:16:23 juju-84a348-0 mongod.37017[20713]: [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs
Oct 10 15:16:23 juju-84a348-0 mongod.37017[20713]: [initandlisten] The size storer reports that the oplog contains 1378374 records totaling to 391729955 bytes
Oct 10 15:16:23 juju-84a348-0 mongod.37017[20713]: [initandlisten] Sampling from the oplog between Oct  7 03:34:26:1 and Oct  8 14:56:18:2 to determine where to place markers for truncation
Oct 10 15:16:23 juju-84a348-0 mongod.37017[20713]: [initandlisten] Taking 229 samples and assuming that each section of oplog contains approximately 59971 records totaling to 17043586 bytes
Oct 10 15:16:23 juju-84a348-0 mongod.37017[20713]: [initandlisten] WiredTiger (5) [1476112583:118688][20713:0x3ff8fa6ab50], file:collection-4-3785058392042379666.wt, WT_CURSOR.next: /var/lib/juju/db/collection-4-3785058392042379666.wt: handle-read: pread: failed to read 8192 bytes at offset 140378112: Input/output error
Oct 10 15:16:23 juju-84a348-0 mongod.37017[20713]: [initandlisten] Invariant failure: advanceRet resulted in status UnknownError: 5: Input/output error at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 674
Oct 10 15:16:23 juju-84a348-0 mongod.37017[20713]: [initandlisten] 
                                                    0x2aa188e64cc 0x2aa1887e90a 0x2aa18867fa4 0x2aa18621b96 0x2aa1861f18a 0x2aa1861fb2a 0x2aa1861fdd0 0x2aa1862058e 0x2aa1860bd8e 0x2aa1853642c 0x2aa1853c1da 0x2aa186093d6 0x2aa184ea01c 0x2aa17cdf55a 0x2aa17ce24ca 0x2aa17c9578e 0x3ff8e922ece 0x2aa17cdb184
                                                   ----- BEGIN BACKTRACE -----
                                                   {"backtrace":[{"b":"2AA17900000","o":"FE64CC","s":"_ZN5mongo15printStackTraceERSo"},{"b":"2AA17900000","o":"F7E90A","s":"_ZN5mongo10logContextEPKc"},{"b":"2AA17900000","o":"F67FA4","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"2AA17900000","o":"D21B96","s":"_ZN5mongo21WiredTigerRecordStore12RandomCursor4nextEv"},{"b":"2AA17900000","o":"D1F18A","s":"_ZN5mongo21WiredTigerRecordStore11OplogStones26_calculateStonesBySamplingEPNS_16OperationContextEll"},{"b":"2AA17900000","o":"D1FB2A","s":"_ZN5mongo21WiredTigerRecordStore11OplogStones16_calculateStonesEPNS_16OperationContextE"},{"b":"2AA17900000","o":"D1FDD0","s":"_ZN5mongo21WiredTigerRecordStore11OplogStonesC1EPNS_16OperationContextEPS0_"},{"b":"2AA17900000","o":"D2058E","s":"_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextENS_10StringDataES3_NSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEbbllPNS_14CappedCallbackEPNS_20WiredTigerSizeStorerE"},{"b":"2AA17900000","o":"D0BD8E","s":"_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsE"},{"b":"2AA17900000","o":"C3642C","s":"_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb"},{"b":"2AA17900000","o":"C3C1DA","s":"_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE"},{"b":"2AA17900000","o":"D093D6"},{"b":"2AA17900000","o":"BEA01C","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"2AA17900000","o":"3DF55A"},{"b":"2AA17900000","o":"3E24CA","s":"_ZN5mongo13initAndListenEi"},{"b":"2AA17900000","o":"39578E","s":"main"},{"b":"3FF8E900000","o":"22ECE","s":"__libc_start_main"},{"b":"2AA17900000","o":"3DB184"}],"processInfo":{ "mongodbVersion" : "3.2.9", "gitVersion" : "22ec9e93b40c85fc7cae7d56e7d6a02fd811088c", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-38-generic", "version" : "#57-Ubuntu SMP Tue Sep 6 15:47:15 UTC 2016", "machine" : "s390x" }, "somap" : [ { "b" : "2AA17900000", "elfType" : 3, "buildId" : "31054E6B093666B39652D82A92821416B63DCF85" }, { "b" : "3FF8FA7E000", "elfType" : 3, "buildId" : "BE870A96C8A763E6A833FB0E1089F43D51A763BD" }, { "b" : "3FF8F880000", "path" : "/usr/lib/s390x-linux-gnu/libboost_program_options.so.1.58.0", "elfType" : 3, "buildId" : "5E5E392A01D68D1293B8217B4856404884BBA87F" }, { "b" : "3FF8F800000", "path" : "/usr/lib/s390x-linux-gnu/libboost_filesystem.so.1.58.0", "elfType" : 3, "buildId" : "FDE3947CAD68B78BAF80A7BA55FC4965A6E1D244" }, { "b" : "3FF8F780000", "path" : "/usr/lib/s390x-linux-gnu/libboost_thread.so.1.58.0", "elfType" : 3, "buildId" : "F5C15B03A95392CD92E1E5678485F5EB3BB7873E" }, { "b" : "3FF8F700000", "path" : "/usr/lib/s390x-linux-gnu/libboost_system.so.1.58.0", "elfType" : 3, "buildId" : "1C9354143944280EFA3B9BB310DD27020B5713D9" }, { "b" : "3FF8F680000", "path" : "/usr/lib/s390x-linux-gnu/libboost_chrono.so.1.58.0", "elfType" : 3, "buildId" : "E6851F9FFB38D29D4BE6FA32A244B16F376429E1" }, { "b" : "3FF8F500000", "path" : "/usr/lib/s390x-linux-gnu/libboost_regex.so.1.58.0", "elfType" : 3, "buildId" : "A2669E1230272504E67D4C753067B9ADA2AF2193" }, { "b" : "3FF8F480000", "path" : "/lib/s390x-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "509C86154DA660B3DD34F9A2DA6FA4D406CC1A2D" }, { "b" : "3FF8F400000", "path" : "/usr/lib/s390x-linux-gnu/libsnappy.so.1", "elfType" : 3, "buildId" : "6922CA9AC421DF51DF02C8CFCA4FD1379B36C864" }, { "b" : "3FF8F380000", "path" : "/usr/lib/s390x-linux-gnu/libpcrecpp.so.0", "elfType" : 3, "buildId" : "8D5828036A127B76D9647389065C4266D5F45A9F" }, { "b" : "3FF8F280000", "path" : "/usr/lib/s390x-linux-gnu/libyaml-cpp.so.0.5", "elfType" : 3, "buildId" : "B8FA50FE8D1DB24958C40A0EE4F0DD69BF50A42C" }, { "b" : "3FF8F200000", "path" : "/lib/s390x-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "ED4741B6EA59F9B2F4169C5446C73947FF852B32" }, { "b" : "3FF8F000000", "path" : "/lib/s390x-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "D21E67ED5D6BE438E96EA5C78DDF5AC2E5F90203" }, { "b" : "3FF8EF80000", "path" : "/lib/s390x-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F2D002F02CE52324F0E369264186287EB0E67BE8" }, { "b" : "3FF8EF00000", "path" : "/lib/s390x-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0F93456E45EED7284C7C1534E6E34A53AAF60AA1" }, { "b" : "3FF8ED00000", "path" : "/usr/lib/s390x-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "58E1D5C6F900D3AB50E04A0CD561BC7C21D846F8" }, { "b" : "3FF8EC00000", "path" : "/lib/s390x-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "0E62A558BADA56B5CDDC57F6BFF89A3E1A8553BA" }, { "b" : "3FF8EB80000", "path" : "/lib/s390x-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "F0DD559EE2315FF87DE5C371D021145F28A2853B" }, { "b" : "3FF8EB00000", "path" : "/lib/s390x-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "96C573EE24AE2FBC716E23F0AACEB4B533C4C568" }, { "b" : "3FF8E900000", "path" : "/lib/s390x-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "3304BE1E66F0BDD1D099F3CC42CB115A86D6F0BF" }, { "b" : "3FF8FA00000", "path" : "/lib/ld64.so.1", "elfType" : 3, "buildId" : "72B051E559F3B8BD14A2E039AB96674A00C4044C" }, { "b" : "3FF8E600000", "path" : "/usr/lib/s390x-linux-gnu/libicui18n.so.55", "elfType" : 3, "buildId" : "279786FB95006B699A007D3CF076F74E514787F5" }, { "b" : "3FF8E400000", "path" : "/usr/lib/s390x-linux-gnu/libicuuc.so.55", "elfType" : 3, "buildId" : "2A96356099364B49A8561D88950E36942C792058" }, { "b" : "3FF8E380000", "path" : "/lib/s390x-linux-gnu/libpcre.so.3", "elfType" : 3, "buildId" : "13C8CFDFF43447A2E26D8AAA0A78AE7BF1A10C24" }, { "b" : "3FF8CA80000", "path" : "/usr/lib/s390x-linux-gnu/libicudata.so.55", "elfType" : 3, "buildId" : "0FE729E8345CB044E32F7797B343241C32457640" } ] }}
                                                    mongod(_ZN5mongo15printStackTraceERSo+0x5C) [0x2aa188e64cc]
                                                    mongod(_ZN5mongo10logContextEPKc+0x1A2) [0x2aa1887e90a]
                                                    mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0x10C) [0x2aa18867fa4]
                                                    mongod(_ZN5mongo21WiredTigerRecordStore12RandomCursor4nextEv+0x166) [0x2aa18621b96]
                                                    mongod(_ZN5mongo21WiredTigerRecordStore11OplogStones26_calculateStonesBySamplingEPNS_16OperationContextEll+0x50A) [0x2aa1861f18a]
                                                    mongod(_ZN5mongo21WiredTigerRecordStore11OplogStones16_calculateStonesEPNS_16OperationContextE+0x19A) [0x2aa1861fb2a]
                                                    mongod(_ZN5mongo21WiredTigerRecordStore11OplogStonesC1EPNS_16OperationContextEPS0_+0x1E0) [0x2aa1861fdd0]
                                                    mongod(_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextENS_10StringDataES3_NSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEbbllPNS_14CappedCallbackEPNS_20WiredTigerSizeStorerE+0x676) [0x2aa1862058e]
                                                    mongod(_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsE+0x146) [0x2aa1860bd8e]
                                                    mongod(_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb+0x334) [0x2aa1853642c]
                                                    mongod(_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE+0x8EA) [0x2aa1853c1da]
                                                    mongod(+0xD093D6) [0x2aa186093d6]
                                                    mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x3FC) [0x2aa184ea01c]
                                                    mongod(+0x3DF55A) [0x2aa17cdf55a]
                                                    mongod(_ZN5mongo13initAndListenEi+0x2A) [0x2aa17ce24ca]
                                                    mongod(main+0x71E) [0x2aa17c9578e]
                                                    libc.so.6(__libc_start_main+0x10E) [0x3ff8e922ece]
                                                    mongod(+0x3DB184) [0x2aa17cdb184]
                                                   -----  END BACKTRACE  -----
Oct 10 15:16:23 juju-84a348-0 mongod.37017[20713]: [initandlisten] 
                                                   
                                                   ***aborting after invariant() failure
Oct 10 15:16:23 juju-84a348-0 systemd[1]: juju-db.service: Main process exited, code=exited, status=14/n/a
Oct 10 15:16:23 juju-84a348-0 systemd[1]: juju-db.service: Unit entered failed state.
Oct 10 15:16:23 juju-84a348-0 systemd[1]: juju-db.service: Failed with result 'exit-code'.
Oct 10 15:16:23 juju-84a348-0 systemd[1]: juju-db.service: Service hold-off time over, scheduling restart.
Oct 10 15:16:23 juju-84a348-0 systemd[1]: Stopped juju state database.
-- Subject: Unit juju-db.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit juju-db.service has finished shutting down.
Oct 10 15:16:23 juju-84a348-0 systemd[1]: juju-db.service: Start request repeated too quickly.
Oct 10 15:16:23 juju-84a348-0 systemd[1]: Failed to start juju state database.
-- Subject: Unit juju-db.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit juju-db.service has failed.
-- 
-- The result is failed.