Skip to content

Instantly share code, notes, and snippets.

@Firefishy
Created September 22, 2018 18:07
Show Gist options
  • Save Firefishy/a1bf8806ea60561fb77e02f877dffc4f to your computer and use it in GitHub Desktop.
Save Firefishy/a1bf8806ea60561fb77e02f877dffc4f to your computer and use it in GitHub Desktop.
osd.5 crash
-2> 2018-09-22 17:56:11.830 7fe76e603700 5 -- 10.89.121.203:6809/8690 >> 10.89.121.202:6810/3429 conn(0x55fe26068300 :6809 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=118 cs=1 l=0). rx osd.10 s
eq 57 0x55fe25e41600 pg_query(1.1b4s5 epoch 41227) v4
-1> 2018-09-22 17:56:11.830 7fe76e603700 1 -- 10.89.121.203:6809/8690 <== osd.10 10.89.121.202:6810/3429 57 ==== pg_query(1.1b4s5 epoch 41227) v4 ==== 162+0+0 (2599670235 0 0) 0x55fe25e41600 con 0x55
fe26068300
0> 2018-09-22 17:56:11.830 7fe7533ca700 -1 /build/ceph-13.2.1/src/osd/PGLog.h: In function 'static void PGLog::_merge_object_divergent_entries(const PGLog::IndexedLog&, const hobject_t&, mempool::osd
_pglog::list<pg_log_entry_t>&, const pg_info_t&, eversion_t, missing_type&, PGLog::LogEntryHandler*, const DoutPrefixProvider*) [with missing_type = pg_missing_set<true>; mempool::osd_pglog::list<pg_log_e
ntry_t> = std::__cxx11::list<pg_log_entry_t, mempool::pool_allocator<(mempool::pool_index_t)14, pg_log_entry_t> >]' thread 7fe7533ca700 time 2018-09-22 17:56:11.833432
/build/ceph-13.2.1/src/osd/PGLog.h: 921: FAILED assert(objiter->second->version > last_divergent_update)
ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7fe7745594e2]
2: (()+0x2e26a7) [0x7fe7745596a7]
3: (void PGLog::_merge_object_divergent_entries<pg_missing_set<true> >(PGLog::IndexedLog const&, hobject_t const&, std::__cxx11::list<pg_log_entry_t, mempool::pool_allocator<(mempool::pool_index_t)14, pg
_log_entry_t> > const&, pg_info_t const&, eversion_t, pg_missing_set<true>&, PGLog::LogEntryHandler*, DoutPrefixProvider const*)+0x39b0) [0x55fda8e47c20]
4: (PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0xe30) [0x55fda8e41510]
5: (PG::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_shard_t)+0x64) [0x55fda8d9eeb4]
6: (PG::RecoveryState::Stray::react(MLogRec const&)+0xef) [0x55fda8de6b3f]
7: (boost::statechart::simple_state<PG::RecoveryState::Stray, PG::RecoveryState::Started, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::
na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+
0x1a8) [0x55fda8e34ba8]
8: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::
event_base const&)+0x6b) [0x55fda8e07edb]
9: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x143) [0x55fda8dedd53]
10: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xcf) [0x55fda8d2f64f]
11: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x50) [0x55fda8f99350]
12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x590) [0x55fda8d3f000]
13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x46e) [0x7fe77455e41e]
14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x7fe7745604a0]
15: (()+0x76ba) [0x7fe772be46ba]
16: (clone()+0x6d) [0x7fe7721f341d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment