past_interval start interval mismatch

BUG #49689.md

past_intervals: start interval does not contain the required bound start, Bug #49689

More importantly, we haven't be able to pin down the root cause of this issue. If anyone is able to reproduce this with logs you will be my hero forever! sage

Log files

Error message is:

2021-03-20T05:10:47.151+0000 7f29a5cd4700 -1 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 peering mbc={}] 3.11s0 past_intervals [362,375) start interval does not contain the required bound [151,375) start

The mismatch occurring is more broad and is happening in most of the OSDs with the same starting epoch of 362.

Taking a closer look into the relevant pg values (osd.11):

pg 3.11s0	val
past_intervals.first	362
past_intervals.last	375
ec (epoch_pool_created)	27
lec (last epoch clean)	115

Abort message is triggred from within check_past_interval_bounds():

auto oldest_epoch = pl->oldest_stored_osdmap();
auto rpib = get_required_past_interval_bounds(info, oldest_epoch);

...

auto apib = past_intervals.get_bounds();
if (apib.first > rpib.first) {
  ceph_abort_msg("past_interval start interval mismatch");
}

get_required_past_interval_bounds() is setting past interval first value in the following way:

 epoch_t start = std::max(
    info.history.last_epoch_clean ? info.history.last_epoch_clean :
    info.history.epoch_pool_created,
    oldest_map);

Resulting in the following values:

apib = [362,375)

rpib = [151,375)

Superblock Oldest Map

oldest_map is set by the osd's superblock: osd->get_superblock().oldest_map

2021-03-20T05:10:47.147+0000 7f29b34ef700 10 osd.11 374 write_superblock sb(0437ecdc-8b43-4f3f-ab79-d7775b8c351f osd.11 de2bf8dc-46eb-4787-ad9f-794f5c25bfc7 e375 [151,375] lci=[18,375])

superblock	val
Oldest map we have (sb.oldest_map)	151
Newest map we have (sb.newest_map)	75
Last epoch active and clean thru	375

Pg 3.11s0 Past Intervals at creation

2021-03-20T05:10:47.150+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s0 no pg, should create on OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)
2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 375 _make_pg 3.11s0

pg 3.11	val
first	362
last	374

// OSDMap Trimming

Latest osdmap epoch is trimmed up to 361 and is being removed from osd 10 and 5:

2021-03-20T05:10:56.416+0000 7f2a0358d700 10 osd.5 381 handle_osd_map  got inc map for epoch 382
2021-03-20T05:10:56.416+0000 7f2a0358d700 20 osd.5 381 got_full_map 382, nothing requested
2021-03-20T05:10:56.416+0000 7f2a0358d700 20 osd.5 381  removing old osdmap epoch 361
2021-03-20T05:10:56.416+0000 7f2a0358d700 10 osd.5 381 write_superblock sb(0437ecdc-8b43-4f3f-ab79-d7775b8c351f osd.5 37657fce-de27-4acb-b831-0cbb03ad507b e381 [362,381] lci=[18,381])

2021-03-20T05:10:59.420+0000 7f04e1332700  3 osd.10 382 handle_osd_map epochs [383,383], i have 382, src has [362,383]
2021-03-20T05:10:59.420+0000 7f04e1332700 10 osd.10 382 handle_osd_map  got inc map for epoch 383
2021-03-20T05:10:59.420+0000 7f04e1332700 20 osd.10 382 got_full_map 383, nothing requested
2021-03-20T05:10:59.420+0000 7f04e1332700 20 osd.10 382  removing old osdmap epoch 361
2021-03-20T05:10:59.420+0000 7f04e1332700 10 osd.10 382 write_superblock sb(0437ecdc-8b43-4f3f-ab79-d7775b8c351f osd.10 5abc33f5-2896-4d8d-a33d-8c85a0d39a5b e382 [362,382] lci=[18,382])

Latest trim from osd.11 (150):

2021-03-20T05:10:47.145+0000 7f29b34ef700  3 osd.11 374 handle_osd_map epochs [375,375], i have 374, src has [362,375]
2021-03-20T05:10:47.145+0000 7f29b34ef700 10 osd.11 374 handle_osd_map  got inc map for epoch 375
2021-03-20T05:10:47.145+0000 7f29b34ef700 20 osd.11 374 got_full_map 375, nothing requested
2021-03-20T05:10:47.145+0000 7f29b34ef700 20 osd.11 374  removing old osdmap epoch 121
2021-03-20T05:10:47.145+0000 7f29b34ef700 20 osd.11 374  removing old osdmap epoch 122
...
2021-03-20T05:10:47.145+0000 7f29b34ef700 20 osd.11 374  removing old osdmap epoch 149
2021-03-20T05:10:47.145+0000 7f29b34ef700 20 osd.11 374  removing old osdmap epoch 150
2021-03-20T05:10:47.145+0000 7f29b34ef700 10 osd.11 374 write_superblock sb(0437ecdc-8b43-4f3f-ab79-d7775b8c351f osd.11 de2bf8dc-46eb-4787-ad9f-794f5c25bfc7 e374 [151,374] lci=[18,374])

handle_osd_map epochs	val
first	375
last	375
superblock.newest_map	374
m->newest_map	375
m->oldest_map	362

if (superblock.oldest_map) {
    // make sure we at least keep pace with incoming maps
    trim_maps(m->oldest_map, last - first + 1, skip_maps);
    pg_num_history.prune(superblock.oldest_map);
}

OSD::trim_maps arguments	val
nreceived	1
oldest	362
superblock.oldest_map	121
osd_target_transaction_size	30
skip_maps	false

void OSD::trim_maps(epoch_t oldest, int nreceived, bool skip_maps)
{
  epoch_t min = std::min(oldest, service.map_cache.cached_key_lower_bound());
  if (min <= superblock.oldest_map)
    return;

  int num = 0;
  ObjectStore::Transaction t;
  for (epoch_t e = superblock.oldest_map; e < min; ++e) {
    dout(20) << " removing old osdmap epoch " << e << dendl;
    t.remove(coll_t::meta(), get_osdmap_pobject_name(e));
    t.remove(coll_t::meta(), get_inc_osdmap_pobject_name(e));
    superblock.oldest_map = e + 1;
    num++;
    if (num >= cct->_conf->osd_target_transaction_size && num >= nreceived) {
      service.publish_superblock(superblock);
      write_superblock(t);
      int tr = store->queue_transaction(service.meta_ch, std::move(t), nullptr);
      ceph_assert(tr == 0);
      num = 0;
      if (!skip_maps) {
	// skip_maps leaves us with a range of old maps if we fail to remove all
	// of them before moving superblock.oldest_map forward to the first map
	// in the incoming MOSDMap msg. so we should continue removing them in
	// this case, even we could do huge series of delete transactions all at
	// once.
	break;
      }
    }
  }
  if (num > 0) {
    service.publish_superblock(superblock);
    write_superblock(t);
    int tr = store->queue_transaction(service.meta_ch, std::move(t), nullptr);
    ceph_assert(tr == 0);
  }
  // we should not remove the cached maps
  ceph_assert(min <= service.map_cache.cached_key_lower_bound());
}

After reaching osd_target_transaction_size which is 30 we publish and write into the super block which result in superblock's oldest_map of 151. skip_maps is set to false, so we will break out of the foor loop and won't remove the remaining osdmaps (151 to 362). skip_maps usage while trimming is introduced here: https://github.com/ceph/ceph/commit/369db9930887d75b498927da9c97733bff4472b6

• trim all stale osdmaps in batch of conf->osd_target_transaction_size if skip_maps is true. in my test, it happens when the osd only receives the osdmap from monitor occasionally because the osd happens to be chosen when monitor wants to share a new osdmap with a random osd.

althouh osd_target_transaction_size in handle_osd_map is introduced beforehand: https://github.com/ceph/ceph/commit/35753eb7761cd37d8c20b808e390df2b67a4eb64

• (Should we remove all stale osdmaps if start interval does not contain the required bound start?)

Thesis #2 (WIP)

When assigning required past intervals bounds (rpib) with get_required_past_interval_bounds() we need to verify that info.history.last_epoch_clean is in sync. Since it looks like oldest_map > info.history.last_epoch_clean (!) and we set rpib.start by the osd's oldest map.

get_required_past_interval_bounds() is setting rpib.start to be 151, bases on the following values:

epoch_t start = std::max(
  info.history.last_epoch_clean (115) ? info.history.last_epoch_clean (115) :
  info.history.epoch_pool_created (27),
  oldest_map(151);

Should last_epoch_clean be equal/greater than pi.start (Compared to a happy flow)? If so, are we updating history.last_epoch_clean correctly?

Comparing to Unreleated (successful) pg creations (3.c,3.5):

2021-03-20T05:09:40.131+0000 7f29a44d1700 10 osd.11 308 handle_pg_create_info new pg pg[3.cs4( empty local-lis/les=0/0 n=0 ec=27/27 lis/c=257/257 les/c/f=258/258/0 sis=307) [0,3,6,10,11,2]/[0,3,6,10,NONE,2]p0(0) r=-1 lpr=307 pi=[257,307)/1 crt=0'0 mlcod 0'0 unknown mbc={}]

2021-03-20T05:06:23.076+0000 7f29b34ef700 10 osd.11 111 write_superblock sb(0437ecdc-8b43-4f3f-ab79-d7775b8c351f osd.11 de2bf8dc-46eb-4787-ad9f-794f5c25bfc7 e112 [1,112] lci=[18,112])

pg 3.c	val
pi.start	257
lec (last epoch clean)	258
superblock's Oldest map we have (sb.oldest_map)	1

2021-03-20T05:08:39.437+0000 7f29a3cd0700 10 osd.11 247 handle_pg_create_info new pg pg[3.5s0( empty local-lis/les=0/0 n=0 ec=27/27 lis/c=109/109 les/c/f=110/110/0 sis=247) [11,3,2,6,7,1]p11(0) r=0 lpr=247 pi=[109,247)/1 crt=0'0 mlcod 0'0 peering mbc={}]

pg 3.5	val
pi.start	109
lec (last epoch clean)	110

In both cases, last_epoch_clean is equal/greater than pi.start. While Superblock's oldestmap is not being used. (Unlike our case) We are not setting the required past intervals bound (rpib) by the oldestmap.

// WIP: Is last_epoch_clean up to date?

"no pg, should create on " --> pg = osd->handle_pg_create_info(osdmap, create_info); --> pg->init(info->history, info->past_intervals,);

void PeeringState::init(const pg_history_t& history) {
  info.history = history;
}

// WIP2

pg 3.11s5	val
first	114
last	362

2021-03-20T05:10:46.202+0000 7f2a0fea3700 10 osd.9 pg_epoch: 362 pg[3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=114) [10,4,6,0,3,7]p10(0) r=-1 lpr=362 pi=[114,362)/1 crt=304'349 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[e1~1,f8~1,fc~2,10a~1]] start_peering_interval: check_new_interval output: check_new_interval interval(114-361 up [10,4,6,0,3,9](10) acting [10,4,6,0,3,9](10)) up_thru 247 up_from 18 last_epoch_clean 115 interval(114-361 up [10,4,6,0,3,9](10) acting [10,4,6,0,3,9](10) maybe_went_rw) : primary up 18-247 includes interval

2021-03-20T05:10:46.202+0000 7f2a0fea3700 10 osd.9 pg_epoch: 362 pg[3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=114) [10,4,6,0,3,7]p10(0) r=-1 lpr=362 pi=[114,362)/1 crt=304'349 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[e1~1,f8~1,fc~2,10a~1]]  map gap, clearing past_intervals and faking
2021-03-20T05:10:46.202+0000 7f2a0fea3700 20 osd.9 pg_epoch: 362 pg[3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=362) [10,4,6,0,3,7]p10(0) r=-1 lpr=362 crt=304'349 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[e1~1,f8~1,fc~2,10a~1]] on_new_interval

2021-03-20T05:10:46.585+0000 7f2a2ef26700 20 osd.9 374 scrub_purged_snaps pg 3.11 not found
2021-03-20T05:10:47.139+0000 7f2a214fc700 20 osd.9 375 identify_splits_and_merges 3.11s5 e374 to e375 pg_nums {109=26}
2021-03-20T05:10:47.139+0000 7f2a214fc700 20 osd.9:1.consume_map consume_map 3.11s5

2021-03-20T05:10:47.141+0000 7f2a0fea3700 10 osd.9 pg_epoch: 375 pg[3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=362) [11,4,6,0,3,7]p11(0) r=-1 lpr=375 pi=[362,375)/0 crt=304'349 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[e1~1,f8~1,fc~2,10a~1]] start_peering_interval: check_new_interval output: check_new_interval interval(362-374 up [10,4,6,0,3,7](10) acting [10,4,6,0,3,7](10)) up_thru 309 up_from 18 last_epoch_clean 115 interval(362-374 up [10,4,6,0,3,7](10) acting [10,4,6,0,3,7](10)) : primary up 18-309 does not include interval
2021-03-20T05:10:47.141+0000 7f2a0fea3700 10 osd.9 pg_epoch: 375 pg[3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=362) [11,4,6,0,3,7]p11(0) r=-1 lpr=375 pi=[362,375)/0 crt=304'349 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[e1~1,f8~1,fc~2,10a~1]]  noting past ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)
2021-03-20T05:10:47.141+0000 7f2a0fea3700 20 osd.9 pg_epoch: 375 pg[3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=-1 lpr=375 pi=[362,375)/0 crt=304'349 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[e1~1,f8~1,fc~2,10a~1]] on_new_interval
2021-03-20T05:10:47.141+0000 7f2a0fea3700 20 osd.9 pg_epoch: 375 pg[3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=-1 lpr=375 pi=[362,375)/0 crt=304'349 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[e1~1,f8~1,fc~2,10a~1]] on_new_interval upacting_features 0x3f01cfbb7ffdffff from [11,4,6,0,3,7]+[11,4,6,0,3,7]
2021-03-20T05:10:47.141+0000 7f2a0fea3700 20 osd.9 pg_epoch: 375 pg[3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=-1 lpr=375 pi=[362,375)/0 crt=304'349 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[e1~1,f8~1,fc~2,10a~1]] on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1)
2021-03-20T05:10:47.141+0000 7f2a0fea3700 10 osd.9 pg_epoch: 375 pg[3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=-1 lpr=375 pi=[362,375)/0 crt=304'349 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[e1~1,f8~1,fc~2,10a~1]] init_hb_stamps now []
2021-03-20T05:10:47.141+0000 7f2a0fea3700 10 osd.9 pg_epoch: 375 pg[3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=-1 lpr=375 pi=[362,375)/0 crt=304'349 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[e1~1,f8~1,fc~2,10a~1]] on_new_interval prior_readable_until_ub 0.000000000s (mnow 1.677365899s + 0.000000000s)
2021-03-20T05:10:47.141+0000 7f2a0fea3700 20 osd.9 pg_epoch: 375 pg[3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=-1 lpr=375 pi=[362,375)/0 crt=304'349 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[e1~1,f8~1,fc~2,10a~1]] on_new_interval scrub_queued was 0 flags: 

2021-03-20T05:10:47.141+0000 7f2a0fea3700  1 -- [v2:172.21.15.109:6804/57610,v1:172.21.15.109:6806/57610] --> [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] -- pg_notify2(3.11s0 3.11s0 (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) e375/375) v1 -- 0x55964dd49500 con 0x55964dd73c00
2021-03-20T05:10:47.141+0000 7f2a0fea3700 10 bluestore(/var/lib/ceph/osd/ceph-9) queue_transactions ch 0x55963ecb1860 3.11s5_head

All OSDs

Multiple OSDs start interval mismatch

osd.0, osd.1, osd.2, osd.3, osd.4, osd.6, osd.7, osd.8, osd.11

3.4s2/s4, 3.fs5, 1.1, 3.7s4, 3.17s2, 3.0s1, 3.11s0, 3.es1

 /ceph-osd.0.log:1314997:2021-03-20T05:10:48.134+0000 7f3cdf572700 -1 log_channel(cluster) log [ERR] : 3.4s2 past_intervals [362,376) start interval does not contain the required bound [255,376) start
./ceph-osd.0.log:1314998:2021-03-20T05:10:48.134+0000 7f3cdf572700 -1 osd.0 pg_epoch: 376 pg[3.4s2( empty local-lis/les=0/0 n=0 ec=27/27 lis/c=254/254 les/c/f=255/255/0 sis=376) [1,NONE,0,8,3,2]p0(2) r=2 lpr=376 pi=[362,376)/0 crt=0'0 mlcod 0'0 peering mbc={}] 3.4s2 past_intervals [362,376) start interval does not contain the required bound [255,376) start
./ceph-osd.1.log:1051995:2021-03-20T05:10:50.150+0000 7fe929d45700 -1 log_channel(cluster) log [ERR] : 3.fs5 past_intervals [362,378) start interval does not contain the required bound [241,378) start
./ceph-osd.1.log:1051996:2021-03-20T05:10:50.150+0000 7fe929d45700 -1 osd.1 pg_epoch: 378 pg[3.fs5( empty local-lis/les=0/0 n=0 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=378) [NONE,NONE,NONE,NONE,NONE,1]p1(5) r=5 lpr=378 pi=[362,378)/0 crt=0'0 mlcod 0'0 peering mbc={}] 3.fs5 past_intervals [362,378) start interval does not contain the required bound [241,378) start
./ceph-osd.2.log:1017561:2021-03-20T05:10:49.149+0000 7f0a4b3ee700 -1 log_channel(cluster) log [ERR] : 1.1 past_intervals [362,377) start interval does not contain the required bound [227,377) start
./ceph-osd.2.log:1017562:2021-03-20T05:10:49.149+0000 7f0a4b3ee700 -1 osd.2 pg_epoch: 377 pg[1.1( empty local-lis/les=0/0 n=0 ec=147/19 lis/c=226/226 les/c/f=227/227/0 sis=377) [2] r=0 lpr=377 pi=[362,377)/1 crt=0'0 mlcod 0'0 peering mbc={}] 1.1 past_intervals [362,377) start interval does not contain the required bound [227,377) start
./ceph-osd.3.log:1335457:2021-03-20T05:10:49.139+0000 7f6009204700 -1 log_channel(cluster) log [ERR] : 3.4s4 past_intervals [362,377) start interval does not contain the required bound [255,377) start
./ceph-osd.3.log:1335461:2021-03-20T05:10:49.139+0000 7f6009204700 -1 osd.3 pg_epoch: 377 pg[3.4s4( empty local-lis/les=0/0 n=0 ec=27/27 lis/c=254/254 les/c/f=255/255/0 sis=377) [1,NONE,NONE,NONE,3,2]p3(4) r=4 lpr=377 pi=[362,377)/0 crt=0'0 mlcod 0'0 peering mbc={}] 3.4s4 past_intervals [362,377) start interval does not contain the required bound [255,377) start
./ceph-osd.4.log:1229067:2021-03-20T05:10:50.142+0000 7fe785e25700 -1 log_channel(cluster) log [ERR] : 3.7s4 past_intervals [362,378) start interval does not contain the required bound [241,378) start
./ceph-osd.4.log:1229072:2021-03-20T05:10:50.142+0000 7fe785e25700 -1 osd.4 pg_epoch: 378 pg[3.7s4( empty local-lis/les=0/0 n=0 ec=27/27 lis/c=109/109 les/c/f=110/110/0 sis=378) [NONE,NONE,NONE,NONE,4,NONE]p4(4) r=4 lpr=378 pi=[362,378)/0 crt=0'0 mlcod 0'0 peering mbc={}] 3.7s4 past_intervals [362,378) start interval does not contain the required bound [241,378) start
./ceph-osd.6.log:1418181:2021-03-20T05:10:51.147+0000 7f4bcc959700 -1 log_channel(cluster) log [ERR] : 3.17s2 past_intervals [362,379) start interval does not contain the required bound [271,379) start
./ceph-osd.6.log:1418191:2021-03-20T05:10:51.147+0000 7f4bcc959700 -1 osd.6 pg_epoch: 379 pg[3.17s2( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=259/259 les/c/f=260/260/0 sis=379) [NONE,NONE,6,NONE,NONE,NONE]p6(2) r=2 lpr=379 pi=[362,379)/0 crt=0'0 mlcod 0'0 peering mbc={}] 3.17s2 past_intervals [362,379) start interval does not contain the required bound [271,379) start
./ceph-osd.7.log:1441040:2021-03-20T05:10:48.132+0000 7ff676b50700 -1 log_channel(cluster) log [ERR] : 3.0s1 past_intervals [362,376) start interval does not contain the required bound [181,376) start
./ceph-osd.7.log:1441043:2021-03-20T05:10:48.132+0000 7ff676b50700 -1 osd.7 pg_epoch: 376 pg[3.0s1( empty local-lis/les=0/0 n=0 ec=27/27 lis/c=109/109 les/c/f=110/110/0 sis=376) [NONE,7,6,1,2,8]p7(1) r=1 lpr=376 pi=[362,376)/0 crt=0'0 mlcod 0'0 peering mbc={}] 3.0s1 past_intervals [362,376) start interval does not contain the required bound [181,376) start
/ceph-osd.11.log:1704220:2021-03-20T05:10:47.151+0000 7f29a5cd4700 -1 log_channel(cluster) log [ERR] : 3.11s0 past_intervals [362,375) start interval does not contain the required bound [151,375) start
/ceph-osd.11.log:1704225:2021-03-20T05:10:47.151+0000 7f29a5cd4700 -1 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 peering mbc={}] 3.11s0 past_intervals [362,375) start interval does not contain the required bound [151,375) start
./ceph-osd.8.log:1179234:2021-03-20T05:10:48.131+0000 7f61a5942700 -1 log_channel(cluster) log [ERR] : 3.es1 past_intervals [362,376) start interval does not contain the required bound [181,376) start
./ceph-osd.8.log:1179237:2021-03-20T05:10:48.131+0000 7f61a5942700 -1 osd.8 pg_epoch: 376 pg[3.es1( empty local-lis/les=0/0 n=0 ec=27/27 lis/c=80/80 les/c/f=81/82/0 sis=376) [NONE,8,6,4,0,7]p8(1) r=1 lpr=376 pi=[362,376)/0 crt=0'0 mlcod 0'0 peering mbc={}] 3.es1 past_intervals [362,376) start interval does not contain the required bound [181,376) start

Map Gaps

Pgs: 1.1 (Start Interval mismatch) 3.15s5 3.13s1 3.cs4 3.4s4 (Start Interval mismatch) 3.0s1 (Start Interval mismatch) 3.8s5 3.es5 (Start Interval mismatch) 3.6s5 1.6 3.7s5 (Start Interval mismatch) 3.11s5 (Start Interval mismatch) 3.10s1 3.fs4 (Start Interval mismatch) 3.18s1 1.9 3.ds2 3.2s1 3.1s5 3.17s2 (Start Interval mismatch) 1.a 3.9s3

./ceph-osd.9.log:1030096:2021-03-20T05:10:46.188+0000 7f2a0fea3700 10 osd.9 pg_epoch: 362 pg[1.1( empty local-lis/les=226/227 n=0 ec=147/19 lis/c=226/226 les/c/f=227/227/0 sis=226) [10,0] r=-1 lpr=362 pi=[226,362)/1 crt=0'0 mlcod 0'0 unknown mbc={}]  map gap, clearing past_intervals and faking
./ceph-osd.9.log:1030608:2021-03-20T05:10:46.189+0000 7f2a11ea7700 10 osd.9 pg_epoch: 362 pg[3.15s5( v 106'54 (0'0,106'54] local-lis/les=111/112 n=2 ec=109/27 lis/c=111/111 les/c/f=112/112/0 sis=111) [8,11,7,4,6,10]p8(0) r=-1 lpr=362 pi=[111,362)/1 crt=106'54 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[e1~1,f8~1,fc~2,10a~1]]  map gap, clearing past_intervals and faking
./ceph-osd.9.log:1031079:2021-03-20T05:10:46.190+0000 7f2a12ea9700 10 osd.9 pg_epoch: 362 pg[3.13s1( v 101'38 (0'0,101'38] local-lis/les=111/112 n=1 ec=109/27 lis/c=111/111 les/c/f=112/112/0 sis=111) [7,3,4,2,6,0]p7(0) r=-1 lpr=362 pi=[111,362)/1 crt=101'38 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[e1~1,f8~1,fc~2,10a~1]]  map gap, clearing past_intervals and faking
./ceph-osd.9.log:1031582:2021-03-20T05:10:46.191+0000 7f2a126a8700 10 osd.9 pg_epoch: 362 pg[3.cs4( v 304'377 (0'0,304'377] local-lis/les=257/258 n=14 ec=27/27 lis/c=257/257 les/c/f=258/258/0 sis=257) [0,3,6,10,11,2]p0(0) r=-1 lpr=362 pi=[257,362)/1 crt=304'377 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[e1~1,f8~1,fc~2,10a~1]]  map gap, clearing past_intervals and faking
./ceph-osd.9.log:1032107:2021-03-20T05:10:46.193+0000 7f2a0e6a0700 10 osd.9 pg_epoch: 362 pg[3.4s4( v 304'208 (0'0,304'208] local-lis/les=254/255 n=5 ec=27/27 lis/c=254/254 les/c/f=255/255/0 sis=254) [1,11,10,8,0,2]p11(1) r=-1 lpr=362 pi=[254,362)/1 crt=304'208 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[e1~1,f8~1,fc~2,10a~1]]  map gap, clearing past_intervals and faking
./ceph-osd.9.log:1032574:2021-03-20T05:10:46.195+0000 7f2a146ac700 10 osd.9 pg_epoch: 362 pg[3.0s1( v 304'240 (0'0,304'240] local-lis/les=109/110 n=6 ec=27/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [11,10,6,1,2,8]p11(0) r=-1 lpr=362 pi=[109,362)/1 crt=304'240 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[e1~1,f8~1,fc~2,10a~1]]  map gap, clearing past_intervals and faking
./ceph-osd.9.log:1033060:2021-03-20T05:10:46.196+0000 7f2a146ac700 10 osd.9 pg_epoch: 362 pg[3.8s5( v 298'191 (0'0,298'191] local-lis/les=250/251 n=9 ec=27/27 lis/c=250/250 les/c/f=251/251/0 sis=250) [11,1,8,4,10,3]p11(0) r=-1 lpr=362 pi=[250,362)/1 crt=298'191 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[e1~1,f8~1,fc~2,10a~1]]  map gap, clearing past_intervals and faking
./ceph-osd.9.log:1033601:2021-03-20T05:10:46.197+0000 7f2a0d69e700 10 osd.9 pg_epoch: 362 pg[3.es5( v 304'275 (0'0,304'275] local-lis/les=80/81 n=13 ec=27/27 lis/c=80/80 les/c/f=81/82/0 sis=80) [11,10,6,4,0,7]p11(0) r=-1 lpr=362 pi=[80,362)/1 crt=304'275 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[e1~1,f8~1,fc~2,10a~1]]  map gap, clearing past_intervals and faking
./ceph-osd.9.log:1034117:2021-03-20T05:10:46.198+0000 7f2a0d69e700 10 osd.9 pg_epoch: 362 pg[3.6s5( v 304'144 (0'0,304'144] local-lis/les=255/256 n=3 ec=27/27 lis/c=255/255 les/c/f=256/256/0 sis=255) [0,1,4,7,8,2]p0(0) r=-1 lpr=362 pi=[255,362)/1 crt=304'144 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[e1~1,f8~1,fc~2,10a~1]]  map gap, clearing past_intervals and faking
./ceph-osd.9.log:1034834:2021-03-20T05:10:46.200+0000 7f2a116a6700 10 osd.9 pg_epoch: 362 pg[1.6( empty local-lis/les=226/227 n=0 ec=147/19 lis/c=226/226 les/c/f=227/227/0 sis=226) [10,8] r=-1 lpr=362 pi=[226,362)/1 crt=0'0 mlcod 0'0 unknown mbc={}]  map gap, clearing past_intervals and faking
./ceph-osd.9.log:1035557:2021-03-20T05:10:46.202+0000 7f2a10ea5700 10 osd.9 pg_epoch: 362 pg[3.7s5( v 101'54 (0'0,101'54] local-lis/les=109/110 n=1 ec=27/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [3,11,7,0,10,2]p3(0) r=-1 lpr=362 pi=[109,362)/1 crt=101'54 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[e1~1,f8~1,fc~2,10a~1]]  map gap, clearing past_intervals and faking
./ceph-osd.9.log:1036035:2021-03-20T05:10:46.202+0000 7f2a0fea3700 10 osd.9 pg_epoch: 362 pg[3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=114) [10,4,6,0,3,7]p10(0) r=-1 lpr=362 pi=[114,362)/1 crt=304'349 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[e1~1,f8~1,fc~2,10a~1]]  map gap, clearing past_intervals and faking
./ceph-osd.9.log:1036215:2021-03-20T05:10:46.203+0000 7f2a106a4700 10 osd.9 pg_epoch: 362 pg[3.10s1( v 304'578 (0'0,304'578] local-lis/les=250/251 n=18 ec=109/27 lis/c=250/250 les/c/f=251/251/0 sis=250) [6,4,10,3,0,2]p6(0) r=-1 lpr=362 pi=[250,362)/1 crt=304'578 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[e1~1,f8~1,fc~2,10a~1]]  map gap, clearing past_intervals and faking
./ceph-osd.9.log:1036986:2021-03-20T05:10:46.204+0000 7f2a0ce9d700 10 osd.9 pg_epoch: 362 pg[3.fs4( v 304'784 (0'0,304'784] local-lis/les=27/28 n=35 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) [7,11,8,3,2,10]p7(0) r=-1 lpr=362 pi=[27,362)/1 crt=304'784 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[e1~1,f8~1,fc~2,10a~1]]  map gap, clearing past_intervals and faking
./ceph-osd.9.log:1037547:2021-03-20T05:10:46.205+0000 7f2a106a4700 10 osd.9 pg_epoch: 362 pg[3.18s1( v 107'46 (0'0,107'46] local-lis/les=111/112 n=1 ec=109/27 lis/c=111/111 les/c/f=112/112/0 sis=111) [3,10,1,2,8,7]p3(0) r=-1 lpr=362 pi=[111,362)/1 crt=107'46 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[e1~1,f8~1,fc~2,10a~1]]  map gap, clearing past_intervals and faking
./ceph-osd.9.log:1038040:2021-03-20T05:10:46.205+0000 7f2a13eab700 10 osd.9 pg_epoch: 362 pg[1.9( empty local-lis/les=149/150 n=0 ec=147/19 lis/c=149/149 les/c/f=150/150/0 sis=149) [7,2] r=-1 lpr=362 pi=[149,362)/1 crt=0'0 mlcod 0'0 unknown mbc={}]  map gap, clearing past_intervals and faking
./ceph-osd.9.log:1038841:2021-03-20T05:10:46.207+0000 7f2a0de9f700 10 osd.9 pg_epoch: 362 pg[3.ds2( v 304'475 (0'0,304'475] local-lis/les=254/255 n=16 ec=27/27 lis/c=254/254 les/c/f=255/255/0 sis=254) [7,0,2,6,11,3]p0(1) r=-1 lpr=362 pi=[254,362)/1 crt=304'475 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[e1~1,f8~1,fc~2,10a~1]]  map gap, clearing past_intervals and faking
./ceph-osd.9.log:1038872:2021-03-20T05:10:46.207+0000 7f2a136aa700 10 osd.9 pg_epoch: 362 pg[3.2s1( v 282'168 (0'0,282'168] local-lis/les=250/251 n=2 ec=27/27 lis/c=250/250 les/c/f=251/251/0 sis=250) [3,7,6,4,8,1]p3(0) r=-1 lpr=362 pi=[250,362)/1 crt=282'168 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[e1~1,f8~1,fc~2,10a~1]]  map gap, clearing past_intervals and faking
./ceph-osd.9.log:1039963:2021-03-20T05:10:46.208+0000 7f2a13eab700 10 osd.9 pg_epoch: 362 pg[3.1s5( v 304'361 (0'0,304'361] local-lis/les=259/260 n=8 ec=27/27 lis/c=259/259 les/c/f=260/260/0 sis=259) [0,4,10,11,1,6]p0(0) r=-1 lpr=362 pi=[259,362)/1 crt=304'361 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[e1~1,f8~1,fc~2,10a~1]]  map gap, clearing past_intervals and faking
./ceph-osd.9.log:1040500:2021-03-20T05:10:46.209+0000 7f2a10ea5700 10 osd.9 pg_epoch: 362 pg[3.17s2( v 304'260 (0'0,304'260] local-lis/les=259/260 n=6 ec=109/27 lis/c=259/259 les/c/f=260/260/0 sis=259) [0,1,10,3,2,8]p0(0) r=-1 lpr=362 pi=[259,362)/1 crt=304'260 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[e1~1,f8~1,fc~2,10a~1]]  map gap, clearing past_intervals and faking
./ceph-osd.9.log:1040784:2021-03-20T05:10:46.210+0000 7f2a0f6a2700 10 osd.9 pg_epoch: 362 pg[1.a( empty local-lis/les=149/150 n=0 ec=147/19 lis/c=149/149 les/c/f=150/150/0 sis=149) [6,4] r=-1 lpr=362 pi=[149,362)/1 crt=0'0 mlcod 0'0 unknown mbc={}]  map gap, clearing past_intervals and faking
./ceph-osd.9.log:1041408:2021-03-20T05:10:46.211+0000 7f2a0fea3700 10 osd.9 pg_epoch: 362 pg[3.9s3( v 299'356 (0'0,299'356] local-lis/les=250/251 n=11 ec=27/27 lis/c=250/250 les/c/f=251/251/0 sis=250) [8,11,7,3,6,1]p8(0) r=-1 lpr=362 pi=[250,362)/1 crt=299'356 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[e1~1,f8~1,fc~2,10a~1]]  map gap, clearing past_intervals and faking

messages skips

./ceph-osd.6.log:1290568:2021-03-20T05:10:27.321+0000 7f4bdf17e700 10 osd.6 353 handle_osd_map message skips epochs 354..354
./ceph-osd.7.log:475600:2021-03-20T05:07:07.093+0000 7ff681b66700 10 osd.7 154 handle_osd_map message skips epochs 155..155
./ceph-osd.7.log:1344529:2021-03-20T05:10:27.323+0000 7ff681b66700 10 osd.7 353 handle_osd_map message skips epochs 354..354
./ceph-osd.11.log:134042:2021-03-20T05:05:27.068+0000 7f29b54f3700 10 osd.11 54 handle_osd_map message skips epochs 55..55
./ceph-osd.11.log:543949:2021-03-20T05:07:07.093+0000 7f29b54f3700 10 osd.11 154 handle_osd_map message skips epochs 155..155
./ceph-osd.11.log:1584333:2021-03-20T05:10:27.321+0000 7f29b54f3700 10 osd.11 353 handle_osd_map message skips epochs 354..354
./ceph-osd.8.log:1093278:2021-03-20T05:10:27.320+0000 7f61b7966700 10 osd.8 353 handle_osd_map message skips epochs 354..354
./ceph-osd.9.log:1011588:2021-03-20T05:10:46.158+0000 7f2a1f6c2700 10 osd.9 304 handle_osd_map message skips epochs 305..361

osd.11_pg3.11

2021-03-20T05:05:19.124+0000 7f29b7b2e700 20 bluestore(/var/lib/ceph/osd/ceph-11) _kv_sync_thread committed 3 cleaned 0 in 0.000473011s (0.000006486s flush + 0.000466525s kv commit)
2021-03-20T05:05:36.254+0000 7f29b7b2e700 20 bluestore(/var/lib/ceph/osd/ceph-11) _kv_sync_thread committed 1 cleaned 0 in 0.001968219s (0.000004708s flush + 0.001963511s kv commit)
2021-03-20T05:05:42.700+0000 7f29c7dda700  1 -- [v2:172.21.15.109:6827/35285,v1:172.21.15.109:6829/35285] --> 172.21.15.109:0/35282 -- osd_ping(ping_reply e71 up_from 18 ping_stamp 2021-03-20T05:05:42.701301+0000/66.019752502s send_stamp 66.002883911s delta_ub -0.016868329s) v5 -- 0x556323397680 con 0x556316b15c00
2021-03-20T05:06:16.197+0000 7f29b7b2e700 20 bluestore(/var/lib/ceph/osd/ceph-11) _kv_sync_thread committed 1 cleaned 1 in 0.000703311s (0.000055082s flush + 0.000648229s kv commit)
2021-03-20T05:06:17.217+0000 7f29b7b2e700 20 bluestore(/var/lib/ceph/osd/ceph-11) _kv_sync_thread committed 1 cleaned 0 in 0.001311975s (0.000078764s flush + 0.001233211s kv commit)
2021-03-20T05:06:20.082+0000 7f29b732d700 20 osd.11 109 identify_splits_and_merges 3.1s3 e109 pg_num 16 -> 26 children 3.11s3
2021-03-20T05:06:20.082+0000 7f29b732d700 20 osd.11 109 identify_splits_and_merges 3.11s3 e109 pg_num 16 -> 26 is a child
2021-03-20T05:06:20.082+0000 7f29b732d700 10 osd.11:0._prime_splits 3.10s0,109,3.11s3,109,3.13s4,109,3.14s1,109,3.17s1,109,3.18s0,109,3.19s1,109
2021-03-20T05:06:20.082+0000 7f29b732d700 10 osd.11:1._prime_splits 3.11s3,109,3.13s4,109,3.14s1,109,3.17s1,109,3.19s1,109
2021-03-20T05:06:20.082+0000 7f29b732d700 10 osd.11:1._prime_splits priming slot 3.11s3 e109
2021-03-20T05:06:20.082+0000 7f29b732d700 20 osd.11:1.consume_map consume_map 3.11s3
2021-03-20T05:06:20.082+0000 7f29b732d700 20 osd.11:1.consume_map consume_map  3.11s3 waiting for split 109
2021-03-20T05:06:20.083+0000 7f29a5cd4700 10 osd.11 109 split_pgs splitting pg[3.1s3( v 107'142 (0'0,107'142] local-lis/les=83/84 n=6 ec=27/27 lis/c=83/83 les/c/f=84/84/0 sis=109 pruub=15.001996994s) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY pruub 118.387443542s@ mbc={} ps=[42~1,49~2]] into 3.11s3
2021-03-20T05:06:20.083+0000 7f29a5cd4700 10 osd.11 109 _make_pg 3.11s3
2021-03-20T05:06:20.083+0000 7f29a5cd4700 20 osd.11 pg_epoch: 109 pg[3.11(unlocked)]  scrubber pg(3.11s3)  creating PgScrubber for 3.11s3 / 11(3)
2021-03-20T05:06:20.083+0000 7f29a5cd4700 15 osd.11 pg_epoch: 109 pg[3.11(unlocked)]  scrubberFSM pg(3.11s3) ScrubMachine created 3.11s3
2021-03-20T05:06:20.083+0000 7f29a5cd4700 10 osd.11 pg_epoch: 109 pg[3.11(unlocked)]  scrubberFSM pg(3.11s3) -- state -->> NotActive
2021-03-20T05:06:20.083+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) _osr_attach 3.11s3_head fresh osr 0x556328c3ac60
2021-03-20T05:06:20.083+0000 7f29a5cd4700 20 osd.11 pg_epoch: 109 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=6 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109 pruub=15.001996994s) [0,4,10,11,5,9]p0(0) r=3 lpr=0 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] on_new_interval
2021-03-20T05:06:20.083+0000 7f29a5cd4700 20 osd.11 pg_epoch: 109 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=6 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109 pruub=15.001996994s) [0,4,10,11,5,9]p0(0) r=3 lpr=0 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] on_new_interval upacting_features 0x3f01cfbb7ffdffff from [0,4,10,11,5,9]+[0,4,10,11,5,9]
2021-03-20T05:06:20.083+0000 7f29a5cd4700 20 osd.11 pg_epoch: 109 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=6 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109 pruub=15.001996994s) [0,4,10,11,5,9]p0(0) r=3 lpr=0 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1)
2021-03-20T05:06:20.084+0000 7f29a5cd4700 10 osd.11 pg_epoch: 109 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=6 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109 pruub=15.001996994s) [0,4,10,11,5,9]p0(0) r=3 lpr=0 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] init_hb_stamps now [0x5563168d98f0]
2021-03-20T05:06:20.084+0000 7f29a5cd4700 10 osd.11 pg_epoch: 109 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=6 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=0 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] on_new_interval prior_readable_until_ub 0.000000000s (mnow 103.386146545s + 0.000000000s)
2021-03-20T05:06:20.084+0000 7f29a5cd4700 20 osd.11 pg_epoch: 109 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=6 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=0 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] on_new_interval scrub_queued was 0 flags: 
2021-03-20T05:06:20.084+0000 7f29a5cd4700 10 osd.11 pg_epoch: 109 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=6 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=0 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] cancel_recovery
2021-03-20T05:06:20.084+0000 7f29a5cd4700 10 osd.11 pg_epoch: 109 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=6 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=0 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] clear_recovery_state
2021-03-20T05:06:20.084+0000 7f29a5cd4700 15 osd.11 pg_epoch: 109 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=6 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=0 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] _clear_recovery_state flags: 
2021-03-20T05:06:20.084+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) set_collection_opts 3.11s3_head options 
2021-03-20T05:06:20.085+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) _create_collection 3.11s3_head bits 5
2021-03-20T05:06:20.085+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) _create_collection 3.11s3_head bits 5 = 0
2021-03-20T05:06:20.085+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) _split_collection 3.1s3_head to 3.11s3_head  bits 5
2021-03-20T05:06:20.086+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) _split_collection 3.1s3_head to 3.11s3_head  bits 5 = 0
2021-03-20T05:06:20.086+0000 7f29a5cd4700 20 bluestore(/var/lib/ceph/osd/ceph-11).collection(3.11s3_head 0x5563259141e0) get_onode oid 3#3:88000000::::head# key 0x83800000000000000388000000'!!='0xFFFFFFFFFFFFFFFEFFFFFFFFFFFFFFFF6F
2021-03-20T05:06:20.086+0000 7f29a5cd4700 20 bluestore(/var/lib/ceph/osd/ceph-11).collection(3.11s3_head 0x5563259141e0)  r -2 v.len 0
2021-03-20T05:06:20.086+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) _touch 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:20.086+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) _touch 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:20.086+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:20.086+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:20.086+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) _touch 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:20.086+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) _touch 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:20.086+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkey_range 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:20.086+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkey_range 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:20.086+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkey_range 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:20.086+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkey_range 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:20.086+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:20.086+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:20.087+0000 7f29a9cdc700 10 osd.11 109 _finish_splits pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=0 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]]
2021-03-20T05:06:20.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 109 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=0 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] handle_initialize
2021-03-20T05:06:20.087+0000 7f29a9cdc700  5 osd.11 pg_epoch: 109 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=0 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] exit Initial 0.004153 0 0.000000
2021-03-20T05:06:20.087+0000 7f29a9cdc700  5 osd.11 pg_epoch: 109 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=0 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] enter Reset
2021-03-20T05:06:20.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 109 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=0 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] set_last_peering_reset 109
2021-03-20T05:06:20.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 109 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] Clearing blocked outgoing recovery messages
2021-03-20T05:06:20.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 109 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] Not blocking outgoing recovery messages
2021-03-20T05:06:20.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 109 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] null
2021-03-20T05:06:20.087+0000 7f29a9cdc700 15 osd.11 109 enqueue_peering_evt 3.11s3 epoch_sent: 109 epoch_requested: 109 NullEvt
2021-03-20T05:06:20.087+0000 7f29a9cdc700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 109 epoch_requested: 109 NullEvt) prio 255 cost 10 e109)
2021-03-20T05:06:20.087+0000 7f29a9cdc700 10 osd.11:1.register_and_wake_split_child 3.11s3 0x5563156cb000
2021-03-20T05:06:20.087+0000 7f29a9cdc700 20 osd.11:1.register_and_wake_split_child 3.11s3 waiting_for_split 109
2021-03-20T05:06:20.087+0000 7f29a9cdc700 10 osd.11:1._attach_pg 3.11s3 0x5563156cb000
2021-03-20T05:06:20.087+0000 7f29a9cdc700 20 osd.11:1._wake_pg_slot _wake_pg_slot 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:20.087+0000 7f29a9cdc700 15 osd.11 109 enqueue_peering_evt 3.11s3 epoch_sent: 109 epoch_requested: 109 NullEvt
2021-03-20T05:06:20.087+0000 7f29a9cdc700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 109 epoch_requested: 109 NullEvt) prio 255 cost 10 e109)
2021-03-20T05:06:20.087+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:20.087+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 109 epoch_requested: 109 NullEvt) prio 255 cost 10 e109) queued
2021-03-20T05:06:20.087+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 109 epoch_requested: 109 NullEvt) prio 255 cost 10 e109)> waiting <> waiting_peering {}
2021-03-20T05:06:20.087+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 109 epoch_requested: 109 NullEvt) prio 255 cost 10 e109) pg 0x5563156cb000
2021-03-20T05:06:20.087+0000 7f29a5cd4700 10 osd.11 pg_epoch: 109 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 109 epoch_requested: 109 NullEvt
2021-03-20T05:06:20.087+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:20.087+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 109 epoch_requested: 109 NullEvt) prio 255 cost 10 e109) queued
2021-03-20T05:06:20.087+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 109 epoch_requested: 109 NullEvt) prio 255 cost 10 e109)> waiting <> waiting_peering {}
2021-03-20T05:06:20.087+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 109 epoch_requested: 109 NullEvt) prio 255 cost 10 e109) pg 0x5563156cb000
2021-03-20T05:06:20.087+0000 7f29a5cd4700 10 osd.11 pg_epoch: 109 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 109 epoch_requested: 109 NullEvt
2021-03-20T05:06:21.079+0000 7f29b732d700 20 osd.11 110 identify_splits_and_merges 3.11s3 e109 to e110 pg_nums {27=16,109=26}
2021-03-20T05:06:21.080+0000 7f29b732d700 20 osd.11:1.consume_map consume_map 3.11s3
2021-03-20T05:06:21.080+0000 7f29b732d700 15 osd.11 110 enqueue_peering_evt 3.11s3 epoch_sent: 110 epoch_requested: 110 NullEvt
2021-03-20T05:06:21.080+0000 7f29b732d700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 NullEvt) prio 255 cost 10 e110)
2021-03-20T05:06:21.080+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:21.080+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 NullEvt) prio 255 cost 10 e110) queued
2021-03-20T05:06:21.080+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 NullEvt) prio 255 cost 10 e110)> waiting <> waiting_peering {}
2021-03-20T05:06:21.080+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 NullEvt) prio 255 cost 10 e110) pg 0x5563156cb000
2021-03-20T05:06:21.080+0000 7f29a9cdc700 10 osd.11 pg_epoch: 109 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] handle_advance_map: 110
2021-03-20T05:06:21.080+0000 7f29a9cdc700 20 osd.11:1.update_pg_epoch 3.11s3 109 -> 110
2021-03-20T05:06:21.080+0000 7f29a9cdc700 10 osd.11 pg_epoch: 109 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] handle_advance_map [0,4,10,11,5,9]/[0,4,10,11,5,9] -- 0/0
2021-03-20T05:06:21.080+0000 7f29a9cdc700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] state<Reset>: Reset advmap
2021-03-20T05:06:21.080+0000 7f29a9cdc700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] check_recovery_sources no source osds () went down
2021-03-20T05:06:21.080+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) set_collection_opts 3.11s3_head options 
2021-03-20T05:06:21.080+0000 7f29a9cdc700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] plpg_on_pool_change
2021-03-20T05:06:21.080+0000 7f29a9cdc700 20 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] hit_set_clear
2021-03-20T05:06:21.080+0000 7f29a9cdc700 20 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] agent_stop
2021-03-20T05:06:21.080+0000 7f29a9cdc700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] handle_activate_map: 110
2021-03-20T05:06:21.080+0000 7f29a9cdc700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] activate_map
2021-03-20T05:06:21.080+0000 7f29a9cdc700  5 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] exit Reset 0.992762 4 0.000043
2021-03-20T05:06:21.080+0000 7f29a9cdc700  5 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] enter Started
2021-03-20T05:06:21.080+0000 7f29a9cdc700  5 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] enter Start
2021-03-20T05:06:21.080+0000 7f29a9cdc700  1 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] state<Start>: transitioning to Stray
2021-03-20T05:06:21.080+0000 7f29a9cdc700  5 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] exit Start 0.000011 0 0.000000
2021-03-20T05:06:21.080+0000 7f29a9cdc700  5 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] enter Started/Stray
2021-03-20T05:06:21.080+0000 7f29a9cdc700 20 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] activate_map: Not dirtying info: last_persisted is 109 while current is 110
2021-03-20T05:06:21.080+0000 7f29a9cdc700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 110 epoch_requested: 110 NullEvt
2021-03-20T05:06:21.080+0000 7f29a9cdc700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.33:6810/35149,v1:172.21.15.33:6811/35149] -- pg_notify2(3.11s0 3.11s0 (query:110 sent:110 3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) 3->0 ([83,108] all_participants=0(0),4(1),5(4),9(5),10(2),11(3) intervals=([83,108] acting 0(0),4(1),5(4),9(5),10(2),11(3)))) e110/110) v1 -- 0x5563264cd100 con 0x556316b6ec00
2021-03-20T05:06:21.080+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) queue_transactions ch 0x5563259141e0 3.11s3_head
2021-03-20T05:06:21.087+0000 7f29c85db700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.0 v2:172.21.15.33:6810/35149 204 ==== pg_query2(3.11s3 3.11s3 query(info 0'0 epoch_sent 110) e110/110) v1 ==== 158+0+0 (unknown 0 0 0) 0x556316c98780 con 0x556316b6ec00
2021-03-20T05:06:21.087+0000 7f29c85db700 15 osd.11 110 enqueue_peering_evt 3.11s3 epoch_sent: 110 epoch_requested: 110 MQuery 3.11s3 from 0(0) query_epoch 110 query: query(info 0'0 epoch_sent 110)
2021-03-20T05:06:21.087+0000 7f29c85db700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MQuery 3.11s3 from 0(0) query_epoch 110 query: query(info 0'0 epoch_sent 110)) prio 255 cost 10 e110)
2021-03-20T05:06:21.087+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:21.087+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MQuery 3.11s3 from 0(0) query_epoch 110 query: query(info 0'0 epoch_sent 110)) prio 255 cost 10 e110) queued
2021-03-20T05:06:21.087+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MQuery 3.11s3 from 0(0) query_epoch 110 query: query(info 0'0 epoch_sent 110)) prio 255 cost 10 e110)> waiting <> waiting_peering {}
2021-03-20T05:06:21.087+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MQuery 3.11s3 from 0(0) query_epoch 110 query: query(info 0'0 epoch_sent 110)) prio 255 cost 10 e110) pg 0x5563156cb000
2021-03-20T05:06:21.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 110 epoch_requested: 110 MQuery 3.11s3 from 0(0) query_epoch 110 query: query(info 0'0 epoch_sent 110)
2021-03-20T05:06:21.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) unreg_next_scrub existing-0.000000. was registered? 0
2021-03-20T05:06:21.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] sending info
2021-03-20T05:06:21.087+0000 7f29a9cdc700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.33:6810/35149,v1:172.21.15.33:6811/35149] -- pg_notify2(3.11s0 3.11s0 (query:110 sent:110 3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) 3->0 ([83,108] all_participants=0(0),4(1),5(4),9(5),10(2),11(3) intervals=([83,108] acting 0(0),4(1),5(4),9(5),10(2),11(3)))) e110/110) v1 -- 0x556329864700 con 0x556316b6ec00
2021-03-20T05:06:21.094+0000 7f29c85db700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.0 v2:172.21.15.33:6810/35149 207 ==== pg_query2(3.11s3 3.11s3 query(log 84'0 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109 epoch_sent 110) e110/110) v1 ==== 158+0+0 (unknown 0 0 0) 0x556316c98780 con 0x556316b6ec00
2021-03-20T05:06:21.094+0000 7f29c85db700 15 osd.11 110 enqueue_peering_evt 3.11s3 epoch_sent: 110 epoch_requested: 110 MQuery 3.11s3 from 0(0) query_epoch 110 query: query(log 84'0 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109 epoch_sent 110)
2021-03-20T05:06:21.094+0000 7f29c85db700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MQuery 3.11s3 from 0(0) query_epoch 110 query: query(log 84'0 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109 epoch_sent 110)) prio 255 cost 10 e110)
2021-03-20T05:06:21.094+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:21.094+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MQuery 3.11s3 from 0(0) query_epoch 110 query: query(log 84'0 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109 epoch_sent 110)) prio 255 cost 10 e110) queued
2021-03-20T05:06:21.094+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MQuery 3.11s3 from 0(0) query_epoch 110 query: query(log 84'0 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109 epoch_sent 110)) prio 255 cost 10 e110)> waiting <> waiting_peering {}
2021-03-20T05:06:21.094+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MQuery 3.11s3 from 0(0) query_epoch 110 query: query(log 84'0 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109 epoch_sent 110)) prio 255 cost 10 e110) pg 0x5563156cb000
2021-03-20T05:06:21.094+0000 7f29a5cd4700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 110 epoch_requested: 110 MQuery 3.11s3 from 0(0) query_epoch 110 query: query(log 84'0 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109 epoch_sent 110)
2021-03-20T05:06:21.094+0000 7f29a5cd4700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) unreg_next_scrub existing-0.000000. was registered? 0
2021-03-20T05:06:21.094+0000 7f29a5cd4700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] log request from 0(0)
2021-03-20T05:06:21.094+0000 7f29a5cd4700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]]  sending info+missing+log since 84'0
2021-03-20T05:06:21.094+0000 7f29a5cd4700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]]  sending log((82'118,107'142], crt=107'142) missing(0 may_include_deletes = 1)
2021-03-20T05:06:21.094+0000 7f29a5cd4700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.33:6810/35149,v1:172.21.15.33:6811/35149] -- PGlog(3.11s0 log log((82'118,107'142], crt=107'142) pi ([0,0] all_participants= intervals=) e110/110) v6 -- 0x556325dd9800 con 0x556316b6ec00
2021-03-20T05:06:21.095+0000 7f29c85db700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.0 v2:172.21.15.33:6810/35149 208 ==== pg_info2(3.11s3 3.11s3 3.11s0( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) e110/110) v1 ==== 1096+0+0 (unknown 0 0 0) 0x556326814300 con 0x556316b6ec00
2021-03-20T05:06:21.095+0000 7f29c85db700 15 osd.11 110 enqueue_peering_evt 3.11s3 epoch_sent: 110 epoch_requested: 110 MInfoRec from 0(0) info: 3.11s0( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) pg_lease(ru 0.000000000s ub 120.436172485s int 16.000000000s)
2021-03-20T05:06:21.095+0000 7f29c85db700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MInfoRec from 0(0) info: 3.11s0( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) pg_lease(ru 0.000000000s ub 120.436172485s int 16.000000000s)) prio 255 cost 10 e110)
2021-03-20T05:06:21.095+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:21.095+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MInfoRec from 0(0) info: 3.11s0( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) pg_lease(ru 0.000000000s ub 120.436172485s int 16.000000000s)) prio 255 cost 10 e110) queued
2021-03-20T05:06:21.095+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MInfoRec from 0(0) info: 3.11s0( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) pg_lease(ru 0.000000000s ub 120.436172485s int 16.000000000s)) prio 255 cost 10 e110)> waiting <> waiting_peering {}
2021-03-20T05:06:21.095+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MInfoRec from 0(0) info: 3.11s0( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) pg_lease(ru 0.000000000s ub 120.436172485s int 16.000000000s)) prio 255 cost 10 e110) pg 0x5563156cb000
2021-03-20T05:06:21.095+0000 7f29a5cd4700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 110 epoch_requested: 110 MInfoRec from 0(0) info: 3.11s0( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) pg_lease(ru 0.000000000s ub 120.436172485s int 16.000000000s)
2021-03-20T05:06:21.095+0000 7f29a5cd4700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] state<Started/Stray>: got info from osd.0(0) 3.11s0( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109)
2021-03-20T05:06:21.095+0000 7f29a5cd4700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] proc_lease pg_lease(ru 0.000000000s ub 120.436172485s int 16.000000000s)
2021-03-20T05:06:21.096+0000 7f29a5cd4700 20 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]]  peer_clock_delta_lb 0.038306735s -> ruub 120.397865295s
2021-03-20T05:06:21.096+0000 7f29a5cd4700 20 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] proc_lease readable_until_ub now 120.397865295s
2021-03-20T05:06:21.096+0000 7f29a5cd4700  5 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] exit Started/Stray 0.015447 4 0.000228
2021-03-20T05:06:21.096+0000 7f29a5cd4700  5 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] enter Started/ReplicaActive
2021-03-20T05:06:21.096+0000 7f29a5cd4700  5 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] enter Started/ReplicaActive/RepNotRecovering
2021-03-20T05:06:21.096+0000 7f29a5cd4700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=83/84 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] state<Started/ReplicaActive>: In ReplicaActive, about to call activate
2021-03-20T05:06:21.096+0000 7f29a5cd4700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 lc 0'0 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] activate - no missing, moving last_complete 0'0 -> 107'142
2021-03-20T05:06:21.096+0000 7f29a5cd4700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] state<Started/ReplicaActive>: Activate Finished
2021-03-20T05:06:21.096+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) queue_transactions ch 0x5563259141e0 3.11s3_head
2021-03-20T05:06:21.096+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:21.096+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:21.096+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkeys 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:21.096+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkeys 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:21.101+0000 7f29a9cdc700 15 osd.11 110 enqueue_peering_evt 3.11s3 epoch_sent: 110 epoch_requested: 110 ActivateCommitted from 110 processed at 110
2021-03-20T05:06:21.101+0000 7f29a9cdc700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 ActivateCommitted from 110 processed at 110) prio 255 cost 10 e110)
2021-03-20T05:06:21.101+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:21.101+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 ActivateCommitted from 110 processed at 110) prio 255 cost 10 e110) queued
2021-03-20T05:06:21.101+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 ActivateCommitted from 110 processed at 110) prio 255 cost 10 e110)> waiting <> waiting_peering {}
2021-03-20T05:06:21.101+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 ActivateCommitted from 110 processed at 110) prio 255 cost 10 e110) pg 0x5563156cb000
2021-03-20T05:06:21.101+0000 7f29a9cdc700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 110 epoch_requested: 110 ActivateCommitted from 110 processed at 110
2021-03-20T05:06:21.101+0000 7f29a9cdc700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] state<Started/ReplicaActive>: react 110 telling primary
2021-03-20T05:06:21.101+0000 7f29a9cdc700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.33:6810/35149,v1:172.21.15.33:6811/35149] -- pg_info2(3.11s0 3.11s0 3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/83 les/c/f=110/84/0 sis=109) e110/110) v1 -- 0x556317017c00 con 0x556316b6ec00
2021-03-20T05:06:21.104+0000 7f29c85db700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.0 v2:172.21.15.33:6810/35149 212 ==== pg_lease(3.11s3 pg_lease(ru 120.436172485s ub 120.445404053s int 16.000000000s) e110/110) v1 ==== 58+0+0 (unknown 0 0 0) 0x5563169b0600 con 0x556316b6ec00
2021-03-20T05:06:21.104+0000 7f29c85db700 15 osd.11 110 enqueue_peering_evt 3.11s3 epoch_sent: 110 epoch_requested: 110 MLease epoch 110 from osd.0 pg_lease(ru 120.436172485s ub 120.445404053s int 16.000000000s)
2021-03-20T05:06:21.104+0000 7f29c85db700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MLease epoch 110 from osd.0 pg_lease(ru 120.436172485s ub 120.445404053s int 16.000000000s)) prio 255 cost 10 e110)
2021-03-20T05:06:21.104+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:21.104+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MLease epoch 110 from osd.0 pg_lease(ru 120.436172485s ub 120.445404053s int 16.000000000s)) prio 255 cost 10 e110) queued
2021-03-20T05:06:21.104+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MLease epoch 110 from osd.0 pg_lease(ru 120.436172485s ub 120.445404053s int 16.000000000s)) prio 255 cost 10 e110)> waiting <> waiting_peering {}
2021-03-20T05:06:21.104+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MLease epoch 110 from osd.0 pg_lease(ru 120.436172485s ub 120.445404053s int 16.000000000s)) prio 255 cost 10 e110) pg 0x5563156cb000
2021-03-20T05:06:21.104+0000 7f29a5cd4700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 110 epoch_requested: 110 MLease epoch 110 from osd.0 pg_lease(ru 120.436172485s ub 120.445404053s int 16.000000000s)
2021-03-20T05:06:21.104+0000 7f29a5cd4700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] proc_lease pg_lease(ru 120.436172485s ub 120.445404053s int 16.000000000s)
2021-03-20T05:06:21.104+0000 7f29a5cd4700 20 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]]  peer_clock_delta_ub 0.039170153s -> ru 120.397003174s
2021-03-20T05:06:21.104+0000 7f29a5cd4700 20 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] proc_lease readable_until now 120.397003174s
2021-03-20T05:06:21.104+0000 7f29a5cd4700 20 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]]  peer_clock_delta_lb 0.038306735s -> ruub 120.407096863s
2021-03-20T05:06:21.104+0000 7f29a5cd4700 20 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] proc_lease readable_until_ub now 120.407096863s
2021-03-20T05:06:21.104+0000 7f29a5cd4700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.33:6810/35149,v1:172.21.15.33:6811/35149] -- pg_lease_ack(3.11s0 pg_lease_ack(ruub 120.445404053s) e110/110) v1 -- 0x556325a974a0 con 0x556316b6ec00
2021-03-20T05:06:21.105+0000 7f29c85db700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.0 v2:172.21.15.33:6810/35149 213 ==== pg_info2(3.11s3 3.11s3 3.11s0( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/83 les/c/f=110/84/0 sis=109) e110/110) v1 ==== 1216+0+0 (unknown 0 0 0) 0x5563264ff100 con 0x556316b6ec00
2021-03-20T05:06:21.105+0000 7f29c85db700 15 osd.11 110 enqueue_peering_evt 3.11s3 epoch_sent: 110 epoch_requested: 110 MInfoRec from 0(0) info: 3.11s0( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/83 les/c/f=110/84/0 sis=109) pg_lease(ru 120.436172485s ub 120.445404053s int 16.000000000s)
2021-03-20T05:06:21.105+0000 7f29c85db700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MInfoRec from 0(0) info: 3.11s0( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/83 les/c/f=110/84/0 sis=109) pg_lease(ru 120.436172485s ub 120.445404053s int 16.000000000s)) prio 255 cost 10 e110)
2021-03-20T05:06:21.105+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:21.105+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MInfoRec from 0(0) info: 3.11s0( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/83 les/c/f=110/84/0 sis=109) pg_lease(ru 120.436172485s ub 120.445404053s int 16.000000000s)) prio 255 cost 10 e110) queued
2021-03-20T05:06:21.105+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MInfoRec from 0(0) info: 3.11s0( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/83 les/c/f=110/84/0 sis=109) pg_lease(ru 120.436172485s ub 120.445404053s int 16.000000000s)) prio 255 cost 10 e110)> waiting <> waiting_peering {}
2021-03-20T05:06:21.105+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MInfoRec from 0(0) info: 3.11s0( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/83 les/c/f=110/84/0 sis=109) pg_lease(ru 120.436172485s ub 120.445404053s int 16.000000000s)) prio 255 cost 10 e110) pg 0x5563156cb000
2021-03-20T05:06:21.105+0000 7f29a9cdc700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=83/83 les/c/f=84/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 110 epoch_requested: 110 MInfoRec from 0(0) info: 3.11s0( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/83 les/c/f=110/84/0 sis=109) pg_lease(ru 120.436172485s ub 120.445404053s int 16.000000000s)
2021-03-20T05:06:21.105+0000 7f29a9cdc700 20 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/83 les/c/f=110/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] update_history advanced history from ec=109/27 lis/c=109/83 les/c/f=110/84/0 sis=109
2021-03-20T05:06:21.105+0000 7f29a9cdc700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/83 les/c/f=110/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) unreg_next_scrub existing-0.000000. was registered? 0
2021-03-20T05:06:21.105+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) queue_transactions ch 0x5563259141e0 3.11s3_head
2021-03-20T05:06:21.105+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:21.105+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:21.105+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkeys 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:21.105+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkeys 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:21.107+0000 7f29c85db700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.0 v2:172.21.15.33:6810/35149 214 ==== pg_info2(3.11s3 3.11s3 3.11s0( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) e110/110) v1 ==== 1216+0+0 (unknown 0 0 0) 0x5563264ff100 con 0x556316b6ec00
2021-03-20T05:06:21.107+0000 7f29c85db700 15 osd.11 110 enqueue_peering_evt 3.11s3 epoch_sent: 110 epoch_requested: 110 MInfoRec from 0(0) info: 3.11s0( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) pg_lease(ru 120.436172485s ub 120.445404053s int 16.000000000s)
2021-03-20T05:06:21.107+0000 7f29c85db700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MInfoRec from 0(0) info: 3.11s0( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) pg_lease(ru 120.436172485s ub 120.445404053s int 16.000000000s)) prio 255 cost 10 e110)
2021-03-20T05:06:21.107+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:21.107+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MInfoRec from 0(0) info: 3.11s0( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) pg_lease(ru 120.436172485s ub 120.445404053s int 16.000000000s)) prio 255 cost 10 e110) queued
2021-03-20T05:06:21.107+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MInfoRec from 0(0) info: 3.11s0( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) pg_lease(ru 120.436172485s ub 120.445404053s int 16.000000000s)) prio 255 cost 10 e110)> waiting <> waiting_peering {}
2021-03-20T05:06:21.107+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 110 epoch_requested: 110 MInfoRec from 0(0) info: 3.11s0( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) pg_lease(ru 120.436172485s ub 120.445404053s int 16.000000000s)) prio 255 cost 10 e110) pg 0x5563156cb000
2021-03-20T05:06:21.107+0000 7f29a5cd4700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/83 les/c/f=110/84/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 110 epoch_requested: 110 MInfoRec from 0(0) info: 3.11s0( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) pg_lease(ru 120.436172485s ub 120.445404053s int 16.000000000s)
2021-03-20T05:06:21.107+0000 7f29a5cd4700 20 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] update_history advanced history from ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109
2021-03-20T05:06:21.107+0000 7f29a5cd4700 20 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 pi=[83,109)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] update_history clearing past_intervals
2021-03-20T05:06:21.107+0000 7f29a5cd4700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) unreg_next_scrub existing-0.000000. was registered? 0
2021-03-20T05:06:21.107+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) queue_transactions ch 0x5563259141e0 3.11s3_head
2021-03-20T05:06:21.107+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:21.107+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:21.107+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkeys 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:21.107+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkeys 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:22.083+0000 7f29c75d9700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.10 v2:172.21.15.109:6818/35284 183 ==== pg_query2(3.11s3 3.11s3 query(info 0'0 epoch_sent 111) e111/111) v1 ==== 158+0+0 (unknown 0 0 0) 0x556326681680 con 0x5563169b7c00
2021-03-20T05:06:22.083+0000 7f29c75d9700 15 osd.11 110 enqueue_peering_evt 3.11s3 epoch_sent: 111 epoch_requested: 111 MQuery 3.11s3 from 10(0) query_epoch 111 query: query(info 0'0 epoch_sent 111)
2021-03-20T05:06:22.083+0000 7f29c75d9700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 111 epoch_requested: 111 MQuery 3.11s3 from 10(0) query_epoch 111 query: query(info 0'0 epoch_sent 111)) prio 255 cost 10 e111)
2021-03-20T05:06:22.083+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:22.083+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 111 epoch_requested: 111 MQuery 3.11s3 from 10(0) query_epoch 111 query: query(info 0'0 epoch_sent 111)) prio 255 cost 10 e111) queued
2021-03-20T05:06:22.083+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 111 epoch_requested: 111 MQuery 3.11s3 from 10(0) query_epoch 111 query: query(info 0'0 epoch_sent 111)) prio 255 cost 10 e111)> waiting <> waiting_peering {}
2021-03-20T05:06:22.083+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 111 epoch_requested: 111 MQuery 3.11s3 from 10(0) query_epoch 111 query: query(info 0'0 epoch_sent 111)) prio 255 cost 10 e111) pg 0x5563156cb000
2021-03-20T05:06:22.083+0000 7f29a9cdc700 20 osd.11 op_wq _add_slot_waiter 3.11s3 peering, item epoch is 111, will wait on OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 111 epoch_requested: 111 MQuery 3.11s3 from 10(0) query_epoch 111 query: query(info 0'0 epoch_sent 111)) prio 255 cost 10 e111)
2021-03-20T05:06:22.085+0000 7f29b732d700 20 osd.11 111 identify_splits_and_merges 3.11s3 e110 to e111 pg_nums {27=16,109=26}
2021-03-20T05:06:22.085+0000 7f29b732d700 20 osd.11:1.consume_map consume_map 3.11s3
2021-03-20T05:06:22.085+0000 7f29b732d700 20 osd.11:1.consume_map consume_map  3.11s3 pending_peering first epoch 111 <= 111, requeueing
2021-03-20T05:06:22.085+0000 7f29b732d700 20 osd.11:1._wake_pg_slot _wake_pg_slot 3.11s3 to_process <> waiting <> waiting_peering {111=<OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 111 epoch_requested: 111 MQuery 3.11s3 from 10(0) query_epoch 111 query: query(info 0'0 epoch_sent 111)) prio 255 cost 10 e111)>}
2021-03-20T05:06:22.085+0000 7f29b732d700 15 osd.11 111 enqueue_peering_evt 3.11s3 epoch_sent: 111 epoch_requested: 111 NullEvt
2021-03-20T05:06:22.085+0000 7f29b732d700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 111 epoch_requested: 111 NullEvt) prio 255 cost 10 e111)
2021-03-20T05:06:22.086+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:22.086+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 111 epoch_requested: 111 MQuery 3.11s3 from 10(0) query_epoch 111 query: query(info 0'0 epoch_sent 111)) prio 255 cost 10 e111) queued
2021-03-20T05:06:22.086+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 111 epoch_requested: 111 MQuery 3.11s3 from 10(0) query_epoch 111 query: query(info 0'0 epoch_sent 111)) prio 255 cost 10 e111)> waiting <> waiting_peering {}
2021-03-20T05:06:22.086+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 111 epoch_requested: 111 MQuery 3.11s3 from 10(0) query_epoch 111 query: query(info 0'0 epoch_sent 111)) prio 255 cost 10 e111) pg 0x5563156cb000
2021-03-20T05:06:22.086+0000 7f29a9cdc700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] handle_advance_map: 111
2021-03-20T05:06:22.086+0000 7f29a9cdc700 20 osd.11:1.update_pg_epoch 3.11s3 110 -> 111
2021-03-20T05:06:22.086+0000 7f29a9cdc700 10 osd.11 pg_epoch: 110 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] handle_advance_map [10,4,6,0,3,9]/[10,4,6,0,3,9] -- 10/10
2021-03-20T05:06:22.086+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] state<Started>: Started advmap
2021-03-20T05:06:22.086+0000 7f29a9cdc700 20 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] new interval newup [10,4,6,0,3,9] newacting [10,4,6,0,3,9]
2021-03-20T05:06:22.086+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] state<Started>: should_restart_peering, transitioning to Reset
2021-03-20T05:06:22.087+0000 7f29a9cdc700  5 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] exit Started/ReplicaActive/RepNotRecovering 0.990969 5 0.000175
2021-03-20T05:06:22.087+0000 7f29a9cdc700  5 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] exit Started/ReplicaActive 0.990985 0 0.000000
2021-03-20T05:06:22.087+0000 7f29a9cdc700  5 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] exit Started 1.006476 0 0.000000
2021-03-20T05:06:22.087+0000 7f29a9cdc700  5 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] enter Reset
2021-03-20T05:06:22.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=109 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] set_last_peering_reset 111
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=111 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] Clearing blocked outgoing recovery messages
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=111 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] Not blocking outgoing recovery messages
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=111 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] state<Reset>: Reset advmap
2021-03-20T05:06:22.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=111 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] new interval newup [10,4,6,0,3,9] newacting [10,4,6,0,3,9]
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=111 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] state<Reset>: should restart peering, calling start_peering_interval again
2021-03-20T05:06:22.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [0,4,10,11,5,9]p0(0) r=3 lpr=111 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] set_last_peering_reset 111
2021-03-20T05:06:22.087+0000 7f29a9cdc700 15 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [10,4,6,0,3,9]p10(0) r=3 lpr=111 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] clear_stats
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] start_peering_interval: check_new_interval output: check_new_interval interval(109-110 up [0,4,10,11,5,9](0) acting [0,4,10,11,5,9](0)) up_thru 109 up_from 18 last_epoch_clean 110 interval(109-110 up [0,4,10,11,5,9](0) acting [0,4,10,11,5,9](0) maybe_went_rw) : primary up 18-109 includes interval
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=109) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]]  noting past ([109,110] all_participants=0(0),4(1),5(4),9(5),10(2),11(3) intervals=([109,110] acting 0(0),4(1),5(4),9(5),10(2),11(3)))
2021-03-20T05:06:22.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] on_new_interval
2021-03-20T05:06:22.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] on_new_interval upacting_features 0x3f01cfbb7ffdffff from [10,4,6,0,3,9]+[10,4,6,0,3,9]
2021-03-20T05:06:22.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1)
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] init_hb_stamps now []
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] on_new_interval prior_readable_until_ub 120.407096863s (mnow 105.389289856s + 15.017798424s)
2021-03-20T05:06:22.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] on_new_interval scrub_queued was 0 flags: 
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] cancel_recovery
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] clear_recovery_state
2021-03-20T05:06:22.087+0000 7f29a9cdc700 15 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] _clear_recovery_state flags: 
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active pruub 120.407096863s@ mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) unreg_next_scrub existing-0.000000. was registered? 0
2021-03-20T05:06:22.087+0000 7f29a9cdc700  1 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] start_peering_interval up [0,4,10,11,5,9] -> [10,4,6,0,3,9], acting [0,4,10,11,5,9] -> [10,4,6,0,3,9], acting_primary 0(0) -> 10, up_primary 0(0) -> 10, role 3 -> -1, features acting 4540138303579357183 upacting 4540138303579357183
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] clear_primary_state
2021-03-20T05:06:22.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] clear_primary_state
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] release_backoffs [3:88000000::::0,3:90000000::::head)
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown pruub 120.407096863s@ mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) discard_replica_reservations
2021-03-20T05:06:22.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] agent_stop
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] on_change
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] cancel_copy_ops
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] cancel_flush_ops
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] cancel_proxy_ops
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] cancel_manifest_ops
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown pruub 120.407096863s@ mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) scrub_clear_state
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown pruub 120.407096863s@ mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) clear_pgscrub_state
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown pruub 120.407096863s@ mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) clear_scrub_reservations
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown pruub 120.407096863s@ mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) reset_internal_state
2021-03-20T05:06:22.087+0000 7f29a9cdc700 15 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown pruub 120.407096863s@ mbc={} ps=[42~1,49~2]]  PrimaryLog scrubber pg(3.11s3) _scrub_clear_state
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] on_change_cleanup
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] on_change
2021-03-20T05:06:22.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] exit NotTrimming
2021-03-20T05:06:22.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] enter NotTrimming
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] plpg_on_role_change
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] check_recovery_sources no source osds () went down
2021-03-20T05:06:22.087+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) set_collection_opts 3.11s3_head options 
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] plpg_on_pool_change
2021-03-20T05:06:22.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] hit_set_clear
2021-03-20T05:06:22.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] agent_stop
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] handle_activate_map: 111
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017798424s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] activate_map
2021-03-20T05:06:22.087+0000 7f29a9cdc700  5 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017585754s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] exit Reset 0.000343 1 0.000374
2021-03-20T05:06:22.087+0000 7f29a9cdc700  5 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017585754s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] enter Started
2021-03-20T05:06:22.087+0000 7f29a9cdc700  5 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017585754s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] enter Start
2021-03-20T05:06:22.087+0000 7f29a9cdc700  1 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017585754s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] state<Start>: transitioning to Stray
2021-03-20T05:06:22.087+0000 7f29a9cdc700  5 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017585754s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] exit Start 0.000008 0 0.000000
2021-03-20T05:06:22.087+0000 7f29a9cdc700  5 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017585754s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] enter Started/Stray
2021-03-20T05:06:22.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017585754s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] activate_map: Not dirtying info: last_persisted is 110 while current is 111
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017585754s) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 111 epoch_requested: 111 MQuery 3.11s3 from 10(0) query_epoch 111 query: query(info 0'0 epoch_sent 111)
2021-03-20T05:06:22.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 120.407096863s@ mbc={} ps=[42~1,49~2]] update_history advanced history from ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) unreg_next_scrub existing-0.000000. was registered? 0
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] sending info
2021-03-20T05:06:22.087+0000 7f29a9cdc700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.109:6818/35284,v1:172.21.15.109:6820/35284] -- pg_notify2(3.11s0 3.11s0 (query:111 sent:111 3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111 pruub=15.017585754s) 3->0 ([109,110] all_participants=0(0),4(1),5(4),9(5),10(2),11(3) intervals=([109,110] acting 0(0),4(1),5(4),9(5),10(2),11(3)))) e111/111) v1 -- 0x556317016700 con 0x5563169b7c00
2021-03-20T05:06:22.087+0000 7f29a9cdc700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.109:6818/35284,v1:172.21.15.109:6820/35284] -- pg_notify2(3.11s0 3.11s0 (query:111 sent:111 3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) 3->0 ([109,110] all_participants=0(0),4(1),5(4),9(5),10(2),11(3) intervals=([109,110] acting 0(0),4(1),5(4),9(5),10(2),11(3)))) e111/111) v1 -- 0x556325710300 con 0x5563169b7c00
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) queue_transactions ch 0x5563259141e0 3.11s3_head
2021-03-20T05:06:22.087+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:22.087+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkeys 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkeys 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:22.087+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:22.087+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkeys 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:22.087+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkeys 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:22.087+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:22.088+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 111 epoch_requested: 111 NullEvt) prio 255 cost 10 e111) queued
2021-03-20T05:06:22.088+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 111 epoch_requested: 111 NullEvt) prio 255 cost 10 e111)> waiting <> waiting_peering {}
2021-03-20T05:06:22.088+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 111 epoch_requested: 111 NullEvt) prio 255 cost 10 e111) pg 0x5563156cb000
2021-03-20T05:06:22.088+0000 7f29a5cd4700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 111 epoch_requested: 111 NullEvt
2021-03-20T05:06:23.076+0000 7f29c7dda700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.4 v2:172.21.15.74:6818/34910 356 ==== pg_query2(3.11s3 3.11s3 query(info 0'0 epoch_sent 112) e112/112) v1 ==== 158+0+0 (crc 0 0 0) 0x556326681a40 con 0x556316bb9800
2021-03-20T05:06:23.076+0000 7f29c7dda700 15 osd.11 111 enqueue_peering_evt 3.11s3 epoch_sent: 112 epoch_requested: 112 MQuery 3.11s3 from 4(1) query_epoch 112 query: query(info 0'0 epoch_sent 112)
2021-03-20T05:06:23.076+0000 7f29c7dda700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 112 epoch_requested: 112 MQuery 3.11s3 from 4(1) query_epoch 112 query: query(info 0'0 epoch_sent 112)) prio 255 cost 10 e112)
2021-03-20T05:06:23.076+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:23.076+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 112 epoch_requested: 112 MQuery 3.11s3 from 4(1) query_epoch 112 query: query(info 0'0 epoch_sent 112)) prio 255 cost 10 e112) queued
2021-03-20T05:06:23.076+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 112 epoch_requested: 112 MQuery 3.11s3 from 4(1) query_epoch 112 query: query(info 0'0 epoch_sent 112)) prio 255 cost 10 e112)> waiting <> waiting_peering {}
2021-03-20T05:06:23.076+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 112 epoch_requested: 112 MQuery 3.11s3 from 4(1) query_epoch 112 query: query(info 0'0 epoch_sent 112)) prio 255 cost 10 e112) pg 0x5563156cb000
2021-03-20T05:06:23.076+0000 7f29a9cdc700 20 osd.11 op_wq _add_slot_waiter 3.11s3 peering, item epoch is 112, will wait on OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 112 epoch_requested: 112 MQuery 3.11s3 from 4(1) query_epoch 112 query: query(info 0'0 epoch_sent 112)) prio 255 cost 10 e112)
2021-03-20T05:06:23.078+0000 7f29b732d700 20 osd.11 112 identify_splits_and_merges 3.11s3 e111 to e112 pg_nums {27=16,109=26}
2021-03-20T05:06:23.078+0000 7f29b732d700 20 osd.11:1.consume_map consume_map 3.11s3
2021-03-20T05:06:23.078+0000 7f29b732d700 20 osd.11:1.consume_map consume_map  3.11s3 pending_peering first epoch 112 <= 112, requeueing
2021-03-20T05:06:23.078+0000 7f29b732d700 20 osd.11:1._wake_pg_slot _wake_pg_slot 3.11s3 to_process <> waiting <> waiting_peering {112=<OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 112 epoch_requested: 112 MQuery 3.11s3 from 4(1) query_epoch 112 query: query(info 0'0 epoch_sent 112)) prio 255 cost 10 e112)>}
2021-03-20T05:06:23.078+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:23.078+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 112 epoch_requested: 112 MQuery 3.11s3 from 4(1) query_epoch 112 query: query(info 0'0 epoch_sent 112)) prio 255 cost 10 e112) queued
2021-03-20T05:06:23.078+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 112 epoch_requested: 112 MQuery 3.11s3 from 4(1) query_epoch 112 query: query(info 0'0 epoch_sent 112)) prio 255 cost 10 e112)> waiting <> waiting_peering {}
2021-03-20T05:06:23.078+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 112 epoch_requested: 112 MQuery 3.11s3 from 4(1) query_epoch 112 query: query(info 0'0 epoch_sent 112)) prio 255 cost 10 e112) pg 0x5563156cb000
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] handle_advance_map: 112
2021-03-20T05:06:23.078+0000 7f29a9cdc700 20 osd.11:1.update_pg_epoch 3.11s3 111 -> 112
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 111 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] handle_advance_map [10,4,6,0,3,9]/[2147483647,4,2147483647,0,3,9] -- 10/4
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] state<Started>: Started advmap
2021-03-20T05:06:23.078+0000 7f29b732d700 15 osd.11 112 enqueue_peering_evt 3.11s3 epoch_sent: 112 epoch_requested: 112 NullEvt
2021-03-20T05:06:23.078+0000 7f29b732d700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 112 epoch_requested: 112 NullEvt) prio 255 cost 10 e112)
2021-03-20T05:06:23.078+0000 7f29a9cdc700 20 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] new interval newup [10,4,6,0,3,9] newacting [2147483647,4,2147483647,0,3,9]
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] state<Started>: should_restart_peering, transitioning to Reset
2021-03-20T05:06:23.078+0000 7f29a9cdc700  5 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] exit Started/Stray 0.991144 3 0.000049
2021-03-20T05:06:23.078+0000 7f29a9cdc700  5 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] exit Started 0.991180 0 0.000000
2021-03-20T05:06:23.078+0000 7f29a9cdc700  5 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] enter Reset
2021-03-20T05:06:23.078+0000 7f29a9cdc700 20 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]p10(0) r=-1 lpr=111 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] set_last_peering_reset 112
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]p10(0) r=-1 lpr=112 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] Clearing blocked outgoing recovery messages
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]p10(0) r=-1 lpr=112 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] Not blocking outgoing recovery messages
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]p10(0) r=-1 lpr=112 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] state<Reset>: Reset advmap
2021-03-20T05:06:23.078+0000 7f29a9cdc700 20 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]p10(0) r=-1 lpr=112 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] new interval newup [10,4,6,0,3,9] newacting [2147483647,4,2147483647,0,3,9]
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]p10(0) r=-1 lpr=112 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] state<Reset>: should restart peering, calling start_peering_interval again
2021-03-20T05:06:23.078+0000 7f29a9cdc700 20 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]p10(0) r=-1 lpr=112 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] set_last_peering_reset 112
2021-03-20T05:06:23.078+0000 7f29a9cdc700 15 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,111)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] clear_stats
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] start_peering_interval: check_new_interval output: check_new_interval interval(111-111 up [10,4,6,0,3,9](10) acting [10,4,6,0,3,9](10)) up_thru 97 up_from 18 last_epoch_clean 110 interval(111-111 up [10,4,6,0,3,9](10) acting [10,4,6,0,3,9](10)) : primary up 18-97 does not include interval
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=111) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]]  noting past ([109,111] all_participants=0(0),0(3),3(4),4(1),5(4),6(2),9(5),10(0),10(2),11(3) intervals=([109,110] acting 0(0),4(1),5(4),9(5),10(2),11(3)))
2021-03-20T05:06:23.078+0000 7f29a9cdc700 20 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] on_new_interval
2021-03-20T05:06:23.078+0000 7f29a9cdc700 20 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] on_new_interval upacting_features 0x3f01cfbb7ffdffff from [2147483647,4,2147483647,0,3,9]+[10,4,6,0,3,9]
2021-03-20T05:06:23.078+0000 7f29a9cdc700 20 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1)
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] init_hb_stamps now []
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] on_new_interval prior_readable_until_ub 0.000000000s (mnow 106.380859375s + 0.000000000s)
2021-03-20T05:06:23.078+0000 7f29a9cdc700 20 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] on_new_interval scrub_queued was 0 flags: 
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] cancel_recovery
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] clear_recovery_state
2021-03-20T05:06:23.078+0000 7f29a9cdc700 15 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] _clear_recovery_state flags: 
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) unreg_next_scrub existing-0.000000. was registered? 0
2021-03-20T05:06:23.078+0000 7f29a9cdc700  1 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] start_peering_interval up [10,4,6,0,3,9] -> [10,4,6,0,3,9], acting [10,4,6,0,3,9] -> [2147483647,4,2147483647,0,3,9], acting_primary 10(0) -> 4, up_primary 10(0) -> 10, role -1 -> -1, features acting 4540138303579357183 upacting 4540138303579357183
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] clear_primary_state
2021-03-20T05:06:23.078+0000 7f29a9cdc700 20 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] clear_primary_state
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] release_backoffs [3:88000000::::0,3:90000000::::head)
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) discard_replica_reservations
2021-03-20T05:06:23.078+0000 7f29a9cdc700 20 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] agent_stop
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] on_change
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] cancel_copy_ops
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] cancel_flush_ops
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] cancel_proxy_ops
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] cancel_manifest_ops
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) scrub_clear_state
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) clear_pgscrub_state
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) clear_scrub_reservations
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) reset_internal_state
2021-03-20T05:06:23.078+0000 7f29a9cdc700 15 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]]  PrimaryLog scrubber pg(3.11s3) _scrub_clear_state
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] on_change_cleanup
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] on_change
2021-03-20T05:06:23.078+0000 7f29a9cdc700 20 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] exit NotTrimming
2021-03-20T05:06:23.078+0000 7f29a9cdc700 20 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] enter NotTrimming
2021-03-20T05:06:23.078+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] [10,4,6,0,3,9] -> [2147483647,4,2147483647,0,3,9], acting primary 10(0) -> 4(1)
2021-03-20T05:06:23.079+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] check_recovery_sources no source osds () went down
2021-03-20T05:06:23.079+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) set_collection_opts 3.11s3_head options 
2021-03-20T05:06:23.079+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] plpg_on_pool_change
2021-03-20T05:06:23.079+0000 7f29a9cdc700 20 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] hit_set_clear
2021-03-20T05:06:23.079+0000 7f29a9cdc700 20 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] agent_stop
2021-03-20T05:06:23.079+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] handle_activate_map: 112
2021-03-20T05:06:23.079+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] activate_map
2021-03-20T05:06:23.079+0000 7f29a9cdc700  5 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] exit Reset 0.000612 1 0.000618
2021-03-20T05:06:23.079+0000 7f29a9cdc700  5 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] enter Started
2021-03-20T05:06:23.079+0000 7f29a9cdc700  5 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] enter Start
2021-03-20T05:06:23.079+0000 7f29a9cdc700  1 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] state<Start>: transitioning to Stray
2021-03-20T05:06:23.079+0000 7f29a9cdc700  5 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] exit Start 0.000013 0 0.000000
2021-03-20T05:06:23.079+0000 7f29a9cdc700  5 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] enter Started/Stray
2021-03-20T05:06:23.079+0000 7f29a9cdc700 20 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] activate_map: Not dirtying info: last_persisted is 111 while current is 112
2021-03-20T05:06:23.079+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 112 epoch_requested: 112 MQuery 3.11s3 from 4(1) query_epoch 112 query: query(info 0'0 epoch_sent 112)
2021-03-20T05:06:23.079+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) unreg_next_scrub existing-0.000000. was registered? 0
2021-03-20T05:06:23.079+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] sending info
2021-03-20T05:06:23.079+0000 7f29a9cdc700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.74:6818/34910,v1:172.21.15.74:6819/34910] -- pg_notify2(3.11s1 3.11s1 (query:112 sent:112 3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) 3->1 ([109,111] all_participants=0(0),0(3),3(4),4(1),5(4),6(2),9(5),10(0),10(2),11(3) intervals=([109,110] acting 0(0),4(1),5(4),9(5),10(2),11(3)))) e112/112) v1 -- 0x5563264fe300 con 0x556316bb9800
2021-03-20T05:06:23.079+0000 7f29a9cdc700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.74:6818/34910,v1:172.21.15.74:6819/34910] -- pg_notify2(3.11s1 3.11s1 (query:112 sent:112 3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) 3->1 ([109,111] all_participants=0(0),0(3),3(4),4(1),5(4),6(2),9(5),10(0),10(2),11(3) intervals=([109,110] acting 0(0),4(1),5(4),9(5),10(2),11(3)))) e112/112) v1 -- 0x556328c03800 con 0x556316bb9800
2021-03-20T05:06:23.079+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) queue_transactions ch 0x5563259141e0 3.11s3_head
2021-03-20T05:06:23.079+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:23.079+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:23.079+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkeys 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:23.079+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkeys 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:23.080+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:23.080+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 112 epoch_requested: 112 NullEvt) prio 255 cost 10 e112) queued
2021-03-20T05:06:23.080+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 112 epoch_requested: 112 NullEvt) prio 255 cost 10 e112)> waiting <> waiting_peering {}
2021-03-20T05:06:23.080+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 112 epoch_requested: 112 NullEvt) prio 255 cost 10 e112) pg 0x5563156cb000
2021-03-20T05:06:23.080+0000 7f29a9cdc700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 112 epoch_requested: 112 NullEvt
2021-03-20T05:06:24.088+0000 7f29c7dda700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.4 v2:172.21.15.74:6818/34910 362 ==== pg_query2(3.11s3 3.11s3 query(fulllog 0'0 epoch_sent 113) e113/113) v1 ==== 158+0+0 (crc 0 0 0) 0x55632991f2c0 con 0x556316bb9800
2021-03-20T05:06:24.088+0000 7f29c7dda700 15 osd.11 112 enqueue_peering_evt 3.11s3 epoch_sent: 113 epoch_requested: 113 MQuery 3.11s3 from 4(1) query_epoch 113 query: query(fulllog 0'0 epoch_sent 113)
2021-03-20T05:06:24.088+0000 7f29c7dda700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 113 epoch_requested: 113 MQuery 3.11s3 from 4(1) query_epoch 113 query: query(fulllog 0'0 epoch_sent 113)) prio 255 cost 10 e113)
2021-03-20T05:06:24.088+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:24.088+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 113 epoch_requested: 113 MQuery 3.11s3 from 4(1) query_epoch 113 query: query(fulllog 0'0 epoch_sent 113)) prio 255 cost 10 e113) queued
2021-03-20T05:06:24.088+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 113 epoch_requested: 113 MQuery 3.11s3 from 4(1) query_epoch 113 query: query(fulllog 0'0 epoch_sent 113)) prio 255 cost 10 e113)> waiting <> waiting_peering {}
2021-03-20T05:06:24.088+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 113 epoch_requested: 113 MQuery 3.11s3 from 4(1) query_epoch 113 query: query(fulllog 0'0 epoch_sent 113)) prio 255 cost 10 e113) pg 0x5563156cb000
2021-03-20T05:06:24.088+0000 7f29a5cd4700 20 osd.11 op_wq _add_slot_waiter 3.11s3 peering, item epoch is 113, will wait on OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 113 epoch_requested: 113 MQuery 3.11s3 from 4(1) query_epoch 113 query: query(fulllog 0'0 epoch_sent 113)) prio 255 cost 10 e113)
2021-03-20T05:06:24.090+0000 7f29b732d700 20 osd.11 113 identify_splits_and_merges 3.11s3 e112 to e113 pg_nums {27=16,109=26}
2021-03-20T05:06:24.090+0000 7f29b732d700 20 osd.11:1.consume_map consume_map 3.11s3
2021-03-20T05:06:24.090+0000 7f29b732d700 20 osd.11:1.consume_map consume_map  3.11s3 pending_peering first epoch 113 <= 113, requeueing
2021-03-20T05:06:24.090+0000 7f29b732d700 20 osd.11:1._wake_pg_slot _wake_pg_slot 3.11s3 to_process <> waiting <> waiting_peering {113=<OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 113 epoch_requested: 113 MQuery 3.11s3 from 4(1) query_epoch 113 query: query(fulllog 0'0 epoch_sent 113)) prio 255 cost 10 e113)>}
2021-03-20T05:06:24.090+0000 7f29b732d700 15 osd.11 113 enqueue_peering_evt 3.11s3 epoch_sent: 113 epoch_requested: 113 NullEvt
2021-03-20T05:06:24.090+0000 7f29b732d700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 113 epoch_requested: 113 NullEvt) prio 255 cost 10 e113)
2021-03-20T05:06:24.092+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:24.092+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 113 epoch_requested: 113 MQuery 3.11s3 from 4(1) query_epoch 113 query: query(fulllog 0'0 epoch_sent 113)) prio 255 cost 10 e113) queued
2021-03-20T05:06:24.092+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 113 epoch_requested: 113 MQuery 3.11s3 from 4(1) query_epoch 113 query: query(fulllog 0'0 epoch_sent 113)) prio 255 cost 10 e113)> waiting <> waiting_peering {}
2021-03-20T05:06:24.092+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 113 epoch_requested: 113 MQuery 3.11s3 from 4(1) query_epoch 113 query: query(fulllog 0'0 epoch_sent 113)) prio 255 cost 10 e113) pg 0x5563156cb000
2021-03-20T05:06:24.092+0000 7f29a5cd4700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] handle_advance_map: 113
2021-03-20T05:06:24.092+0000 7f29a5cd4700 20 osd.11:1.update_pg_epoch 3.11s3 112 -> 113
2021-03-20T05:06:24.092+0000 7f29a5cd4700 10 osd.11 pg_epoch: 112 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] handle_advance_map [10,4,6,0,3,9]/[2147483647,4,2147483647,0,3,9] -- 10/4
2021-03-20T05:06:24.092+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] state<Started>: Started advmap
2021-03-20T05:06:24.092+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] check_recovery_sources no source osds () went down
2021-03-20T05:06:24.093+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) set_collection_opts 3.11s3_head options 
2021-03-20T05:06:24.093+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] plpg_on_pool_change
2021-03-20T05:06:24.093+0000 7f29a5cd4700 20 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] hit_set_clear
2021-03-20T05:06:24.093+0000 7f29a5cd4700 20 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] agent_stop
2021-03-20T05:06:24.093+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] handle_activate_map: 113
2021-03-20T05:06:24.093+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] activate_map
2021-03-20T05:06:24.093+0000 7f29a5cd4700 20 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] activate_map: Not dirtying info: last_persisted is 112 while current is 113
2021-03-20T05:06:24.093+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 113 epoch_requested: 113 MQuery 3.11s3 from 4(1) query_epoch 113 query: query(fulllog 0'0 epoch_sent 113)
2021-03-20T05:06:24.093+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) unreg_next_scrub existing-0.000000. was registered? 0
2021-03-20T05:06:24.093+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] log request from 4(1)
2021-03-20T05:06:24.093+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]]  sending info+missing+full log
2021-03-20T05:06:24.094+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]]  sending log((0'0,107'142], crt=107'142) missing(0 may_include_deletes = 1)
2021-03-20T05:06:24.094+0000 7f29a5cd4700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.74:6818/34910,v1:172.21.15.74:6819/34910] -- PGlog(3.11s1 log log((0'0,107'142], crt=107'142) pi ([0,0] all_participants= intervals=) e113/113) v6 -- 0x556316d17800 con 0x556316bb9800
2021-03-20T05:06:24.094+0000 7f29a5cd4700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.74:6818/34910,v1:172.21.15.74:6819/34910] -- pg_notify2(3.11s1 3.11s1 (query:113 sent:113 3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) 3->1 ([109,111] all_participants=0(0),0(3),3(4),4(1),5(4),6(2),9(5),10(0),10(2),11(3) intervals=([109,110] acting 0(0),4(1),5(4),9(5),10(2),11(3)))) e113/113) v1 -- 0x55632346b100 con 0x556316bb9800
2021-03-20T05:06:24.094+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:24.094+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 113 epoch_requested: 113 NullEvt) prio 255 cost 10 e113) queued
2021-03-20T05:06:24.094+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 113 epoch_requested: 113 NullEvt) prio 255 cost 10 e113)> waiting <> waiting_peering {}
2021-03-20T05:06:24.094+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 113 epoch_requested: 113 NullEvt) prio 255 cost 10 e113) pg 0x5563156cb000
2021-03-20T05:06:24.094+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 113 epoch_requested: 113 NullEvt
2021-03-20T05:06:24.097+0000 7f29c7dda700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.4 v2:172.21.15.74:6818/34910 365 ==== pg_info2(3.11s3 3.11s3 3.11s1( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) e113/113) v1 ==== 1046+0+0 (crc 0 0 0) 0x55632346b100 con 0x556316bb9800
2021-03-20T05:06:24.097+0000 7f29c7dda700 15 osd.11 113 enqueue_peering_evt 3.11s3 epoch_sent: 113 epoch_requested: 113 MInfoRec from 4(1) info: 3.11s1( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112)
2021-03-20T05:06:24.097+0000 7f29c7dda700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 113 epoch_requested: 113 MInfoRec from 4(1) info: 3.11s1( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112)) prio 255 cost 10 e113)
2021-03-20T05:06:24.097+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:24.097+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 113 epoch_requested: 113 MInfoRec from 4(1) info: 3.11s1( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112)) prio 255 cost 10 e113) queued
2021-03-20T05:06:24.097+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 113 epoch_requested: 113 MInfoRec from 4(1) info: 3.11s1( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112)) prio 255 cost 10 e113)> waiting <> waiting_peering {}
2021-03-20T05:06:24.097+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 113 epoch_requested: 113 MInfoRec from 4(1) info: 3.11s1( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112)) prio 255 cost 10 e113) pg 0x5563156cb000
2021-03-20T05:06:24.097+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 113 epoch_requested: 113 MInfoRec from 4(1) info: 3.11s1( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112)
2021-03-20T05:06:24.097+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] state<Started/Stray>: got info from osd.4(1) 3.11s1( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112)
2021-03-20T05:06:24.097+0000 7f29a9cdc700  5 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] exit Started/Stray 1.018002 7 0.001208
2021-03-20T05:06:24.097+0000 7f29a9cdc700  5 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] enter Started/ReplicaActive
2021-03-20T05:06:24.097+0000 7f29a9cdc700  5 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] enter Started/ReplicaActive/RepNotRecovering
2021-03-20T05:06:24.097+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped NOTIFY mbc={} ps=[42~1,49~2]] state<Started/ReplicaActive>: In ReplicaActive, about to call activate
2021-03-20T05:06:24.097+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped mbc={} ps=[42~1,49~2]] activate - no missing, moving last_complete 107'142 -> 107'142
2021-03-20T05:06:24.097+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped mbc={} ps=[42~1,49~2]] state<Started/ReplicaActive>: Activate Finished
2021-03-20T05:06:24.097+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) queue_transactions ch 0x5563259141e0 3.11s3_head
2021-03-20T05:06:24.097+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:24.097+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:24.097+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkeys 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:24.097+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkeys 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:24.100+0000 7f29a9cdc700 15 osd.11 113 enqueue_peering_evt 3.11s3 epoch_sent: 113 epoch_requested: 113 ActivateCommitted from 110 processed at 113
2021-03-20T05:06:24.100+0000 7f29a9cdc700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 113 epoch_requested: 113 ActivateCommitted from 110 processed at 113) prio 255 cost 10 e113)
2021-03-20T05:06:24.100+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:24.100+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 113 epoch_requested: 113 ActivateCommitted from 110 processed at 113) prio 255 cost 10 e113) queued
2021-03-20T05:06:24.100+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 113 epoch_requested: 113 ActivateCommitted from 110 processed at 113) prio 255 cost 10 e113)> waiting <> waiting_peering {}
2021-03-20T05:06:24.100+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 113 epoch_requested: 113 ActivateCommitted from 110 processed at 113) prio 255 cost 10 e113) pg 0x5563156cb000
2021-03-20T05:06:24.100+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 113 epoch_requested: 113 ActivateCommitted from 110 processed at 113
2021-03-20T05:06:24.100+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 crt=107'142 lcod 0'0 mlcod 0'0 remapped mbc={} ps=[42~1,49~2]] state<Started/ReplicaActive>: react 113 telling primary
2021-03-20T05:06:24.100+0000 7f29a9cdc700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.74:6818/34910,v1:172.21.15.74:6819/34910] -- pg_info2(3.11s1 3.11s1 3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=110/110/0 sis=112) e113/113) v1 -- 0x556328c02300 con 0x556316bb9800
2021-03-20T05:06:24.256+0000 7f29c7dda700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.4 v2:172.21.15.74:6818/34910 368 ==== MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=82, to_read={3:8b5265c1:::smithi03341006-20:head=0,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3 ==== 241+0+0 (crc 0 0 0) 0x556325562e00 con 0x556316bb9800
2021-03-20T05:06:24.256+0000 7f29c7dda700 15 osd.11 113 enqueue_op 0x5563256bba20 prio 3 type 110 cost 0 latency 0.000021 epoch 113 MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=82, to_read={3:8b5265c1:::smithi03341006-20:head=0,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3
2021-03-20T05:06:24.256+0000 7f29c7dda700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=82, to_read={3:8b5265c1:::smithi03341006-20:head=0,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3) prio 3 cost 0 e113)
2021-03-20T05:06:24.256+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:24.256+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=82, to_read={3:8b5265c1:::smithi03341006-20:head=0,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3) prio 3 cost 0 e113) queued
2021-03-20T05:06:24.256+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=82, to_read={3:8b5265c1:::smithi03341006-20:head=0,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3) prio 3 cost 0 e113)> waiting <> waiting_peering {}
2021-03-20T05:06:24.256+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=82, to_read={3:8b5265c1:::smithi03341006-20:head=0,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3) prio 3 cost 0 e113) pg 0x5563156cb000
2021-03-20T05:06:24.256+0000 7f29a9cdc700 10 osd.11 113 dequeue_op 0x5563256bba20 prio 3 cost 0 latency 0.000081 MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=82, to_read={3:8b5265c1:::smithi03341006-20:head=0,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3 pg pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]]
2021-03-20T05:06:24.256+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] _handle_message: MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=82, to_read={3:8b5265c1:::smithi03341006-20:head=0,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3
2021-03-20T05:06:24.256+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) read 3.11s3_head 3#3:8b5265c1:::smithi03341006-20:head# 0x0~40000
2021-03-20T05:06:24.256+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) read 3.11s3_head 3#3:8b5265c1:::smithi03341006-20:head# 0x0~40000 = 262144
2021-03-20T05:06:24.256+0000 7f29a9cdc700 20 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] handle_sub_read read request=262144 r=262144 len=262144
2021-03-20T05:06:24.256+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: Getting attr on 3:8b5265c1:::smithi03341006-20:head
2021-03-20T05:06:24.256+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: not in cache 3:8b5265c1:::smithi03341006-20:head
2021-03-20T05:06:24.256+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) stat 3.11s3_head 3#3:8b5265c1:::smithi03341006-20:head#
2021-03-20T05:06:24.256+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: found on disk, size 667648
2021-03-20T05:06:24.256+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) getattr 3.11s3_head 3#3:8b5265c1:::smithi03341006-20:head# hinfo_key
2021-03-20T05:06:24.256+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) getattr 3.11s3_head 3#3:8b5265c1:::smithi03341006-20:head# hinfo_key = 0
2021-03-20T05:06:24.256+0000 7f29a9cdc700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.74:6818/34910,v1:172.21.15.74:6819/34910] -- MOSDECSubOpReadReply(3.11s1 113/112 ECSubReadReply(tid=82, attrs_read=0)) v2 -- 0x5563234e6a00 con 0x556316bb9800
2021-03-20T05:06:24.260+0000 7f29c7dda700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.4 v2:172.21.15.74:6818/34910 369 ==== MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=83, to_read={3:8b5265c1:::smithi03341006-20:head=262144,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3 ==== 241+0+0 (crc 0 0 0) 0x5563234e6a00 con 0x556316bb9800
2021-03-20T05:06:24.260+0000 7f29c7dda700 15 osd.11 113 enqueue_op 0x5563256bbce0 prio 127 type 110 cost 0 latency 0.000024 epoch 113 MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=83, to_read={3:8b5265c1:::smithi03341006-20:head=262144,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3
2021-03-20T05:06:24.260+0000 7f29c7dda700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=83, to_read={3:8b5265c1:::smithi03341006-20:head=262144,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113)
2021-03-20T05:06:24.260+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:24.260+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=83, to_read={3:8b5265c1:::smithi03341006-20:head=262144,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113) queued
2021-03-20T05:06:24.260+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=83, to_read={3:8b5265c1:::smithi03341006-20:head=262144,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113)> waiting <> waiting_peering {}
2021-03-20T05:06:24.260+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=83, to_read={3:8b5265c1:::smithi03341006-20:head=262144,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113) pg 0x5563156cb000
2021-03-20T05:06:24.260+0000 7f29a5cd4700 10 osd.11 113 dequeue_op 0x5563256bbce0 prio 127 cost 0 latency 0.000062 MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=83, to_read={3:8b5265c1:::smithi03341006-20:head=262144,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3 pg pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]]
2021-03-20T05:06:24.260+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] _handle_message: MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=83, to_read={3:8b5265c1:::smithi03341006-20:head=262144,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3
2021-03-20T05:06:24.260+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) read 3.11s3_head 3#3:8b5265c1:::smithi03341006-20:head# 0x40000~40000
2021-03-20T05:06:24.260+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) read 3.11s3_head 3#3:8b5265c1:::smithi03341006-20:head# 0x40000~40000 = 262144
2021-03-20T05:06:24.260+0000 7f29a5cd4700 20 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] handle_sub_read read request=262144 r=262144 len=262144
2021-03-20T05:06:24.260+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: Getting attr on 3:8b5265c1:::smithi03341006-20:head
2021-03-20T05:06:24.260+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: not in cache 3:8b5265c1:::smithi03341006-20:head
2021-03-20T05:06:24.260+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) stat 3.11s3_head 3#3:8b5265c1:::smithi03341006-20:head#
2021-03-20T05:06:24.260+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: found on disk, size 667648
2021-03-20T05:06:24.260+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) getattr 3.11s3_head 3#3:8b5265c1:::smithi03341006-20:head# hinfo_key
2021-03-20T05:06:24.260+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) getattr 3.11s3_head 3#3:8b5265c1:::smithi03341006-20:head# hinfo_key = 0
2021-03-20T05:06:24.260+0000 7f29a5cd4700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.74:6818/34910,v1:172.21.15.74:6819/34910] -- MOSDECSubOpReadReply(3.11s1 113/112 ECSubReadReply(tid=83, attrs_read=0)) v2 -- 0x556328c76200 con 0x556316bb9800
2021-03-20T05:06:24.264+0000 7f29c7dda700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.4 v2:172.21.15.74:6818/34910 370 ==== MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=84, to_read={3:8b5265c1:::smithi03341006-20:head=524288,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3 ==== 241+0+0 (crc 0 0 0) 0x556328c76200 con 0x556316bb9800
2021-03-20T05:06:24.264+0000 7f29c7dda700 15 osd.11 113 enqueue_op 0x5563259bb600 prio 127 type 110 cost 0 latency 0.000023 epoch 113 MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=84, to_read={3:8b5265c1:::smithi03341006-20:head=524288,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3
2021-03-20T05:06:24.264+0000 7f29c7dda700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=84, to_read={3:8b5265c1:::smithi03341006-20:head=524288,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113)
2021-03-20T05:06:24.264+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:24.264+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=84, to_read={3:8b5265c1:::smithi03341006-20:head=524288,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113) queued
2021-03-20T05:06:24.264+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=84, to_read={3:8b5265c1:::smithi03341006-20:head=524288,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113)> waiting <> waiting_peering {}
2021-03-20T05:06:24.264+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=84, to_read={3:8b5265c1:::smithi03341006-20:head=524288,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113) pg 0x5563156cb000
2021-03-20T05:06:24.264+0000 7f29a9cdc700 10 osd.11 113 dequeue_op 0x5563259bb600 prio 127 cost 0 latency 0.000065 MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=84, to_read={3:8b5265c1:::smithi03341006-20:head=524288,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3 pg pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]]
2021-03-20T05:06:24.264+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] _handle_message: MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=84, to_read={3:8b5265c1:::smithi03341006-20:head=524288,262144,0}, subchunks={3:8b5265c1:::smithi03341006-20:head=[0,1]}, attrs_to_read=)) v3
2021-03-20T05:06:24.264+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) read 3.11s3_head 3#3:8b5265c1:::smithi03341006-20:head# 0x80000~40000
2021-03-20T05:06:24.264+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) read 3.11s3_head 3#3:8b5265c1:::smithi03341006-20:head# 0x80000~40000 = 143360
2021-03-20T05:06:24.264+0000 7f29a9cdc700 20 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] handle_sub_read read request=262144 r=143360 len=143360
2021-03-20T05:06:24.264+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: Getting attr on 3:8b5265c1:::smithi03341006-20:head
2021-03-20T05:06:24.264+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: not in cache 3:8b5265c1:::smithi03341006-20:head
2021-03-20T05:06:24.264+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) stat 3.11s3_head 3#3:8b5265c1:::smithi03341006-20:head#
2021-03-20T05:06:24.264+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: found on disk, size 667648
2021-03-20T05:06:24.264+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) getattr 3.11s3_head 3#3:8b5265c1:::smithi03341006-20:head# hinfo_key
2021-03-20T05:06:24.264+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) getattr 3.11s3_head 3#3:8b5265c1:::smithi03341006-20:head# hinfo_key = 0
2021-03-20T05:06:24.264+0000 7f29a9cdc700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.74:6818/34910,v1:172.21.15.74:6819/34910] -- MOSDECSubOpReadReply(3.11s1 113/112 ECSubReadReply(tid=84, attrs_read=0)) v2 -- 0x5563232cdc00 con 0x556316bb9800
2021-03-20T05:06:24.269+0000 7f29c7dda700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.4 v2:172.21.15.74:6818/34910 371 ==== MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=85, to_read={3:8d775aef:::smithi03341006-38:4a=0,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3 ==== 241+0+0 (crc 0 0 0) 0x5563232cdc00 con 0x556316bb9800
2021-03-20T05:06:24.269+0000 7f29c7dda700 15 osd.11 113 enqueue_op 0x5563259bb080 prio 3 type 110 cost 0 latency 0.000021 epoch 113 MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=85, to_read={3:8d775aef:::smithi03341006-38:4a=0,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3
2021-03-20T05:06:24.269+0000 7f29c7dda700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=85, to_read={3:8d775aef:::smithi03341006-38:4a=0,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3) prio 3 cost 0 e113)
2021-03-20T05:06:24.269+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:24.269+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=85, to_read={3:8d775aef:::smithi03341006-38:4a=0,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3) prio 3 cost 0 e113) queued
2021-03-20T05:06:24.269+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=85, to_read={3:8d775aef:::smithi03341006-38:4a=0,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3) prio 3 cost 0 e113)> waiting <> waiting_peering {}
2021-03-20T05:06:24.269+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=85, to_read={3:8d775aef:::smithi03341006-38:4a=0,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3) prio 3 cost 0 e113) pg 0x5563156cb000
2021-03-20T05:06:24.269+0000 7f29a5cd4700 10 osd.11 113 dequeue_op 0x5563259bb080 prio 3 cost 0 latency 0.000066 MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=85, to_read={3:8d775aef:::smithi03341006-38:4a=0,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3 pg pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]]
2021-03-20T05:06:24.269+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] _handle_message: MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=85, to_read={3:8d775aef:::smithi03341006-38:4a=0,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3
2021-03-20T05:06:24.269+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) read 3.11s3_head 3#3:8d775aef:::smithi03341006-38:4a# 0x0~40000
2021-03-20T05:06:24.269+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) read 3.11s3_head 3#3:8d775aef:::smithi03341006-38:4a# 0x0~40000 = 262144
2021-03-20T05:06:24.269+0000 7f29a5cd4700 20 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] handle_sub_read read request=262144 r=262144 len=262144
2021-03-20T05:06:24.269+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: Getting attr on 3:8d775aef:::smithi03341006-38:4a
2021-03-20T05:06:24.269+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: not in cache 3:8d775aef:::smithi03341006-38:4a
2021-03-20T05:06:24.269+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) stat 3.11s3_head 3#3:8d775aef:::smithi03341006-38:4a#
2021-03-20T05:06:24.269+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: found on disk, size 593920
2021-03-20T05:06:24.269+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) getattr 3.11s3_head 3#3:8d775aef:::smithi03341006-38:4a# hinfo_key
2021-03-20T05:06:24.269+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) getattr 3.11s3_head 3#3:8d775aef:::smithi03341006-38:4a# hinfo_key = 0
2021-03-20T05:06:24.269+0000 7f29a5cd4700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.74:6818/34910,v1:172.21.15.74:6819/34910] -- MOSDECSubOpReadReply(3.11s1 113/112 ECSubReadReply(tid=85, attrs_read=0)) v2 -- 0x5563155f5a00 con 0x556316bb9800
2021-03-20T05:06:24.272+0000 7f29c7dda700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.4 v2:172.21.15.74:6818/34910 372 ==== MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=86, to_read={3:8d775aef:::smithi03341006-38:4a=262144,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3 ==== 241+0+0 (crc 0 0 0) 0x5563155f5a00 con 0x556316bb9800
2021-03-20T05:06:24.272+0000 7f29c7dda700 15 osd.11 113 enqueue_op 0x556316b5c840 prio 127 type 110 cost 0 latency 0.000027 epoch 113 MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=86, to_read={3:8d775aef:::smithi03341006-38:4a=262144,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3
2021-03-20T05:06:24.272+0000 7f29c7dda700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=86, to_read={3:8d775aef:::smithi03341006-38:4a=262144,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113)
2021-03-20T05:06:24.272+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:24.272+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=86, to_read={3:8d775aef:::smithi03341006-38:4a=262144,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113) queued
2021-03-20T05:06:24.272+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=86, to_read={3:8d775aef:::smithi03341006-38:4a=262144,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113)> waiting <> waiting_peering {}
2021-03-20T05:06:24.272+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=86, to_read={3:8d775aef:::smithi03341006-38:4a=262144,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113) pg 0x5563156cb000
2021-03-20T05:06:24.272+0000 7f29a9cdc700 10 osd.11 113 dequeue_op 0x556316b5c840 prio 127 cost 0 latency 0.000074 MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=86, to_read={3:8d775aef:::smithi03341006-38:4a=262144,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3 pg pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]]
2021-03-20T05:06:24.272+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] _handle_message: MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=86, to_read={3:8d775aef:::smithi03341006-38:4a=262144,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3
2021-03-20T05:06:24.272+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) read 3.11s3_head 3#3:8d775aef:::smithi03341006-38:4a# 0x40000~40000
2021-03-20T05:06:24.272+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) read 3.11s3_head 3#3:8d775aef:::smithi03341006-38:4a# 0x40000~40000 = 262144
2021-03-20T05:06:24.272+0000 7f29a9cdc700 20 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] handle_sub_read read request=262144 r=262144 len=262144
2021-03-20T05:06:24.272+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: Getting attr on 3:8d775aef:::smithi03341006-38:4a
2021-03-20T05:06:24.272+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: not in cache 3:8d775aef:::smithi03341006-38:4a
2021-03-20T05:06:24.272+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) stat 3.11s3_head 3#3:8d775aef:::smithi03341006-38:4a#
2021-03-20T05:06:24.272+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: found on disk, size 593920
2021-03-20T05:06:24.272+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) getattr 3.11s3_head 3#3:8d775aef:::smithi03341006-38:4a# hinfo_key
2021-03-20T05:06:24.272+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) getattr 3.11s3_head 3#3:8d775aef:::smithi03341006-38:4a# hinfo_key = 0
2021-03-20T05:06:24.272+0000 7f29a9cdc700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.74:6818/34910,v1:172.21.15.74:6819/34910] -- MOSDECSubOpReadReply(3.11s1 113/112 ECSubReadReply(tid=86, attrs_read=0)) v2 -- 0x5563232cd200 con 0x556316bb9800
2021-03-20T05:06:24.278+0000 7f29c7dda700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.4 v2:172.21.15.74:6818/34910 373 ==== MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=87, to_read={3:8d775aef:::smithi03341006-38:4a=524288,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3 ==== 241+0+0 (crc 0 0 0) 0x5563232cd200 con 0x556316bb9800
2021-03-20T05:06:24.278+0000 7f29c7dda700 15 osd.11 113 enqueue_op 0x556316564f20 prio 127 type 110 cost 0 latency 0.000020 epoch 113 MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=87, to_read={3:8d775aef:::smithi03341006-38:4a=524288,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3
2021-03-20T05:06:24.278+0000 7f29c7dda700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=87, to_read={3:8d775aef:::smithi03341006-38:4a=524288,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113)
2021-03-20T05:06:24.278+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:24.278+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=87, to_read={3:8d775aef:::smithi03341006-38:4a=524288,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113) queued
2021-03-20T05:06:24.278+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=87, to_read={3:8d775aef:::smithi03341006-38:4a=524288,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113)> waiting <> waiting_peering {}
2021-03-20T05:06:24.278+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=87, to_read={3:8d775aef:::smithi03341006-38:4a=524288,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113) pg 0x5563156cb000
2021-03-20T05:06:24.278+0000 7f29a9cdc700 10 osd.11 113 dequeue_op 0x556316564f20 prio 127 cost 0 latency 0.000070 MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=87, to_read={3:8d775aef:::smithi03341006-38:4a=524288,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3 pg pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]]
2021-03-20T05:06:24.278+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] _handle_message: MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=87, to_read={3:8d775aef:::smithi03341006-38:4a=524288,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:4a=[0,1]}, attrs_to_read=)) v3
2021-03-20T05:06:24.278+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) read 3.11s3_head 3#3:8d775aef:::smithi03341006-38:4a# 0x80000~40000
2021-03-20T05:06:24.278+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) read 3.11s3_head 3#3:8d775aef:::smithi03341006-38:4a# 0x80000~40000 = 69632
2021-03-20T05:06:24.278+0000 7f29a9cdc700 20 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] handle_sub_read read request=262144 r=69632 len=69632
2021-03-20T05:06:24.278+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: Getting attr on 3:8d775aef:::smithi03341006-38:4a
2021-03-20T05:06:24.278+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: not in cache 3:8d775aef:::smithi03341006-38:4a
2021-03-20T05:06:24.278+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) stat 3.11s3_head 3#3:8d775aef:::smithi03341006-38:4a#
2021-03-20T05:06:24.278+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: found on disk, size 593920
2021-03-20T05:06:24.278+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) getattr 3.11s3_head 3#3:8d775aef:::smithi03341006-38:4a# hinfo_key
2021-03-20T05:06:24.278+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) getattr 3.11s3_head 3#3:8d775aef:::smithi03341006-38:4a# hinfo_key = 0
2021-03-20T05:06:24.278+0000 7f29a9cdc700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.74:6818/34910,v1:172.21.15.74:6819/34910] -- MOSDECSubOpReadReply(3.11s1 113/112 ECSubReadReply(tid=87, attrs_read=0)) v2 -- 0x5563232cde00 con 0x556316bb9800
2021-03-20T05:06:24.282+0000 7f29c7dda700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.4 v2:172.21.15.74:6818/34910 374 ==== MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=88, to_read={3:8d775aef:::smithi03341006-38:head=0,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3 ==== 241+0+0 (crc 0 0 0) 0x5563232cde00 con 0x556316bb9800
2021-03-20T05:06:24.282+0000 7f29c7dda700 15 osd.11 113 enqueue_op 0x5563234c69a0 prio 3 type 110 cost 0 latency 0.000018 epoch 113 MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=88, to_read={3:8d775aef:::smithi03341006-38:head=0,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3
2021-03-20T05:06:24.282+0000 7f29c7dda700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=88, to_read={3:8d775aef:::smithi03341006-38:head=0,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3) prio 3 cost 0 e113)
2021-03-20T05:06:24.282+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:24.282+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=88, to_read={3:8d775aef:::smithi03341006-38:head=0,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3) prio 3 cost 0 e113) queued
2021-03-20T05:06:24.282+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=88, to_read={3:8d775aef:::smithi03341006-38:head=0,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3) prio 3 cost 0 e113)> waiting <> waiting_peering {}
2021-03-20T05:06:24.282+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=88, to_read={3:8d775aef:::smithi03341006-38:head=0,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3) prio 3 cost 0 e113) pg 0x5563156cb000
2021-03-20T05:06:24.282+0000 7f29a5cd4700 10 osd.11 113 dequeue_op 0x5563234c69a0 prio 3 cost 0 latency 0.000062 MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=88, to_read={3:8d775aef:::smithi03341006-38:head=0,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3 pg pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]]
2021-03-20T05:06:24.282+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] _handle_message: MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=88, to_read={3:8d775aef:::smithi03341006-38:head=0,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3
2021-03-20T05:06:24.282+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) read 3.11s3_head 3#3:8d775aef:::smithi03341006-38:head# 0x0~40000
2021-03-20T05:06:24.282+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) read 3.11s3_head 3#3:8d775aef:::smithi03341006-38:head# 0x0~40000 = 262144
2021-03-20T05:06:24.282+0000 7f29a5cd4700 20 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] handle_sub_read read request=262144 r=262144 len=262144
2021-03-20T05:06:24.282+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: Getting attr on 3:8d775aef:::smithi03341006-38:head
2021-03-20T05:06:24.282+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: not in cache 3:8d775aef:::smithi03341006-38:head
2021-03-20T05:06:24.282+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) stat 3.11s3_head 3#3:8d775aef:::smithi03341006-38:head#
2021-03-20T05:06:24.282+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: found on disk, size 847872
2021-03-20T05:06:24.282+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) getattr 3.11s3_head 3#3:8d775aef:::smithi03341006-38:head# hinfo_key
2021-03-20T05:06:24.282+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) getattr 3.11s3_head 3#3:8d775aef:::smithi03341006-38:head# hinfo_key = 0
2021-03-20T05:06:24.282+0000 7f29a5cd4700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.74:6818/34910,v1:172.21.15.74:6819/34910] -- MOSDECSubOpReadReply(3.11s1 113/112 ECSubReadReply(tid=88, attrs_read=0)) v2 -- 0x556325e4fe00 con 0x556316bb9800
2021-03-20T05:06:24.285+0000 7f29c7dda700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.4 v2:172.21.15.74:6818/34910 375 ==== MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=89, to_read={3:8d775aef:::smithi03341006-38:head=262144,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3 ==== 241+0+0 (crc 0 0 0) 0x556325e4fe00 con 0x556316bb9800
2021-03-20T05:06:24.285+0000 7f29c7dda700 15 osd.11 113 enqueue_op 0x556325e2b8c0 prio 127 type 110 cost 0 latency 0.000024 epoch 113 MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=89, to_read={3:8d775aef:::smithi03341006-38:head=262144,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3
2021-03-20T05:06:24.285+0000 7f29c7dda700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=89, to_read={3:8d775aef:::smithi03341006-38:head=262144,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113)
2021-03-20T05:06:24.285+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:24.285+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=89, to_read={3:8d775aef:::smithi03341006-38:head=262144,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113) queued
2021-03-20T05:06:24.285+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=89, to_read={3:8d775aef:::smithi03341006-38:head=262144,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113)> waiting <> waiting_peering {}
2021-03-20T05:06:24.285+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=89, to_read={3:8d775aef:::smithi03341006-38:head=262144,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113) pg 0x5563156cb000
2021-03-20T05:06:24.285+0000 7f29a9cdc700 10 osd.11 113 dequeue_op 0x556325e2b8c0 prio 127 cost 0 latency 0.000067 MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=89, to_read={3:8d775aef:::smithi03341006-38:head=262144,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3 pg pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]]
2021-03-20T05:06:24.285+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] _handle_message: MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=89, to_read={3:8d775aef:::smithi03341006-38:head=262144,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3
2021-03-20T05:06:24.285+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) read 3.11s3_head 3#3:8d775aef:::smithi03341006-38:head# 0x40000~40000
2021-03-20T05:06:24.285+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) read 3.11s3_head 3#3:8d775aef:::smithi03341006-38:head# 0x40000~40000 = 262144
2021-03-20T05:06:24.285+0000 7f29a9cdc700 20 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] handle_sub_read read request=262144 r=262144 len=262144
2021-03-20T05:06:24.285+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: Getting attr on 3:8d775aef:::smithi03341006-38:head
2021-03-20T05:06:24.285+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: not in cache 3:8d775aef:::smithi03341006-38:head
2021-03-20T05:06:24.285+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) stat 3.11s3_head 3#3:8d775aef:::smithi03341006-38:head#
2021-03-20T05:06:24.285+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: found on disk, size 847872
2021-03-20T05:06:24.285+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) getattr 3.11s3_head 3#3:8d775aef:::smithi03341006-38:head# hinfo_key
2021-03-20T05:06:24.285+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) getattr 3.11s3_head 3#3:8d775aef:::smithi03341006-38:head# hinfo_key = 0
2021-03-20T05:06:24.285+0000 7f29a9cdc700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.74:6818/34910,v1:172.21.15.74:6819/34910] -- MOSDECSubOpReadReply(3.11s1 113/112 ECSubReadReply(tid=89, attrs_read=0)) v2 -- 0x556325b89800 con 0x556316bb9800
2021-03-20T05:06:24.288+0000 7f29c7dda700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.4 v2:172.21.15.74:6818/34910 376 ==== MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=90, to_read={3:8d775aef:::smithi03341006-38:head=524288,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3 ==== 241+0+0 (crc 0 0 0) 0x556325b89800 con 0x556316bb9800
2021-03-20T05:06:24.288+0000 7f29c7dda700 15 osd.11 113 enqueue_op 0x5563233a02c0 prio 127 type 110 cost 0 latency 0.000019 epoch 113 MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=90, to_read={3:8d775aef:::smithi03341006-38:head=524288,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3
2021-03-20T05:06:24.288+0000 7f29c7dda700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=90, to_read={3:8d775aef:::smithi03341006-38:head=524288,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113)
2021-03-20T05:06:24.288+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:24.288+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=90, to_read={3:8d775aef:::smithi03341006-38:head=524288,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113) queued
2021-03-20T05:06:24.288+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=90, to_read={3:8d775aef:::smithi03341006-38:head=524288,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113)> waiting <> waiting_peering {}
2021-03-20T05:06:24.288+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=90, to_read={3:8d775aef:::smithi03341006-38:head=524288,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113) pg 0x5563156cb000
2021-03-20T05:06:24.288+0000 7f29a5cd4700 10 osd.11 113 dequeue_op 0x5563233a02c0 prio 127 cost 0 latency 0.000050 MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=90, to_read={3:8d775aef:::smithi03341006-38:head=524288,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3 pg pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]]
2021-03-20T05:06:24.288+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] _handle_message: MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=90, to_read={3:8d775aef:::smithi03341006-38:head=524288,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3
2021-03-20T05:06:24.288+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) read 3.11s3_head 3#3:8d775aef:::smithi03341006-38:head# 0x80000~40000
2021-03-20T05:06:24.289+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) read 3.11s3_head 3#3:8d775aef:::smithi03341006-38:head# 0x80000~40000 = 262144
2021-03-20T05:06:24.289+0000 7f29a5cd4700 20 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] handle_sub_read read request=262144 r=262144 len=262144
2021-03-20T05:06:24.289+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: Getting attr on 3:8d775aef:::smithi03341006-38:head
2021-03-20T05:06:24.289+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: not in cache 3:8d775aef:::smithi03341006-38:head
2021-03-20T05:06:24.289+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) stat 3.11s3_head 3#3:8d775aef:::smithi03341006-38:head#
2021-03-20T05:06:24.289+0000 7f29a5cd4700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: found on disk, size 847872
2021-03-20T05:06:24.289+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) getattr 3.11s3_head 3#3:8d775aef:::smithi03341006-38:head# hinfo_key
2021-03-20T05:06:24.289+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) getattr 3.11s3_head 3#3:8d775aef:::smithi03341006-38:head# hinfo_key = 0
2021-03-20T05:06:24.289+0000 7f29a5cd4700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.74:6818/34910,v1:172.21.15.74:6819/34910] -- MOSDECSubOpReadReply(3.11s1 113/112 ECSubReadReply(tid=90, attrs_read=0)) v2 -- 0x556325563600 con 0x556316bb9800
2021-03-20T05:06:24.293+0000 7f29c7dda700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.4 v2:172.21.15.74:6818/34910 377 ==== MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=91, to_read={3:8d775aef:::smithi03341006-38:head=786432,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3 ==== 241+0+0 (crc 0 0 0) 0x556325563600 con 0x556316bb9800
2021-03-20T05:06:24.293+0000 7f29c7dda700 15 osd.11 113 enqueue_op 0x5563259baf20 prio 127 type 110 cost 0 latency 0.000018 epoch 113 MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=91, to_read={3:8d775aef:::smithi03341006-38:head=786432,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3
2021-03-20T05:06:24.293+0000 7f29c7dda700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=91, to_read={3:8d775aef:::smithi03341006-38:head=786432,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113)
2021-03-20T05:06:24.293+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:24.293+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=91, to_read={3:8d775aef:::smithi03341006-38:head=786432,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113) queued
2021-03-20T05:06:24.293+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=91, to_read={3:8d775aef:::smithi03341006-38:head=786432,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113)> waiting <> waiting_peering {}
2021-03-20T05:06:24.293+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGOpItem(op=MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=91, to_read={3:8d775aef:::smithi03341006-38:head=786432,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3) prio 127 cost 0 e113) pg 0x5563156cb000
2021-03-20T05:06:24.293+0000 7f29a9cdc700 10 osd.11 113 dequeue_op 0x5563259baf20 prio 127 cost 0 latency 0.000069 MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=91, to_read={3:8d775aef:::smithi03341006-38:head=786432,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3 pg pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]]
2021-03-20T05:06:24.293+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] _handle_message: MOSDECSubOpRead(3.11s3 113/112 ECSubRead(tid=91, to_read={3:8d775aef:::smithi03341006-38:head=786432,262144,0}, subchunks={3:8d775aef:::smithi03341006-38:head=[0,1]}, attrs_to_read=)) v3
2021-03-20T05:06:24.293+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) read 3.11s3_head 3#3:8d775aef:::smithi03341006-38:head# 0xc0000~40000
2021-03-20T05:06:24.294+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) read 3.11s3_head 3#3:8d775aef:::smithi03341006-38:head# 0xc0000~40000 = 61440
2021-03-20T05:06:24.294+0000 7f29a9cdc700 20 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] handle_sub_read read request=262144 r=61440 len=61440
2021-03-20T05:06:24.294+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: Getting attr on 3:8d775aef:::smithi03341006-38:head
2021-03-20T05:06:24.294+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: not in cache 3:8d775aef:::smithi03341006-38:head
2021-03-20T05:06:24.294+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) stat 3.11s3_head 3#3:8d775aef:::smithi03341006-38:head#
2021-03-20T05:06:24.294+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] get_hash_info: found on disk, size 847872
2021-03-20T05:06:24.294+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) getattr 3.11s3_head 3#3:8d775aef:::smithi03341006-38:head# hinfo_key
2021-03-20T05:06:24.294+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) getattr 3.11s3_head 3#3:8d775aef:::smithi03341006-38:head# hinfo_key = 0
2021-03-20T05:06:24.294+0000 7f29a9cdc700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.74:6818/34910,v1:172.21.15.74:6819/34910] -- MOSDECSubOpReadReply(3.11s1 113/112 ECSubReadReply(tid=91, attrs_read=0)) v2 -- 0x5563168e9e00 con 0x556316bb9800
2021-03-20T05:06:25.086+0000 7f29b732d700 20 osd.11 114 identify_splits_and_merges 3.11s3 e113 to e114 pg_nums {27=16,109=26}
2021-03-20T05:06:25.086+0000 7f29b732d700 20 osd.11:1.consume_map consume_map 3.11s3
2021-03-20T05:06:25.086+0000 7f29b732d700 15 osd.11 114 enqueue_peering_evt 3.11s3 epoch_sent: 114 epoch_requested: 114 NullEvt
2021-03-20T05:06:25.086+0000 7f29b732d700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 114 epoch_requested: 114 NullEvt) prio 255 cost 10 e114)
2021-03-20T05:06:25.087+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:25.087+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 114 epoch_requested: 114 NullEvt) prio 255 cost 10 e114) queued
2021-03-20T05:06:25.087+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 114 epoch_requested: 114 NullEvt) prio 255 cost 10 e114)> waiting <> waiting_peering {}
2021-03-20T05:06:25.087+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 114 epoch_requested: 114 NullEvt) prio 255 cost 10 e114) pg 0x5563156cb000
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] handle_advance_map: 114
2021-03-20T05:06:25.087+0000 7f29a9cdc700 20 osd.11:1.update_pg_epoch 3.11s3 113 -> 114
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 113 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] handle_advance_map [10,4,6,0,3,9]/[10,4,6,0,3,9] -- 10/10
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] state<Started>: Started advmap
2021-03-20T05:06:25.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] new interval newup [10,4,6,0,3,9] newacting [10,4,6,0,3,9]
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] state<Started>: should_restart_peering, transitioning to Reset
2021-03-20T05:06:25.087+0000 7f29a9cdc700  5 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] exit Started/ReplicaActive/RepNotRecovering 0.990355 2 0.000056
2021-03-20T05:06:25.087+0000 7f29a9cdc700  5 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] exit Started/ReplicaActive 0.990371 0 0.000000
2021-03-20T05:06:25.087+0000 7f29a9cdc700  5 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] exit Started 2.008418 0 0.000000
2021-03-20T05:06:25.087+0000 7f29a9cdc700  5 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] enter Reset
2021-03-20T05:06:25.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=112 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] set_last_peering_reset 114
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=114 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] Clearing blocked outgoing recovery messages
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=114 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] Not blocking outgoing recovery messages
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=114 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] state<Reset>: Reset advmap
2021-03-20T05:06:25.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=114 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] new interval newup [10,4,6,0,3,9] newacting [10,4,6,0,3,9]
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=114 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] state<Reset>: should restart peering, calling start_peering_interval again
2021-03-20T05:06:25.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]/[NONE,4,NONE,0,3,9]p4(1) r=-1 lpr=114 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] set_last_peering_reset 114
2021-03-20T05:06:25.087+0000 7f29a9cdc700 15 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,112)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active+remapped mbc={} ps=[42~1,49~2]] clear_stats
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] start_peering_interval: check_new_interval output: check_new_interval interval(112-113 up [10,4,6,0,3,9](10) acting [2147483647,4,2147483647,0,3,9](4)) : acting set is too small
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=112) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]]  noting past ([109,113] all_participants=0(0),0(3),3(4),4(1),5(4),6(2),9(5),10(0),10(2),11(3) intervals=([109,110] acting 0(0),4(1),5(4),9(5),10(2),11(3)))
2021-03-20T05:06:25.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] on_new_interval
2021-03-20T05:06:25.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] on_new_interval upacting_features 0x3f01cfbb7ffdffff from [10,4,6,0,3,9]+[10,4,6,0,3,9]
2021-03-20T05:06:25.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1)
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] init_hb_stamps now []
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] on_new_interval prior_readable_until_ub 0.000000000s (mnow 108.389907837s + 0.000000000s)
2021-03-20T05:06:25.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] on_new_interval scrub_queued was 0 flags: 
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] cancel_recovery
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] clear_recovery_state
2021-03-20T05:06:25.087+0000 7f29a9cdc700 15 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] _clear_recovery_state flags: 
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) unreg_next_scrub existing-0.000000. was registered? 0
2021-03-20T05:06:25.087+0000 7f29a9cdc700  1 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 luod=0'0 crt=107'142 lcod 0'0 mlcod 0'0 active mbc={} ps=[42~1,49~2]] start_peering_interval up [10,4,6,0,3,9] -> [10,4,6,0,3,9], acting [2147483647,4,2147483647,0,3,9] -> [10,4,6,0,3,9], acting_primary 4(1) -> 10, up_primary 10(0) -> 10, role -1 -> -1, features acting 4540138303579357183 upacting 4540138303579357183
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] clear_primary_state
2021-03-20T05:06:25.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] clear_primary_state
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] release_backoffs [3:88000000::::0,3:90000000::::head)
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) discard_replica_reservations
2021-03-20T05:06:25.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] agent_stop
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] on_change
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] cancel_copy_ops
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] cancel_flush_ops
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] cancel_proxy_ops
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] cancel_manifest_ops
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) scrub_clear_state
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) clear_pgscrub_state
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) clear_scrub_reservations
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) reset_internal_state
2021-03-20T05:06:25.087+0000 7f29a9cdc700 15 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]]  PrimaryLog scrubber pg(3.11s3) _scrub_clear_state
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] on_change_cleanup
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] on_change
2021-03-20T05:06:25.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] exit NotTrimming
2021-03-20T05:06:25.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[42~1,49~2]] enter NotTrimming
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] [2147483647,4,2147483647,0,3,9] -> [10,4,6,0,3,9], acting primary 4(1) -> 10(0)
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] check_recovery_sources no source osds () went down
2021-03-20T05:06:25.087+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) set_collection_opts 3.11s3_head options 
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] plpg_on_pool_change
2021-03-20T05:06:25.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] hit_set_clear
2021-03-20T05:06:25.087+0000 7f29a9cdc700 20 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] agent_stop
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] handle_activate_map: 114
2021-03-20T05:06:25.087+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] activate_map
2021-03-20T05:06:25.087+0000 7f29a9cdc700  5 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] exit Reset 0.000333 1 0.000352
2021-03-20T05:06:25.087+0000 7f29a9cdc700  5 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] enter Started
2021-03-20T05:06:25.087+0000 7f29a9cdc700  5 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] enter Start
2021-03-20T05:06:25.087+0000 7f29a9cdc700  1 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] state<Start>: transitioning to Stray
2021-03-20T05:06:25.088+0000 7f29a9cdc700  5 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] exit Start 0.000022 0 0.000000
2021-03-20T05:06:25.088+0000 7f29a9cdc700  5 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] enter Started/Stray
2021-03-20T05:06:25.088+0000 7f29a9cdc700 20 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] activate_map: Not dirtying info: last_persisted is 113 while current is 114
2021-03-20T05:06:25.088+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 114 epoch_requested: 114 NullEvt
2021-03-20T05:06:25.088+0000 7f29a9cdc700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.109:6818/35284,v1:172.21.15.109:6820/35284] -- pg_notify2(3.11s0 3.11s0 (query:114 sent:114 3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) 3->0 ([109,113] all_participants=0(0),0(3),3(4),4(1),5(4),6(2),9(5),10(0),10(2),11(3) intervals=([109,110] acting 0(0),4(1),5(4),9(5),10(2),11(3)))) e114/114) v1 -- 0x556326562700 con 0x5563169b7c00
2021-03-20T05:06:25.088+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) queue_transactions ch 0x5563259141e0 3.11s3_head
2021-03-20T05:06:25.088+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:25.088+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:25.088+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkeys 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:25.088+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkeys 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:25.088+0000 7f29c75d9700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.10 v2:172.21.15.109:6818/35284 187 ==== pg_query2(3.11s3 3.11s3 query(info 0'0 epoch_sent 114) e114/114) v1 ==== 158+0+0 (unknown 0 0 0) 0x556326681680 con 0x5563169b7c00
2021-03-20T05:06:25.088+0000 7f29c75d9700 15 osd.11 114 enqueue_peering_evt 3.11s3 epoch_sent: 114 epoch_requested: 114 MQuery 3.11s3 from 10(0) query_epoch 114 query: query(info 0'0 epoch_sent 114)
2021-03-20T05:06:25.088+0000 7f29c75d9700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 114 epoch_requested: 114 MQuery 3.11s3 from 10(0) query_epoch 114 query: query(info 0'0 epoch_sent 114)) prio 255 cost 10 e114)
2021-03-20T05:06:25.089+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:25.089+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 114 epoch_requested: 114 MQuery 3.11s3 from 10(0) query_epoch 114 query: query(info 0'0 epoch_sent 114)) prio 255 cost 10 e114) queued
2021-03-20T05:06:25.089+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 114 epoch_requested: 114 MQuery 3.11s3 from 10(0) query_epoch 114 query: query(info 0'0 epoch_sent 114)) prio 255 cost 10 e114)> waiting <> waiting_peering {}
2021-03-20T05:06:25.089+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 114 epoch_requested: 114 MQuery 3.11s3 from 10(0) query_epoch 114 query: query(info 0'0 epoch_sent 114)) prio 255 cost 10 e114) pg 0x5563156cb000
2021-03-20T05:06:25.089+0000 7f29a5cd4700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=109/109 les/c/f=110/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 114 epoch_requested: 114 MQuery 3.11s3 from 10(0) query_epoch 114 query: query(info 0'0 epoch_sent 114)
2021-03-20T05:06:25.089+0000 7f29a5cd4700 20 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] update_history advanced history from ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114
2021-03-20T05:06:25.089+0000 7f29a5cd4700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) unreg_next_scrub existing-0.000000. was registered? 0
2021-03-20T05:06:25.089+0000 7f29a5cd4700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] sending info
2021-03-20T05:06:25.089+0000 7f29a5cd4700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.109:6818/35284,v1:172.21.15.109:6820/35284] -- pg_notify2(3.11s0 3.11s0 (query:114 sent:114 3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) 3->0 ([109,113] all_participants=0(0),0(3),3(4),4(1),5(4),6(2),9(5),10(0),10(2),11(3) intervals=([109,110] acting 0(0),4(1),5(4),9(5),10(2),11(3)))) e114/114) v1 -- 0x556328eab100 con 0x5563169b7c00
2021-03-20T05:06:25.089+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) queue_transactions ch 0x5563259141e0 3.11s3_head
2021-03-20T05:06:25.089+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:25.089+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:25.089+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkeys 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:25.089+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkeys 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:26.084+0000 7f29b732d700 20 osd.11 115 identify_splits_and_merges 3.11s3 e114 to e115 pg_nums {27=16,109=26}
2021-03-20T05:06:26.084+0000 7f29b732d700 20 osd.11:1.consume_map consume_map 3.11s3
2021-03-20T05:06:26.084+0000 7f29b732d700 15 osd.11 115 enqueue_peering_evt 3.11s3 epoch_sent: 115 epoch_requested: 115 NullEvt
2021-03-20T05:06:26.084+0000 7f29b732d700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 115 epoch_requested: 115 NullEvt) prio 255 cost 10 e115)
2021-03-20T05:06:26.085+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:26.085+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 115 epoch_requested: 115 NullEvt) prio 255 cost 10 e115) queued
2021-03-20T05:06:26.085+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 115 epoch_requested: 115 NullEvt) prio 255 cost 10 e115)> waiting <> waiting_peering {}
2021-03-20T05:06:26.085+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 115 epoch_requested: 115 NullEvt) prio 255 cost 10 e115) pg 0x5563156cb000
2021-03-20T05:06:26.085+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] handle_advance_map: 115
2021-03-20T05:06:26.085+0000 7f29a9cdc700 20 osd.11:1.update_pg_epoch 3.11s3 114 -> 115
2021-03-20T05:06:26.085+0000 7f29a9cdc700 10 osd.11 pg_epoch: 114 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] handle_advance_map [10,4,6,0,3,9]/[10,4,6,0,3,9] -- 10/10
2021-03-20T05:06:26.085+0000 7f29a9cdc700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] state<Started>: Started advmap
2021-03-20T05:06:26.085+0000 7f29a9cdc700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] check_recovery_sources no source osds () went down
2021-03-20T05:06:26.085+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) set_collection_opts 3.11s3_head options 
2021-03-20T05:06:26.085+0000 7f29a9cdc700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] plpg_on_pool_change
2021-03-20T05:06:26.085+0000 7f29a9cdc700 20 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] hit_set_clear
2021-03-20T05:06:26.085+0000 7f29a9cdc700 20 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] agent_stop
2021-03-20T05:06:26.085+0000 7f29a9cdc700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] handle_activate_map: 115
2021-03-20T05:06:26.085+0000 7f29a9cdc700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] activate_map
2021-03-20T05:06:26.085+0000 7f29a9cdc700 20 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] activate_map: Not dirtying info: last_persisted is 114 while current is 115
2021-03-20T05:06:26.085+0000 7f29a9cdc700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 115 epoch_requested: 115 NullEvt
2021-03-20T05:06:26.085+0000 7f29a9cdc700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.109:6818/35284,v1:172.21.15.109:6820/35284] -- pg_notify2(3.11s0 3.11s0 (query:115 sent:115 3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) 3->0 ([109,113] all_participants=0(0),0(3),3(4),4(1),5(4),6(2),9(5),10(0),10(2),11(3) intervals=([109,110] acting 0(0),4(1),5(4),9(5),10(2),11(3)))) e115/115) v1 -- 0x55632638a300 con 0x5563169b7c00
2021-03-20T05:06:26.104+0000 7f29c75d9700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.10 v2:172.21.15.109:6818/35284 192 ==== osd pg remove(epoch 115; pg3.11s3; ) v3 ==== 32+0+0 (unknown 0 0 0) 0x5563259c3b80 con 0x5563169b7c00
2021-03-20T05:06:26.104+0000 7f29c75d9700  7 osd.11 115 handle_fast_pg_remove osd pg remove(epoch 115; pg3.11s3; ) v3 from osd.10
2021-03-20T05:06:26.104+0000 7f29c75d9700 15 osd.11 115 enqueue_peering_evt 3.11s3 epoch_sent: 115 epoch_requested: 115 DeleteStart
2021-03-20T05:06:26.104+0000 7f29c75d9700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 115 epoch_requested: 115 DeleteStart) prio 255 cost 10 e115)
2021-03-20T05:06:26.104+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:26.104+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 115 epoch_requested: 115 DeleteStart) prio 255 cost 10 e115) queued
2021-03-20T05:06:26.104+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 115 epoch_requested: 115 DeleteStart) prio 255 cost 10 e115)> waiting <> waiting_peering {}
2021-03-20T05:06:26.104+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 115 epoch_requested: 115 DeleteStart) prio 255 cost 10 e115) pg 0x5563156cb000
2021-03-20T05:06:26.104+0000 7f29a5cd4700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 115 epoch_requested: 115 DeleteStart
2021-03-20T05:06:26.104+0000 7f29a5cd4700  5 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] exit Started/Stray 1.016539 6 0.000126
2021-03-20T05:06:26.104+0000 7f29a5cd4700  5 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] enter Started/ToDelete
2021-03-20T05:06:26.104+0000 7f29a5cd4700  5 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] enter Started/ToDelete/WaitDeleteReseved
2021-03-20T05:06:26.104+0000 7f29ad4e3700 15 osd.11 115 enqueue_peering_evt 3.11s3 epoch_sent: 115 epoch_requested: 115 DeleteReserved
2021-03-20T05:06:26.104+0000 7f29ad4e3700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 115 epoch_requested: 115 DeleteReserved) prio 255 cost 10 e115)
2021-03-20T05:06:26.104+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:26.104+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 115 epoch_requested: 115 DeleteReserved) prio 255 cost 10 e115) queued
2021-03-20T05:06:26.104+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 115 epoch_requested: 115 DeleteReserved) prio 255 cost 10 e115)> waiting <> waiting_peering {}
2021-03-20T05:06:26.104+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGPeeringEvent(epoch_sent: 115 epoch_requested: 115 DeleteReserved) prio 255 cost 10 e115) pg 0x5563156cb000
2021-03-20T05:06:26.104+0000 7f29a5cd4700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 115 epoch_requested: 115 DeleteReserved
2021-03-20T05:06:26.104+0000 7f29a5cd4700  5 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] exit Started/ToDelete/WaitDeleteReseved 0.000072 1 0.000040
2021-03-20T05:06:26.104+0000 7f29a5cd4700  5 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] enter Started/ToDelete/Deleting
2021-03-20T05:06:26.104+0000 7f29a5cd4700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] on_removal
2021-03-20T05:06:26.104+0000 7f29a5cd4700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] on_shutdown
2021-03-20T05:06:26.104+0000 7f29a5cd4700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) scrub_clear_state
2021-03-20T05:06:26.104+0000 7f29a5cd4700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) clear_pgscrub_state
2021-03-20T05:06:26.104+0000 7f29a5cd4700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) clear_scrub_reservations
2021-03-20T05:06:26.104+0000 7f29a5cd4700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) reset_internal_state
2021-03-20T05:06:26.104+0000 7f29a5cd4700 15 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]]  PrimaryLog scrubber pg(3.11s3) _scrub_clear_state
2021-03-20T05:06:26.104+0000 7f29a5cd4700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) unreg_next_scrub existing-0.000000. was registered? 0
2021-03-20T05:06:26.104+0000 7f29a5cd4700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] cancel_copy_ops
2021-03-20T05:06:26.104+0000 7f29a5cd4700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] cancel_flush_ops
2021-03-20T05:06:26.104+0000 7f29a5cd4700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] cancel_proxy_ops
2021-03-20T05:06:26.104+0000 7f29a5cd4700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] cancel_manifest_ops
2021-03-20T05:06:26.104+0000 7f29a5cd4700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] clear_backoffs 
2021-03-20T05:06:26.104+0000 7f29a5cd4700 20 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] exit NotTrimming
2021-03-20T05:06:26.104+0000 7f29a5cd4700 20 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] enter NotTrimming
2021-03-20T05:06:26.104+0000 7f29a5cd4700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] on_change
2021-03-20T05:06:26.104+0000 7f29a5cd4700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] clear_async_reads
2021-03-20T05:06:26.104+0000 7f29a5cd4700 20 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] clear_primary_state
2021-03-20T05:06:26.104+0000 7f29a5cd4700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] release_backoffs [3:88000000::::0,3:90000000::::head)
2021-03-20T05:06:26.104+0000 7f29a5cd4700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]]  scrubber pg(3.11s3) discard_replica_reservations
2021-03-20T05:06:26.104+0000 7f29a5cd4700 20 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] agent_stop
2021-03-20T05:06:26.104+0000 7f29a5cd4700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] cancel_recovery
2021-03-20T05:06:26.104+0000 7f29a5cd4700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] clear_recovery_state
2021-03-20T05:06:26.104+0000 7f29a5cd4700 15 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] _clear_recovery_state flags: 
2021-03-20T05:06:26.104+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) queue_transactions ch 0x5563259141e0 3.11s3_head
2021-03-20T05:06:26.104+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:26.104+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_setkeys 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:26.104+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkeys 3.11s3_head 3#3:88000000::::head#
2021-03-20T05:06:26.104+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) _omap_rmkeys 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:26.107+0000 7f29a9cdc700 10 osd.11 115 queue_for_pg_delete on 3.11s3 e 115
2021-03-20T05:06:26.107+0000 7f29a9cdc700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGDelete(3.11s3 e115) prio 5 cost 1048576 e115)
2021-03-20T05:06:26.107+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:26.107+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGDelete(3.11s3 e115) prio 5 cost 1048576 e115) queued
2021-03-20T05:06:26.107+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGDelete(3.11s3 e115) prio 5 cost 1048576 e115)> waiting <> waiting_peering {}
2021-03-20T05:06:26.107+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGDelete(3.11s3 e115) prio 5 cost 1048576 e115) pg 0x5563156cb000
2021-03-20T05:06:26.107+0000 7f29a9cdc700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 115 epoch_requested: 115 DeleteSome
2021-03-20T05:06:26.107+0000 7f29a9cdc700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] do_delete_work
2021-03-20T05:06:26.107+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) collection_list 3.11s3_head start GHMIN end GHMAX max 30
2021-03-20T05:06:26.107+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) collection_list 3.11s3_head start GHMIN end GHMAX max 30 = 0, ls.size() = 4, next = GHMAX
2021-03-20T05:06:26.107+0000 7f29a9cdc700 20 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] do_delete_work [3#3:88000000::::head#,3#3:8b5265c1:::smithi03341006-20:head#,3#3:8d775aef:::smithi03341006-38:4a#,3#3:8d775aef:::smithi03341006-38:head#]
2021-03-20T05:06:26.107+0000 7f29a9cdc700 20 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] do_delete_work deleting 3 objects
2021-03-20T05:06:26.107+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) queue_transactions ch 0x5563259141e0 3.11s3_head
2021-03-20T05:06:26.108+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) _remove 3.11s3_head 3#3:8b5265c1:::smithi03341006-20:head# onode 0x556326436280 txc 0x556326224e00
2021-03-20T05:06:26.108+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) _do_truncate 3.11s3_head 3#3:8b5265c1:::smithi03341006-20:head# 0x0
2021-03-20T05:06:26.108+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) _remove 3.11s3_head 3#3:8b5265c1:::smithi03341006-20:head# = 0
2021-03-20T05:06:26.108+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) _remove 3.11s3_head 3#3:8d775aef:::smithi03341006-38:4a# onode 0x55632962e500 txc 0x556326224e00
2021-03-20T05:06:26.108+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) _do_truncate 3.11s3_head 3#3:8d775aef:::smithi03341006-38:4a# 0x0
2021-03-20T05:06:26.108+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) _remove 3.11s3_head 3#3:8d775aef:::smithi03341006-38:4a# = 0
2021-03-20T05:06:26.108+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) _remove 3.11s3_head 3#3:8d775aef:::smithi03341006-38:head# onode 0x556326111400 txc 0x556326224e00
2021-03-20T05:06:26.108+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) _do_truncate 3.11s3_head 3#3:8d775aef:::smithi03341006-38:head# 0x0
2021-03-20T05:06:26.108+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) _remove 3.11s3_head 3#3:8d775aef:::smithi03341006-38:head# = 0
2021-03-20T05:06:26.113+0000 7f29a9cdc700 10 osd.11 115 queue_for_pg_delete on 3.11s3 e 115
2021-03-20T05:06:26.113+0000 7f29a9cdc700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s3 PGDelete(3.11s3 e115) prio 5 cost 1048576 e115)
2021-03-20T05:06:26.113+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <> waiting <> waiting_peering {}
2021-03-20T05:06:26.113+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGDelete(3.11s3 e115) prio 5 cost 1048576 e115) queued
2021-03-20T05:06:26.113+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s3 to_process <OpSchedulerItem(3.11s3 PGDelete(3.11s3 e115) prio 5 cost 1048576 e115)> waiting <> waiting_peering {}
2021-03-20T05:06:26.113+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s3 PGDelete(3.11s3 e115) prio 5 cost 1048576 e115) pg 0x5563156cb000
2021-03-20T05:06:26.113+0000 7f29a9cdc700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] do_peering_event: epoch_sent: 115 epoch_requested: 115 DeleteSome
2021-03-20T05:06:26.113+0000 7f29a9cdc700 10 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] do_delete_work
2021-03-20T05:06:26.113+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) collection_list 3.11s3_head start GHMAX end GHMAX max 30
2021-03-20T05:06:26.113+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) collection_list 3.11s3_head start GHMAX end GHMAX max 30 = 0, ls.size() = 0, next = GHMAX
2021-03-20T05:06:26.113+0000 7f29a9cdc700 20 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] do_delete_work []
2021-03-20T05:06:26.113+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) collection_list 3.11s3_head start GHMIN end GHMAX max 30
2021-03-20T05:06:26.113+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) collection_list 3.11s3_head start GHMAX end GHMAX max 30 = 0, ls.size() = 1, next = GHMAX
2021-03-20T05:06:26.113+0000 7f29a9cdc700  0 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] do_delete_work additional unexpected onode list (new onodes has appeared since PG removal started[3#3:88000000::::head#]
2021-03-20T05:06:26.113+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) queue_transactions ch 0x5563259141e0 3.11s3_head
2021-03-20T05:06:26.113+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) _remove 3.11s3_head 3#3:88000000::::head# onode 0x556326110f00 txc 0x556326225180
2021-03-20T05:06:26.113+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) _do_truncate 3.11s3_head 3#3:88000000::::head# 0x0
2021-03-20T05:06:26.113+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) _remove 3.11s3_head 3#3:88000000::::head# = 0
2021-03-20T05:06:26.113+0000 7f29a9cdc700 15 bluestore(/var/lib/ceph/osd/ceph-11) _remove_collection 3.11s3_head
2021-03-20T05:06:26.113+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) _osr_register_zombie 0x556328c3ac60 3.11s3_head
2021-03-20T05:06:26.113+0000 7f29a9cdc700 10 bluestore(/var/lib/ceph/osd/ceph-11) _remove_collection 3.11s3_head = 0
2021-03-20T05:06:26.114+0000 7f29a9cdc700 20 osd.11 115 try_finish_pg_delete 3.11s3 0x5563156cb000
2021-03-20T05:06:26.114+0000 7f29a9cdc700 10 osd.11:1._detach_pg 3.11s3 0x5563156cb000
2021-03-20T05:06:26.114+0000 7f29a9cdc700  5 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 DELETING pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] exit Started/ToDelete/Deleting 0.010244 2 0.000457
2021-03-20T05:06:26.114+0000 7f29a9cdc700  5 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] exit Started/ToDelete 0.010350 0 0.000000
2021-03-20T05:06:26.114+0000 7f29a9cdc700  5 osd.11 pg_epoch: 115 pg[3.11s3( v 107'142 (0'0,107'142] lb MIN local-lis/les=109/110 n=3 ec=109/27 lis/c=112/109 les/c/f=113/110/0 sis=114) [10,4,6,0,3,9]p10(0) r=-1 lpr=114 pi=[109,114)/1 crt=107'142 lcod 0'0 mlcod 0'0 unknown NOTIFY mbc={} ps=[42~1,49~2]] exit Started 1.026945 0 0.000000
2021-03-20T05:06:26.119+0000 7f29bb335700 10 bluestore(/var/lib/ceph/osd/ceph-11) _queue_reap_collection 0x5563259141e0 3.11s3_head
2021-03-20T05:06:26.119+0000 7f29bb335700 10 bluestore(/var/lib/ceph/osd/ceph-11) _reap_collections 0x5563259141e0 3.11s3_head
2021-03-20T05:06:26.119+0000 7f29bb335700 10 bluestore(/var/lib/ceph/osd/ceph-11) _reap_collections 0x5563259141e0 3.11s3_head done
2021-03-20T05:06:27.094+0000 7f29b732d700 20 osd.11:1.identify_splits_and_merges identify_splits_and_merges slot 3.11s3 has no pg and waiting_for_split 
2021-03-20T05:06:27.094+0000 7f29b732d700 20 osd.11:1.consume_map consume_map 3.11s3
2021-03-20T05:06:27.094+0000 7f29b732d700 20 osd.11:1.consume_map consume_map  3.11s3 empty, pruning
2021-03-20T05:06:35.326+0000 7f29c85db700  1 -- 172.21.15.109:0/35285 <== osd.7 v2:172.21.15.74:6830/34923 38 ==== osd_ping(ping_reply e124 up_from 18 ping_stamp 2021-03-20T05:06:35.326379+0000/118.627532959s send_stamp 118.607223511s delta_ub -0.020309417s) v5 ==== 2033+0+0 (crc 0 0 0) 0x556328e68600 con 0x5563169e0c00
2021-03-20T05:06:58.129+0000 7f29c85db700  1 -- 172.21.15.109:0/35285 <== osd.4 v2:172.21.15.74:6820/34910 44 ==== osd_ping(ping_reply e146 up_from 18 ping_stamp 2021-03-20T05:06:58.129663+0000/141.430816650s send_stamp 141.435760498s delta_ub 0.004943711s) v5 ==== 2033+0+0 (crc 0 0 0) 0x556326553080 con 0x556316a97000
2021-03-20T05:06:58.129+0000 7f29c85db700 20 osd.11 146 handle_osd_ping new stamps hbstamp(osd.4 up_from 18 peer_clock_delta [0.004356929s,0.004943711s])
2021-03-20T05:06:59.693+0000 7f29c7dda700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.6 v2:172.21.15.74:6802/34892 667 ==== pg_lease(3.as3 pg_lease(ru 151.023391724s ub 159.023803711s int 16.000000000s) e148/148) v1 ==== 58+0+0 (crc 0 0 0) 0x55632927bb00 con 0x5563169b6000
2021-03-20T05:06:59.693+0000 7f29c7dda700 15 osd.11 148 enqueue_peering_evt 3.as3 epoch_sent: 148 epoch_requested: 148 MLease epoch 148 from osd.6 pg_lease(ru 151.023391724s ub 159.023803711s int 16.000000000s)
2021-03-20T05:06:59.693+0000 7f29c7dda700 20 osd.11 op_wq(2) _enqueue OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 148 epoch_requested: 148 MLease epoch 148 from osd.6 pg_lease(ru 151.023391724s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e148)
2021-03-20T05:06:59.693+0000 7f29a54d3700 20 osd.11 op_wq(2) _process OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 148 epoch_requested: 148 MLease epoch 148 from osd.6 pg_lease(ru 151.023391724s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e148) queued
2021-03-20T05:06:59.693+0000 7f29a54d3700 20 osd.11 op_wq(2) _process 3.as3 to_process <OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 148 epoch_requested: 148 MLease epoch 148 from osd.6 pg_lease(ru 151.023391724s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e148)> waiting <> waiting_peering {}
2021-03-20T05:06:59.693+0000 7f29a54d3700 20 osd.11 op_wq(2) _process OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 148 epoch_requested: 148 MLease epoch 148 from osd.6 pg_lease(ru 151.023391724s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e148) pg 0x556316d2e000
2021-03-20T05:06:59.693+0000 7f29a54d3700 10 osd.11 pg_epoch: 148 pg[3.as3( v 144'241 (0'0,144'241] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) [6,4,1,11,8,10]p6(0) r=3 lpr=27 luod=0'0 crt=144'241 mlcod 144'241 active mbc={} ps=[60~1,69~1,6f~1]] do_peering_event: epoch_sent: 148 epoch_requested: 148 MLease epoch 148 from osd.6 pg_lease(ru 151.023391724s ub 159.023803711s int 16.000000000s)
2021-03-20T05:06:59.693+0000 7f29a54d3700 10 osd.11 pg_epoch: 148 pg[3.as3( v 144'241 (0'0,144'241] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) [6,4,1,11,8,10]p6(0) r=3 lpr=27 luod=0'0 crt=144'241 mlcod 144'241 active mbc={} ps=[60~1,69~1,6f~1]] proc_lease pg_lease(ru 151.023391724s ub 159.023803711s int 16.000000000s)
2021-03-20T05:06:59.693+0000 7f29a54d3700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.74:6802/34892,v1:172.21.15.74:6803/34892] -- pg_lease_ack(3.as0 pg_lease_ack(ruub 159.023803711s) e148/148) v1 -- 0x556316e38580 con 0x5563169b6000
2021-03-20T05:07:00.231+0000 7f29c7dda700 15 osd.11 149 enqueue_peering_evt 3.as3 epoch_sent: 149 epoch_requested: 149 MInfoRec from 6(0) info: 3.as0( v 144'241 (0'0,144'241] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)
2021-03-20T05:07:00.231+0000 7f29c7dda700 20 osd.11 op_wq(2) _enqueue OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 149 epoch_requested: 149 MInfoRec from 6(0) info: 3.as0( v 144'241 (0'0,144'241] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e149)
2021-03-20T05:07:00.231+0000 7f29a54d3700 20 osd.11 op_wq(2) _process OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 149 epoch_requested: 149 MInfoRec from 6(0) info: 3.as0( v 144'241 (0'0,144'241] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e149) queued
2021-03-20T05:07:00.231+0000 7f29a54d3700 20 osd.11 op_wq(2) _process 3.as3 to_process <OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 149 epoch_requested: 149 MInfoRec from 6(0) info: 3.as0( v 144'241 (0'0,144'241] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e149)> waiting <> waiting_peering {}
2021-03-20T05:07:00.231+0000 7f29a54d3700 20 osd.11 op_wq(2) _process OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 149 epoch_requested: 149 MInfoRec from 6(0) info: 3.as0( v 144'241 (0'0,144'241] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e149) pg 0x556316d2e000
2021-03-20T05:07:00.231+0000 7f29a54d3700 10 osd.11 pg_epoch: 149 pg[3.as3( v 144'241 (0'0,144'241] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) [6,4,1,11,8,10]p6(0) r=3 lpr=27 luod=0'0 crt=144'241 mlcod 144'241 active mbc={} ps=[60~1,69~1,6f~1]] do_peering_event: epoch_sent: 149 epoch_requested: 149 MInfoRec from 6(0) info: 3.as0( v 144'241 (0'0,144'241] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)
2021-03-20T05:07:02.251+0000 7f29c7dda700 15 osd.11 151 enqueue_peering_evt 3.as3 epoch_sent: 151 epoch_requested: 151 MInfoRec from 6(0) info: 3.as0( v 151'243 (0'0,151'243] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)
2021-03-20T05:07:02.251+0000 7f29c7dda700 20 osd.11 op_wq(2) _enqueue OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 151 epoch_requested: 151 MInfoRec from 6(0) info: 3.as0( v 151'243 (0'0,151'243] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e151)
2021-03-20T05:07:02.251+0000 7f29a94db700 20 osd.11 op_wq(2) _process OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 151 epoch_requested: 151 MInfoRec from 6(0) info: 3.as0( v 151'243 (0'0,151'243] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e151) queued
2021-03-20T05:07:02.251+0000 7f29a94db700 20 osd.11 op_wq(2) _process 3.as3 to_process <OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 151 epoch_requested: 151 MInfoRec from 6(0) info: 3.as0( v 151'243 (0'0,151'243] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e151)> waiting <> waiting_peering {}
2021-03-20T05:07:02.251+0000 7f29a94db700 20 osd.11 op_wq(2) _process OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 151 epoch_requested: 151 MInfoRec from 6(0) info: 3.as0( v 151'243 (0'0,151'243] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e151) pg 0x556316d2e000
2021-03-20T05:07:02.251+0000 7f29a94db700 10 osd.11 pg_epoch: 151 pg[3.as3( v 151'243 (0'0,151'243] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) [6,4,1,11,8,10]p6(0) r=3 lpr=27 luod=0'0 crt=151'243 mlcod 151'243 active mbc={} ps=[60~1,69~1,6f~1,74~1]] do_peering_event: epoch_sent: 151 epoch_requested: 151 MInfoRec from 6(0) info: 3.as0( v 151'243 (0'0,151'243] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)
2021-03-20T05:07:03.082+0000 7f29c7dda700 15 osd.11 151 enqueue_peering_evt 3.as3 epoch_sent: 152 epoch_requested: 152 MInfoRec from 6(0) info: 3.as0( v 151'243 (0'0,151'243] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)
2021-03-20T05:07:03.082+0000 7f29c7dda700 20 osd.11 op_wq(2) _enqueue OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 152 epoch_requested: 152 MInfoRec from 6(0) info: 3.as0( v 151'243 (0'0,151'243] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e152)
2021-03-20T05:07:03.082+0000 7f29a54d3700 20 osd.11 op_wq(2) _process OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 152 epoch_requested: 152 MInfoRec from 6(0) info: 3.as0( v 151'243 (0'0,151'243] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e152) queued
2021-03-20T05:07:03.082+0000 7f29a54d3700 20 osd.11 op_wq(2) _process 3.as3 to_process <OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 152 epoch_requested: 152 MInfoRec from 6(0) info: 3.as0( v 151'243 (0'0,151'243] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e152)> waiting <> waiting_peering {}
2021-03-20T05:07:03.082+0000 7f29a54d3700 20 osd.11 op_wq(2) _process OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 152 epoch_requested: 152 MInfoRec from 6(0) info: 3.as0( v 151'243 (0'0,151'243] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e152) pg 0x556316d2e000
2021-03-20T05:07:03.082+0000 7f29a54d3700 20 osd.11 op_wq _add_slot_waiter 3.as3 peering, item epoch is 152, will wait on OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 152 epoch_requested: 152 MInfoRec from 6(0) info: 3.as0( v 151'243 (0'0,151'243] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e152)
2021-03-20T05:07:03.082+0000 7f29b732d700 20 osd.11:2._wake_pg_slot _wake_pg_slot 3.as3 to_process <> waiting <> waiting_peering {152=<OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 152 epoch_requested: 152 MInfoRec from 6(0) info: 3.as0( v 151'243 (0'0,151'243] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e152)>}
2021-03-20T05:07:03.082+0000 7f29a54d3700 20 osd.11 op_wq(2) _process OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 152 epoch_requested: 152 MInfoRec from 6(0) info: 3.as0( v 151'243 (0'0,151'243] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e152) queued
2021-03-20T05:07:03.082+0000 7f29a54d3700 20 osd.11 op_wq(2) _process 3.as3 to_process <OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 152 epoch_requested: 152 MInfoRec from 6(0) info: 3.as0( v 151'243 (0'0,151'243] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e152)> waiting <> waiting_peering {}
2021-03-20T05:07:03.082+0000 7f29a54d3700 20 osd.11 op_wq(2) _process OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 152 epoch_requested: 152 MInfoRec from 6(0) info: 3.as0( v 151'243 (0'0,151'243] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e152) pg 0x556316d2e000
2021-03-20T05:07:03.082+0000 7f29a54d3700 10 osd.11 pg_epoch: 152 pg[3.as3( v 151'243 (0'0,151'243] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) [6,4,1,11,8,10]p6(0) r=3 lpr=27 luod=0'0 crt=151'243 mlcod 151'243 active mbc={} ps=[60~1,69~1,6f~1,72~1,74~1]] do_peering_event: epoch_sent: 152 epoch_requested: 152 MInfoRec from 6(0) info: 3.as0( v 151'243 (0'0,151'243] local-lis/les=27/28 n=14 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)
2021-03-20T05:07:04.122+0000 7f29c7dda700 15 osd.11 153 enqueue_peering_evt 3.as3 epoch_sent: 153 epoch_requested: 153 MInfoRec from 6(0) info: 3.as0( v 153'249 (0'0,153'249] local-lis/les=27/28 n=13 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)
2021-03-20T05:07:04.122+0000 7f29c7dda700 20 osd.11 op_wq(2) _enqueue OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 153 epoch_requested: 153 MInfoRec from 6(0) info: 3.as0( v 153'249 (0'0,153'249] local-lis/les=27/28 n=13 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e153)
2021-03-20T05:07:04.122+0000 7f29a94db700 20 osd.11 op_wq(2) _process OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 153 epoch_requested: 153 MInfoRec from 6(0) info: 3.as0( v 153'249 (0'0,153'249] local-lis/les=27/28 n=13 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e153) queued
2021-03-20T05:07:04.122+0000 7f29a94db700 20 osd.11 op_wq(2) _process 3.as3 to_process <OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 153 epoch_requested: 153 MInfoRec from 6(0) info: 3.as0( v 153'249 (0'0,153'249] local-lis/les=27/28 n=13 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e153)> waiting <> waiting_peering {}
2021-03-20T05:07:04.122+0000 7f29a94db700 20 osd.11 op_wq(2) _process OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 153 epoch_requested: 153 MInfoRec from 6(0) info: 3.as0( v 153'249 (0'0,153'249] local-lis/les=27/28 n=13 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e153) pg 0x556316d2e000
2021-03-20T05:07:04.122+0000 7f29a94db700 10 osd.11 pg_epoch: 153 pg[3.as3( v 153'249 (0'0,153'249] local-lis/les=27/28 n=13 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) [6,4,1,11,8,10]p6(0) r=3 lpr=27 luod=0'0 crt=153'249 mlcod 153'249 active mbc={} ps=[72~1,74~1]] do_peering_event: epoch_sent: 153 epoch_requested: 153 MInfoRec from 6(0) info: 3.as0( v 153'249 (0'0,153'249] local-lis/les=27/28 n=13 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)
2021-03-20T05:07:07.109+0000 7f29c7dda700 15 osd.11 156 enqueue_peering_evt 3.as3 epoch_sent: 156 epoch_requested: 156 MInfoRec from 6(0) info: 3.as0( v 153'249 (0'0,153'249] local-lis/les=27/28 n=13 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)
2021-03-20T05:07:07.109+0000 7f29c7dda700 20 osd.11 op_wq(2) _enqueue OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 156 epoch_requested: 156 MInfoRec from 6(0) info: 3.as0( v 153'249 (0'0,153'249] local-lis/les=27/28 n=13 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e156)
2021-03-20T05:07:07.109+0000 7f29a94db700 20 osd.11 op_wq(2) _process OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 156 epoch_requested: 156 MInfoRec from 6(0) info: 3.as0( v 153'249 (0'0,153'249] local-lis/les=27/28 n=13 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e156) queued
2021-03-20T05:07:07.109+0000 7f29a94db700 20 osd.11 op_wq(2) _process 3.as3 to_process <OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 156 epoch_requested: 156 MInfoRec from 6(0) info: 3.as0( v 153'249 (0'0,153'249] local-lis/les=27/28 n=13 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e156)> waiting <> waiting_peering {}
2021-03-20T05:07:07.109+0000 7f29a94db700 20 osd.11 op_wq(2) _process OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 156 epoch_requested: 156 MInfoRec from 6(0) info: 3.as0( v 153'249 (0'0,153'249] local-lis/les=27/28 n=13 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)) prio 255 cost 10 e156) pg 0x556316d2e000
2021-03-20T05:07:07.109+0000 7f29a94db700 10 osd.11 pg_epoch: 156 pg[3.as3( v 153'249 (0'0,153'249] local-lis/les=27/28 n=13 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) [6,4,1,11,8,10]p6(0) r=3 lpr=27 luod=0'0 crt=153'249 mlcod 153'249 active mbc={} ps=[64~1,72~1,74~1]] do_peering_event: epoch_sent: 156 epoch_requested: 156 MInfoRec from 6(0) info: 3.as0( v 153'249 (0'0,153'249] local-lis/les=27/28 n=13 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) pg_lease(ru 159.023803711s ub 159.023803711s int 16.000000000s)
2021-03-20T05:07:07.693+0000 7f29c7dda700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.6 v2:172.21.15.74:6802/34892 700 ==== pg_lease(3.as3 pg_lease(ru 159.023803711s ub 167.024139404s int 16.000000000s) e156/156) v1 ==== 58+0+0 (crc 0 0 0) 0x55632a393c80 con 0x5563169b6000
2021-03-20T05:07:07.693+0000 7f29c7dda700 15 osd.11 156 enqueue_peering_evt 3.as3 epoch_sent: 156 epoch_requested: 156 MLease epoch 156 from osd.6 pg_lease(ru 159.023803711s ub 167.024139404s int 16.000000000s)
2021-03-20T05:07:07.693+0000 7f29c7dda700 20 osd.11 op_wq(2) _enqueue OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 156 epoch_requested: 156 MLease epoch 156 from osd.6 pg_lease(ru 159.023803711s ub 167.024139404s int 16.000000000s)) prio 255 cost 10 e156)
2021-03-20T05:07:07.693+0000 7f29a94db700 20 osd.11 op_wq(2) _process OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 156 epoch_requested: 156 MLease epoch 156 from osd.6 pg_lease(ru 159.023803711s ub 167.024139404s int 16.000000000s)) prio 255 cost 10 e156) queued
2021-03-20T05:07:07.693+0000 7f29a94db700 20 osd.11 op_wq(2) _process 3.as3 to_process <OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 156 epoch_requested: 156 MLease epoch 156 from osd.6 pg_lease(ru 159.023803711s ub 167.024139404s int 16.000000000s)) prio 255 cost 10 e156)> waiting <> waiting_peering {}
2021-03-20T05:07:07.693+0000 7f29a94db700 20 osd.11 op_wq(2) _process OpSchedulerItem(3.as3 PGPeeringEvent(epoch_sent: 156 epoch_requested: 156 MLease epoch 156 from osd.6 pg_lease(ru 159.023803711s ub 167.024139404s int 16.000000000s)) prio 255 cost 10 e156) pg 0x556316d2e000
2021-03-20T05:07:07.693+0000 7f29a94db700 10 osd.11 pg_epoch: 156 pg[3.as3( v 153'249 (0'0,153'249] local-lis/les=27/28 n=13 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) [6,4,1,11,8,10]p6(0) r=3 lpr=27 luod=0'0 crt=153'249 mlcod 153'249 active mbc={} ps=[64~1,72~1,74~1,76~1]] do_peering_event: epoch_sent: 156 epoch_requested: 156 MLease epoch 156 from osd.6 pg_lease(ru 159.023803711s ub 167.024139404s int 16.000000000s)
2021-03-20T05:07:07.693+0000 7f29a94db700 10 osd.11 pg_epoch: 156 pg[3.as3( v 153'249 (0'0,153'249] local-lis/les=27/28 n=13 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) [6,4,1,11,8,10]p6(0) r=3 lpr=27 luod=0'0 crt=153'249 mlcod 153'249 active mbc={} ps=[64~1,72~1,74~1,76~1]] proc_lease pg_lease(ru 159.023803711s ub 167.024139404s int 16.000000000s)
2021-03-20T05:07:09.106+0000 7f29b7b2e700 20 bluestore(/var/lib/ceph/osd/ceph-11) _kv_sync_thread committed 1 cleaned 0 in 0.000753906s (0.000000095s flush + 0.000753811s kv commit)
2021-03-20T05:07:11.531+0000 7f29c85db700  1 -- 172.21.15.109:0/35285 <== osd.7 v2:172.21.15.74:6830/34923 48 ==== osd_ping(ping_reply e160 up_from 18 ping_stamp 2021-03-20T05:07:11.531691+0000/154.832839966s send_stamp 154.812591553s delta_ub -0.020253211s) v5 ==== 2033+0+0 (crc 0 0 0) 0x556328ca1b00 con 0x5563169e0c00
2021-03-20T05:07:11.531+0000 7f29c85db700 20 osd.11 160 handle_osd_ping new stamps hbstamp(osd.7 up_from 18 peer_clock_delta [-0.021026958s,-0.020253211s])
2021-03-20T05:07:11.549+0000 7f29c85db700  1 -- [v2:172.21.15.109:6827/35285,v1:172.21.15.109:6829/35285] <== osd.7 172.21.15.74:0/34923 42 ==== osd_ping(ping e160 up_from 18 ping_stamp 2021-03-20T05:07:11.548998+0000/154.829421997s send_stamp 154.829421997s delta_ub -0.020253211s) v5 ==== 2033+0+0 (crc 0 0 0) 0x556328ca1b00 con 0x556316b6e400
2021-03-20T05:07:11.549+0000 7f29c85db700 20 osd.11 160 handle_osd_ping new stamps hbstamp(osd.7 up_from 18 peer_clock_delta [-0.021284657s,-0.020253211s])
2021-03-20T05:07:11.549+0000 7f29c7dda700  1 -- [v2:172.21.15.109:6830/35285,v1:172.21.15.109:6831/35285] <== osd.7 172.21.15.74:0/34923 42 ==== osd_ping(ping e160 up_from 18 ping_stamp 2021-03-20T05:07:11.548998+0000/154.829421997s send_stamp 154.829421997s delta_ub -0.020253211s) v5 ==== 2033+0+0 (crc 0 0 0) 0x556329960d80 con 0x556316b15000
2021-03-20T05:07:11.549+0000 7f29c7dda700 20 osd.11 160 handle_osd_ping new stamps hbstamp(osd.7 up_from 18 peer_clock_delta [-0.021327237s,-0.020253211s])
2021-03-20T05:07:18.120+0000 7f29aa4dd700  1 -- 172.21.15.109:0/35285 --> [v2:172.21.15.109:6816/35284,v1:172.21.15.109:6817/35284] -- osd_op(unknown.0.18:11 3.11s0 3:8b5265c1:::smithi03341006-20:head [copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+known_if_redirected e167) v8 -- 0x55632650c000 con 0x556316bb8c00
2021-03-20T05:07:22.032+0000 7f29c75d9700  1 -- 172.21.15.109:0/35285 <== osd.1 v2:172.21.15.33:6804/35148 20 ==== osd_ping(ping_reply e170 up_from 17 ping_stamp 2021-03-20T05:07:22.033429+0000/165.334579468s send_stamp 165.584716797s delta_ub 0.250143111s) v5 ==== 2033+0+0 (crc 0 0 0) 0x55632564cf00 con 0x556316ddc800
2021-03-20T05:07:22.032+0000 7f29c75d9700 20 osd.11 170 handle_osd_ping new stamps hbstamp(osd.1 up_from 17 peer_clock_delta [0.249825329s,0.250143111s])
2021-03-20T05:07:25.430+0000 7f29c75d9700  1 -- [v2:172.21.15.109:6827/35285,v1:172.21.15.109:6829/35285] --> 172.21.15.74:0/34910 -- osd_ping(ping_reply e174 up_from 18 ping_stamp 2021-03-20T05:07:25.430357+0000/168.736175537s send_stamp 168.732223511s delta_ub -0.003940225s) v5 -- 0x556323181080 con 0x556325e67400
2021-03-20T05:07:33.134+0000 7f29c75d9700  1 -- 172.21.15.109:0/35285 <== osd.4 v2:172.21.15.74:6823/34910 54 ==== osd_ping(ping_reply e182 up_from 18 ping_stamp 2021-03-20T05:07:33.134830+0000/176.435989380s send_stamp 176.440765381s delta_ub 0.004783311s) v5 ==== 2033+0+0 (crc 0 0 0) 0x55632a393500 con 0x556316a96c00
2021-03-20T05:07:33.134+0000 7f29c75d9700 20 osd.11 182 handle_osd_ping new stamps hbstamp(osd.4 up_from 18 peer_clock_delta [0.004407435s,0.004783311s])
2021-03-20T05:07:46.430+0000 7f29c85db700  1 -- [v2:172.21.15.109:6830/35285,v1:172.21.15.109:6831/35285] --> 172.21.15.109:0/35284 -- osd_ping(ping_reply e195 up_from 18 ping_stamp 2021-03-20T05:07:46.430695+0000/189.739700317s send_stamp 189.732223511s delta_ub -0.007482170s) v5 -- 0x5563266a8c00 con 0x556316abc400
2021-03-20T05:08:23.735+0000 7f29c85db700  1 -- [v2:172.21.15.109:6830/35285,v1:172.21.15.109:6831/35285] <== osd.10 172.21.15.109:0/35284 28 ==== osd_ping(ping e231 up_from 18 ping_stamp 2021-03-20T05:08:23.735709+0000/227.044723511s send_stamp 227.044723511s delta_ub 0.008201444s) v5 ==== 2033+0+0 (crc 0 0 0) 0x556329477980 con 0x556316abc400
2021-03-20T05:08:23.735+0000 7f29c85db700  1 -- [v2:172.21.15.109:6830/35285,v1:172.21.15.109:6831/35285] --> 172.21.15.109:0/35284 -- osd_ping(ping_reply e231 up_from 18 ping_stamp 2021-03-20T05:08:23.735709+0000/227.044723511s send_stamp 227.037414551s delta_ub -0.007298081s) v5 -- 0x556326279380 con 0x556316abc400
2021-03-20T05:08:23.735+0000 7f29c85db700  1 -- [v2:172.21.15.109:6827/35285,v1:172.21.15.109:6829/35285] <== osd.10 172.21.15.109:0/35284 28 ==== osd_ping(ping e231 up_from 18 ping_stamp 2021-03-20T05:08:23.735709+0000/227.044723511s send_stamp 227.044723511s delta_ub 0.008201444s) v5 ==== 2033+0+0 (crc 0 0 0) 0x556326279380 con 0x5563297af000
2021-03-20T05:08:23.735+0000 7f29c85db700  1 -- [v2:172.21.15.109:6827/35285,v1:172.21.15.109:6829/35285] --> 172.21.15.109:0/35284 -- osd_ping(ping_reply e231 up_from 18 ping_stamp 2021-03-20T05:08:23.735709+0000/227.044723511s send_stamp 227.037689209s delta_ub -0.007025901s) v5 -- 0x556329477980 con 0x5563297af000
2021-03-20T05:08:38.139+0000 7f29b7b2e700 20 bluestore(/var/lib/ceph/osd/ceph-11) _kv_sync_thread committed 1 cleaned 0 in 0.000953711s (0.000000051s flush + 0.000953660s kv commit)
2021-03-20T05:09:09.849+0000 7f29c85db700  1 -- 172.21.15.109:0/35285 <== osd.6 v2:172.21.15.74:6804/34892 61 ==== osd_ping(ping_reply e277 up_from 18 ping_stamp 2021-03-20T05:09:09.849454+0000/273.150604248s send_stamp 273.180053711s delta_ub 0.029436674s) v5 ==== 2033+0+0 (crc 0 0 0) 0x5563259eb380 con 0x556325bba400
2021-03-20T05:09:18.124+0000 7f29b7b2e700 20 bluestore(/var/lib/ceph/osd/ceph-11) _kv_sync_thread committed 3 cleaned 0 in 0.004443711s (0.000007353s flush + 0.004436358s kv commit)
2021-03-20T05:09:18.138+0000 7f29c85db700  1 -- [v2:172.21.15.109:6830/35285,v1:172.21.15.109:6831/35285] --> 172.21.15.74:0/34893 -- osd_ping(ping_reply e286 up_from 18 ping_stamp 2021-03-20T05:09:18.138856+0000/281.454956055s send_stamp 281.440643311s delta_ub -0.014325837s) v5 -- 0x556323396d80 con 0x5563298e7000
2021-03-20T05:09:26.235+0000 7f29b7b2e700 20 bluestore(/var/lib/ceph/osd/ceph-11) _kv_sync_thread committed 1 cleaned 0 in 0.000903815s (0.000000104s flush + 0.000903711s kv commit)
2021-03-20T05:09:31.153+0000 7f29b7b2e700 20 bluestore(/var/lib/ceph/osd/ceph-11) _kv_sync_thread committed 4 cleaned 0 in 0.000793694s (0.000000083s flush + 0.000793611s kv commit)
2021-03-20T05:09:54.076+0000 7f29c7dda700  1 -- [v2:172.21.15.109:6830/35285,v1:172.21.15.109:6831/35285] --> 172.21.15.74:0/34923 -- osd_ping(ping_reply e321 up_from 18 ping_stamp 2021-03-20T05:09:54.076530+0000/317.356964111s send_stamp 317.378143311s delta_ub 0.021186609s) v5 -- 0x556316c4a600 con 0x556316b15000
2021-03-20T05:10:06.260+0000 7f29b7b2e700 20 bluestore(/var/lib/ceph/osd/ceph-11) _kv_sync_thread committed 4 cleaned 0 in 0.000843511s (0.000000090s flush + 0.000843421s kv commit)
2021-03-20T05:10:07.359+0000 7f29c85db700  1 -- 172.21.15.109:0/35285 <== osd.6 v2:172.21.15.74:6804/34892 80 ==== osd_ping(ping_reply e335 up_from 18 ping_stamp 2021-03-20T05:10:07.360053+0000/330.661193848s send_stamp 330.690643311s delta_ub 0.029439000s) v5 ==== 2033+0+0 (crc 0 0 0) 0x556323395980 con 0x556325bba400
2021-03-20T05:10:16.314+0000 7f29b7b2e700 20 bluestore(/var/lib/ceph/osd/ceph-11) _kv_sync_thread committed 1 cleaned 0 in 0.001093798s (0.000000087s flush + 0.001093711s kv commit)
2021-03-20T05:10:29.357+0000 7f29a8cda700 20 osd.11 pg_epoch: 357 pg[1.13( empty local-lis/les=228/229 n=0 ec=226/19 lis/c=228/228 les/c/f=229/229/0 sis=228) [11,1] r=0 lpr=228 crt=0'0 mlcod 0'0 active+clean] recalc_readable_until readable_until[_ub] 360.659118652s (sent 368.659393311s)
2021-03-20T05:10:29.357+0000 7f29a8cda700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.33:6802/35148,v1:172.21.15.33:6803/35148] -- pg_lease(1.13 pg_lease(ru 360.659118652s ub 368.659393311s int 16.000000000s) e357/357) v1 -- 0x556328ca0a80 con 0x556316be8000
2021-03-20T05:10:29.357+0000 7f29c75d9700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.1 v2:172.21.15.33:6802/35148 1838 ==== pg_lease_ack(1.13 pg_lease_ack(ruub 368.659393311s) e357/357) v1 ==== 42+0+0 (crc 0 0 0) 0x5563234c6000 con 0x556316be8000
2021-03-20T05:10:29.357+0000 7f29c75d9700 15 osd.11 357 enqueue_peering_evt 1.13 epoch_sent: 357 epoch_requested: 357 MLeaseAck epoch 357 from osd.1 pg_lease_ack(ruub 368.659393311s)
2021-03-20T05:10:29.357+0000 7f29c75d9700 20 osd.11 op_wq(3) _enqueue OpSchedulerItem(1.13 PGPeeringEvent(epoch_sent: 357 epoch_requested: 357 MLeaseAck epoch 357 from osd.1 pg_lease_ack(ruub 368.659393311s)) prio 255 cost 10 e357)
2021-03-20T05:10:29.357+0000 7f29a8cda700 20 osd.11 op_wq(3) _process OpSchedulerItem(1.13 PGPeeringEvent(epoch_sent: 357 epoch_requested: 357 MLeaseAck epoch 357 from osd.1 pg_lease_ack(ruub 368.659393311s)) prio 255 cost 10 e357) queued
2021-03-20T05:10:29.357+0000 7f29a8cda700 20 osd.11 op_wq(3) _process 1.13 to_process <OpSchedulerItem(1.13 PGPeeringEvent(epoch_sent: 357 epoch_requested: 357 MLeaseAck epoch 357 from osd.1 pg_lease_ack(ruub 368.659393311s)) prio 255 cost 10 e357)> waiting <> waiting_peering {}
2021-03-20T05:10:29.357+0000 7f29a8cda700 20 osd.11 op_wq(3) _process OpSchedulerItem(1.13 PGPeeringEvent(epoch_sent: 357 epoch_requested: 357 MLeaseAck epoch 357 from osd.1 pg_lease_ack(ruub 368.659393311s)) prio 255 cost 10 e357) pg 0x556331b22000
2021-03-20T05:10:29.357+0000 7f29a8cda700 10 osd.11 pg_epoch: 357 pg[1.13( empty local-lis/les=228/229 n=0 ec=226/19 lis/c=228/228 les/c/f=229/229/0 sis=228) [11,1] r=0 lpr=228 crt=0'0 mlcod 0'0 active+clean] do_peering_event: epoch_sent: 357 epoch_requested: 357 MLeaseAck epoch 357 from osd.1 pg_lease_ack(ruub 368.659393311s)
2021-03-20T05:10:29.357+0000 7f29a8cda700 20 osd.11 pg_epoch: 357 pg[1.13( empty local-lis/les=228/229 n=0 ec=226/19 lis/c=228/228 les/c/f=229/229/0 sis=228) [11,1] r=0 lpr=228 crt=0'0 mlcod 0'0 active+clean] recalc_readable_until peer osd.1 ruub 368.659393311s
2021-03-20T05:10:29.357+0000 7f29a8cda700 20 osd.11 pg_epoch: 357 pg[1.13( empty local-lis/les=228/229 n=0 ec=226/19 lis/c=228/228 les/c/f=229/229/0 sis=228) [11,1] r=0 lpr=228 crt=0'0 mlcod 0'0 active+clean] recalc_readable_until readable_until[_ub] 368.659393311s (sent 368.659393311s)
2021-03-20T05:10:37.357+0000 7f29a4cd2700 20 osd.11 pg_epoch: 365 pg[1.13( empty local-lis/les=228/229 n=0 ec=226/19 lis/c=228/228 les/c/f=229/229/0 sis=228) [11,1] r=0 lpr=228 crt=0'0 mlcod 0'0 active+clean] recalc_readable_until peer osd.1 ruub 368.659393311s
2021-03-20T05:10:37.357+0000 7f29a4cd2700 20 osd.11 pg_epoch: 365 pg[1.13( empty local-lis/les=228/229 n=0 ec=226/19 lis/c=228/228 les/c/f=229/229/0 sis=228) [11,1] r=0 lpr=228 crt=0'0 mlcod 0'0 active+clean] recalc_readable_until readable_until[_ub] 368.659393311s (sent 376.659576416s)
2021-03-20T05:10:37.357+0000 7f29a4cd2700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] --> [v2:172.21.15.33:6802/35148,v1:172.21.15.33:6803/35148] -- pg_lease(1.13 pg_lease(ru 368.659393311s ub 376.659576416s int 16.000000000s) e365/365) v1 -- 0x556328ca1e00 con 0x556316be8000
2021-03-20T05:10:39.151+0000 7f29b7b2e700 20 bluestore(/var/lib/ceph/osd/ceph-11) _kv_sync_thread committed 1 cleaned 0 in 0.001113111s (0.000000132s flush + 0.001112979s kv commit)
2021-03-20T05:10:47.144+0000 7f29c7dda700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.9 v2:172.21.15.109:6804/57610 4 ==== pg_notify2(3.11s0 3.11s0 (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) e375/375) v1 ==== 1179+0+0 (crc 0 0 0) 0x556326726300 con 0x55633ee15000
2021-03-20T05:10:47.144+0000 7f29c7dda700 15 osd.11 374 enqueue_peering_evt 3.11s0 epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info
2021-03-20T05:10:47.144+0000 7f29c7dda700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)
2021-03-20T05:10:47.144+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s0 (new) to_process <> waiting <> waiting_peering {}
2021-03-20T05:10:47.144+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375) queued
2021-03-20T05:10:47.144+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s0 to_process <OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)> waiting <> waiting_peering {}
2021-03-20T05:10:47.144+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375) pg 0
2021-03-20T05:10:47.144+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s0 map 375 > 374
2021-03-20T05:10:47.144+0000 7f29a9cdc700 20 osd.11 op_wq _add_slot_waiter 3.11s0 peering, item epoch is 375, will wait on OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)
2021-03-20T05:10:47.147+0000 7f29c85db700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.4 v2:172.21.15.74:6818/34910 2490 ==== pg_notify2(3.11s0 3.11s0 (query:375 sent:375 3.11s1( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982177734s) 1->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) e375/375) v1 ==== 1327+0+0 (crc 0 0 0) 0x5563234be700 con 0x556316bb9800
2021-03-20T05:10:47.147+0000 7f29c85db700 15 osd.11 374 enqueue_peering_evt 3.11s0 epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 4(1) notify: (query:375 sent:375 3.11s1( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982177734s) 1->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info
2021-03-20T05:10:47.147+0000 7f29c85db700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 4(1) notify: (query:375 sent:375 3.11s1( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982177734s) 1->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)
2021-03-20T05:10:47.147+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s0 to_process <> waiting <> waiting_peering {375=<OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)>}
2021-03-20T05:10:47.147+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 4(1) notify: (query:375 sent:375 3.11s1( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982177734s) 1->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375) queued
2021-03-20T05:10:47.147+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s0 to_process <OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 4(1) notify: (query:375 sent:375 3.11s1( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982177734s) 1->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)> waiting <> waiting_peering {375=<OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)>}
2021-03-20T05:10:47.147+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 4(1) notify: (query:375 sent:375 3.11s1( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982177734s) 1->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375) pg 0
2021-03-20T05:10:47.147+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s0 map 375 > 374
2021-03-20T05:10:47.147+0000 7f29a9cdc700 20 osd.11 op_wq _add_slot_waiter 3.11s0 peering, item epoch is 375, will wait on OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 4(1) notify: (query:375 sent:375 3.11s1( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982177734s) 1->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)
2021-03-20T05:10:47.147+0000 7f29c7dda700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.6 v2:172.21.15.74:6802/34892 2125 ==== pg_notify2(3.11s0 3.11s0 (query:375 sent:375 3.11s2( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982148170s) 2->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) e375/375) v1 ==== 1327+0+0 (crc 0 0 0) 0x55632911e000 con 0x5563169b6000
2021-03-20T05:10:47.147+0000 7f29c7dda700 15 osd.11 374 enqueue_peering_evt 3.11s0 epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 6(2) notify: (query:375 sent:375 3.11s2( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982148170s) 2->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info
2021-03-20T05:10:47.147+0000 7f29c7dda700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 6(2) notify: (query:375 sent:375 3.11s2( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982148170s) 2->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)
2021-03-20T05:10:47.149+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s0 to_process <> waiting <> waiting_peering {375=<OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375),OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 4(1) notify: (query:375 sent:375 3.11s1( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982177734s) 1->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)>}
2021-03-20T05:10:47.149+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 6(2) notify: (query:375 sent:375 3.11s2( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982148170s) 2->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375) queued
2021-03-20T05:10:47.149+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s0 to_process <OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 6(2) notify: (query:375 sent:375 3.11s2( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982148170s) 2->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)> waiting <> waiting_peering {375=<OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375),OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 4(1) notify: (query:375 sent:375 3.11s1( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982177734s) 1->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)>}
2021-03-20T05:10:47.149+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 6(2) notify: (query:375 sent:375 3.11s2( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982148170s) 2->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375) pg 0
2021-03-20T05:10:47.149+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s0 map 375 > 374
2021-03-20T05:10:47.149+0000 7f29a5cd4700 20 osd.11 op_wq _add_slot_waiter 3.11s0 peering, item epoch is 375, will wait on OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 6(2) notify: (query:375 sent:375 3.11s2( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982148170s) 2->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)
2021-03-20T05:10:47.150+0000 7f29b732d700 20 osd.11:1.identify_splits_and_merges identify_splits_and_merges slot 3.11s0 has no pg and waiting_for_split 
2021-03-20T05:10:47.150+0000 7f29b732d700 20 osd.11:1.consume_map consume_map 3.11s0
2021-03-20T05:10:47.150+0000 7f29b732d700 20 osd.11:1.consume_map consume_map  3.11s0 pending_peering first epoch 375 <= 375, requeueing
2021-03-20T05:10:47.150+0000 7f29b732d700 20 osd.11:1._wake_pg_slot _wake_pg_slot 3.11s0 to_process <> waiting <> waiting_peering {375=<OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375),OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 4(1) notify: (query:375 sent:375 3.11s1( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982177734s) 1->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375),OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 6(2) notify: (query:375 sent:375 3.11s2( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982148170s) 2->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)>}
2021-03-20T05:10:47.150+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s0 to_process <> waiting <> waiting_peering {}
2021-03-20T05:10:47.150+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375) queued
2021-03-20T05:10:47.150+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s0 to_process <OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)> waiting <> waiting_peering {}
2021-03-20T05:10:47.150+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375) pg 0
2021-03-20T05:10:47.150+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s0 no pg, should create on OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)
2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 375 _make_pg 3.11s0
2021-03-20T05:10:47.151+0000 7f29a5cd4700 20 osd.11 pg_epoch: 375 pg[3.11(unlocked)]  scrubber pg(3.11s0)  creating PgScrubber for 3.11s0 / 11(0)
2021-03-20T05:10:47.151+0000 7f29a5cd4700 15 osd.11 pg_epoch: 375 pg[3.11(unlocked)]  scrubberFSM pg(3.11s0) ScrubMachine created 3.11s0
2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11(unlocked)]  scrubberFSM pg(3.11s0) -- state -->> NotActive
2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) _osr_attach 3.11s0_head fresh osr 0x5563256bb4a0
2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11s0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) []p? r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown mbc={}] init role 0 up [11,4,6,0,3,7] acting [11,4,6,0,3,7] history ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375 past_intervals ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)
2021-03-20T05:10:47.151+0000 7f29a5cd4700 20 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] on_new_interval
2021-03-20T05:10:47.151+0000 7f29a5cd4700 20 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] on_new_interval upacting_features 0x3f01cfbb7ffdffff from [11,4,6,0,3,7]+[11,4,6,0,3,7]
2021-03-20T05:10:47.151+0000 7f29a5cd4700 20 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1)
2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] init_hb_stamps now [0x556316a51420,0x556316a51500,0x5563168d98f0,0x5563168d9960,0x5563168d89a0]
2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] on_new_interval prior_readable_until_ub 0.000000000s (mnow 370.453277588s + 0.000000000s)
2021-03-20T05:10:47.151+0000 7f29a5cd4700 20 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] on_new_interval scrub_queued was 0 flags: 
2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] cancel_recovery
2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] clear_recovery_state
2021-03-20T05:10:47.151+0000 7f29a5cd4700 15 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] _clear_recovery_state flags: 
2021-03-20T05:10:47.151+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) set_collection_opts 3.11s0_head options 
2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] handle_initialize
2021-03-20T05:10:47.151+0000 7f29a5cd4700  5 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] exit Initial 0.000154 0 0.000000
2021-03-20T05:10:47.151+0000 7f29a5cd4700  5 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] enter Reset
2021-03-20T05:10:47.151+0000 7f29a5cd4700 20 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] set_last_peering_reset 375
2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] Clearing blocked outgoing recovery messages
2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] Not blocking outgoing recovery messages
2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] handle_activate_map: 375
2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] activate_map
2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] update_heartbeat_peers  -> 0,3,4,6,7,11
2021-03-20T05:10:47.151+0000 7f29a5cd4700  5 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] exit Reset 0.000040 1 0.000030
2021-03-20T05:10:47.151+0000 7f29a5cd4700  5 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] enter Started
2021-03-20T05:10:47.151+0000 7f29a5cd4700  5 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] enter Start
2021-03-20T05:10:47.151+0000 7f29a5cd4700  1 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary
2021-03-20T05:10:47.151+0000 7f29a5cd4700  5 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] exit Start 0.000009 0 0.000000
2021-03-20T05:10:47.151+0000 7f29a5cd4700  5 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] enter Started/Primary
2021-03-20T05:10:47.151+0000 7f29a5cd4700  5 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] enter Started/Primary/Peering
2021-03-20T05:10:47.151+0000 7f29a5cd4700  5 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 peering mbc={}] enter Started/Primary/Peering/GetInfo
2021-03-20T05:10:47.151+0000 7f29a5cd4700 -1 log_channel(cluster) log [ERR] : 3.11s0 past_intervals [362,375) start interval does not contain the required bound [151,375) start
2021-03-20T05:10:47.151+0000 7f29a5cd4700 -1 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 peering mbc={}] 3.11s0 past_intervals [362,375) start interval does not contain the required bound [151,375) start
2021-03-20T05:10:47.156+0000 7f29c75d9700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.7 v2:172.21.15.74:6826/34923 3094 ==== pg_notify2(3.11s0 3.11s0 (query:375 sent:375 3.11s5( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.972459793s) 5->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) e375/375) v1 ==== 1327+0+0 (crc 0 0 0) 0x5563265c5800 con 0x5563169b6400
2021-03-20T05:10:47.156+0000 7f29c75d9700 15 osd.11 375 enqueue_peering_evt 3.11s0 epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 7(5) notify: (query:375 sent:375 3.11s5( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.972459793s) 5->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info
2021-03-20T05:10:47.156+0000 7f29c75d9700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 7(5) notify: (query:375 sent:375 3.11s5( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.972459793s) 5->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)
 -3445> 2021-03-20T05:10:47.144+0000 7f29c7dda700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.9 v2:172.21.15.109:6804/57610 4 ==== pg_notify2(3.11s0 3.11s0 (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) e375/375) v1 ==== 1179+0+0 (crc 0 0 0) 0x556326726300 con 0x55633ee15000
 -3444> 2021-03-20T05:10:47.144+0000 7f29c7dda700 15 osd.11 374 enqueue_peering_evt 3.11s0 epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info
 -3443> 2021-03-20T05:10:47.144+0000 7f29c7dda700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)
 -3442> 2021-03-20T05:10:47.144+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s0 (new) to_process <> waiting <> waiting_peering {}
 -3441> 2021-03-20T05:10:47.144+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375) queued
 -3440> 2021-03-20T05:10:47.144+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s0 to_process <OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)> waiting <> waiting_peering {}
 -3439> 2021-03-20T05:10:47.144+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375) pg 0
 -3437> 2021-03-20T05:10:47.144+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s0 map 375 > 374
 -3436> 2021-03-20T05:10:47.144+0000 7f29a9cdc700 20 osd.11 op_wq _add_slot_waiter 3.11s0 peering, item epoch is 375, will wait on OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)
 -2065> 2021-03-20T05:10:47.147+0000 7f29c85db700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.4 v2:172.21.15.74:6818/34910 2490 ==== pg_notify2(3.11s0 3.11s0 (query:375 sent:375 3.11s1( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982177734s) 1->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) e375/375) v1 ==== 1327+0+0 (crc 0 0 0) 0x5563234be700 con 0x556316bb9800
 -2043> 2021-03-20T05:10:47.147+0000 7f29c85db700 15 osd.11 374 enqueue_peering_evt 3.11s0 epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 4(1) notify: (query:375 sent:375 3.11s1( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982177734s) 1->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info
 -2042> 2021-03-20T05:10:47.147+0000 7f29c85db700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 4(1) notify: (query:375 sent:375 3.11s1( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982177734s) 1->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)
 -2021> 2021-03-20T05:10:47.147+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s0 to_process <> waiting <> waiting_peering {375=<OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)>}
 -2020> 2021-03-20T05:10:47.147+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 4(1) notify: (query:375 sent:375 3.11s1( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982177734s) 1->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375) queued
 -2019> 2021-03-20T05:10:47.147+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s0 to_process <OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 4(1) notify: (query:375 sent:375 3.11s1( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982177734s) 1->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)> waiting <> waiting_peering {375=<OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)>}
 -2018> 2021-03-20T05:10:47.147+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 4(1) notify: (query:375 sent:375 3.11s1( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982177734s) 1->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375) pg 0
 -2017> 2021-03-20T05:10:47.147+0000 7f29a9cdc700 20 osd.11 op_wq(1) _process 3.11s0 map 375 > 374
 -2016> 2021-03-20T05:10:47.147+0000 7f29a9cdc700 20 osd.11 op_wq _add_slot_waiter 3.11s0 peering, item epoch is 375, will wait on OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 4(1) notify: (query:375 sent:375 3.11s1( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982177734s) 1->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)
 -1998> 2021-03-20T05:10:47.147+0000 7f29c7dda700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.6 v2:172.21.15.74:6802/34892 2125 ==== pg_notify2(3.11s0 3.11s0 (query:375 sent:375 3.11s2( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982148170s) 2->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) e375/375) v1 ==== 1327+0+0 (crc 0 0 0) 0x55632911e000 con 0x5563169b6000
 -1995> 2021-03-20T05:10:47.147+0000 7f29c7dda700 15 osd.11 374 enqueue_peering_evt 3.11s0 epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 6(2) notify: (query:375 sent:375 3.11s2( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982148170s) 2->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info
 -1994> 2021-03-20T05:10:47.147+0000 7f29c7dda700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 6(2) notify: (query:375 sent:375 3.11s2( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982148170s) 2->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)
 -1914> 2021-03-20T05:10:47.149+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s0 to_process <> waiting <> waiting_peering {375=<OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375),OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 4(1) notify: (query:375 sent:375 3.11s1( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982177734s) 1->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)>}
 -1913> 2021-03-20T05:10:47.149+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 6(2) notify: (query:375 sent:375 3.11s2( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982148170s) 2->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375) queued
 -1912> 2021-03-20T05:10:47.149+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s0 to_process <OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 6(2) notify: (query:375 sent:375 3.11s2( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982148170s) 2->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)> waiting <> waiting_peering {375=<OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375),OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 4(1) notify: (query:375 sent:375 3.11s1( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982177734s) 1->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)>}
 -1911> 2021-03-20T05:10:47.149+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 6(2) notify: (query:375 sent:375 3.11s2( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982148170s) 2->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375) pg 0
 -1910> 2021-03-20T05:10:47.149+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s0 map 375 > 374
 -1909> 2021-03-20T05:10:47.149+0000 7f29a5cd4700 20 osd.11 op_wq _add_slot_waiter 3.11s0 peering, item epoch is 375, will wait on OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 6(2) notify: (query:375 sent:375 3.11s2( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982148170s) 2->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)
 -1710> 2021-03-20T05:10:47.150+0000 7f29b732d700 20 osd.11:1.identify_splits_and_merges identify_splits_and_merges slot 3.11s0 has no pg and waiting_for_split 
 -1659> 2021-03-20T05:10:47.150+0000 7f29b732d700 20 osd.11:1.consume_map consume_map 3.11s0
 -1658> 2021-03-20T05:10:47.150+0000 7f29b732d700 20 osd.11:1.consume_map consume_map  3.11s0 pending_peering first epoch 375 <= 375, requeueing
 -1657> 2021-03-20T05:10:47.150+0000 7f29b732d700 20 osd.11:1._wake_pg_slot _wake_pg_slot 3.11s0 to_process <> waiting <> waiting_peering {375=<OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375),OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 4(1) notify: (query:375 sent:375 3.11s1( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982177734s) 1->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375),OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 6(2) notify: (query:375 sent:375 3.11s2( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.982148170s) 2->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)>}
 -1612> 2021-03-20T05:10:47.150+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s0 to_process <> waiting <> waiting_peering {}
 -1609> 2021-03-20T05:10:47.150+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375) queued
 -1606> 2021-03-20T05:10:47.150+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s0 to_process <OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)> waiting <> waiting_peering {}
 -1604> 2021-03-20T05:10:47.150+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375) pg 0
 -1597> 2021-03-20T05:10:47.150+0000 7f29a5cd4700 20 osd.11 op_wq(1) _process 3.11s0 no pg, should create on OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 9(5) notify: (query:375 sent:375 3.11s5( v 304'349 (0'0,304'349] local-lis/les=114/115 n=11 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) 5->0 ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)
 -1576> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 375 _make_pg 3.11s0
 -1513> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 20 osd.11 pg_epoch: 375 pg[3.11(unlocked)]  scrubber pg(3.11s0)  creating PgScrubber for 3.11s0 / 11(0)
 -1512> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 15 osd.11 pg_epoch: 375 pg[3.11(unlocked)]  scrubberFSM pg(3.11s0) ScrubMachine created 3.11s0
 -1511> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11(unlocked)]  scrubberFSM pg(3.11s0) -- state -->> NotActive
 -1509> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 bluestore(/var/lib/ceph/osd/ceph-11) _osr_attach 3.11s0_head fresh osr 0x5563256bb4a0
 -1499> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11s0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) []p? r=-1 lpr=0 crt=0'0 mlcod 0'0 unknown mbc={}] init role 0 up [11,4,6,0,3,7] acting [11,4,6,0,3,7] history ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375 past_intervals ([362,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=)
 -1491> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 20 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] on_new_interval
 -1486> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 20 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] on_new_interval upacting_features 0x3f01cfbb7ffdffff from [11,4,6,0,3,7]+[11,4,6,0,3,7]
 -1479> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 20 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1)
 -1473> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] init_hb_stamps now [0x556316a51420,0x556316a51500,0x5563168d98f0,0x5563168d9960,0x5563168d89a0]
 -1466> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] on_new_interval prior_readable_until_ub 0.000000000s (mnow 370.453277588s + 0.000000000s)
 -1462> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 20 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] on_new_interval scrub_queued was 0 flags: 
 -1459> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] cancel_recovery
 -1456> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] clear_recovery_state
 -1451> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 15 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] _clear_recovery_state flags: 
 -1435> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 15 bluestore(/var/lib/ceph/osd/ceph-11) set_collection_opts 3.11s0_head options 
 -1429> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] handle_initialize
 -1424> 2021-03-20T05:10:47.151+0000 7f29a5cd4700  5 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] exit Initial 0.000154 0 0.000000
 -1419> 2021-03-20T05:10:47.151+0000 7f29a5cd4700  5 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] enter Reset
 -1415> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 20 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=0 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] set_last_peering_reset 375
 -1410> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] Clearing blocked outgoing recovery messages
 -1404> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] Not blocking outgoing recovery messages
 -1401> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] handle_activate_map: 375
 -1398> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] activate_map
 -1392> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 10 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] update_heartbeat_peers  -> 0,3,4,6,7,11
 -1386> 2021-03-20T05:10:47.151+0000 7f29a5cd4700  5 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] exit Reset 0.000040 1 0.000030
 -1384> 2021-03-20T05:10:47.151+0000 7f29a5cd4700  5 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] enter Started
 -1381> 2021-03-20T05:10:47.151+0000 7f29a5cd4700  5 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] enter Start
 -1380> 2021-03-20T05:10:47.151+0000 7f29a5cd4700  1 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary
 -1377> 2021-03-20T05:10:47.151+0000 7f29a5cd4700  5 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] exit Start 0.000009 0 0.000000
 -1375> 2021-03-20T05:10:47.151+0000 7f29a5cd4700  5 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] enter Started/Primary
 -1373> 2021-03-20T05:10:47.151+0000 7f29a5cd4700  5 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 unknown mbc={}] enter Started/Primary/Peering
 -1372> 2021-03-20T05:10:47.151+0000 7f29a5cd4700  5 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 peering mbc={}] enter Started/Primary/Peering/GetInfo
 -1368> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 -1 log_channel(cluster) log [ERR] : 3.11s0 past_intervals [362,375) start interval does not contain the required bound [151,375) start
 -1363> 2021-03-20T05:10:47.151+0000 7f29a5cd4700 -1 osd.11 pg_epoch: 375 pg[3.11s0( empty local-lis/les=0/0 n=0 ec=109/27 lis/c=114/114 les/c/f=115/115/0 sis=375) [11,4,6,0,3,7]p11(0) r=0 lpr=375 pi=[362,375)/0 crt=0'0 mlcod 0'0 peering mbc={}] 3.11s0 past_intervals [362,375) start interval does not contain the required bound [151,375) start
  -468> 2021-03-20T05:10:47.156+0000 7f29c75d9700  1 -- [v2:172.21.15.109:6823/35285,v1:172.21.15.109:6825/35285] <== osd.7 v2:172.21.15.74:6826/34923 3094 ==== pg_notify2(3.11s0 3.11s0 (query:375 sent:375 3.11s5( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.972459793s) 5->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) e375/375) v1 ==== 1327+0+0 (crc 0 0 0) 0x5563265c5800 con 0x5563169b6400
  -460> 2021-03-20T05:10:47.156+0000 7f29c75d9700 15 osd.11 375 enqueue_peering_evt 3.11s0 epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 7(5) notify: (query:375 sent:375 3.11s5( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.972459793s) 5->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info
  -459> 2021-03-20T05:10:47.156+0000 7f29c75d9700 20 osd.11 op_wq(1) _enqueue OpSchedulerItem(3.11s0 PGPeeringEvent(epoch_sent: 375 epoch_requested: 375 MNotifyRec 3.11s0 from 7(5) notify: (query:375 sent:375 3.11s5( v 373'447 (0'0,373'447] local-lis/les=309/310 n=15 ec=109/27 lis/c=309/309 les/c/f=310/310/0 sis=375 pruub=14.972459793s) 5->0 ([309,374] all_participants=0(3),3(4),4(1),6(2),7(5),10(0) intervals=([309,374] acting 0(3),3(4),4(1),6(2),7(5),10(0)))) features: 0x3f01cfbb7ffdffff +create_info) prio 255 cost 10 e375)