wip-16878-test-1.log

2016-10-25 08:51:30.940709 7f210cc63700  5 osd.3 23 heartbeat: osd_stat(9515 MB used, 713 MB avail, 10228 MB total, peers [2]/[] op hist [0,0,0,0,0,0,0,0,0,0,16,1])
2016-10-25 08:51:30.940733 7f210cc63700  1 -- 167.114.251.239:0/12812 --> 167.114.251.239:6802/12736 -- osd_ping(ping e23 stamp 2016-10-25 08:51:30.940721) v2 -- 0x7f213ec77800 con 0
2016-10-25 08:51:30.940832 7f210cc63700  1 -- 167.114.251.239:0/12812 --> 167.114.251.239:6803/12736 -- osd_ping(ping e23 stamp 2016-10-25 08:51:30.940721) v2 -- 0x7f213ec78c00 con 0
2016-10-25 08:51:30.940881 7f210cc63700 25 osd.3 23 heartbeat_check osd.2 first_tx 2016-10-25 08:34:57.103177 last_tx 2016-10-25 08:51:30.940721 last_rx_back 2016-10-25 08:51:27.440394 last_rx_front 2016-10-25 08:51:27.440394
2016-10-25 08:51:30.941334 7f2128b4c700  1 -- 167.114.251.239:0/12812 <== osd.2 167.114.251.239:6803/12736 324 ==== osd_ping(ping_reply e23 stamp 2016-10-25 08:51:30.940721) v2 ==== 47+0+0 (664087836 0 0) 0x7f2139679600 con 0x7f2139591000
2016-10-25 08:51:30.941369 7f2128b4c700 25 osd.3 23 handle_osd_ping got reply from osd.2 first_tx 2016-10-25 08:34:57.103177 last_tx 2016-10-25 08:51:30.940721 last_rx_back 2016-10-25 08:51:27.440394 last_rx_front 2016-10-25 08:51:27.440394 -> 2016-10-25 08:51:30.940721
2016-10-25 08:51:30.941394 7f2128b4c700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:30.941565 7f212934d700  1 -- 167.114.251.239:0/12812 <== osd.2 167.114.251.239:6802/12736 324 ==== osd_ping(ping_reply e23 stamp 2016-10-25 08:51:30.940721) v2 ==== 47+0+0 (664087836 0 0) 0x7f2139505800 con 0x7f213958e000
2016-10-25 08:51:30.941587 7f212934d700 25 osd.3 23 handle_osd_ping got reply from osd.2 first_tx 2016-10-25 08:34:57.103177 last_tx 2016-10-25 08:51:30.940721 last_rx_back 2016-10-25 08:51:27.440394 -> 2016-10-25 08:51:30.940721 last_rx_front 2016-10-25 08:51:30.940721
2016-10-25 08:51:30.941600 7f212934d700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:30.980384 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.1_head/#1:9669e639:::benchmark_data_target167114251239.teuthology_13304_object840:head# 0~4194304 = 4194304
2016-10-25 08:51:30.980503 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.1_head/#1:9669e639:::benchmark_data_target167114251239.teuthology_13304_object840:head#
2016-10-25 08:51:30.980576 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.1_head/#1:9669e639:::benchmark_data_target167114251239.teuthology_13304_object840:head# = 0
2016-10-25 08:51:30.980615 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.1_head/#1:80000000::::head#
2016-10-25 08:51:30.980638 7f2121400700 10 filestore oid: #1:80000000::::head# not skipping op, *spos 2324.0.2
2016-10-25 08:51:30.980652 7f2121400700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:30.980727 7f2121400700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.1_head/#1:80000000::::head# = 0
2016-10-25 08:51:30.980750 7f2121400700 10 journal op_apply_finish 2324 open_ops 1 -> 0, max_applied_seq 2327 -> 2327
2016-10-25 08:51:30.980753 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f214141b0e0 seq 2324 r = 0, finisher 0x7f213ece0520 0x7f213eaeffc0
2016-10-25 08:51:30.980823 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f214141b0e0 seq 2324 osr(1.1 0x7f21394758a0)/0x7f21394758a0 lat 1.181094
2016-10-25 08:51:30.980884 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'320 lua=23'319 crt=23'318 lcod 23'319 mlcod 23'318 active+clean] op_applied: 1690
2016-10-25 08:51:30.980950 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'320 lua=23'319 crt=23'318 lcod 23'319 mlcod 23'318 active+clean] op_applied version 23'320
2016-10-25 08:51:30.980968 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'320 lua=23'320 crt=23'318 lcod 23'319 mlcod 23'318 active+clean] repop_all_applied: repop tid 1690 all applied 
2016-10-25 08:51:30.980984 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'320 lua=23'320 crt=23'318 lcod 23'319 mlcod 23'318 active+clean] eval_repop repgather(0x7f2139880fc0 23'320 rep_tid=1690 committed?=1 applied?=1) wants=d
2016-10-25 08:51:30.981001 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'320 lua=23'320 crt=23'318 lcod 23'319 mlcod 23'318 active+clean]  commit: repgather(0x7f2139880fc0 23'320 rep_tid=1690 committed?=1 applied?=1)
2016-10-25 08:51:30.981017 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'320 lua=23'320 crt=23'318 lcod 23'319 mlcod 23'318 active+clean]  applied: repgather(0x7f2139880fc0 23'320 rep_tid=1690 committed?=1 applied?=1) 
2016-10-25 08:51:30.981040 7f211e3fa700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'320 lua=23'320 crt=23'318 lcod 23'319 mlcod 23'318 active+clean] publish_stats_to_osd 23:162
2016-10-25 08:51:30.981060 7f211e3fa700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'320 lua=23'320 crt=23'318 lcod 23'319 mlcod 23'319 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:30.981078 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'320 lua=23'320 crt=23'318 lcod 23'319 mlcod 23'319 active+clean]  removing repgather(0x7f2139880fc0 23'320 rep_tid=1690 committed?=1 applied?=1)
2016-10-25 08:51:30.981094 7f211e3fa700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'320 lua=23'320 crt=23'318 lcod 23'319 mlcod 23'319 active+clean]    q front is repgather(0x7f2139880fc0 23'320 rep_tid=1690 committed?=1 applied?=1)
2016-10-25 08:51:30.981110 7f211e3fa700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'320 lua=23'320 crt=23'318 lcod 23'319 mlcod 23'319 active+clean] remove_repop repgather(0x7f2139880fc0 23'320 rep_tid=1690 committed?=1 applied?=1)
2016-10-25 08:51:30.981133 7f211e3fa700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'320 lua=23'320 crt=23'318 lcod 23'319 mlcod 23'319 active+clean]  requeue_ops 
2016-10-25 08:51:31.021897 7f2125c09700 10 osd.3 23 tick
2016-10-25 08:51:31.021929 7f2125c09700 10 osd.3 23 do_waiters -- start
2016-10-25 08:51:31.021932 7f2125c09700 10 osd.3 23 do_waiters -- finish
2016-10-25 08:51:31.278702 7f2122c03700 10 journal write_finish_thread_entry aio 853377024~4202496 done
2016-10-25 08:51:31.278724 7f2122c03700 10 journal write_finish_thread_entry aio 857579520~8404992 done
2016-10-25 08:51:31.278726 7f2122c03700 10 journal write_finish_thread_entry aio 865984512~4202496 done
2016-10-25 08:51:31.278728 7f2122c03700 10 journal write_finish_thread_entry aio 870187008~4202496 done
2016-10-25 08:51:31.278729 7f2122c03700 10 journal write_finish_thread_entry aio 874389504~4202496 done
2016-10-25 08:51:31.278730 7f2122c03700 10 journal write_finish_thread_entry aio 878592000~4202496 done
2016-10-25 08:51:31.278732 7f2122c03700 10 journal write_finish_thread_entry aio 882794496~4202496 done
2016-10-25 08:51:31.278733 7f2122c03700 10 journal write_finish_thread_entry aio 886996992~4202496 done
2016-10-25 08:51:31.278735 7f2122c03700 20 journal check_aio_completion
2016-10-25 08:51:31.278737 7f2122c03700 20 journal check_aio_completion completed seq 2328 853377024~4202496
2016-10-25 08:51:31.278747 7f2122c03700 20 journal check_aio_completion completed seq 2330 857579520~8404992
2016-10-25 08:51:31.278754 7f2122c03700 20 journal check_aio_completion completed seq 2331 865984512~4202496
2016-10-25 08:51:31.278758 7f2122c03700 20 journal check_aio_completion completed seq 2332 870187008~4202496
2016-10-25 08:51:31.278779 7f2122c03700 20 journal check_aio_completion completed seq 2333 874389504~4202496
2016-10-25 08:51:31.278783 7f2122c03700 20 journal check_aio_completion completed seq 2334 878592000~4202496
2016-10-25 08:51:31.278788 7f2122c03700 20 journal check_aio_completion completed seq 2335 882794496~4202496
2016-10-25 08:51:31.278791 7f2122c03700 20 journal check_aio_completion completed seq 2336 886996992~4202496
2016-10-25 08:51:31.278796 7f2122c03700 20 journal check_aio_completion queueing finishers through seq 2336
2016-10-25 08:51:31.278799 7f2122c03700 10 journal queue_completions_thru seq 2336 queueing seq 2328 0x7f214811fcc0 lat 1.093519
2016-10-25 08:51:31.278828 7f2122c03700 10 journal queue_completions_thru seq 2336 queueing seq 2329 0x7f214123d840 lat 1.061426
2016-10-25 08:51:31.278838 7f2122c03700 10 journal queue_completions_thru seq 2336 queueing seq 2330 0x7f214811ce70 lat 1.059571
2016-10-25 08:51:31.278846 7f2122c03700 10 journal queue_completions_thru seq 2336 queueing seq 2331 0x7f2142e4e8b0 lat 1.046322
2016-10-25 08:51:31.278854 7f2122c03700 10 journal queue_completions_thru seq 2336 queueing seq 2332 0x7f214811c720 lat 1.039445
2016-10-25 08:51:31.278862 7f2122c03700 10 journal queue_completions_thru seq 2336 queueing seq 2333 0x7f2140ed5aa0 lat 1.038452
2016-10-25 08:51:31.278870 7f2122c03700 10 journal queue_completions_thru seq 2336 queueing seq 2334 0x7f214808b4b0 lat 1.033778
2016-10-25 08:51:31.278878 7f2122c03700 10 journal queue_completions_thru seq 2336 queueing seq 2335 0x7f2140fb54a0 lat 1.029202
2016-10-25 08:51:31.278885 7f2122c03700 10 journal queue_completions_thru seq 2336 queueing seq 2336 0x7f2140ed4690 lat 1.017599
2016-10-25 08:51:31.278895 7f2122c03700 20 journal write_finish_thread_entry waiting for aio(s)
2016-10-25 08:51:31.279064 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f21397fe6a0 seq 2328 osr(1.3 0x7f2139475570) [Transaction(0x7f2147f48680)]
2016-10-25 08:51:31.279071 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f21397fe6a0 seq 2328 osr(1.3 0x7f2139475570) 4195584 bytes   (queue has 15 ops and 62933830 bytes)
2016-10-25 08:51:31.279078 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f21401ee160
2016-10-25 08:51:31.279083 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f2142ec9620 seq 2329 osr(1.3 0x7f2139475570) [Transaction(0x7f2147f06a80)]
2016-10-25 08:51:31.279086 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f2142ec9620 seq 2329 osr(1.3 0x7f2139475570) 4195584 bytes   (queue has 15 ops and 62933830 bytes)
2016-10-25 08:51:31.279091 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f2142e09580
2016-10-25 08:51:31.279093 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f2142f70300 seq 2330 osr(1.1 0x7f21394758a0) [Transaction(0x7f21415bb340)]
2016-10-25 08:51:31.279095 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f2142f70300 seq 2330 osr(1.1 0x7f21394758a0) 4195584 bytes   (queue has 15 ops and 62933830 bytes)
2016-10-25 08:51:31.279098 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f2147e84040
2016-10-25 08:51:31.279108 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f213ecd3260 seq 2331 osr(1.0 0x7f2139475d80) [Transaction(0x7f2147e34a00)]
2016-10-25 08:51:31.279111 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f213ecd3260 seq 2331 osr(1.0 0x7f2139475d80) 4195584 bytes   (queue has 15 ops and 62933830 bytes)
2016-10-25 08:51:31.279113 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f2139916900
2016-10-25 08:51:31.279118 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f214133de40 seq 2332 osr(1.7 0x7f2139474be0) [Transaction(0x7f2147e65100)]
2016-10-25 08:51:31.279121 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f214133de40 seq 2332 osr(1.7 0x7f2139474be0) 4195584 bytes   (queue has 15 ops and 62933830 bytes)
2016-10-25 08:51:31.279123 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f2140e14940
2016-10-25 08:51:31.279125 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f2142f71860 seq 2333 osr(1.4 0x7f2139475240) [Transaction(0x7f2147e55480)]
2016-10-25 08:51:31.279127 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f2142f71860 seq 2333 osr(1.4 0x7f2139475240) 4195584 bytes   (queue has 15 ops and 62933830 bytes)
2016-10-25 08:51:31.279130 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f213ecdf940
2016-10-25 08:51:31.279131 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f2143016220 seq 2334 osr(1.1 0x7f21394758a0) [Transaction(0x7f2147e6e1c0)]
2016-10-25 08:51:31.279133 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f2143016220 seq 2334 osr(1.1 0x7f21394758a0) 4195584 bytes   (queue has 15 ops and 62933830 bytes)
2016-10-25 08:51:31.279136 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f213a53c320
2016-10-25 08:51:31.279138 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f213ed5df00 seq 2335 osr(1.6 0x7f2139474f10) [Transaction(0x7f2147e62e00)]
2016-10-25 08:51:31.279140 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f213ed5df00 seq 2335 osr(1.6 0x7f2139474f10) 4195584 bytes   (queue has 15 ops and 62933830 bytes)
2016-10-25 08:51:31.279143 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f2147e84dc0
2016-10-25 08:51:31.279145 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f213ed5cee0 seq 2336 osr(1.4 0x7f2139475240) [Transaction(0x7f2147f68380)]
2016-10-25 08:51:31.279148 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f213ed5cee0 seq 2336 osr(1.4 0x7f2139475240) 4195584 bytes   (queue has 15 ops and 62933830 bytes)
2016-10-25 08:51:31.279152 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f2148113d40
2016-10-25 08:51:31.279172 7f2121c01700 10 journal op_apply_start 2328 open_ops 0 -> 1
2016-10-25 08:51:31.279177 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f21397fe6a0 seq 2328 osr(1.3 0x7f2139475570)/0x7f2139475570 start
2016-10-25 08:51:31.279182 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147f48680
2016-10-25 08:51:31.279201 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.3_head/#1:c47708d3:::benchmark_data_target167114251239.teuthology_13304_object844:head# 0~4194304
2016-10-25 08:51:31.279416 7f2120bff700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'278 lua=23'278 crt=23'277 lcod 23'277 mlcod 23'277 active+clean] op_commit: 1693
2016-10-25 08:51:31.279470 7f2120bff700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'278 lua=23'278 crt=23'277 lcod 23'277 mlcod 23'277 active+clean] op_commit: 1695
2016-10-25 08:51:31.279554 7f2120bff700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'264 lua=23'264 crt=23'263 lcod 23'263 mlcod 23'263 active+clean] op_commit: 1698
2016-10-25 08:51:31.279594 7f21203fe700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] op_commit: 1696
2016-10-25 08:51:31.279627 7f21203fe700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'278 lua=23'279 crt=23'277 lcod 23'277 mlcod 23'277 active+clean] op_commit: 1697
2016-10-25 08:51:31.279655 7f21203fe700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'278 lua=23'279 crt=23'277 lcod 23'277 mlcod 23'277 active+clean] op_commit: 1703
2016-10-25 08:51:31.279793 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2267 ==== osd_repop_reply(client.4144.0:844 1.4) v1 ==== 83+0+0 (2703907601 0 0) 0x7f213ea48800 con 0x7f2139592800
2016-10-25 08:51:31.279822 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:844 1.4) v1 epoch 23
2016-10-25 08:51:31.279831 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:31.279843 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f213981de60 prio 196 cost 0 latency 0.000102 osd_repop_reply(client.4144.0:844 1.4) v1
2016-10-25 08:51:31.279936 7f2110c6b700 10 osd.3 23 dequeue_op 0x7f213981de60 prio 196 cost 0 latency 0.000194 osd_repop_reply(client.4144.0:844 1.4) v1 pg pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'278 lua=23'279 crt=23'277 lcod 23'277 mlcod 23'277 active+clean]
2016-10-25 08:51:31.279972 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'278 lua=23'279 crt=23'277 lcod 23'277 mlcod 23'277 active+clean] handle_message: 0x7f213981de60
2016-10-25 08:51:31.280000 7f2110c6b700  7 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'278 lua=23'279 crt=23'277 lcod 23'277 mlcod 23'277 active+clean] sub_op_modify_reply: tid 1692 op  ack_type 4 from 2
2016-10-25 08:51:31.280038 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'278 lua=23'279 crt=23'277 lcod 23'277 mlcod 23'277 active+clean] repop_all_applied: repop tid 1692 all applied 
2016-10-25 08:51:31.280054 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'278 lua=23'279 crt=23'277 lcod 23'277 mlcod 23'277 active+clean] eval_repop repgather(0x7f213983f960 23'279 rep_tid=1692 committed?=0 applied?=1) wants=d
2016-10-25 08:51:31.280071 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'278 lua=23'279 crt=23'277 lcod 23'277 mlcod 23'277 active+clean]  applied: repgather(0x7f213983f960 23'279 rep_tid=1692 committed?=0 applied?=1) 
2016-10-25 08:51:31.280091 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'278 lua=23'279 crt=23'277 lcod 23'277 mlcod 23'277 active+clean] repop_all_committed: repop tid 1692 all committed 
2016-10-25 08:51:31.280106 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'277 active+clean] eval_repop repgather(0x7f213983f960 23'279 rep_tid=1692 committed?=1 applied?=1) wants=d
2016-10-25 08:51:31.280121 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'277 active+clean]  commit: repgather(0x7f213983f960 23'279 rep_tid=1692 committed?=1 applied?=1)
2016-10-25 08:51:31.280174 7f2110c6b700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'277 active+clean] log_op_stats osd_op(client.4144.0:844 1.7666b10c benchmark_data_target167114251239.teuthology_13304_object843 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 1.120837 lat 1.120919
2016-10-25 08:51:31.280175 7f211fbfd700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'320 lua=23'320 crt=23'318 lcod 23'319 mlcod 23'319 active+clean] op_commit: 1694
2016-10-25 08:51:31.280200 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'277 active+clean]  sending commit on osd_op(client.4144.0:844 1.7666b10c benchmark_data_target167114251239.teuthology_13304_object843 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f21480aa3c0
2016-10-25 08:51:31.280222 7f2110c6b700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(844 benchmark_data_target167114251239.teuthology_13304_object843 [write 0~4194304] v23'279 uv279 ondisk = 0) v7 -- 0x7f21480aa3c0 con 0
2016-10-25 08:51:31.280249 7f211fbfd700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'320 lua=23'320 crt=23'318 lcod 23'319 mlcod 23'319 active+clean] op_commit: 1699
2016-10-25 08:51:31.280271 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'277 active+clean]  applied: repgather(0x7f213983f960 23'279 rep_tid=1692 committed?=1 applied?=1) 
2016-10-25 08:51:31.280295 7f2110c6b700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'277 active+clean] publish_stats_to_osd 23:143
2016-10-25 08:51:31.280295 7f211fbfd700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'273 lua=23'273 crt=23'272 lcod 23'272 mlcod 23'272 active+clean] op_commit: 1700
2016-10-25 08:51:31.280313 7f2110c6b700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:31.280330 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean]  removing repgather(0x7f213983f960 23'279 rep_tid=1692 committed?=1 applied?=1)
2016-10-25 08:51:31.280344 7f2110c6b700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean]    q front is repgather(0x7f213983f960 23'279 rep_tid=1692 committed?=1 applied?=1)
2016-10-25 08:51:31.280360 7f2110c6b700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] remove_repop repgather(0x7f213983f960 23'279 rep_tid=1692 committed?=1 applied?=1)
2016-10-25 08:51:31.280384 7f2110c6b700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean]  requeue_ops 
2016-10-25 08:51:31.280420 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2268 ==== osd_repop_reply(client.4144.0:845 1.3) v1 ==== 83+0+0 (583404959 0 0) 0x7f213935a580 con 0x7f2139592800
2016-10-25 08:51:31.280434 7f2110c6b700 10 osd.3 23 dequeue_op 0x7f213981de60 finish
2016-10-25 08:51:31.280440 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:845 1.3) v1 epoch 23
2016-10-25 08:51:31.280445 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:31.280454 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f213981f3c0 prio 196 cost 0 latency 0.000072 osd_repop_reply(client.4144.0:845 1.3) v1
2016-10-25 08:51:31.280511 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2269 ==== osd_repop_reply(client.4144.0:846 1.3) v1 ==== 83+0+0 (3406220449 0 0) 0x7f213935b480 con 0x7f2139592800
2016-10-25 08:51:31.280529 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:846 1.3) v1 epoch 23
2016-10-25 08:51:31.280535 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:31.280543 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f213981cea0 prio 196 cost 0 latency 0.000054 osd_repop_reply(client.4144.0:846 1.3) v1
2016-10-25 08:51:31.280599 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2270 ==== osd_repop_reply(client.4144.0:847 1.1) v1 ==== 83+0+0 (1531324597 0 0) 0x7f213eb37900 con 0x7f2139592800
2016-10-25 08:51:31.280613 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:847 1.1) v1 epoch 23
2016-10-25 08:51:31.280617 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:31.280622 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f213981cfc0 prio 196 cost 0 latency 0.000044 osd_repop_reply(client.4144.0:847 1.1) v1
2016-10-25 08:51:31.280653 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2271 ==== osd_repop_reply(client.4144.0:848 1.0) v1 ==== 83+0+0 (94859199 0 0) 0x7f213eb37e00 con 0x7f2139592800
2016-10-25 08:51:31.280669 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:848 1.0) v1 epoch 23
2016-10-25 08:51:31.280678 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:31.280685 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f213981e640 prio 196 cost 0 latency 0.000047 osd_repop_reply(client.4144.0:848 1.0) v1
2016-10-25 08:51:31.280864 7f211146c700 10 osd.3 23 dequeue_op 0x7f213981f3c0 prio 196 cost 0 latency 0.000482 osd_repop_reply(client.4144.0:845 1.3) v1 pg pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'278 lua=23'278 crt=23'277 lcod 23'277 mlcod 23'277 active+clean]
2016-10-25 08:51:31.280893 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'278 lua=23'278 crt=23'277 lcod 23'277 mlcod 23'277 active+clean] handle_message: 0x7f213981f3c0
2016-10-25 08:51:31.280910 7f211146c700  7 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'278 lua=23'278 crt=23'277 lcod 23'277 mlcod 23'277 active+clean] sub_op_modify_reply: tid 1693 op  ack_type 4 from 2
2016-10-25 08:51:31.280931 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'278 lua=23'278 crt=23'277 lcod 23'277 mlcod 23'277 active+clean] repop_all_committed: repop tid 1693 all committed 
2016-10-25 08:51:31.280940 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'278 crt=23'277 lcod 23'278 mlcod 23'277 active+clean] eval_repop repgather(0x7f213985c6c0 23'279 rep_tid=1693 committed?=1 applied?=0) wants=d
2016-10-25 08:51:31.280950 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'278 crt=23'277 lcod 23'278 mlcod 23'277 active+clean]  commit: repgather(0x7f213985c6c0 23'279 rep_tid=1693 committed?=1 applied?=0)
2016-10-25 08:51:31.280960 7f211146c700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'278 crt=23'277 lcod 23'278 mlcod 23'277 active+clean] log_op_stats osd_op(client.4144.0:845 1.cb10ee23 benchmark_data_target167114251239.teuthology_13304_object844 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 0.000000 lat 1.115187
2016-10-25 08:51:31.280973 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'278 crt=23'277 lcod 23'278 mlcod 23'277 active+clean]  sending commit on osd_op(client.4144.0:845 1.cb10ee23 benchmark_data_target167114251239.teuthology_13304_object844 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f213e5e8000
2016-10-25 08:51:31.280984 7f211146c700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(845 benchmark_data_target167114251239.teuthology_13304_object844 [write 0~4194304] v23'279 uv279 ondisk = 0) v7 -- 0x7f213e5e8000 con 0
2016-10-25 08:51:31.281007 7f211146c700 10 osd.3 23 dequeue_op 0x7f213981f3c0 finish
2016-10-25 08:51:31.281025 7f211146c700 10 osd.3 23 dequeue_op 0x7f213981cea0 prio 196 cost 0 latency 0.000537 osd_repop_reply(client.4144.0:846 1.3) v1 pg pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'278 crt=23'277 lcod 23'278 mlcod 23'277 active+clean]
2016-10-25 08:51:31.281038 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'278 crt=23'277 lcod 23'278 mlcod 23'277 active+clean] handle_message: 0x7f213981cea0
2016-10-25 08:51:31.281051 7f211146c700  7 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'278 crt=23'277 lcod 23'278 mlcod 23'277 active+clean] sub_op_modify_reply: tid 1695 op  ack_type 4 from 2
2016-10-25 08:51:31.281067 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'278 crt=23'277 lcod 23'278 mlcod 23'277 active+clean] repop_all_committed: repop tid 1695 all committed 
2016-10-25 08:51:31.281075 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] eval_repop repgather(0x7f213983dd40 23'280 rep_tid=1695 committed?=1 applied?=0) wants=d
2016-10-25 08:51:31.281084 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean]  commit: repgather(0x7f213983dd40 23'280 rep_tid=1695 committed?=1 applied?=0)
2016-10-25 08:51:31.281093 7f211146c700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] log_op_stats osd_op(client.4144.0:846 1.d3d9fa7b benchmark_data_target167114251239.teuthology_13304_object845 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 0.000000 lat 1.095174
2016-10-25 08:51:31.281104 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean]  sending commit on osd_op(client.4144.0:846 1.d3d9fa7b benchmark_data_target167114251239.teuthology_13304_object845 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f213ecbf600
2016-10-25 08:51:31.281114 7f211146c700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(846 benchmark_data_target167114251239.teuthology_13304_object845 [write 0~4194304] v23'280 uv280 ondisk = 0) v7 -- 0x7f213ecbf600 con 0
2016-10-25 08:51:31.281134 7f211146c700 10 osd.3 23 dequeue_op 0x7f213981cea0 finish
2016-10-25 08:51:31.281157 7f211246e700 10 osd.3 23 dequeue_op 0x7f213981cfc0 prio 196 cost 0 latency 0.000580 osd_repop_reply(client.4144.0:847 1.1) v1 pg pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'320 lua=23'320 crt=23'318 lcod 23'319 mlcod 23'319 active+clean]
2016-10-25 08:51:31.281173 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'320 lua=23'320 crt=23'318 lcod 23'319 mlcod 23'319 active+clean] handle_message: 0x7f213981cfc0
2016-10-25 08:51:31.281196 7f211246e700  7 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'320 lua=23'320 crt=23'318 lcod 23'319 mlcod 23'319 active+clean] sub_op_modify_reply: tid 1694 op  ack_type 4 from 2
2016-10-25 08:51:31.281226 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'320 lua=23'320 crt=23'318 lcod 23'319 mlcod 23'319 active+clean] repop_all_committed: repop tid 1694 all committed 
2016-10-25 08:51:31.281242 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'321 lua=23'320 crt=23'318 lcod 23'320 mlcod 23'319 active+clean] eval_repop repgather(0x7f213983cb40 23'321 rep_tid=1694 committed?=1 applied?=0) wants=d
2016-10-25 08:51:31.281258 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'321 lua=23'320 crt=23'318 lcod 23'320 mlcod 23'319 active+clean]  commit: repgather(0x7f213983cb40 23'321 rep_tid=1694 committed?=1 applied?=0)
2016-10-25 08:51:31.281276 7f211246e700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'321 lua=23'320 crt=23'318 lcod 23'320 mlcod 23'319 active+clean] log_op_stats osd_op(client.4144.0:847 1.164bf581 benchmark_data_target167114251239.teuthology_13304_object846 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 0.000000 lat 1.084644
2016-10-25 08:51:31.281293 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'321 lua=23'320 crt=23'318 lcod 23'320 mlcod 23'319 active+clean]  sending commit on osd_op(client.4144.0:847 1.164bf581 benchmark_data_target167114251239.teuthology_13304_object846 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f2142e11340
2016-10-25 08:51:31.281304 7f211246e700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(847 benchmark_data_target167114251239.teuthology_13304_object846 [write 0~4194304] v23'321 uv321 ondisk = 0) v7 -- 0x7f2142e11340 con 0
2016-10-25 08:51:31.281321 7f211246e700 10 osd.3 23 dequeue_op 0x7f213981cfc0 finish
2016-10-25 08:51:31.281360 7f2112c6f700 10 osd.3 23 dequeue_op 0x7f213981e640 prio 196 cost 0 latency 0.000713 osd_repop_reply(client.4144.0:848 1.0) v1 pg pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean]
2016-10-25 08:51:31.281387 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] handle_message: 0x7f213981e640
2016-10-25 08:51:31.281413 7f2112c6f700  7 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] sub_op_modify_reply: tid 1696 op  ack_type 4 from 2
2016-10-25 08:51:31.281442 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] repop_all_committed: repop tid 1696 all committed 
2016-10-25 08:51:31.281451 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] eval_repop repgather(0x7f213983c000 23'280 rep_tid=1696 committed?=1 applied?=0) wants=d
2016-10-25 08:51:31.281461 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean]  commit: repgather(0x7f213983c000 23'280 rep_tid=1696 committed?=1 applied?=0)
2016-10-25 08:51:31.281471 7f2112c6f700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] log_op_stats osd_op(client.4144.0:848 1.7c2cd648 benchmark_data_target167114251239.teuthology_13304_object847 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 0.000000 lat 1.061007
2016-10-25 08:51:31.281490 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean]  sending commit on osd_op(client.4144.0:848 1.7c2cd648 benchmark_data_target167114251239.teuthology_13304_object847 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f2140fcaec0
2016-10-25 08:51:31.281523 7f2112c6f700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(848 benchmark_data_target167114251239.teuthology_13304_object847 [write 0~4194304] v23'280 uv280 ondisk = 0) v7 -- 0x7f2140fcaec0 con 0
2016-10-25 08:51:31.281550 7f2112c6f700 10 osd.3 23 dequeue_op 0x7f213981e640 finish
2016-10-25 08:51:31.289125 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.3_head/#1:c47708d3:::benchmark_data_target167114251239.teuthology_13304_object844:head# 0~4194304 = 4194304
2016-10-25 08:51:31.289253 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.3_head/#1:c47708d3:::benchmark_data_target167114251239.teuthology_13304_object844:head#
2016-10-25 08:51:31.289322 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.3_head/#1:c47708d3:::benchmark_data_target167114251239.teuthology_13304_object844:head# = 0
2016-10-25 08:51:31.289352 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.3_head/#1:c0000000::::head#
2016-10-25 08:51:31.289388 7f2121c01700 10 filestore oid: #1:c0000000::::head# not skipping op, *spos 2328.0.2
2016-10-25 08:51:31.289424 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.289452 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 660 ==== osd_op(client.4144.0:860 1.24824798 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (3413483986 0 2005269816) 0x7f2140fcaec0 con 0x7f2147f52800
2016-10-25 08:51:31.289515 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:31.289529 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f213981f2a0 prio 63 cost 4194304 latency 0.007021 osd_op(client.4144.0:860 1.24824798 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.289597 7f211046a700 10 osd.3 23 dequeue_op 0x7f213981f2a0 prio 63 cost 4194304 latency 0.007090 osd_op(client.4144.0:860 1.24824798 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean]
2016-10-25 08:51:31.289639 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] handle_message: 0x7f213981f2a0
2016-10-25 08:51:31.289673 7f211046a700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:31.289712 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] do_op osd_op(client.4144.0:860 1.24824798 benchmark_data_target167114251239.teuthology_13304_object859 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:31.289753 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] get_object_context: obc NOT found in cache: 1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head
2016-10-25 08:51:31.289767 7f211046a700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head# '_'
2016-10-25 08:51:31.289844 7f211046a700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.0_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject859__head_24824798__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.289859 7f211046a700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head# '_' = -2
2016-10-25 08:51:31.289866 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] get_object_context: no obc for soid 1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head but can_create
2016-10-25 08:51:31.289891 7f211046a700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:snapdir# 'snapset'
2016-10-25 08:51:31.289925 7f211046a700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.0_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject859__snapdir_24824798__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.289933 7f211046a700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:snapdir# 'snapset' = -2
2016-10-25 08:51:31.289967 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] create_object_context 0x7f213ebcb080 1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head 
2016-10-25 08:51:31.289981 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] populate_obc_watchers 1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head
2016-10-25 08:51:31.289991 7f211046a700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head
2016-10-25 08:51:31.290000 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] get_object_context: 0x7f213ebcb080 1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head rwstate(none n=0 w=0) oi: 1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f21394e72c0 snapset: 0=[]:[]
2016-10-25 08:51:31.290015 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] find_object_context 1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head @head oi=1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.290030 7f211046a700 25 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] do_op oi 1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.290044 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] get_object_context: obc NOT found in cache: 1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:snapdir
2016-10-25 08:51:31.290054 7f211046a700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:snapdir# '_'
2016-10-25 08:51:31.290079 7f211046a700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.0_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject859__snapdir_24824798__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.290083 7f211046a700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:snapdir# '_' = -2
2016-10-25 08:51:31.290087 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] get_object_context: no obc for soid 1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:snapdir and !can_create
2016-10-25 08:51:31.290111 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] execute_ctx 0x7f2147e26700
2016-10-25 08:51:31.290122 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] do_op 1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head [write 0~4194304] ov 0'0 av 23'284 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:31.290135 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] do_osd_op 1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head [write 0~4194304]
2016-10-25 08:51:31.290145 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:31.290574 7f211046a700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] make_writeable 1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head snapset=0x7f21394e72f8  snapc=0=[]
2016-10-25 08:51:31.290598 7f211046a700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean]  setting DIRTY flag
2016-10-25 08:51:31.290614 7f211046a700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] make_writeable 1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head done, snapset=0=[]:[]+head
2016-10-25 08:51:31.290624 7f211046a700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] finish_ctx 1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head 0x7f2147e26700 op modify  
2016-10-25 08:51:31.290638 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean]  set mtime to 2016-10-25 08:51:31.281089
2016-10-25 08:51:31.290657 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean]  final snapset 0=[]:[]+head in 1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head
2016-10-25 08:51:31.290687 7f211046a700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean]  zeroing write result code 0
2016-10-25 08:51:31.290707 7f211046a700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean]  op order client.4144 tid 860 (first)
2016-10-25 08:51:31.290723 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] new_repop rep_tid 1707 on osd_op(client.4144.0:860 1.24824798 benchmark_data_target167114251239.teuthology_13304_object859 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.290747 7f211046a700  7 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] issue_repop rep_tid 1707 o 1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head
2016-10-25 08:51:31.290811 7f211046a700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:31.290814 7f211046a700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:860 1.0 1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head v 23'284) v1 -- 0x7f214103f000 con 0
2016-10-25 08:51:31.291279 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'283 (0'0,23'283] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] append_log log((0'0,23'283], crt=23'277) [23'284 (0'0) modify   1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head by client.4144.0:860 2016-10-25 08:51:31.281089 0]
2016-10-25 08:51:31.291302 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'278 active+clean] add_log_entry 23'284 (0'0) modify   1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head by client.4144.0:860 2016-10-25 08:51:31.281089 0
2016-10-25 08:51:31.291316 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'278 lcod 23'279 mlcod 23'278 active+clean] append_log: trimming to 23'278 entries 23'278 (0'0) modify   1:0daf06b2:::benchmark_data_target167114251239.teuthology_13304_object836:head by client.4144.0:837 2016-10-25 08:51:29.651082 0
2016-10-25 08:51:31.291336 7f211046a700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'284, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:31.291672 7f211046a700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139241080 osr(1.0 0x7f2139475d80)
2016-10-25 08:51:31.291681 7f211046a700 10 journal prepare_entry [Transaction(0x7f2147ece8c0)]
2016-10-25 08:51:31.292107 7f211046a700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:31.293319 7f211046a700 10 journal op_submit_start 2343
2016-10-25 08:51:31.293325 7f211046a700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2343 [Transaction(0x7f2147ece8c0)]
2016-10-25 08:51:31.293330 7f211046a700 10 journal op_journal_transactions 2343 reqid_t client.4144.0:860
2016-10-25 08:51:31.293333 7f211046a700  5 journal submit_entry seq 2343 len 4202496 (0x7f214811db90)
2016-10-25 08:51:31.293722 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:31.293729 7f2123404700 20 journal write_thread_entry aio throttle: aio num 3 bytes 25214976 ... exp 6 min_new 64 ... pending 4202496
2016-10-25 08:51:31.293735 7f2123404700 10 journal room 547385343 max_size 1072709632 pos 916414464 header.start 391094272 top 4096
2016-10-25 08:51:31.293739 7f2123404700 10 journal check_for_full at 916414464 : 4202496 < 547385343
2016-10-25 08:51:31.293742 7f2123404700 15 journal prepare_single_write 1 will write 916414464 : seq 2343 len 4195598 -> 4202496
2016-10-25 08:51:31.293765 7f2123404700 20 journal prepare_multi_write queue_pos now 920616960
2016-10-25 08:51:31.293768 7f2123404700 15 journal do_aio_write writing 916414464~4202496
2016-10-25 08:51:31.293772 7f2123404700 20 journal write_aio_bl 916414464~4202496 seq 2343
2016-10-25 08:51:31.293778 7f2123404700 20 journal write_aio_bl .. 916414464~4202496 in 1
2016-10-25 08:51:31.294267 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:31.294273 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:31.294275 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:31.294348 7f211046a700 10 journal op_submit_finish 2343
2016-10-25 08:51:31.294371 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'278 lcod 23'279 mlcod 23'278 active+clean] eval_repop repgather(0x7f2139755b00 23'284 rep_tid=1707 committed?=0 applied?=0) wants=d
2016-10-25 08:51:31.294417 7f211046a700 10 osd.3 23 dequeue_op 0x7f213981f2a0 finish
2016-10-25 08:51:31.289495 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.3_head/#1:c0000000::::head# = 0
2016-10-25 08:51:31.294487 7f2121c01700 10 journal op_apply_finish 2328 open_ops 1 -> 0, max_applied_seq 2327 -> 2328
2016-10-25 08:51:31.294502 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f21397fe6a0 seq 2328 r = 0, finisher 0x7f21401eee80 0x7f214304de00
2016-10-25 08:51:31.294534 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f21397fe6a0 seq 2328 osr(1.3 0x7f2139475570)/0x7f2139475570 lat 1.110922
2016-10-25 08:51:31.294581 7f2121400700 10 journal op_apply_start 2329 open_ops 0 -> 1
2016-10-25 08:51:31.294598 7f2121400700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2142ec9620 seq 2329 osr(1.3 0x7f2139475570)/0x7f2139475570 start
2016-10-25 08:51:31.294619 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147f06a80
2016-10-25 08:51:31.294658 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.3_head/#1:de5f9bcb:::benchmark_data_target167114251239.teuthology_13304_object845:head# 0~4194304
2016-10-25 08:51:31.297083 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 661 ==== osd_op(client.4144.0:861 1.f3447533 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (4148832506 0 179349784) 0x7f2142e11340 con 0x7f2147f52800
2016-10-25 08:51:31.297200 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:31.297247 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f213981d8c0 prio 63 cost 4194304 latency 0.007659 osd_op(client.4144.0:861 1.f3447533 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.297327 7f211146c700 10 osd.3 23 dequeue_op 0x7f213981d8c0 prio 63 cost 4194304 latency 0.007751 osd_op(client.4144.0:861 1.f3447533 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean]
2016-10-25 08:51:31.297367 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] handle_message: 0x7f213981d8c0
2016-10-25 08:51:31.297397 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:31.297432 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] do_op osd_op(client.4144.0:861 1.f3447533 benchmark_data_target167114251239.teuthology_13304_object860 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:31.297469 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] get_object_context: obc NOT found in cache: 1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head
2016-10-25 08:51:31.297489 7f211146c700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head# '_'
2016-10-25 08:51:31.297577 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject860__head_F3447533__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.297592 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head# '_' = -2
2016-10-25 08:51:31.297599 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] get_object_context: no obc for soid 1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head but can_create
2016-10-25 08:51:31.297621 7f211146c700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:snapdir# 'snapset'
2016-10-25 08:51:31.297662 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject860__snapdir_F3447533__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.297670 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:snapdir# 'snapset' = -2
2016-10-25 08:51:31.297698 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] create_object_context 0x7f213ea57e40 1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head 
2016-10-25 08:51:31.297717 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] populate_obc_watchers 1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head
2016-10-25 08:51:31.297731 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head
2016-10-25 08:51:31.297745 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] get_object_context: 0x7f213ea57e40 1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head rwstate(none n=0 w=0) oi: 1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f213946df00 snapset: 0=[]:[]
2016-10-25 08:51:31.297767 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] find_object_context 1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head @head oi=1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.297787 7f211146c700 25 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] do_op oi 1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.297808 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] get_object_context: obc NOT found in cache: 1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:snapdir
2016-10-25 08:51:31.297824 7f211146c700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:snapdir# '_'
2016-10-25 08:51:31.297860 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject860__snapdir_F3447533__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.297868 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:snapdir# '_' = -2
2016-10-25 08:51:31.297873 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] get_object_context: no obc for soid 1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:snapdir and !can_create
2016-10-25 08:51:31.297905 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] execute_ctx 0x7f214303d500
2016-10-25 08:51:31.297921 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] do_op 1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head [write 0~4194304] ov 0'0 av 23'282 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:31.297941 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] do_osd_op 1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head [write 0~4194304]
2016-10-25 08:51:31.297955 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:31.298299 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] make_writeable 1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head snapset=0x7f213946df38  snapc=0=[]
2016-10-25 08:51:31.298323 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean]  setting DIRTY flag
2016-10-25 08:51:31.298335 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] make_writeable 1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head done, snapset=0=[]:[]+head
2016-10-25 08:51:31.298350 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] finish_ctx 1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head 0x7f214303d500 op modify  
2016-10-25 08:51:31.298370 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean]  set mtime to 2016-10-25 08:51:31.283863
2016-10-25 08:51:31.298395 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=281 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean]  final snapset 0=[]:[]+head in 1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head
2016-10-25 08:51:31.298424 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean]  zeroing write result code 0
2016-10-25 08:51:31.298441 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean]  op order client.4144 tid 861 (first)
2016-10-25 08:51:31.298456 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] new_repop rep_tid 1708 on osd_op(client.4144.0:861 1.f3447533 benchmark_data_target167114251239.teuthology_13304_object860 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.298476 7f211146c700  7 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] issue_repop rep_tid 1708 o 1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head
2016-10-25 08:51:31.298536 7f211146c700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:31.298541 7f211146c700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:861 1.3 1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head v 23'282) v1 -- 0x7f2140e67600 con 0
2016-10-25 08:51:31.298904 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'281 (0'0,23'281] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] append_log log((0'0,23'281], crt=23'277) [23'282 (0'0) modify   1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head by client.4144.0:861 2016-10-25 08:51:31.283863 0]
2016-10-25 08:51:31.298935 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] add_log_entry 23'282 (0'0) modify   1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head by client.4144.0:861 2016-10-25 08:51:31.283863 0
2016-10-25 08:51:31.298952 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] append_log: trimming to 23'277 entries 
2016-10-25 08:51:31.298974 7f211146c700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'282, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:31.299266 7f211146c700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139241e40 osr(1.3 0x7f2139475570)
2016-10-25 08:51:31.299276 7f211146c700 10 journal prepare_entry [Transaction(0x7f2147f67480)]
2016-10-25 08:51:31.299555 7f2121c01700 10 journal op_apply_start 2330 open_ops 1 -> 2
2016-10-25 08:51:31.299563 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2142f70300 seq 2330 osr(1.1 0x7f21394758a0)/0x7f21394758a0 start
2016-10-25 08:51:31.299568 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f21415bb340
2016-10-25 08:51:31.299580 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.1_head/#1:81afd268:::benchmark_data_target167114251239.teuthology_13304_object846:head# 0~4194304
2016-10-25 08:51:31.299976 7f211146c700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:31.300987 7f211146c700 10 journal op_submit_start 2344
2016-10-25 08:51:31.300992 7f211146c700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2344 [Transaction(0x7f2147f67480)]
2016-10-25 08:51:31.300997 7f211146c700 10 journal op_journal_transactions 2344 reqid_t client.4144.0:861
2016-10-25 08:51:31.301000 7f211146c700  5 journal submit_entry seq 2344 len 4202496 (0x7f2142e4e220)
2016-10-25 08:51:31.301039 7f211146c700 10 journal op_submit_finish 2344
2016-10-25 08:51:31.301048 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:31.301055 7f2123404700 20 journal write_thread_entry aio throttle: aio num 4 bytes 29417472 ... exp 8 min_new 256 ... pending 4202496
2016-10-25 08:51:31.301060 7f2123404700 10 journal room 543182847 max_size 1072709632 pos 920616960 header.start 391094272 top 4096
2016-10-25 08:51:31.301063 7f2123404700 10 journal check_for_full at 920616960 : 4202496 < 543182847
2016-10-25 08:51:31.301065 7f2123404700 15 journal prepare_single_write 1 will write 920616960 : seq 2344 len 4195598 -> 4202496
2016-10-25 08:51:31.301046 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] eval_repop repgather(0x7f213981cea0 23'282 rep_tid=1708 committed?=0 applied?=0) wants=d
2016-10-25 08:51:31.301073 7f211146c700 10 osd.3 23 dequeue_op 0x7f213981d8c0 finish
2016-10-25 08:51:31.301085 7f2123404700 20 journal prepare_multi_write queue_pos now 924819456
2016-10-25 08:51:31.301087 7f2123404700 15 journal do_aio_write writing 920616960~4202496
2016-10-25 08:51:31.301090 7f2123404700 20 journal write_aio_bl 920616960~4202496 seq 2344
2016-10-25 08:51:31.301093 7f2123404700 20 journal write_aio_bl .. 920616960~4202496 in 1
2016-10-25 08:51:31.301091 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] op_applied: 1693
2016-10-25 08:51:31.301121 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'278 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] op_applied version 23'279
2016-10-25 08:51:31.301135 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] repop_all_applied: repop tid 1693 all applied 
2016-10-25 08:51:31.301147 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] eval_repop repgather(0x7f213985c6c0 23'279 rep_tid=1693 committed?=1 applied?=1) wants=d
2016-10-25 08:51:31.301161 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'277 active+clean]  commit: repgather(0x7f213985c6c0 23'279 rep_tid=1693 committed?=1 applied?=1)
2016-10-25 08:51:31.301173 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'277 active+clean]  applied: repgather(0x7f213985c6c0 23'279 rep_tid=1693 committed?=1 applied?=1) 
2016-10-25 08:51:31.301192 7f211f3fc700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'277 active+clean] publish_stats_to_osd 23:150
2016-10-25 08:51:31.301208 7f211f3fc700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:31.301222 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'279 active+clean]  removing repgather(0x7f213985c6c0 23'279 rep_tid=1693 committed?=1 applied?=1)
2016-10-25 08:51:31.301235 7f211f3fc700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'279 active+clean]    q front is repgather(0x7f213985c6c0 23'279 rep_tid=1693 committed?=1 applied?=1)
2016-10-25 08:51:31.301248 7f211f3fc700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] remove_repop repgather(0x7f213985c6c0 23'279 rep_tid=1693 committed?=1 applied?=1)
2016-10-25 08:51:31.301267 7f211f3fc700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'279 active+clean]  requeue_ops 
2016-10-25 08:51:31.301515 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:31.301518 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:31.301520 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:31.302585 7f2125408700 10 osd.3 23 tick_without_osd_lock
2016-10-25 08:51:31.302621 7f2125408700 25 osd.3 23 heartbeat_check osd.2 first_tx 2016-10-25 08:34:57.103177 last_tx 2016-10-25 08:51:30.940721 last_rx_back 2016-10-25 08:51:30.940721 last_rx_front 2016-10-25 08:51:30.940721
2016-10-25 08:51:31.302644 7f2125408700 20 osd.3 23 scrub_random_backoff lost coin flip, randomly backing off
2016-10-25 08:51:31.302648 7f2125408700 10 osd.3 23 promote_throttle_recalibrate 0 attempts, promoted 0 objects and 0  bytes; target 25 obj/sec or 5120 k bytes/sec
2016-10-25 08:51:31.302654 7f2125408700 20 osd.3 23 promote_throttle_recalibrate  new_prob 1000
2016-10-25 08:51:31.302657 7f2125408700 10 osd.3 23 promote_throttle_recalibrate  actual 0, actual/prob ratio 1, adjusted new_prob 1000, prob 1000 -> 1000
2016-10-25 08:51:31.304587 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.1_head/#1:81afd268:::benchmark_data_target167114251239.teuthology_13304_object846:head# 0~4194304 = 4194304
2016-10-25 08:51:31.304661 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.1_head/#1:81afd268:::benchmark_data_target167114251239.teuthology_13304_object846:head#
2016-10-25 08:51:31.304714 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.1_head/#1:81afd268:::benchmark_data_target167114251239.teuthology_13304_object846:head# = 0
2016-10-25 08:51:31.304731 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.1_head/#1:80000000::::head#
2016-10-25 08:51:31.304751 7f2121c01700 10 filestore oid: #1:80000000::::head# not skipping op, *spos 2330.0.2
2016-10-25 08:51:31.304757 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.304808 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.1_head/#1:80000000::::head# = 0
2016-10-25 08:51:31.304818 7f2121c01700 10 journal op_apply_finish 2330 open_ops 2 -> 1, max_applied_seq 2328 -> 2330
2016-10-25 08:51:31.304821 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2142f70300 seq 2330 r = 0, finisher 0x7f2147e84ce0 0x7f213e5faf40
2016-10-25 08:51:31.304837 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f2142f70300 seq 2330 osr(1.1 0x7f21394758a0)/0x7f21394758a0 lat 1.086765
2016-10-25 08:51:31.305556 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 662 ==== osd_op(client.4144.0:862 1.afa14904 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (870827760 0 3238537819) 0x7f213ecbf600 con 0x7f2147f52800
2016-10-25 08:51:31.305629 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:31.305660 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f213981cd80 prio 63 cost 4194304 latency 0.004039 osd_op(client.4144.0:862 1.afa14904 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.305729 7f210e466700 10 osd.3 23 dequeue_op 0x7f213981cd80 prio 63 cost 4194304 latency 0.004108 osd_op(client.4144.0:862 1.afa14904 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean]
2016-10-25 08:51:31.305770 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] handle_message: 0x7f213981cd80
2016-10-25 08:51:31.305797 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:31.305826 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] do_op osd_op(client.4144.0:862 1.afa14904 benchmark_data_target167114251239.teuthology_13304_object861 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:31.305859 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] get_object_context: obc NOT found in cache: 1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head
2016-10-25 08:51:31.305878 7f210e466700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.4_head/#1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head# '_'
2016-10-25 08:51:31.305975 7f210e466700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.4_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject861__head_AFA14904__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.305988 7f210e466700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.4_head/#1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head# '_' = -2
2016-10-25 08:51:31.306005 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] get_object_context: no obc for soid 1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head but can_create
2016-10-25 08:51:31.306027 7f210e466700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.4_head/#1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:snapdir# 'snapset'
2016-10-25 08:51:31.306062 7f210e466700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.4_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject861__snapdir_AFA14904__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.306068 7f210e466700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.4_head/#1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:snapdir# 'snapset' = -2
2016-10-25 08:51:31.306096 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] create_object_context 0x7f2142e36680 1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head 
2016-10-25 08:51:31.306112 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] populate_obc_watchers 1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head
2016-10-25 08:51:31.306125 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head
2016-10-25 08:51:31.306137 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] get_object_context: 0x7f2142e36680 1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head rwstate(none n=0 w=0) oi: 1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f2139879ce0 snapset: 0=[]:[]
2016-10-25 08:51:31.306161 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] find_object_context 1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head @head oi=1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.306181 7f210e466700 25 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] do_op oi 1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.306201 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] get_object_context: obc NOT found in cache: 1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:snapdir
2016-10-25 08:51:31.306215 7f210e466700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.4_head/#1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:snapdir# '_'
2016-10-25 08:51:31.306244 7f210e466700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.4_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject861__snapdir_AFA14904__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.306251 7f210e466700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.4_head/#1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:snapdir# '_' = -2
2016-10-25 08:51:31.306256 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] get_object_context: no obc for soid 1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:snapdir and !can_create
2016-10-25 08:51:31.306285 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] execute_ctx 0x7f2139420e00
2016-10-25 08:51:31.306301 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] do_op 1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head [write 0~4194304] ov 0'0 av 23'283 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:31.306319 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] do_osd_op 1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head [write 0~4194304]
2016-10-25 08:51:31.306333 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:31.306661 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] make_writeable 1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head snapset=0x7f2139879d18  snapc=0=[]
2016-10-25 08:51:31.306682 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean]  setting DIRTY flag
2016-10-25 08:51:31.306694 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] make_writeable 1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head done, snapset=0=[]:[]+head
2016-10-25 08:51:31.306709 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] finish_ctx 1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head 0x7f2139420e00 op modify  
2016-10-25 08:51:31.306727 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean]  set mtime to 2016-10-25 08:51:31.284082
2016-10-25 08:51:31.306750 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean]  final snapset 0=[]:[]+head in 1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head
2016-10-25 08:51:31.306777 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean]  zeroing write result code 0
2016-10-25 08:51:31.306792 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean]  op order client.4144 tid 862 (first)
2016-10-25 08:51:31.306806 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] new_repop rep_tid 1709 on osd_op(client.4144.0:862 1.afa14904 benchmark_data_target167114251239.teuthology_13304_object861 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.306825 7f210e466700  7 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] issue_repop rep_tid 1709 o 1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head
2016-10-25 08:51:31.306889 7f210e466700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:31.306893 7f210e466700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:862 1.4 1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head v 23'283) v1 -- 0x7f2140f90600 con 0
2016-10-25 08:51:31.306976 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.3_head/#1:de5f9bcb:::benchmark_data_target167114251239.teuthology_13304_object845:head# 0~4194304 = 4194304
2016-10-25 08:51:31.307010 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.3_head/#1:de5f9bcb:::benchmark_data_target167114251239.teuthology_13304_object845:head#
2016-10-25 08:51:31.307043 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.3_head/#1:de5f9bcb:::benchmark_data_target167114251239.teuthology_13304_object845:head# = 0
2016-10-25 08:51:31.307053 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.3_head/#1:c0000000::::head#
2016-10-25 08:51:31.307062 7f2121400700 10 filestore oid: #1:c0000000::::head# not skipping op, *spos 2329.0.2
2016-10-25 08:51:31.307064 7f2121400700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.307095 7f2121400700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.3_head/#1:c0000000::::head# = 0
2016-10-25 08:51:31.307100 7f2121400700 10 journal op_apply_finish 2329 open_ops 1 -> 0, max_applied_seq 2330 -> 2330
2016-10-25 08:51:31.307102 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2142ec9620 seq 2329 r = 0, finisher 0x7f2142e08d00 0x7f2141250c00
2016-10-25 08:51:31.307165 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'321 lua=23'320 crt=23'318 lcod 23'320 mlcod 23'319 active+clean] op_applied: 1694
2016-10-25 08:51:31.307196 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'321 lua=23'320 crt=23'318 lcod 23'320 mlcod 23'319 active+clean] op_applied version 23'321
2016-10-25 08:51:31.307205 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'321 lua=23'321 crt=23'318 lcod 23'320 mlcod 23'319 active+clean] repop_all_applied: repop tid 1694 all applied 
2016-10-25 08:51:31.307213 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'321 lua=23'321 crt=23'318 lcod 23'320 mlcod 23'319 active+clean] eval_repop repgather(0x7f213983cb40 23'321 rep_tid=1694 committed?=1 applied?=1) wants=d
2016-10-25 08:51:31.307222 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'321 lua=23'321 crt=23'318 lcod 23'320 mlcod 23'319 active+clean]  commit: repgather(0x7f213983cb40 23'321 rep_tid=1694 committed?=1 applied?=1)
2016-10-25 08:51:31.307231 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'321 lua=23'321 crt=23'318 lcod 23'320 mlcod 23'319 active+clean]  applied: repgather(0x7f213983cb40 23'321 rep_tid=1694 committed?=1 applied?=1) 
2016-10-25 08:51:31.307244 7f211e3fa700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'321 lua=23'321 crt=23'318 lcod 23'320 mlcod 23'319 active+clean] publish_stats_to_osd 23: no change since 2016-10-25 08:51:30.981036
2016-10-25 08:51:31.307233 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'282 (0'0,23'282] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] append_log log((0'0,23'282], crt=23'277) [23'283 (0'0) modify   1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head by client.4144.0:862 2016-10-25 08:51:31.284082 0]
2016-10-25 08:51:31.307255 7f211e3fa700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'321 lua=23'321 crt=23'318 lcod 23'320 mlcod 23'320 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:31.307264 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'321 lua=23'321 crt=23'318 lcod 23'320 mlcod 23'320 active+clean]  removing repgather(0x7f213983cb40 23'321 rep_tid=1694 committed?=1 applied?=1)
2016-10-25 08:51:31.307262 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'277 lcod 23'278 mlcod 23'278 active+clean] add_log_entry 23'283 (0'0) modify   1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head by client.4144.0:862 2016-10-25 08:51:31.284082 0
2016-10-25 08:51:31.307272 7f211e3fa700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'321 lua=23'321 crt=23'318 lcod 23'320 mlcod 23'320 active+clean]    q front is repgather(0x7f213983cb40 23'321 rep_tid=1694 committed?=1 applied?=1)
2016-10-25 08:51:31.307281 7f211e3fa700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'321 lua=23'321 crt=23'318 lcod 23'320 mlcod 23'320 active+clean] remove_repop repgather(0x7f213983cb40 23'321 rep_tid=1694 committed?=1 applied?=1)
2016-10-25 08:51:31.307283 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'278 lcod 23'278 mlcod 23'278 active+clean] append_log: trimming to 23'278 entries 23'278 (0'0) modify   1:2d740813:::benchmark_data_target167114251239.teuthology_13304_object835:head by client.4144.0:836 2016-10-25 08:51:29.651017 0
2016-10-25 08:51:31.307293 7f211e3fa700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'321 lua=23'321 crt=23'318 lcod 23'320 mlcod 23'320 active+clean]  requeue_ops 
2016-10-25 08:51:31.307315 7f210e466700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'283, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:31.307336 7f2121c01700 10 journal op_apply_start 2331 open_ops 0 -> 1
2016-10-25 08:51:31.307338 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f213ecd3260 seq 2331 osr(1.0 0x7f2139475d80)/0x7f2139475d80 start
2016-10-25 08:51:31.307341 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147e34a00
2016-10-25 08:51:31.307349 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.0_head/#1:126b343e:::benchmark_data_target167114251239.teuthology_13304_object847:head# 0~4194304
2016-10-25 08:51:31.308093 7f210e466700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139242940 osr(1.4 0x7f2139475240)
2016-10-25 08:51:31.308107 7f210e466700 10 journal prepare_entry [Transaction(0x7f2147f61500)]
2016-10-25 08:51:31.308442 7f210e466700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:31.308983 7f2122c03700 10 journal write_finish_thread_entry aio 891199488~4202496 done
2016-10-25 08:51:31.308989 7f2122c03700 10 journal write_finish_thread_entry aio 895401984~12607488 done
2016-10-25 08:51:31.308991 7f2122c03700 10 journal write_finish_thread_entry aio 908009472~8404992 done
2016-10-25 08:51:31.308993 7f2122c03700 20 journal check_aio_completion
2016-10-25 08:51:31.308995 7f2122c03700 20 journal check_aio_completion completed seq 2337 891199488~4202496
2016-10-25 08:51:31.309000 7f2122c03700 20 journal check_aio_completion completed seq 2340 895401984~12607488
2016-10-25 08:51:31.309006 7f2122c03700 20 journal check_aio_completion completed seq 2342 908009472~8404992
2016-10-25 08:51:31.309010 7f2122c03700 20 journal check_aio_completion queueing finishers through seq 2342
2016-10-25 08:51:31.309013 7f2122c03700 10 journal queue_completions_thru seq 2342 queueing seq 2337 0x7f2141346900 lat 1.037769
2016-10-25 08:51:31.309039 7f2122c03700 10 journal queue_completions_thru seq 2342 queueing seq 2338 0x7f2141348040 lat 1.035918
2016-10-25 08:51:31.309051 7f2122c03700 10 journal queue_completions_thru seq 2342 queueing seq 2339 0x7f21414aaab0 lat 1.034789
2016-10-25 08:51:31.309062 7f2122c03700 10 journal queue_completions_thru seq 2342 queueing seq 2340 0x7f2140fb5950 lat 1.032431
2016-10-25 08:51:31.309072 7f2122c03700 10 journal queue_completions_thru seq 2342 queueing seq 2341 0x7f2140ed4540 lat 1.030314
2016-10-25 08:51:31.309084 7f2122c03700 10 journal queue_completions_thru seq 2342 queueing seq 2342 0x7f2140fb4240 lat 1.010056
2016-10-25 08:51:31.309097 7f2122c03700 20 journal write_finish_thread_entry waiting for aio(s)
2016-10-25 08:51:31.309108 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f214141a960 seq 2337 osr(1.3 0x7f2139475570) [Transaction(0x7f2147ee7340)]
2016-10-25 08:51:31.309113 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f214141a960 seq 2337 osr(1.3 0x7f2139475570) 4195584 bytes   (queue has 15 ops and 62933830 bytes)
2016-10-25 08:51:31.309118 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f2148111e20
2016-10-25 08:51:31.309124 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f2139545c20 seq 2338 osr(1.4 0x7f2139475240) [Transaction(0x7f2147f05b80)]
2016-10-25 08:51:31.309128 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f2139545c20 seq 2338 osr(1.4 0x7f2139475240) 4195584 bytes   (queue has 15 ops and 62933830 bytes)
2016-10-25 08:51:31.309132 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f213ece0e00
2016-10-25 08:51:31.309137 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f2142f72d60 seq 2339 osr(1.0 0x7f2139475d80) [Transaction(0x7f2147e48300)]
2016-10-25 08:51:31.309141 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f2142f72d60 seq 2339 osr(1.0 0x7f2139475d80) 4195584 bytes   (queue has 15 ops and 62933830 bytes)
2016-10-25 08:51:31.309145 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f213ecdf3a0
2016-10-25 08:51:31.309147 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f21397fd260 seq 2340 osr(1.0 0x7f2139475d80) [Transaction(0x7f2147e4b340)]
2016-10-25 08:51:31.309150 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f21397fd260 seq 2340 osr(1.0 0x7f2139475d80) 4195584 bytes   (queue has 15 ops and 62933830 bytes)
2016-10-25 08:51:31.309154 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f213a53c380
2016-10-25 08:51:31.309156 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f21397feb80 seq 2341 osr(1.0 0x7f2139475d80) [Transaction(0x7f2147e33c00)]
2016-10-25 08:51:31.309159 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f21397feb80 seq 2341 osr(1.0 0x7f2139475d80) 4195584 bytes   (queue has 15 ops and 62933830 bytes)
2016-10-25 08:51:31.309162 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f213ecde380
2016-10-25 08:51:31.309164 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f213ed5c220 seq 2342 osr(1.2 0x7f2139474dc0) [Transaction(0x7f2147f6f480),Transaction(0x7f2147f6f640)]
2016-10-25 08:51:31.309169 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f213ed5c220 seq 2342 osr(1.2 0x7f2139474dc0) 4195654 bytes   (queue has 15 ops and 62933830 bytes)
2016-10-25 08:51:31.309173 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f21395013c0
2016-10-25 08:51:31.309183 7f2120bff700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] op_commit: 1701
2016-10-25 08:51:31.310025 7f210e466700 10 journal op_submit_start 2345
2016-10-25 08:51:31.310065 7f210e466700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2345 [Transaction(0x7f2147f61500)]
2016-10-25 08:51:31.310085 7f210e466700 10 journal op_journal_transactions 2345 reqid_t client.4144.0:862
2016-10-25 08:51:31.310103 7f210e466700  5 journal submit_entry seq 2345 len 4202496 (0x7f2147fd2790)
2016-10-25 08:51:31.310159 7f210e466700 10 journal op_submit_finish 2345
2016-10-25 08:51:31.310183 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'278 lcod 23'278 mlcod 23'278 active+clean] eval_repop repgather(0x7f213985d320 23'283 rep_tid=1709 committed?=0 applied?=0) wants=d
2016-10-25 08:51:31.310237 7f210e466700 10 osd.3 23 dequeue_op 0x7f213981cd80 finish
2016-10-25 08:51:31.310262 7f21203fe700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'278 lcod 23'278 mlcod 23'278 active+clean] op_commit: 1704
2016-10-25 08:51:31.310319 7f21203fe700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'278 lcod 23'279 mlcod 23'278 active+clean] op_commit: 1702
2016-10-25 08:51:31.310350 7f21203fe700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'278 lcod 23'279 mlcod 23'278 active+clean] op_commit: 1705
2016-10-25 08:51:31.310378 7f21203fe700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'278 lcod 23'279 mlcod 23'278 active+clean] op_commit: 1706
2016-10-25 08:51:31.310468 7f21203fe700 10 osd.3 pg_epoch: 23 pg[1.2( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'275 crt=23'274 lcod 23'274 active] sub_op_modify_commit on op osd_repop(client.4144.0:849 1.2 1:4229aed4:::benchmark_data_target167114251239.teuthology_13304_object848:head v 23'276) v1, sending commit to osd.2
2016-10-25 08:51:31.310493 7f21203fe700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:31.310496 7f21203fe700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop_reply(client.4144.0:849 1.2 ondisk, result = 0) v1 -- 0x7f213935a080 con 0
2016-10-25 08:51:31.310830 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:31.310860 7f2123404700 20 journal write_thread_entry aio throttle: aio num 2 bytes 8404992 ... exp 4 min_new 16 ... pending 4202496
2016-10-25 08:51:31.310869 7f2123404700 10 journal room 538980351 max_size 1072709632 pos 924819456 header.start 391094272 top 4096
2016-10-25 08:51:31.310875 7f2123404700 10 journal  passing half full mark, triggering commit
2016-10-25 08:51:31.310884 7f2123404700 10 journal check_for_full at 924819456 : 4202496 < 538980351
2016-10-25 08:51:31.310887 7f2123404700 15 journal prepare_single_write 1 will write 924819456 : seq 2345 len 4195598 -> 4202496
2016-10-25 08:51:31.310928 7f2123404700 20 journal prepare_multi_write queue_pos now 929021952
2016-10-25 08:51:31.310939 7f2123404700 15 journal do_aio_write writing 924819456~4202496
2016-10-25 08:51:31.310944 7f2123404700 20 journal write_aio_bl 924819456~4202496 seq 2345
2016-10-25 08:51:31.310952 7f2123404700 20 journal write_aio_bl .. 924819456~4202496 in 1
2016-10-25 08:51:31.311392 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:31.311403 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:31.311405 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:31.311421 7f2123c05700 20 filestore(/var/lib/ceph/osd/ceph-3) sync_entry woke after 4.465346
2016-10-25 08:51:31.311431 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f2142ec9620 seq 2329 osr(1.3 0x7f2139475570)/0x7f2139475570 lat 1.095201
2016-10-25 08:51:31.311474 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] op_applied: 1695
2016-10-25 08:51:31.311567 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] op_applied version 23'280
2016-10-25 08:51:31.311585 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] repop_all_applied: repop tid 1695 all applied 
2016-10-25 08:51:31.311597 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] eval_repop repgather(0x7f213983dd40 23'280 rep_tid=1695 committed?=1 applied?=1) wants=d
2016-10-25 08:51:31.311612 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean]  commit: repgather(0x7f213983dd40 23'280 rep_tid=1695 committed?=1 applied?=1)
2016-10-25 08:51:31.311625 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean]  applied: repgather(0x7f213983dd40 23'280 rep_tid=1695 committed?=1 applied?=1) 
2016-10-25 08:51:31.311643 7f211f3fc700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] publish_stats_to_osd 23: no change since 2016-10-25 08:51:31.301188
2016-10-25 08:51:31.311660 7f211f3fc700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:31.311674 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean]  removing repgather(0x7f213983dd40 23'280 rep_tid=1695 committed?=1 applied?=1)
2016-10-25 08:51:31.311693 7f211f3fc700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean]    q front is repgather(0x7f213983dd40 23'280 rep_tid=1695 committed?=1 applied?=1)
2016-10-25 08:51:31.311706 7f211f3fc700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] remove_repop repgather(0x7f213983dd40 23'280 rep_tid=1695 committed?=1 applied?=1)
2016-10-25 08:51:31.311726 7f211f3fc700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean]  requeue_ops 
2016-10-25 08:51:31.315007 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 663 ==== osd_op(client.4144.0:863 1.85982bc3 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (1719965833 0 2553141615) 0x7f213e5e8000 con 0x7f2147f52800
2016-10-25 08:51:31.315165 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:31.315204 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f213981ebe0 prio 63 cost 4194304 latency 0.004604 osd_op(client.4144.0:863 1.85982bc3 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.315325 7f210ec67700 10 osd.3 23 dequeue_op 0x7f213981ebe0 prio 63 cost 4194304 latency 0.004725 osd_op(client.4144.0:863 1.85982bc3 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean]
2016-10-25 08:51:31.315381 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] handle_message: 0x7f213981ebe0
2016-10-25 08:51:31.315403 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:31.315427 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] do_op osd_op(client.4144.0:863 1.85982bc3 benchmark_data_target167114251239.teuthology_13304_object862 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:31.315458 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] get_object_context: obc NOT found in cache: 1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head
2016-10-25 08:51:31.315476 7f210ec67700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head# '_'
2016-10-25 08:51:31.315575 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject862__head_85982BC3__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.315590 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head# '_' = -2
2016-10-25 08:51:31.315595 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] get_object_context: no obc for soid 1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head but can_create
2016-10-25 08:51:31.315610 7f210ec67700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:snapdir# 'snapset'
2016-10-25 08:51:31.315633 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject862__snapdir_85982BC3__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.315641 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:snapdir# 'snapset' = -2
2016-10-25 08:51:31.315667 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] create_object_context 0x7f213e5e9b80 1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head 
2016-10-25 08:51:31.315682 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] populate_obc_watchers 1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head
2016-10-25 08:51:31.315690 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head
2016-10-25 08:51:31.315699 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] get_object_context: 0x7f213e5e9b80 1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head rwstate(none n=0 w=0) oi: 1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f21395863e0 snapset: 0=[]:[]
2016-10-25 08:51:31.315716 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] find_object_context 1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head @head oi=1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.315729 7f210ec67700 25 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] do_op oi 1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.315742 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] get_object_context: obc NOT found in cache: 1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:snapdir
2016-10-25 08:51:31.315754 7f210ec67700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:snapdir# '_'
2016-10-25 08:51:31.315773 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject862__snapdir_85982BC3__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.315781 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:snapdir# '_' = -2
2016-10-25 08:51:31.315784 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] get_object_context: no obc for soid 1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:snapdir and !can_create
2016-10-25 08:51:31.315804 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] execute_ctx 0x7f2140fc6300
2016-10-25 08:51:31.315818 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] do_op 1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head [write 0~4194304] ov 0'0 av 23'283 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:31.315831 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] do_osd_op 1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head [write 0~4194304]
2016-10-25 08:51:31.315843 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:31.316232 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] make_writeable 1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head snapset=0x7f2139586418  snapc=0=[]
2016-10-25 08:51:31.316246 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean]  setting DIRTY flag
2016-10-25 08:51:31.316253 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] make_writeable 1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head done, snapset=0=[]:[]+head
2016-10-25 08:51:31.316262 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] finish_ctx 1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head 0x7f2140fc6300 op modify  
2016-10-25 08:51:31.316274 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean]  set mtime to 2016-10-25 08:51:31.284296
2016-10-25 08:51:31.316290 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=282 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean]  final snapset 0=[]:[]+head in 1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head
2016-10-25 08:51:31.316314 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean]  zeroing write result code 0
2016-10-25 08:51:31.316330 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean]  op order client.4144 tid 863 (first)
2016-10-25 08:51:31.316340 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] new_repop rep_tid 1710 on osd_op(client.4144.0:863 1.85982bc3 benchmark_data_target167114251239.teuthology_13304_object862 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.316354 7f210ec67700  7 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] issue_repop rep_tid 1710 o 1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head
2016-10-25 08:51:31.316398 7f210ec67700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:31.316404 7f210ec67700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:863 1.3 1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head v 23'283) v1 -- 0x7f2140ff9800 con 0
2016-10-25 08:51:31.316806 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'282 (0'0,23'282] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] append_log log((0'0,23'282], crt=23'277) [23'283 (0'0) modify   1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head by client.4144.0:863 2016-10-25 08:51:31.284296 0]
2016-10-25 08:51:31.316826 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'277 lcod 23'279 mlcod 23'279 active+clean] add_log_entry 23'283 (0'0) modify   1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head by client.4144.0:863 2016-10-25 08:51:31.284296 0
2016-10-25 08:51:31.316840 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] append_log: trimming to 23'279 entries 23'278 (0'0) modify   1:d9543ca2:::benchmark_data_target167114251239.teuthology_13304_object832:head by client.4144.0:833 2016-10-25 08:51:29.650816 0,23'279 (0'0) modify   1:c47708d3:::benchmark_data_target167114251239.teuthology_13304_object844:head by client.4144.0:845 2016-10-25 08:51:30.150455 0
2016-10-25 08:51:31.316860 7f210ec67700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'283, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:31.317088 7f210ec67700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139241e40 osr(1.3 0x7f2139475570)
2016-10-25 08:51:31.317093 7f210ec67700 10 journal prepare_entry [Transaction(0x7f2147eaba40)]
2016-10-25 08:51:31.317459 7f210ec67700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:31.318582 7f210ec67700 10 journal op_submit_start 2346
2016-10-25 08:51:31.318623 7f210ec67700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2346 [Transaction(0x7f2147eaba40)]
2016-10-25 08:51:31.318640 7f210ec67700 10 journal op_journal_transactions 2346 reqid_t client.4144.0:863
2016-10-25 08:51:31.318654 7f210ec67700  5 journal submit_entry seq 2346 len 4202496 (0x7f2143097bf0)
2016-10-25 08:51:31.318699 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:31.318702 7f2123404700 20 journal write_thread_entry aio throttle: aio num 3 bytes 12607488 ... exp 6 min_new 64 ... pending 4202496
2016-10-25 08:51:31.318706 7f2123404700 10 journal room 534777855 max_size 1072709632 pos 929021952 header.start 391094272 top 4096
2016-10-25 08:51:31.318708 7f2123404700 10 journal check_for_full at 929021952 : 4202496 < 534777855
2016-10-25 08:51:31.318710 7f2123404700 15 journal prepare_single_write 1 will write 929021952 : seq 2346 len 4195598 -> 4202496
2016-10-25 08:51:31.318721 7f2123404700 20 journal prepare_multi_write queue_pos now 933224448
2016-10-25 08:51:31.318723 7f2123404700 15 journal do_aio_write writing 929021952~4202496
2016-10-25 08:51:31.318725 7f2123404700 20 journal write_aio_bl 929021952~4202496 seq 2346
2016-10-25 08:51:31.318729 7f2123404700 20 journal write_aio_bl .. 929021952~4202496 in 1
2016-10-25 08:51:31.318895 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 664 ==== osd_op(client.4144.0:864 1.d38cced3 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (1264590583 0 1402296876) 0x7f21480aa3c0 con 0x7f2147f52800
2016-10-25 08:51:31.318929 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:31.318939 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f213981db00 prio 63 cost 4194304 latency 0.003631 osd_op(client.4144.0:864 1.d38cced3 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.320013 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:31.320023 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:31.320026 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:31.320052 7f210ec67700 10 journal op_submit_finish 2346
2016-10-25 08:51:31.320071 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] eval_repop repgather(0x7f21395d2b40 23'283 rep_tid=1710 committed?=0 applied?=0) wants=d
2016-10-25 08:51:31.320108 7f210ec67700 10 osd.3 23 dequeue_op 0x7f213981ebe0 finish
2016-10-25 08:51:31.320402 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.0_head/#1:126b343e:::benchmark_data_target167114251239.teuthology_13304_object847:head# 0~4194304 = 4194304
2016-10-25 08:51:31.320477 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.0_head/#1:126b343e:::benchmark_data_target167114251239.teuthology_13304_object847:head#
2016-10-25 08:51:31.320534 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.0_head/#1:126b343e:::benchmark_data_target167114251239.teuthology_13304_object847:head# = 0
2016-10-25 08:51:31.320560 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.0_head/#1:00000000::::head#
2016-10-25 08:51:31.320588 7f2121c01700 10 filestore oid: #1:00000000::::head# not skipping op, *spos 2331.0.2
2016-10-25 08:51:31.320601 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.320668 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.0_head/#1:00000000::::head# = 0
2016-10-25 08:51:31.320689 7f2121c01700 10 journal op_apply_finish 2331 open_ops 1 -> 0, max_applied_seq 2330 -> 2331
2016-10-25 08:51:31.320701 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f213ecd3260 seq 2331 r = 0, finisher 0x7f2139916460 0x7f21414c3d80
2016-10-25 08:51:31.320723 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f213ecd3260 seq 2331 osr(1.0 0x7f2139475d80)/0x7f2139475d80 lat 1.089457
2016-10-25 08:51:31.320758 7f2123c05700 10 journal commit_start max_applied_seq 2331, open_ops 0
2016-10-25 08:51:31.320762 7f2123c05700 10 journal commit_start blocked, all open_ops have completed
2016-10-25 08:51:31.320763 7f2123c05700 10 journal commit_start committing 2331, still blocked
2016-10-25 08:51:31.320766 7f2123c05700 10 journal commit_start
2016-10-25 08:51:31.320772 7f2123c05700 15 filestore(/var/lib/ceph/osd/ceph-3) sync_entry committing 2331
2016-10-25 08:51:31.320775 7f2123c05700 10 journal commit_started committing 2331, unblocking
2016-10-25 08:51:31.320817 7f2123c05700 20 filestore dbobjectmap: seq is 9
2016-10-25 08:51:31.320900 7f2121400700 10 journal op_apply_start 2332 open_ops 0 -> 1
2016-10-25 08:51:31.320905 7f2121400700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f214133de40 seq 2332 osr(1.7 0x7f2139474be0)/0x7f2139474be0 start
2016-10-25 08:51:31.320908 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147e65100
2016-10-25 08:51:31.320918 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.7_head/#1:ffe055e2:::benchmark_data_target167114251239.teuthology_13304_object850:head# 0~4194304
2016-10-25 08:51:31.324530 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.7_head/#1:ffe055e2:::benchmark_data_target167114251239.teuthology_13304_object850:head# 0~4194304 = 4194304
2016-10-25 08:51:31.324603 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.7_head/#1:ffe055e2:::benchmark_data_target167114251239.teuthology_13304_object850:head#
2016-10-25 08:51:31.324650 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.7_head/#1:ffe055e2:::benchmark_data_target167114251239.teuthology_13304_object850:head# = 0
2016-10-25 08:51:31.324675 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.7_head/#1:e0000000::::head#
2016-10-25 08:51:31.324700 7f2121c01700 10 journal op_apply_start 2333 open_ops 1 -> 2
2016-10-25 08:51:31.324713 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2142f71860 seq 2333 osr(1.4 0x7f2139475240)/0x7f2139475240 start
2016-10-25 08:51:31.324727 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147e55480
2016-10-25 08:51:31.324745 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.4_head/#1:24bae17a:::benchmark_data_target167114251239.teuthology_13304_object849:head# 0~4194304
2016-10-25 08:51:31.325600 7f211146c700 10 osd.3 23 dequeue_op 0x7f213981db00 prio 63 cost 4194304 latency 0.010291 osd_op(client.4144.0:864 1.d38cced3 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean]
2016-10-25 08:51:31.325658 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] handle_message: 0x7f213981db00
2016-10-25 08:51:31.325719 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'278 lcod 23'279 mlcod 23'278 active+clean] op_applied: 1696
2016-10-25 08:51:31.325754 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'278 lcod 23'279 mlcod 23'278 active+clean] op_applied version 23'280
2016-10-25 08:51:31.325767 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'278 active+clean] repop_all_applied: repop tid 1696 all applied 
2016-10-25 08:51:31.325777 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'278 active+clean] eval_repop repgather(0x7f213983c000 23'280 rep_tid=1696 committed?=1 applied?=1) wants=d
2016-10-25 08:51:31.325789 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'278 active+clean]  commit: repgather(0x7f213983c000 23'280 rep_tid=1696 committed?=1 applied?=1)
2016-10-25 08:51:31.325800 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'278 active+clean]  applied: repgather(0x7f213983c000 23'280 rep_tid=1696 committed?=1 applied?=1) 
2016-10-25 08:51:31.325816 7f211ebfb700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'278 active+clean] publish_stats_to_osd 23:137
2016-10-25 08:51:31.325831 7f211ebfb700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:31.325855 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:31.325885 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] do_op osd_op(client.4144.0:864 1.d38cced3 benchmark_data_target167114251239.teuthology_13304_object863 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:31.325916 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] get_object_context: obc NOT found in cache: 1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head
2016-10-25 08:51:31.325932 7f211146c700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head# '_'
2016-10-25 08:51:31.325988 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject863__head_D38CCED3__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.326001 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head# '_' = -2
2016-10-25 08:51:31.326006 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] get_object_context: no obc for soid 1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head but can_create
2016-10-25 08:51:31.326023 7f211146c700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:snapdir# 'snapset'
2016-10-25 08:51:31.326054 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject863__snapdir_D38CCED3__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.326059 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:snapdir# 'snapset' = -2
2016-10-25 08:51:31.326086 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] create_object_context 0x7f213c577700 1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head 
2016-10-25 08:51:31.326101 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] populate_obc_watchers 1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head
2016-10-25 08:51:31.326115 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head
2016-10-25 08:51:31.326127 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] get_object_context: 0x7f213c577700 1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head rwstate(none n=0 w=0) oi: 1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f213946b6c0 snapset: 0=[]:[]
2016-10-25 08:51:31.326148 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] find_object_context 1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head @head oi=1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.326166 7f211146c700 25 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] do_op oi 1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.326184 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] get_object_context: obc NOT found in cache: 1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:snapdir
2016-10-25 08:51:31.326198 7f211146c700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:snapdir# '_'
2016-10-25 08:51:31.326223 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject863__snapdir_D38CCED3__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.326234 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:snapdir# '_' = -2
2016-10-25 08:51:31.326239 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] get_object_context: no obc for soid 1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:snapdir and !can_create
2016-10-25 08:51:31.326267 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] execute_ctx 0x7f2142ed2300
2016-10-25 08:51:31.326282 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] do_op 1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head [write 0~4194304] ov 0'0 av 23'284 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:31.326299 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] do_osd_op 1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head [write 0~4194304]
2016-10-25 08:51:31.326313 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:31.326625 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] make_writeable 1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head snapset=0x7f213946b6f8  snapc=0=[]
2016-10-25 08:51:31.326644 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean]  setting DIRTY flag
2016-10-25 08:51:31.326655 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] make_writeable 1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head done, snapset=0=[]:[]+head
2016-10-25 08:51:31.326666 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] finish_ctx 1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head 0x7f2142ed2300 op modify  
2016-10-25 08:51:31.326683 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean]  set mtime to 2016-10-25 08:51:31.284487
2016-10-25 08:51:31.326705 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean]  final snapset 0=[]:[]+head in 1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head
2016-10-25 08:51:31.326728 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean]  zeroing write result code 0
2016-10-25 08:51:31.326743 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean]  op order client.4144 tid 864 (first)
2016-10-25 08:51:31.326754 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] new_repop rep_tid 1711 on osd_op(client.4144.0:864 1.d38cced3 benchmark_data_target167114251239.teuthology_13304_object863 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.326770 7f211146c700  7 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] issue_repop rep_tid 1711 o 1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head
2016-10-25 08:51:31.326819 7f211146c700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:31.326823 7f211146c700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:864 1.3 1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head v 23'284) v1 -- 0x7f2147f93600 con 0
2016-10-25 08:51:31.327955 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.4_head/#1:24bae17a:::benchmark_data_target167114251239.teuthology_13304_object849:head# 0~4194304 = 4194304
2016-10-25 08:51:31.327996 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.4_head/#1:24bae17a:::benchmark_data_target167114251239.teuthology_13304_object849:head#
2016-10-25 08:51:31.328030 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.4_head/#1:24bae17a:::benchmark_data_target167114251239.teuthology_13304_object849:head# = 0
2016-10-25 08:51:31.328040 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.4_head/#1:20000000::::head#
2016-10-25 08:51:31.328093 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean]  removing repgather(0x7f213983c000 23'280 rep_tid=1696 committed?=1 applied?=1)
2016-10-25 08:51:31.328113 7f211ebfb700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean]    q front is repgather(0x7f213983c000 23'280 rep_tid=1696 committed?=1 applied?=1)
2016-10-25 08:51:31.328121 7f211ebfb700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] remove_repop repgather(0x7f213983c000 23'280 rep_tid=1696 committed?=1 applied?=1)
2016-10-25 08:51:31.328144 7f211ebfb700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean]  requeue_ops 
2016-10-25 08:51:31.329453 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'283 (0'0,23'283] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] append_log log((0'0,23'283], crt=23'279) [23'284 (0'0) modify   1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head by client.4144.0:864 2016-10-25 08:51:31.284487 0]
2016-10-25 08:51:31.329498 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] add_log_entry 23'284 (0'0) modify   1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head by client.4144.0:864 2016-10-25 08:51:31.284487 0
2016-10-25 08:51:31.329516 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] append_log: trimming to 23'279 entries 
2016-10-25 08:51:31.329537 7f211146c700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'284, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:31.329806 7f211146c700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139241e40 osr(1.3 0x7f2139475570)
2016-10-25 08:51:31.329815 7f211146c700 10 journal prepare_entry [Transaction(0x7f2147e9e000)]
2016-10-25 08:51:31.330107 7f211146c700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:31.331062 7f211146c700 10 journal op_submit_start 2347
2016-10-25 08:51:31.331075 7f211146c700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2347 [Transaction(0x7f2147e9e000)]
2016-10-25 08:51:31.331081 7f211146c700 10 journal op_journal_transactions 2347 reqid_t client.4144.0:864
2016-10-25 08:51:31.331084 7f211146c700  5 journal submit_entry seq 2347 len 4202496 (0x7f21414ac100)
2016-10-25 08:51:31.331090 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 665 ==== osd_op(client.4144.0:865 1.ed9c7da8 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (695657570 0 711343879) 0x7f2142efb180 con 0x7f2147f52800
2016-10-25 08:51:31.331129 7f211146c700 10 journal op_submit_finish 2347
2016-10-25 08:51:31.331136 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] eval_repop repgather(0x7f21397f9e60 23'284 rep_tid=1711 committed?=0 applied?=0) wants=d
2016-10-25 08:51:31.331169 7f211146c700 10 osd.3 23 dequeue_op 0x7f213981db00 finish
2016-10-25 08:51:31.334445 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:31.334456 7f2123404700 20 journal write_thread_entry aio throttle: aio num 4 bytes 16809984 ... exp 8 min_new 256 ... pending 4202496
2016-10-25 08:51:31.334461 7f2123404700 10 journal room 530575359 max_size 1072709632 pos 933224448 header.start 391094272 top 4096
2016-10-25 08:51:31.334463 7f2123404700 10 journal check_for_full at 933224448 : 4202496 < 530575359
2016-10-25 08:51:31.334465 7f2123404700 15 journal prepare_single_write 1 will write 933224448 : seq 2347 len 4195598 -> 4202496
2016-10-25 08:51:31.334487 7f2123404700 20 journal prepare_multi_write queue_pos now 937426944
2016-10-25 08:51:31.334490 7f2123404700 15 journal do_aio_write writing 933224448~4202496
2016-10-25 08:51:31.334491 7f2123404700 20 journal write_aio_bl 933224448~4202496 seq 2347
2016-10-25 08:51:31.334494 7f2123404700 20 journal write_aio_bl .. 933224448~4202496 in 1
2016-10-25 08:51:31.334846 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:31.334849 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:31.334851 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:31.331121 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:31.334869 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f21397717a0 prio 63 cost 4194304 latency 0.005417 osd_op(client.4144.0:865 1.ed9c7da8 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.334903 7f2112c6f700 10 osd.3 23 dequeue_op 0x7f21397717a0 prio 63 cost 4194304 latency 0.005451 osd_op(client.4144.0:865 1.ed9c7da8 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean]
2016-10-25 08:51:31.334927 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] handle_message: 0x7f21397717a0
2016-10-25 08:51:31.334956 7f2112c6f700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:31.334977 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] do_op osd_op(client.4144.0:865 1.ed9c7da8 benchmark_data_target167114251239.teuthology_13304_object864 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:31.335001 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] get_object_context: obc NOT found in cache: 1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head
2016-10-25 08:51:31.335012 7f2112c6f700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head# '_'
2016-10-25 08:51:31.335068 7f2112c6f700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.0_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject864__head_ED9C7DA8__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.335077 7f2112c6f700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head# '_' = -2
2016-10-25 08:51:31.335081 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] get_object_context: no obc for soid 1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head but can_create
2016-10-25 08:51:31.335093 7f2112c6f700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:snapdir# 'snapset'
2016-10-25 08:51:31.335111 7f2112c6f700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.0_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject864__snapdir_ED9C7DA8__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.335114 7f2112c6f700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:snapdir# 'snapset' = -2
2016-10-25 08:51:31.335133 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] create_object_context 0x7f2140eaaec0 1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head 
2016-10-25 08:51:31.335143 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] populate_obc_watchers 1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head
2016-10-25 08:51:31.335150 7f2112c6f700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head
2016-10-25 08:51:31.335157 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] get_object_context: 0x7f2140eaaec0 1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head rwstate(none n=0 w=0) oi: 1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f213976ad80 snapset: 0=[]:[]
2016-10-25 08:51:31.335169 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] find_object_context 1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head @head oi=1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.335181 7f2112c6f700 25 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] do_op oi 1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.335192 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] get_object_context: obc NOT found in cache: 1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:snapdir
2016-10-25 08:51:31.335199 7f2112c6f700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:snapdir# '_'
2016-10-25 08:51:31.335215 7f2112c6f700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.0_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject864__snapdir_ED9C7DA8__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.335218 7f2112c6f700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:snapdir# '_' = -2
2016-10-25 08:51:31.335221 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] get_object_context: no obc for soid 1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:snapdir and !can_create
2016-10-25 08:51:31.335238 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] execute_ctx 0x7f213edb0a00
2016-10-25 08:51:31.335247 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] do_op 1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head [write 0~4194304] ov 0'0 av 23'285 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:31.335257 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] do_osd_op 1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head [write 0~4194304]
2016-10-25 08:51:31.335265 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:31.335622 7f2112c6f700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] make_writeable 1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head snapset=0x7f213976adb8  snapc=0=[]
2016-10-25 08:51:31.335632 7f2112c6f700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean]  setting DIRTY flag
2016-10-25 08:51:31.335639 7f2112c6f700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] make_writeable 1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head done, snapset=0=[]:[]+head
2016-10-25 08:51:31.335647 7f2112c6f700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] finish_ctx 1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head 0x7f213edb0a00 op modify  
2016-10-25 08:51:31.335657 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean]  set mtime to 2016-10-25 08:51:31.320000
2016-10-25 08:51:31.335672 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean]  final snapset 0=[]:[]+head in 1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head
2016-10-25 08:51:31.335688 7f2112c6f700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean]  zeroing write result code 0
2016-10-25 08:51:31.335698 7f2112c6f700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean]  op order client.4144 tid 865 (first)
2016-10-25 08:51:31.335705 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] new_repop rep_tid 1712 on osd_op(client.4144.0:865 1.ed9c7da8 benchmark_data_target167114251239.teuthology_13304_object864 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.335716 7f2112c6f700  7 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] issue_repop rep_tid 1712 o 1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head
2016-10-25 08:51:31.335753 7f2112c6f700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:31.335755 7f2112c6f700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:865 1.0 1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head v 23'285) v1 -- 0x7f2140ff4600 con 0
2016-10-25 08:51:31.336099 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'284 (0'0,23'284] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] append_log log((0'0,23'284], crt=23'278) [23'285 (0'0) modify   1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head by client.4144.0:865 2016-10-25 08:51:31.320000 0]
2016-10-25 08:51:31.336120 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'278 lcod 23'279 mlcod 23'279 active+clean] add_log_entry 23'285 (0'0) modify   1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head by client.4144.0:865 2016-10-25 08:51:31.320000 0
2016-10-25 08:51:31.336132 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] append_log: trimming to 23'279 entries 23'279 (0'0) modify   1:0ab5be29:::benchmark_data_target167114251239.teuthology_13304_object842:head by client.4144.0:843 2016-10-25 08:51:29.812846 0
2016-10-25 08:51:31.336156 7f2112c6f700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'285, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:31.336389 7f2112c6f700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139241080 osr(1.0 0x7f2139475d80)
2016-10-25 08:51:31.336398 7f2112c6f700 10 journal prepare_entry [Transaction(0x7f2147e33500)]
2016-10-25 08:51:31.336741 7f2112c6f700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:31.338506 7f2112c6f700 10 journal op_submit_start 2348
2016-10-25 08:51:31.338535 7f2112c6f700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2348 [Transaction(0x7f2147e33500)]
2016-10-25 08:51:31.338544 7f2112c6f700 10 journal op_journal_transactions 2348 reqid_t client.4144.0:865
2016-10-25 08:51:31.338551 7f2112c6f700  5 journal submit_entry seq 2348 len 4202496 (0x7f2140fda520)
2016-10-25 08:51:31.338801 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:31.338811 7f2123404700 20 journal write_thread_entry aio throttle: aio num 5 bytes 21012480 ... exp 10 min_new 1024 ... pending 4202496
2016-10-25 08:51:31.338818 7f2123404700 10 journal room 526372863 max_size 1072709632 pos 937426944 header.start 391094272 top 4096
2016-10-25 08:51:31.338823 7f2123404700 10 journal check_for_full at 937426944 : 4202496 < 526372863
2016-10-25 08:51:31.338826 7f2123404700 15 journal prepare_single_write 1 will write 937426944 : seq 2348 len 4195598 -> 4202496
2016-10-25 08:51:31.338852 7f2123404700 20 journal prepare_multi_write queue_pos now 941629440
2016-10-25 08:51:31.338855 7f2123404700 15 journal do_aio_write writing 937426944~4202496
2016-10-25 08:51:31.338859 7f2123404700 20 journal write_aio_bl 937426944~4202496 seq 2348
2016-10-25 08:51:31.338866 7f2123404700 20 journal write_aio_bl .. 937426944~4202496 in 1
2016-10-25 08:51:31.339446 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:31.339460 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:31.339463 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:31.339472 7f2112c6f700 10 journal op_submit_finish 2348
2016-10-25 08:51:31.339479 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] eval_repop repgather(0x7f213981e640 23'285 rep_tid=1712 committed?=0 applied?=0) wants=d
2016-10-25 08:51:31.339542 7f2112c6f700 10 osd.3 23 dequeue_op 0x7f21397717a0 finish
2016-10-25 08:51:31.339666 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2272 ==== osd_repop_reply(client.4144.0:852 1.1) v1 ==== 83+0+0 (1169078911 0 0) 0x7f214101ca00 con 0x7f2139592800
2016-10-25 08:51:31.339708 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:852 1.1) v1 epoch 23
2016-10-25 08:51:31.339718 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:31.339730 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139773960 prio 196 cost 0 latency 0.000128 osd_repop_reply(client.4144.0:852 1.1) v1
2016-10-25 08:51:31.339785 7f210fc69700 10 osd.3 23 dequeue_op 0x7f2139773960 prio 196 cost 0 latency 0.000183 osd_repop_reply(client.4144.0:852 1.1) v1 pg pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'321 lua=23'321 crt=23'318 lcod 23'320 mlcod 23'320 active+clean]
2016-10-25 08:51:31.339819 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'321 lua=23'321 crt=23'318 lcod 23'320 mlcod 23'320 active+clean] handle_message: 0x7f2139773960
2016-10-25 08:51:31.339848 7f210fc69700  7 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'321 lua=23'321 crt=23'318 lcod 23'320 mlcod 23'320 active+clean] sub_op_modify_reply: tid 1699 op  ack_type 4 from 2
2016-10-25 08:51:31.339891 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'321 lua=23'321 crt=23'318 lcod 23'320 mlcod 23'320 active+clean] repop_all_committed: repop tid 1699 all committed 
2016-10-25 08:51:31.339931 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] eval_repop repgather(0x7f21398820a0 23'322 rep_tid=1699 committed?=1 applied?=0) wants=d
2016-10-25 08:51:31.339950 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean]  commit: repgather(0x7f21398820a0 23'322 rep_tid=1699 committed?=1 applied?=0)
2016-10-25 08:51:31.339969 7f210fc69700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] log_op_stats osd_op(client.4144.0:852 1.bd0d8051 benchmark_data_target167114251239.teuthology_13304_object851 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 0.000000 lat 1.099077
2016-10-25 08:51:31.339992 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean]  sending commit on osd_op(client.4144.0:852 1.bd0d8051 benchmark_data_target167114251239.teuthology_13304_object851 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f2142e34840
2016-10-25 08:51:31.340008 7f210fc69700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(852 benchmark_data_target167114251239.teuthology_13304_object851 [write 0~4194304] v23'322 uv322 ondisk = 0) v7 -- 0x7f2142e34840 con 0
2016-10-25 08:51:31.340044 7f210fc69700 10 osd.3 23 dequeue_op 0x7f2139773960 finish
2016-10-25 08:51:31.340102 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2273 ==== osd_repop_reply(client.4144.0:853 1.6) v1 ==== 83+0+0 (2516440075 0 0) 0x7f214101ca00 con 0x7f2139592800
2016-10-25 08:51:31.340118 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:853 1.6) v1 epoch 23
2016-10-25 08:51:31.340122 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:31.340129 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139773960 prio 196 cost 0 latency 0.000049 osd_repop_reply(client.4144.0:853 1.6) v1
2016-10-25 08:51:31.340172 7f211246e700 10 osd.3 23 dequeue_op 0x7f2139773960 prio 196 cost 0 latency 0.000092 osd_repop_reply(client.4144.0:853 1.6) v1 pg pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'273 lua=23'273 crt=23'272 lcod 23'272 mlcod 23'272 active+clean]
2016-10-25 08:51:31.340190 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'273 lua=23'273 crt=23'272 lcod 23'272 mlcod 23'272 active+clean] handle_message: 0x7f2139773960
2016-10-25 08:51:31.340204 7f211246e700  7 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'273 lua=23'273 crt=23'272 lcod 23'272 mlcod 23'272 active+clean] sub_op_modify_reply: tid 1700 op  ack_type 4 from 2
2016-10-25 08:51:31.340220 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'273 lua=23'273 crt=23'272 lcod 23'272 mlcod 23'272 active+clean] repop_all_committed: repop tid 1700 all committed 
2016-10-25 08:51:31.340227 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] eval_repop repgather(0x7f2139645180 23'274 rep_tid=1700 committed?=1 applied?=0) wants=d
2016-10-25 08:51:31.340235 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean]  commit: repgather(0x7f2139645180 23'274 rep_tid=1700 committed?=1 applied?=0)
2016-10-25 08:51:31.340244 7f211246e700 15 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] log_op_stats osd_op(client.4144.0:853 1.86c6778e benchmark_data_target167114251239.teuthology_13304_object852 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 0.000000 lat 1.094635
2016-10-25 08:51:31.340253 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean]  sending commit on osd_op(client.4144.0:853 1.86c6778e benchmark_data_target167114251239.teuthology_13304_object852 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f2141377e40
2016-10-25 08:51:31.340261 7f211246e700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(853 benchmark_data_target167114251239.teuthology_13304_object852 [write 0~4194304] v23'274 uv274 ondisk = 0) v7 -- 0x7f2141377e40 con 0
2016-10-25 08:51:31.340275 7f211246e700 10 osd.3 23 dequeue_op 0x7f2139773960 finish
2016-10-25 08:51:31.340307 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2274 ==== osd_repop_reply(client.4144.0:851 1.7) v1 ==== 83+0+0 (3331332328 0 0) 0x7f214101ca00 con 0x7f2139592800
2016-10-25 08:51:31.340315 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:851 1.7) v1 epoch 23
2016-10-25 08:51:31.340318 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:31.340323 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139773960 prio 196 cost 0 latency 0.000029 osd_repop_reply(client.4144.0:851 1.7) v1
2016-10-25 08:51:31.340346 7f210f468700 10 osd.3 23 dequeue_op 0x7f2139773960 prio 196 cost 0 latency 0.000052 osd_repop_reply(client.4144.0:851 1.7) v1 pg pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'264 lua=23'264 crt=23'263 lcod 23'263 mlcod 23'263 active+clean]
2016-10-25 08:51:31.340361 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'264 lua=23'264 crt=23'263 lcod 23'263 mlcod 23'263 active+clean] handle_message: 0x7f2139773960
2016-10-25 08:51:31.340384 7f210f468700  7 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'264 lua=23'264 crt=23'263 lcod 23'263 mlcod 23'263 active+clean] sub_op_modify_reply: tid 1698 op  ack_type 4 from 2
2016-10-25 08:51:31.340406 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'264 lua=23'264 crt=23'263 lcod 23'263 mlcod 23'263 active+clean] repop_all_committed: repop tid 1698 all committed 
2016-10-25 08:51:31.340418 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] eval_repop repgather(0x7f21396426c0 23'265 rep_tid=1698 committed?=1 applied?=0) wants=d
2016-10-25 08:51:31.340431 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean]  commit: repgather(0x7f21396426c0 23'265 rep_tid=1698 committed?=1 applied?=0)
2016-10-25 08:51:31.340448 7f210f468700 15 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] log_op_stats osd_op(client.4144.0:851 1.47aa07ff benchmark_data_target167114251239.teuthology_13304_object850 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 0.000000 lat 1.105500
2016-10-25 08:51:31.340468 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean]  sending commit on osd_op(client.4144.0:851 1.47aa07ff benchmark_data_target167114251239.teuthology_13304_object850 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f2142e54b00
2016-10-25 08:51:31.340486 7f210f468700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(851 benchmark_data_target167114251239.teuthology_13304_object850 [write 0~4194304] v23'265 uv265 ondisk = 0) v7 -- 0x7f2142e54b00 con 0
2016-10-25 08:51:31.340513 7f210f468700 10 osd.3 23 dequeue_op 0x7f2139773960 finish
2016-10-25 08:51:31.340577 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2275 ==== osd_repop_reply(client.4144.0:855 1.3) v1 ==== 83+0+0 (2757545181 0 0) 0x7f214101ca00 con 0x7f2139592800
2016-10-25 08:51:31.340594 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:855 1.3) v1 epoch 23
2016-10-25 08:51:31.340599 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:31.340608 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139773960 prio 196 cost 0 latency 0.000057 osd_repop_reply(client.4144.0:855 1.3) v1
2016-10-25 08:51:31.340646 7f210ec67700 10 osd.3 23 dequeue_op 0x7f2139773960 prio 196 cost 0 latency 0.000095 osd_repop_reply(client.4144.0:855 1.3) v1 pg pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean]
2016-10-25 08:51:31.340671 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] handle_message: 0x7f2139773960
2016-10-25 08:51:31.340697 7f210ec67700  7 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] sub_op_modify_reply: tid 1701 op  ack_type 4 from 2
2016-10-25 08:51:31.340731 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] repop_all_committed: repop tid 1701 all committed 
2016-10-25 08:51:31.340746 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'280 crt=23'279 lcod 23'280 mlcod 23'279 active+clean] eval_repop repgather(0x7f21395d5a80 23'281 rep_tid=1701 committed?=1 applied?=0) wants=d
2016-10-25 08:51:31.340761 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'280 crt=23'279 lcod 23'280 mlcod 23'279 active+clean]  commit: repgather(0x7f21395d5a80 23'281 rep_tid=1701 committed?=1 applied?=0)
2016-10-25 08:51:31.340779 7f210ec67700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'280 crt=23'279 lcod 23'280 mlcod 23'279 active+clean] log_op_stats osd_op(client.4144.0:855 1.da63b8db benchmark_data_target167114251239.teuthology_13304_object854 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 0.000000 lat 1.088804
2016-10-25 08:51:31.340801 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'280 crt=23'279 lcod 23'280 mlcod 23'279 active+clean]  sending commit on osd_op(client.4144.0:855 1.da63b8db benchmark_data_target167114251239.teuthology_13304_object854 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f2142f37180
2016-10-25 08:51:31.340821 7f210ec67700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(855 benchmark_data_target167114251239.teuthology_13304_object854 [write 0~4194304] v23'281 uv281 ondisk = 0) v7 -- 0x7f2142f37180 con 0
2016-10-25 08:51:31.340864 7f210ec67700 10 osd.3 23 dequeue_op 0x7f2139773960 finish
2016-10-25 08:51:31.340938 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2276 ==== osd_repop_reply(client.4144.0:850 1.4) v1 ==== 83+0+0 (505146400 0 0) 0x7f214101ca00 con 0x7f2139592800
2016-10-25 08:51:31.340957 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:850 1.4) v1 epoch 23
2016-10-25 08:51:31.340965 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:31.340976 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139773960 prio 196 cost 0 latency 0.000067 osd_repop_reply(client.4144.0:850 1.4) v1
2016-10-25 08:51:31.341038 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2277 ==== osd_repop_reply(client.4144.0:854 1.4) v1 ==== 83+0+0 (665885267 0 0) 0x7f214101c780 con 0x7f2139592800
2016-10-25 08:51:31.341055 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:854 1.4) v1 epoch 23
2016-10-25 08:51:31.341061 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:31.341069 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139770000 prio 196 cost 0 latency 0.000052 osd_repop_reply(client.4144.0:854 1.4) v1
2016-10-25 08:51:31.341122 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2278 ==== osd_repop_reply(client.4144.0:856 1.0) v1 ==== 83+0+0 (1201156150 0 0) 0x7f214101c500 con 0x7f2139592800
2016-10-25 08:51:31.341139 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:856 1.0) v1 epoch 23
2016-10-25 08:51:31.341145 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:31.341153 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139771680 prio 196 cost 0 latency 0.000051 osd_repop_reply(client.4144.0:856 1.0) v1
2016-10-25 08:51:31.341199 7f211046a700 10 osd.3 23 dequeue_op 0x7f2139771680 prio 196 cost 0 latency 0.000096 osd_repop_reply(client.4144.0:856 1.0) v1 pg pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean]
2016-10-25 08:51:31.341229 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] handle_message: 0x7f2139771680
2016-10-25 08:51:31.341258 7f211046a700  7 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] sub_op_modify_reply: tid 1702 op  ack_type 4 from 2
2016-10-25 08:51:31.341296 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'280 crt=23'279 lcod 23'279 mlcod 23'279 active+clean] repop_all_committed: repop tid 1702 all committed 
2016-10-25 08:51:31.341315 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'280 crt=23'279 lcod 23'280 mlcod 23'279 active+clean] eval_repop repgather(0x7f2139756ac0 23'281 rep_tid=1702 committed?=1 applied?=0) wants=d
2016-10-25 08:51:31.341331 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'280 crt=23'279 lcod 23'280 mlcod 23'279 active+clean]  commit: repgather(0x7f2139756ac0 23'281 rep_tid=1702 committed?=1 applied?=0)
2016-10-25 08:51:31.341349 7f211046a700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'280 crt=23'279 lcod 23'280 mlcod 23'279 active+clean] log_op_stats osd_op(client.4144.0:856 1.71b87dd0 benchmark_data_target167114251239.teuthology_13304_object855 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 0.000000 lat 1.087654
2016-10-25 08:51:31.341371 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'280 crt=23'279 lcod 23'280 mlcod 23'279 active+clean]  sending commit on osd_op(client.4144.0:856 1.71b87dd0 benchmark_data_target167114251239.teuthology_13304_object855 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f2142f96ec0
2016-10-25 08:51:31.341385 7f211046a700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(856 benchmark_data_target167114251239.teuthology_13304_object855 [write 0~4194304] v23'281 uv281 ondisk = 0) v7 -- 0x7f2142f96ec0 con 0
2016-10-25 08:51:31.341406 7f211046a700 10 osd.3 23 dequeue_op 0x7f2139771680 finish
2016-10-25 08:51:31.341451 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2279 ==== osd_repop_reply(client.4144.0:858 1.4) v1 ==== 83+0+0 (704070545 0 0) 0x7f214101c500 con 0x7f2139592800
2016-10-25 08:51:31.341468 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:858 1.4) v1 epoch 23
2016-10-25 08:51:31.341474 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:31.341484 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139771680 prio 196 cost 0 latency 0.000051 osd_repop_reply(client.4144.0:858 1.4) v1
2016-10-25 08:51:31.341520 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2280 ==== osd_repop_reply(client.4144.0:857 1.0) v1 ==== 83+0+0 (1178939148 0 0) 0x7f214101c280 con 0x7f2139592800
2016-10-25 08:51:31.341536 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:857 1.0) v1 epoch 23
2016-10-25 08:51:31.341541 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:31.341550 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139773840 prio 196 cost 0 latency 0.000045 osd_repop_reply(client.4144.0:857 1.0) v1
2016-10-25 08:51:31.341523 7f2110c6b700 10 osd.3 23 dequeue_op 0x7f2139773960 prio 196 cost 0 latency 0.000614 osd_repop_reply(client.4144.0:850 1.4) v1 pg pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'278 lcod 23'278 mlcod 23'278 active+clean]
2016-10-25 08:51:31.341570 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'278 lcod 23'278 mlcod 23'278 active+clean] handle_message: 0x7f2139773960
2016-10-25 08:51:31.341592 7f2110c6b700  7 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'278 lcod 23'278 mlcod 23'278 active+clean] sub_op_modify_reply: tid 1697 op  ack_type 4 from 2
2016-10-25 08:51:31.341596 7f2112c6f700 10 osd.3 23 dequeue_op 0x7f2139773840 prio 196 cost 0 latency 0.000090 osd_repop_reply(client.4144.0:857 1.0) v1 pg pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'280 crt=23'279 lcod 23'280 mlcod 23'279 active+clean]
2016-10-25 08:51:31.341621 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'279 lua=23'279 crt=23'278 lcod 23'278 mlcod 23'278 active+clean] repop_all_committed: repop tid 1697 all committed 
2016-10-25 08:51:31.341627 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'280 crt=23'279 lcod 23'280 mlcod 23'279 active+clean] handle_message: 0x7f2139773840
2016-10-25 08:51:31.341632 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'278 lcod 23'279 mlcod 23'278 active+clean] eval_repop repgather(0x7f213985c7e0 23'280 rep_tid=1697 committed?=1 applied?=0) wants=d
2016-10-25 08:51:31.341645 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'278 lcod 23'279 mlcod 23'278 active+clean]  commit: repgather(0x7f213985c7e0 23'280 rep_tid=1697 committed?=1 applied?=0)
2016-10-25 08:51:31.341651 7f2112c6f700  7 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'280 crt=23'279 lcod 23'280 mlcod 23'279 active+clean] sub_op_modify_reply: tid 1705 op  ack_type 4 from 2
2016-10-25 08:51:31.341659 7f2110c6b700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'278 lcod 23'279 mlcod 23'278 active+clean] log_op_stats osd_op(client.4144.0:850 1.5e875d24 benchmark_data_target167114251239.teuthology_13304_object849 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 0.000000 lat 1.113667
2016-10-25 08:51:31.341680 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'280 crt=23'279 lcod 23'280 mlcod 23'279 active+clean] repop_all_committed: repop tid 1705 all committed 
2016-10-25 08:51:31.341675 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'278 lcod 23'279 mlcod 23'278 active+clean]  sending commit on osd_op(client.4144.0:850 1.5e875d24 benchmark_data_target167114251239.teuthology_13304_object849 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f21398ddb80
2016-10-25 08:51:31.341693 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'280 crt=23'279 lcod 23'281 mlcod 23'279 active+clean] eval_repop repgather(0x7f2139756be0 23'282 rep_tid=1705 committed?=1 applied?=0) wants=d
2016-10-25 08:51:31.341695 7f2110c6b700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(850 benchmark_data_target167114251239.teuthology_13304_object849 [write 0~4194304] v23'280 uv280 ondisk = 0) v7 -- 0x7f21398ddb80 con 0
2016-10-25 08:51:31.341704 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'280 crt=23'279 lcod 23'281 mlcod 23'279 active+clean]  commit: repgather(0x7f2139756be0 23'282 rep_tid=1705 committed?=1 applied?=0)
2016-10-25 08:51:31.341713 7f2112c6f700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'280 crt=23'279 lcod 23'281 mlcod 23'279 active+clean] log_op_stats osd_op(client.4144.0:857 1.f43fc4f8 benchmark_data_target167114251239.teuthology_13304_object856 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 0.000000 lat 1.085492
2016-10-25 08:51:31.341723 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'280 crt=23'279 lcod 23'281 mlcod 23'279 active+clean]  sending commit on osd_op(client.4144.0:857 1.f43fc4f8 benchmark_data_target167114251239.teuthology_13304_object856 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f213ebcd9c0
2016-10-25 08:51:31.341731 7f2112c6f700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(857 benchmark_data_target167114251239.teuthology_13304_object856 [write 0~4194304] v23'282 uv282 ondisk = 0) v7 -- 0x7f213ebcd9c0 con 0
2016-10-25 08:51:31.341745 7f2112c6f700 10 osd.3 23 dequeue_op 0x7f2139773840 finish
2016-10-25 08:51:31.341780 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2281 ==== osd_repop_reply(client.4144.0:859 1.0) v1 ==== 83+0+0 (808247616 0 0) 0x7f214101c280 con 0x7f2139592800
2016-10-25 08:51:31.341790 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:859 1.0) v1 epoch 23
2016-10-25 08:51:31.341794 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:31.341800 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139773840 prio 196 cost 0 latency 0.000035 osd_repop_reply(client.4144.0:859 1.0) v1
2016-10-25 08:51:31.341831 7f211046a700 10 osd.3 23 dequeue_op 0x7f2139773840 prio 196 cost 0 latency 0.000065 osd_repop_reply(client.4144.0:859 1.0) v1 pg pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'280 crt=23'279 lcod 23'281 mlcod 23'279 active+clean]
2016-10-25 08:51:31.341843 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'280 crt=23'279 lcod 23'281 mlcod 23'279 active+clean] handle_message: 0x7f2139773840
2016-10-25 08:51:31.341854 7f211046a700  7 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'280 crt=23'279 lcod 23'281 mlcod 23'279 active+clean] sub_op_modify_reply: tid 1706 op  ack_type 4 from 2
2016-10-25 08:51:31.341866 7f2110c6b700 10 osd.3 23 dequeue_op 0x7f2139773960 finish
2016-10-25 08:51:31.341869 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'280 crt=23'279 lcod 23'281 mlcod 23'279 active+clean] repop_all_committed: repop tid 1706 all committed 
2016-10-25 08:51:31.341876 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'280 crt=23'279 lcod 23'282 mlcod 23'279 active+clean] eval_repop repgather(0x7f21396b9c20 23'283 rep_tid=1706 committed?=1 applied?=0) wants=d
2016-10-25 08:51:31.341884 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'280 crt=23'279 lcod 23'282 mlcod 23'279 active+clean]  commit: repgather(0x7f21396b9c20 23'283 rep_tid=1706 committed?=1 applied?=0)
2016-10-25 08:51:31.341892 7f211046a700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'280 crt=23'279 lcod 23'282 mlcod 23'279 active+clean] log_op_stats osd_op(client.4144.0:859 1.e3ef6c58 benchmark_data_target167114251239.teuthology_13304_object858 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 0.000000 lat 1.082660
2016-10-25 08:51:31.341901 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'280 crt=23'279 lcod 23'282 mlcod 23'279 active+clean]  sending commit on osd_op(client.4144.0:859 1.e3ef6c58 benchmark_data_target167114251239.teuthology_13304_object858 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f2140f4f700
2016-10-25 08:51:31.341900 7f2110c6b700 10 osd.3 23 dequeue_op 0x7f2139770000 prio 196 cost 0 latency 0.000884 osd_repop_reply(client.4144.0:854 1.4) v1 pg pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'278 lcod 23'279 mlcod 23'278 active+clean]
2016-10-25 08:51:31.341910 7f211046a700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(859 benchmark_data_target167114251239.teuthology_13304_object858 [write 0~4194304] v23'283 uv283 ondisk = 0) v7 -- 0x7f2140f4f700 con 0
2016-10-25 08:51:31.341919 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'278 lcod 23'279 mlcod 23'278 active+clean] handle_message: 0x7f2139770000
2016-10-25 08:51:31.341928 7f211046a700 10 osd.3 23 dequeue_op 0x7f2139773840 finish
2016-10-25 08:51:31.341935 7f2110c6b700  7 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'278 lcod 23'279 mlcod 23'278 active+clean] sub_op_modify_reply: tid 1703 op  ack_type 4 from 2
2016-10-25 08:51:31.341956 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'280 lua=23'279 crt=23'278 lcod 23'279 mlcod 23'278 active+clean] repop_all_committed: repop tid 1703 all committed 
2016-10-25 08:51:31.341967 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'279 crt=23'278 lcod 23'280 mlcod 23'278 active+clean] eval_repop repgather(0x7f21398aaac0 23'281 rep_tid=1703 committed?=1 applied?=0) wants=d
2016-10-25 08:51:31.341978 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'279 crt=23'278 lcod 23'280 mlcod 23'278 active+clean]  commit: repgather(0x7f21398aaac0 23'281 rep_tid=1703 committed?=1 applied?=0)
2016-10-25 08:51:31.341989 7f2110c6b700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'279 crt=23'278 lcod 23'280 mlcod 23'278 active+clean] log_op_stats osd_op(client.4144.0:854 1.5e6b0fbc benchmark_data_target167114251239.teuthology_13304_object853 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 0.000000 lat 1.091646
2016-10-25 08:51:31.342003 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'279 crt=23'278 lcod 23'280 mlcod 23'278 active+clean]  sending commit on osd_op(client.4144.0:854 1.5e6b0fbc benchmark_data_target167114251239.teuthology_13304_object853 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f2142e12940
2016-10-25 08:51:31.342015 7f2110c6b700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(854 benchmark_data_target167114251239.teuthology_13304_object853 [write 0~4194304] v23'281 uv281 ondisk = 0) v7 -- 0x7f2142e12940 con 0
2016-10-25 08:51:31.342045 7f2110c6b700 10 osd.3 23 dequeue_op 0x7f2139770000 finish
2016-10-25 08:51:31.342059 7f210e466700 10 osd.3 23 dequeue_op 0x7f2139771680 prio 196 cost 0 latency 0.000626 osd_repop_reply(client.4144.0:858 1.4) v1 pg pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'279 crt=23'278 lcod 23'280 mlcod 23'278 active+clean]
2016-10-25 08:51:31.342076 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'279 crt=23'278 lcod 23'280 mlcod 23'278 active+clean] handle_message: 0x7f2139771680
2016-10-25 08:51:31.342095 7f210e466700  7 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'279 crt=23'278 lcod 23'280 mlcod 23'278 active+clean] sub_op_modify_reply: tid 1704 op  ack_type 4 from 2
2016-10-25 08:51:31.342116 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'279 crt=23'278 lcod 23'280 mlcod 23'278 active+clean] repop_all_committed: repop tid 1704 all committed 
2016-10-25 08:51:31.342127 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'279 crt=23'278 lcod 23'281 mlcod 23'278 active+clean] eval_repop repgather(0x7f213985e400 23'282 rep_tid=1704 committed?=1 applied?=0) wants=d
2016-10-25 08:51:31.342139 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'279 crt=23'278 lcod 23'281 mlcod 23'278 active+clean]  commit: repgather(0x7f213985e400 23'282 rep_tid=1704 committed?=1 applied?=0)
2016-10-25 08:51:31.342151 7f210e466700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'279 crt=23'278 lcod 23'281 mlcod 23'278 active+clean] log_op_stats osd_op(client.4144.0:858 1.ef1eb2bc benchmark_data_target167114251239.teuthology_13304_object857 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 0.000000 lat 1.084390
2016-10-25 08:51:31.342166 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'279 crt=23'278 lcod 23'281 mlcod 23'278 active+clean]  sending commit on osd_op(client.4144.0:858 1.ef1eb2bc benchmark_data_target167114251239.teuthology_13304_object857 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f2148003440
2016-10-25 08:51:31.342179 7f210e466700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(858 benchmark_data_target167114251239.teuthology_13304_object857 [write 0~4194304] v23'282 uv282 ondisk = 0) v7 -- 0x7f2148003440 con 0
2016-10-25 08:51:31.342197 7f210e466700 10 osd.3 23 dequeue_op 0x7f2139771680 finish
2016-10-25 08:51:31.356684 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 666 ==== osd_op(client.4144.0:866 1.3deccf7 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (3125325846 0 3789470788) 0x7f2148003440 con 0x7f2147f52800
2016-10-25 08:51:31.356725 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:31.356733 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139773840 prio 63 cost 4194304 latency 0.005118 osd_op(client.4144.0:866 1.3deccf7 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.356783 7f2111c6d700 10 osd.3 23 dequeue_op 0x7f2139773840 prio 63 cost 4194304 latency 0.005168 osd_op(client.4144.0:866 1.3deccf7 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean]
2016-10-25 08:51:31.356805 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] handle_message: 0x7f2139773840
2016-10-25 08:51:31.356823 7f2111c6d700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:31.356842 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] do_op osd_op(client.4144.0:866 1.3deccf7 benchmark_data_target167114251239.teuthology_13304_object865 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:31.356864 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] get_object_context: obc NOT found in cache: 1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head
2016-10-25 08:51:31.356875 7f2111c6d700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head# '_'
2016-10-25 08:51:31.356938 7f2111c6d700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.7_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject865__head_03DECCF7__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.356946 7f2111c6d700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head# '_' = -2
2016-10-25 08:51:31.356950 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] get_object_context: no obc for soid 1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head but can_create
2016-10-25 08:51:31.356962 7f2111c6d700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:snapdir# 'snapset'
2016-10-25 08:51:31.356979 7f2111c6d700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.7_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject865__snapdir_03DECCF7__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.356982 7f2111c6d700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:snapdir# 'snapset' = -2
2016-10-25 08:51:31.357000 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] create_object_context 0x7f213c574580 1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head 
2016-10-25 08:51:31.357008 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] populate_obc_watchers 1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head
2016-10-25 08:51:31.357015 7f2111c6d700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head
2016-10-25 08:51:31.357022 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] get_object_context: 0x7f213c574580 1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head rwstate(none n=0 w=0) oi: 1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f213946b420 snapset: 0=[]:[]
2016-10-25 08:51:31.357033 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] find_object_context 1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head @head oi=1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.357043 7f2111c6d700 25 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] do_op oi 1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.357054 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] get_object_context: obc NOT found in cache: 1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:snapdir
2016-10-25 08:51:31.357061 7f2111c6d700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:snapdir# '_'
2016-10-25 08:51:31.357075 7f2111c6d700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.7_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject865__snapdir_03DECCF7__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.357078 7f2111c6d700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:snapdir# '_' = -2
2016-10-25 08:51:31.357081 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] get_object_context: no obc for soid 1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:snapdir and !can_create
2016-10-25 08:51:31.357097 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] execute_ctx 0x7f2148072a00
2016-10-25 08:51:31.357104 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] do_op 1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head [write 0~4194304] ov 0'0 av 23'266 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:31.357121 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] do_osd_op 1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head [write 0~4194304]
2016-10-25 08:51:31.357137 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:31.357438 7f2111c6d700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] make_writeable 1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head snapset=0x7f213946b458  snapc=0=[]
2016-10-25 08:51:31.357445 7f2111c6d700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean]  setting DIRTY flag
2016-10-25 08:51:31.357451 7f2111c6d700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] make_writeable 1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head done, snapset=0=[]:[]+head
2016-10-25 08:51:31.357458 7f2111c6d700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] finish_ctx 1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head 0x7f2148072a00 op modify  
2016-10-25 08:51:31.357468 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean]  set mtime to 2016-10-25 08:51:31.343707
2016-10-25 08:51:31.357482 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=265 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean]  final snapset 0=[]:[]+head in 1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head
2016-10-25 08:51:31.357497 7f2111c6d700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean]  zeroing write result code 0
2016-10-25 08:51:31.357507 7f2111c6d700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean]  op order client.4144 tid 866 (first)
2016-10-25 08:51:31.357514 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] new_repop rep_tid 1713 on osd_op(client.4144.0:866 1.3deccf7 benchmark_data_target167114251239.teuthology_13304_object865 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.357523 7f2111c6d700  7 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] issue_repop rep_tid 1713 o 1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head
2016-10-25 08:51:31.357556 7f2111c6d700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:31.357559 7f2111c6d700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:866 1.7 1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head v 23'266) v1 -- 0x7f2140fcd200 con 0
2016-10-25 08:51:31.357868 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'265 (0'0,23'265] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] append_log log((0'0,23'265], crt=23'263) [23'266 (0'0) modify   1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head by client.4144.0:866 2016-10-25 08:51:31.343707 0]
2016-10-25 08:51:31.357880 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] add_log_entry 23'266 (0'0) modify   1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head by client.4144.0:866 2016-10-25 08:51:31.343707 0
2016-10-25 08:51:31.357889 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] append_log: trimming to 23'263 entries 
2016-10-25 08:51:31.357900 7f2111c6d700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'266, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:31.358094 7f2111c6d700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139243c80 osr(1.7 0x7f2139474be0)
2016-10-25 08:51:31.358098 7f2111c6d700 10 journal prepare_entry [Transaction(0x7f2147ea9d40)]
2016-10-25 08:51:31.358392 7f2111c6d700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:31.359307 7f2111c6d700 10 journal op_submit_start 2349
2016-10-25 08:51:31.359311 7f2111c6d700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2349 [Transaction(0x7f2147ea9d40)]
2016-10-25 08:51:31.359315 7f2111c6d700 10 journal op_journal_transactions 2349 reqid_t client.4144.0:866
2016-10-25 08:51:31.359318 7f2111c6d700  5 journal submit_entry seq 2349 len 4202496 (0x7f214811dc20)
2016-10-25 08:51:31.359357 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:31.359360 7f2123404700 20 journal write_thread_entry aio throttle: aio num 6 bytes 25214976 ... exp 12 min_new 4096 ... pending 4202496
2016-10-25 08:51:31.359363 7f2123404700 10 journal room 522170367 max_size 1072709632 pos 941629440 header.start 391094272 top 4096
2016-10-25 08:51:31.359365 7f2123404700 10 journal check_for_full at 941629440 : 4202496 < 522170367
2016-10-25 08:51:31.359367 7f2123404700 15 journal prepare_single_write 1 will write 941629440 : seq 2349 len 4195598 -> 4202496
2016-10-25 08:51:31.359377 7f2123404700 20 journal prepare_multi_write queue_pos now 945831936
2016-10-25 08:51:31.359379 7f2123404700 15 journal do_aio_write writing 941629440~4202496
2016-10-25 08:51:31.359381 7f2123404700 20 journal write_aio_bl 941629440~4202496 seq 2349
2016-10-25 08:51:31.359385 7f2123404700 20 journal write_aio_bl .. 941629440~4202496 in 1
2016-10-25 08:51:31.359739 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:31.359743 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:31.359745 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:31.359750 7f2111c6d700 10 journal op_submit_finish 2349
2016-10-25 08:51:31.359755 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] eval_repop repgather(0x7f21397ebde0 23'266 rep_tid=1713 committed?=0 applied?=0) wants=d
2016-10-25 08:51:31.359776 7f2111c6d700 10 osd.3 23 dequeue_op 0x7f2139773840 finish
2016-10-25 08:51:31.362661 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 667 ==== osd_op(client.4144.0:867 1.7058eed6 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (146539256 0 3103477616) 0x7f2142e12940 con 0x7f2147f52800
2016-10-25 08:51:31.362724 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:31.362732 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139770ea0 prio 63 cost 4194304 latency 0.002929 osd_op(client.4144.0:867 1.7058eed6 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.362774 7f210fc69700 10 osd.3 23 dequeue_op 0x7f2139770ea0 prio 63 cost 4194304 latency 0.002971 osd_op(client.4144.0:867 1.7058eed6 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean]
2016-10-25 08:51:31.362798 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] handle_message: 0x7f2139770ea0
2016-10-25 08:51:31.362817 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:31.362835 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] do_op osd_op(client.4144.0:867 1.7058eed6 benchmark_data_target167114251239.teuthology_13304_object866 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:31.362859 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] get_object_context: obc NOT found in cache: 1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head
2016-10-25 08:51:31.362871 7f210fc69700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head# '_'
2016-10-25 08:51:31.362932 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.6_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject866__head_7058EED6__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.362941 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head# '_' = -2
2016-10-25 08:51:31.362945 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] get_object_context: no obc for soid 1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head but can_create
2016-10-25 08:51:31.362955 7f210fc69700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:snapdir# 'snapset'
2016-10-25 08:51:31.362972 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.6_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject866__snapdir_7058EED6__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.362975 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:snapdir# 'snapset' = -2
2016-10-25 08:51:31.362994 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] create_object_context 0x7f213ebcd700 1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head 
2016-10-25 08:51:31.363003 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] populate_obc_watchers 1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head
2016-10-25 08:51:31.363011 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head
2016-10-25 08:51:31.363018 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] get_object_context: 0x7f213ebcd700 1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head rwstate(none n=0 w=0) oi: 1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f21395864c0 snapset: 0=[]:[]
2016-10-25 08:51:31.363029 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] find_object_context 1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head @head oi=1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.363040 7f210fc69700 25 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] do_op oi 1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.363050 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] get_object_context: obc NOT found in cache: 1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:snapdir
2016-10-25 08:51:31.363057 7f210fc69700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:snapdir# '_'
2016-10-25 08:51:31.363072 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.6_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject866__snapdir_7058EED6__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.363075 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:snapdir# '_' = -2
2016-10-25 08:51:31.363078 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] get_object_context: no obc for soid 1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:snapdir and !can_create
2016-10-25 08:51:31.363093 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] execute_ctx 0x7f213eac7500
2016-10-25 08:51:31.363101 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] do_op 1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head [write 0~4194304] ov 0'0 av 23'275 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:31.363111 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] do_osd_op 1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head [write 0~4194304]
2016-10-25 08:51:31.363118 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:31.363417 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] make_writeable 1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head snapset=0x7f21395864f8  snapc=0=[]
2016-10-25 08:51:31.363425 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean]  setting DIRTY flag
2016-10-25 08:51:31.363430 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] make_writeable 1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head done, snapset=0=[]:[]+head
2016-10-25 08:51:31.363437 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] finish_ctx 1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head 0x7f213eac7500 op modify  
2016-10-25 08:51:31.363447 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean]  set mtime to 2016-10-25 08:51:31.343878
2016-10-25 08:51:31.363460 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=274 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean]  final snapset 0=[]:[]+head in 1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head
2016-10-25 08:51:31.363477 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean]  zeroing write result code 0
2016-10-25 08:51:31.363485 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean]  op order client.4144 tid 867 (first)
2016-10-25 08:51:31.363492 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] new_repop rep_tid 1714 on osd_op(client.4144.0:867 1.7058eed6 benchmark_data_target167114251239.teuthology_13304_object866 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.363528 7f210fc69700  7 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] issue_repop rep_tid 1714 o 1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head
2016-10-25 08:51:31.363563 7f210fc69700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:31.363566 7f210fc69700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:867 1.6 1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head v 23'275) v1 -- 0x7f2141031e00 con 0
2016-10-25 08:51:31.363878 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'274 (0'0,23'274] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] append_log log((0'0,23'274], crt=23'272) [23'275 (0'0) modify   1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head by client.4144.0:867 2016-10-25 08:51:31.343878 0]
2016-10-25 08:51:31.363890 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] add_log_entry 23'275 (0'0) modify   1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head by client.4144.0:867 2016-10-25 08:51:31.343878 0
2016-10-25 08:51:31.363899 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] append_log: trimming to 23'272 entries 
2016-10-25 08:51:31.363910 7f210fc69700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'275, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:31.364113 7f210fc69700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139243700 osr(1.6 0x7f2139474f10)
2016-10-25 08:51:31.364123 7f210fc69700 10 journal prepare_entry [Transaction(0x7f21481f8d80)]
2016-10-25 08:51:31.364429 7f210fc69700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:31.365469 7f210fc69700 10 journal op_submit_start 2350
2016-10-25 08:51:31.365475 7f210fc69700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2350 [Transaction(0x7f21481f8d80)]
2016-10-25 08:51:31.365481 7f210fc69700 10 journal op_journal_transactions 2350 reqid_t client.4144.0:867
2016-10-25 08:51:31.365484 7f210fc69700  5 journal submit_entry seq 2350 len 4202496 (0x7f214811cff0)
2016-10-25 08:51:31.369244 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:31.369255 7f2123404700 20 journal write_thread_entry aio throttle: aio num 7 bytes 29417472 ... exp 14 min_new 16384 ... pending 4202496
2016-10-25 08:51:31.369260 7f2123404700 10 journal room 517967871 max_size 1072709632 pos 945831936 header.start 391094272 top 4096
2016-10-25 08:51:31.369263 7f2123404700 10 journal check_for_full at 945831936 : 4202496 < 517967871
2016-10-25 08:51:31.369265 7f2123404700 15 journal prepare_single_write 1 will write 945831936 : seq 2350 len 4195598 -> 4202496
2016-10-25 08:51:31.369286 7f2123404700 20 journal prepare_multi_write queue_pos now 950034432
2016-10-25 08:51:31.369287 7f2123404700 15 journal do_aio_write writing 945831936~4202496
2016-10-25 08:51:31.369290 7f2123404700 20 journal write_aio_bl 945831936~4202496 seq 2350
2016-10-25 08:51:31.369293 7f2123404700 20 journal write_aio_bl .. 945831936~4202496 in 1
2016-10-25 08:51:31.369632 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:31.369637 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:31.369640 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:31.369648 7f210fc69700 10 journal op_submit_finish 2350
2016-10-25 08:51:31.369654 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] eval_repop repgather(0x7f2139719320 23'275 rep_tid=1714 committed?=0 applied?=0) wants=d
2016-10-25 08:51:31.369680 7f210fc69700 10 osd.3 23 dequeue_op 0x7f2139770ea0 finish
2016-10-25 08:51:31.375838 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 668 ==== osd_op(client.4144.0:868 1.6030949 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (2523291169 0 1933823027) 0x7f2140f4f700 con 0x7f2147f52800
2016-10-25 08:51:31.375939 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:31.375966 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f21397721c0 prio 63 cost 4194304 latency 0.006117 osd_op(client.4144.0:868 1.6030949 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.376031 7f211246e700 10 osd.3 23 dequeue_op 0x7f21397721c0 prio 63 cost 4194304 latency 0.006182 osd_op(client.4144.0:868 1.6030949 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean]
2016-10-25 08:51:31.376173 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] handle_message: 0x7f21397721c0
2016-10-25 08:51:31.376197 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:31.376223 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] do_op osd_op(client.4144.0:868 1.6030949 benchmark_data_target167114251239.teuthology_13304_object867 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:31.376253 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] get_object_context: obc NOT found in cache: 1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head
2016-10-25 08:51:31.376269 7f211246e700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head# '_'
2016-10-25 08:51:31.376349 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.1_head/DIR_9/benchmark\udata\utarget167114251239.teuthology\u13304\uobject867__head_06030949__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.376359 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head# '_' = -2
2016-10-25 08:51:31.376364 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] get_object_context: no obc for soid 1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head but can_create
2016-10-25 08:51:31.376380 7f211246e700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:snapdir# 'snapset'
2016-10-25 08:51:31.376411 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.1_head/DIR_9/benchmark\udata\utarget167114251239.teuthology\u13304\uobject867__snapdir_06030949__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.376416 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:snapdir# 'snapset' = -2
2016-10-25 08:51:31.376442 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] create_object_context 0x7f2142eb9e40 1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head 
2016-10-25 08:51:31.376456 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] populate_obc_watchers 1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head
2016-10-25 08:51:31.376467 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head
2016-10-25 08:51:31.376477 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] get_object_context: 0x7f2142eb9e40 1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head rwstate(none n=0 w=0) oi: 1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f21396debc0 snapset: 0=[]:[]
2016-10-25 08:51:31.376493 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] find_object_context 1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head @head oi=1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.376509 7f211246e700 25 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] do_op oi 1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.376525 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] get_object_context: obc NOT found in cache: 1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:snapdir
2016-10-25 08:51:31.376536 7f211246e700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:snapdir# '_'
2016-10-25 08:51:31.376561 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.1_head/DIR_9/benchmark\udata\utarget167114251239.teuthology\u13304\uobject867__snapdir_06030949__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.376566 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:snapdir# '_' = -2
2016-10-25 08:51:31.376570 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] get_object_context: no obc for soid 1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:snapdir and !can_create
2016-10-25 08:51:31.376594 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] execute_ctx 0x7f2148072300
2016-10-25 08:51:31.376607 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] do_op 1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head [write 0~4194304] ov 0'0 av 23'323 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:31.376622 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] do_osd_op 1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head [write 0~4194304]
2016-10-25 08:51:31.376632 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:31.376979 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] make_writeable 1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head snapset=0x7f21396debf8  snapc=0=[]
2016-10-25 08:51:31.376999 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean]  setting DIRTY flag
2016-10-25 08:51:31.377013 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] make_writeable 1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head done, snapset=0=[]:[]+head
2016-10-25 08:51:31.377028 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] finish_ctx 1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head 0x7f2148072300 op modify  
2016-10-25 08:51:31.377047 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean]  set mtime to 2016-10-25 08:51:31.344016
2016-10-25 08:51:31.377072 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=322 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean]  final snapset 0=[]:[]+head in 1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head
2016-10-25 08:51:31.377103 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean]  zeroing write result code 0
2016-10-25 08:51:31.377121 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean]  op order client.4144 tid 868 (first)
2016-10-25 08:51:31.377135 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] new_repop rep_tid 1715 on osd_op(client.4144.0:868 1.6030949 benchmark_data_target167114251239.teuthology_13304_object867 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.377215 7f211246e700  7 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] issue_repop rep_tid 1715 o 1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head
2016-10-25 08:51:31.377291 7f211246e700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:31.377296 7f211246e700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:868 1.1 1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head v 23'323) v1 -- 0x7f2141055600 con 0
2016-10-25 08:51:31.377622 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'322 (0'0,23'322] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] append_log log((0'0,23'322], crt=23'318) [23'323 (0'0) modify   1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head by client.4144.0:868 2016-10-25 08:51:31.344016 0]
2016-10-25 08:51:31.377653 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'318 lcod 23'321 mlcod 23'320 active+clean] add_log_entry 23'323 (0'0) modify   1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head by client.4144.0:868 2016-10-25 08:51:31.344016 0
2016-10-25 08:51:31.377677 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] append_log: trimming to 23'320 entries 23'319 (0'0) modify   1:9275aa7e:::benchmark_data_target167114251239.teuthology_13304_object839:head by client.4144.0:840 2016-10-25 08:51:29.657243 0,23'320 (0'0) modify   1:9669e639:::benchmark_data_target167114251239.teuthology_13304_object840:head by client.4144.0:841 2016-10-25 08:51:29.783252 0
2016-10-25 08:51:31.377710 7f211246e700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'323, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:31.378025 7f211246e700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139241600 osr(1.1 0x7f21394758a0)
2016-10-25 08:51:31.378111 7f211246e700 10 journal prepare_entry [Transaction(0x7f2147f70380)]
2016-10-25 08:51:31.378508 7f211246e700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:31.381982 7f211246e700 10 journal op_submit_start 2351
2016-10-25 08:51:31.382042 7f211246e700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2351 [Transaction(0x7f2147f70380)]
2016-10-25 08:51:31.382057 7f211246e700 10 journal op_journal_transactions 2351 reqid_t client.4144.0:868
2016-10-25 08:51:31.382068 7f211246e700  5 journal submit_entry seq 2351 len 4202496 (0x7f21414aa720)
2016-10-25 08:51:31.382132 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:31.382136 7f2123404700 20 journal write_thread_entry aio throttle: aio num 8 bytes 33619968 ... exp 16 min_new 65536 ... pending 4202496
2016-10-25 08:51:31.382139 7f2123404700 10 journal room 513765375 max_size 1072709632 pos 950034432 header.start 391094272 top 4096
2016-10-25 08:51:31.382141 7f2123404700 10 journal check_for_full at 950034432 : 4202496 < 513765375
2016-10-25 08:51:31.382143 7f2123404700 15 journal prepare_single_write 1 will write 950034432 : seq 2351 len 4195598 -> 4202496
2016-10-25 08:51:31.382163 7f2123404700 20 journal prepare_multi_write queue_pos now 954236928
2016-10-25 08:51:31.382165 7f2123404700 15 journal do_aio_write writing 950034432~4202496
2016-10-25 08:51:31.382167 7f2123404700 20 journal write_aio_bl 950034432~4202496 seq 2351
2016-10-25 08:51:31.382188 7f2123404700 20 journal write_aio_bl .. 950034432~4202496 in 1
2016-10-25 08:51:31.382613 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:31.382617 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:31.382619 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:31.382695 7f211246e700 10 journal op_submit_finish 2351
2016-10-25 08:51:31.382711 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] eval_repop repgather(0x7f213981cfc0 23'323 rep_tid=1715 committed?=0 applied?=0) wants=d
2016-10-25 08:51:31.382771 7f211246e700 10 osd.3 23 dequeue_op 0x7f21397721c0 finish
2016-10-25 08:51:31.388054 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 669 ==== osd_op(client.4144.0:870 1.7c00d607 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (1810963763 0 2158298725) 0x7f213ebcd9c0 con 0x7f2147f52800
2016-10-25 08:51:31.388205 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:31.388228 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139772be0 prio 63 cost 4194304 latency 0.005414 osd_op(client.4144.0:870 1.7c00d607 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.388293 7f210f468700 10 osd.3 23 dequeue_op 0x7f2139772be0 prio 63 cost 4194304 latency 0.005478 osd_op(client.4144.0:870 1.7c00d607 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean]
2016-10-25 08:51:31.388318 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] handle_message: 0x7f2139772be0
2016-10-25 08:51:31.388354 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:31.388372 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] do_op osd_op(client.4144.0:870 1.7c00d607 benchmark_data_target167114251239.teuthology_13304_object869 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:31.388397 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] get_object_context: obc NOT found in cache: 1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head
2016-10-25 08:51:31.388407 7f210f468700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head# '_'
2016-10-25 08:51:31.388481 7f210f468700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.7_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject869__head_7C00D607__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.388491 7f210f468700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head# '_' = -2
2016-10-25 08:51:31.388494 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] get_object_context: no obc for soid 1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head but can_create
2016-10-25 08:51:31.388530 7f210f468700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:snapdir# 'snapset'
2016-10-25 08:51:31.388548 7f210f468700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.7_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject869__snapdir_7C00D607__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.388551 7f210f468700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:snapdir# 'snapset' = -2
2016-10-25 08:51:31.388571 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] create_object_context 0x7f2148001080 1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head 
2016-10-25 08:51:31.388580 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] populate_obc_watchers 1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head
2016-10-25 08:51:31.388612 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head
2016-10-25 08:51:31.388618 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] get_object_context: 0x7f2148001080 1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head rwstate(none n=0 w=0) oi: 1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f2140ea64c0 snapset: 0=[]:[]
2016-10-25 08:51:31.388628 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] find_object_context 1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head @head oi=1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.388637 7f210f468700 25 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] do_op oi 1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.388646 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] get_object_context: obc NOT found in cache: 1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:snapdir
2016-10-25 08:51:31.388653 7f210f468700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:snapdir# '_'
2016-10-25 08:51:31.388687 7f210f468700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.7_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject869__snapdir_7C00D607__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.388691 7f210f468700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:snapdir# '_' = -2
2016-10-25 08:51:31.388694 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] get_object_context: no obc for soid 1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:snapdir and !can_create
2016-10-25 08:51:31.388710 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] execute_ctx 0x7f213ebb7500
2016-10-25 08:51:31.388717 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] do_op 1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head [write 0~4194304] ov 0'0 av 23'267 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:31.388727 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] do_osd_op 1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head [write 0~4194304]
2016-10-25 08:51:31.388733 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:31.389054 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] make_writeable 1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head snapset=0x7f2140ea64f8  snapc=0=[]
2016-10-25 08:51:31.389063 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean]  setting DIRTY flag
2016-10-25 08:51:31.389067 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] make_writeable 1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head done, snapset=0=[]:[]+head
2016-10-25 08:51:31.389086 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] finish_ctx 1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head 0x7f213ebb7500 op modify  
2016-10-25 08:51:31.389096 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean]  set mtime to 2016-10-25 08:51:31.346527
2016-10-25 08:51:31.389108 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=266 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean]  final snapset 0=[]:[]+head in 1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head
2016-10-25 08:51:31.389122 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean]  zeroing write result code 0
2016-10-25 08:51:31.389132 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean]  op order client.4144 tid 870 (first)
2016-10-25 08:51:31.389138 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] new_repop rep_tid 1716 on osd_op(client.4144.0:870 1.7c00d607 benchmark_data_target167114251239.teuthology_13304_object869 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.389146 7f210f468700  7 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] issue_repop rep_tid 1716 o 1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head
2016-10-25 08:51:31.389217 7f210f468700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:31.389219 7f210f468700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:870 1.7 1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head v 23'267) v1 -- 0x7f2141059600 con 0
2016-10-25 08:51:31.389544 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'266 (0'0,23'266] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] append_log log((0'0,23'266], crt=23'263) [23'267 (0'0) modify   1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head by client.4144.0:870 2016-10-25 08:51:31.346527 0]
2016-10-25 08:51:31.389558 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] add_log_entry 23'267 (0'0) modify   1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head by client.4144.0:870 2016-10-25 08:51:31.346527 0
2016-10-25 08:51:31.389585 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] append_log: trimming to 23'263 entries 
2016-10-25 08:51:31.389597 7f210f468700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'267, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:31.389807 7f210f468700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139243c80 osr(1.7 0x7f2139474be0)
2016-10-25 08:51:31.389831 7f210f468700 10 journal prepare_entry [Transaction(0x7f2147f688c0)]
2016-10-25 08:51:31.390114 7f210f468700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:31.391088 7f210f468700 10 journal op_submit_start 2352
2016-10-25 08:51:31.391094 7f210f468700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2352 [Transaction(0x7f2147f688c0)]
2016-10-25 08:51:31.391099 7f210f468700 10 journal op_journal_transactions 2352 reqid_t client.4144.0:870
2016-10-25 08:51:31.391102 7f210f468700  5 journal submit_entry seq 2352 len 4202496 (0x7f2140fb6970)
2016-10-25 08:51:31.391375 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:31.391382 7f2123404700 20 journal write_thread_entry aio throttle: aio num 9 bytes 37822464 ... exp 18 min_new 262144 ... pending 4202496
2016-10-25 08:51:31.391386 7f2123404700 10 journal room 509562879 max_size 1072709632 pos 954236928 header.start 391094272 top 4096
2016-10-25 08:51:31.391388 7f2123404700 10 journal check_for_full at 954236928 : 4202496 < 509562879
2016-10-25 08:51:31.391389 7f2123404700 15 journal prepare_single_write 1 will write 954236928 : seq 2352 len 4195598 -> 4202496
2016-10-25 08:51:31.391402 7f2123404700 20 journal prepare_multi_write queue_pos now 958439424
2016-10-25 08:51:31.391403 7f2123404700 15 journal do_aio_write writing 954236928~4202496
2016-10-25 08:51:31.391405 7f2123404700 20 journal write_aio_bl 954236928~4202496 seq 2352
2016-10-25 08:51:31.391408 7f2123404700 20 journal write_aio_bl .. 954236928~4202496 in 1
2016-10-25 08:51:31.391648 7f210f468700 10 journal op_submit_finish 2352
2016-10-25 08:51:31.391664 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] eval_repop repgather(0x7f2139643320 23'267 rep_tid=1716 committed?=0 applied?=0) wants=d
2016-10-25 08:51:31.391704 7f210f468700 10 osd.3 23 dequeue_op 0x7f2139772be0 finish
2016-10-25 08:51:31.393944 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 670 ==== osd_op(client.4144.0:871 1.b7f2f276 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (1825840194 0 1481703379) 0x7f21398ddb80 con 0x7f2147f52800
2016-10-25 08:51:31.394030 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:31.394048 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139754fc0 prio 63 cost 4194304 latency 0.002307 osd_op(client.4144.0:871 1.b7f2f276 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.394089 7f210fc69700 10 osd.3 23 dequeue_op 0x7f2139754fc0 prio 63 cost 4194304 latency 0.002348 osd_op(client.4144.0:871 1.b7f2f276 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean]
2016-10-25 08:51:31.394111 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] handle_message: 0x7f2139754fc0
2016-10-25 08:51:31.394129 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:31.394147 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] do_op osd_op(client.4144.0:871 1.b7f2f276 benchmark_data_target167114251239.teuthology_13304_object870 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:31.394171 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] get_object_context: obc NOT found in cache: 1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head
2016-10-25 08:51:31.394181 7f210fc69700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head# '_'
2016-10-25 08:51:31.394233 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.6_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject870__head_B7F2F276__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.394240 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head# '_' = -2
2016-10-25 08:51:31.394243 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] get_object_context: no obc for soid 1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head but can_create
2016-10-25 08:51:31.394253 7f210fc69700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:snapdir# 'snapset'
2016-10-25 08:51:31.394266 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.6_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject870__snapdir_B7F2F276__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.394269 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:snapdir# 'snapset' = -2
2016-10-25 08:51:31.394287 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] create_object_context 0x7f2141378c00 1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head 
2016-10-25 08:51:31.394295 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] populate_obc_watchers 1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head
2016-10-25 08:51:31.394301 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head
2016-10-25 08:51:31.394306 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] get_object_context: 0x7f2141378c00 1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head rwstate(none n=0 w=0) oi: 1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f2139586920 snapset: 0=[]:[]
2016-10-25 08:51:31.394316 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] find_object_context 1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head @head oi=1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.394325 7f210fc69700 25 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] do_op oi 1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.394334 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] get_object_context: obc NOT found in cache: 1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:snapdir
2016-10-25 08:51:31.394340 7f210fc69700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:snapdir# '_'
2016-10-25 08:51:31.394352 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.6_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject870__snapdir_B7F2F276__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.394354 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:snapdir# '_' = -2
2016-10-25 08:51:31.394357 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] get_object_context: no obc for soid 1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:snapdir and !can_create
2016-10-25 08:51:31.394370 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] execute_ctx 0x7f2139417100
2016-10-25 08:51:31.394377 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] do_op 1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head [write 0~4194304] ov 0'0 av 23'276 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:31.394386 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] do_osd_op 1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head [write 0~4194304]
2016-10-25 08:51:31.394393 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:31.394645 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] make_writeable 1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head snapset=0x7f2139586958  snapc=0=[]
2016-10-25 08:51:31.394652 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean]  setting DIRTY flag
2016-10-25 08:51:31.394656 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] make_writeable 1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head done, snapset=0=[]:[]+head
2016-10-25 08:51:31.394663 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] finish_ctx 1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head 0x7f2139417100 op modify  
2016-10-25 08:51:31.394671 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean]  set mtime to 2016-10-25 08:51:31.346684
2016-10-25 08:51:31.394683 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=275 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean]  final snapset 0=[]:[]+head in 1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head
2016-10-25 08:51:31.394697 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean]  zeroing write result code 0
2016-10-25 08:51:31.394706 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean]  op order client.4144 tid 871 (first)
2016-10-25 08:51:31.394711 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] new_repop rep_tid 1717 on osd_op(client.4144.0:871 1.b7f2f276 benchmark_data_target167114251239.teuthology_13304_object870 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.394719 7f210fc69700  7 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] issue_repop rep_tid 1717 o 1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head
2016-10-25 08:51:31.394750 7f210fc69700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:31.394752 7f210fc69700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:871 1.6 1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head v 23'276) v1 -- 0x7f2141031800 con 0
2016-10-25 08:51:31.395014 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'275 (0'0,23'275] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] append_log log((0'0,23'275], crt=23'272) [23'276 (0'0) modify   1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head by client.4144.0:871 2016-10-25 08:51:31.346684 0]
2016-10-25 08:51:31.395025 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] add_log_entry 23'276 (0'0) modify   1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head by client.4144.0:871 2016-10-25 08:51:31.346684 0
2016-10-25 08:51:31.395033 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] append_log: trimming to 23'272 entries 
2016-10-25 08:51:31.395042 7f210fc69700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'276, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:31.395236 7f210fc69700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139243700 osr(1.6 0x7f2139474f10)
2016-10-25 08:51:31.395240 7f210fc69700 10 journal prepare_entry [Transaction(0x7f21415d28c0)]
2016-10-25 08:51:31.395486 7f210fc69700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:31.396397 7f210fc69700 10 journal op_submit_start 2353
2016-10-25 08:51:31.396405 7f210fc69700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2353 [Transaction(0x7f21415d28c0)]
2016-10-25 08:51:31.396410 7f210fc69700 10 journal op_journal_transactions 2353 reqid_t client.4144.0:871
2016-10-25 08:51:31.396413 7f210fc69700  5 journal submit_entry seq 2353 len 4202496 (0x7f2140fb4c00)
2016-10-25 08:51:31.396431 7f210fc69700 10 journal op_submit_finish 2353
2016-10-25 08:51:31.396437 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] eval_repop repgather(0x7f21397f8ea0 23'276 rep_tid=1717 committed?=0 applied?=0) wants=d
2016-10-25 08:51:31.396457 7f210fc69700 10 osd.3 23 dequeue_op 0x7f2139754fc0 finish
2016-10-25 08:51:31.398280 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 671 ==== osd_op(client.4144.0:872 1.406a57b1 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (2156631454 0 2481486992) 0x7f2142f96ec0 con 0x7f2147f52800
2016-10-25 08:51:31.398364 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:31.398381 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139756760 prio 63 cost 4194304 latency 0.001659 osd_op(client.4144.0:872 1.406a57b1 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.398425 7f211246e700 10 osd.3 23 dequeue_op 0x7f2139756760 prio 63 cost 4194304 latency 0.001702 osd_op(client.4144.0:872 1.406a57b1 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean]
2016-10-25 08:51:31.398450 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] handle_message: 0x7f2139756760
2016-10-25 08:51:31.398469 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:31.398489 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] do_op osd_op(client.4144.0:872 1.406a57b1 benchmark_data_target167114251239.teuthology_13304_object871 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:31.398513 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] get_object_context: obc NOT found in cache: 1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head
2016-10-25 08:51:31.398524 7f211246e700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head# '_'
2016-10-25 08:51:31.398590 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.1_head/DIR_1/benchmark\udata\utarget167114251239.teuthology\u13304\uobject871__head_406A57B1__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.398598 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head# '_' = -2
2016-10-25 08:51:31.398601 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] get_object_context: no obc for soid 1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head but can_create
2016-10-25 08:51:31.398611 7f211246e700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:snapdir# 'snapset'
2016-10-25 08:51:31.398628 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.1_head/DIR_1/benchmark\udata\utarget167114251239.teuthology\u13304\uobject871__snapdir_406A57B1__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.398631 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:snapdir# 'snapset' = -2
2016-10-25 08:51:31.398651 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] create_object_context 0x7f213ea56000 1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head 
2016-10-25 08:51:31.398659 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] populate_obc_watchers 1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head
2016-10-25 08:51:31.398665 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head
2016-10-25 08:51:31.398671 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] get_object_context: 0x7f213ea56000 1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head rwstate(none n=0 w=0) oi: 1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f21396dc620 snapset: 0=[]:[]
2016-10-25 08:51:31.398680 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] find_object_context 1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head @head oi=1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.398691 7f211246e700 25 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] do_op oi 1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.398700 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] get_object_context: obc NOT found in cache: 1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:snapdir
2016-10-25 08:51:31.398707 7f211246e700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:snapdir# '_'
2016-10-25 08:51:31.398721 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.1_head/DIR_1/benchmark\udata\utarget167114251239.teuthology\u13304\uobject871__snapdir_406A57B1__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.398724 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:snapdir# '_' = -2
2016-10-25 08:51:31.398726 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] get_object_context: no obc for soid 1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:snapdir and !can_create
2016-10-25 08:51:31.398742 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] execute_ctx 0x7f2148071c00
2016-10-25 08:51:31.398750 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] do_op 1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head [write 0~4194304] ov 0'0 av 23'324 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:31.398759 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] do_osd_op 1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head [write 0~4194304]
2016-10-25 08:51:31.398765 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:31.399022 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] make_writeable 1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head snapset=0x7f21396dc658  snapc=0=[]
2016-10-25 08:51:31.399029 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean]  setting DIRTY flag
2016-10-25 08:51:31.399035 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] make_writeable 1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head done, snapset=0=[]:[]+head
2016-10-25 08:51:31.399041 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] finish_ctx 1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head 0x7f2148071c00 op modify  
2016-10-25 08:51:31.399049 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean]  set mtime to 2016-10-25 08:51:31.346812
2016-10-25 08:51:31.399063 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=323 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean]  final snapset 0=[]:[]+head in 1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head
2016-10-25 08:51:31.399077 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean]  zeroing write result code 0
2016-10-25 08:51:31.399086 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean]  op order client.4144 tid 872 (first)
2016-10-25 08:51:31.399091 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] new_repop rep_tid 1718 on osd_op(client.4144.0:872 1.406a57b1 benchmark_data_target167114251239.teuthology_13304_object871 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.399100 7f211246e700  7 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] issue_repop rep_tid 1718 o 1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head
2016-10-25 08:51:31.399132 7f211246e700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:31.399135 7f211246e700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:872 1.1 1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head v 23'324) v1 -- 0x7f2141055000 con 0
2016-10-25 08:51:31.399400 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'323 (0'0,23'323] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] append_log log((0'0,23'323], crt=23'320) [23'324 (0'0) modify   1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head by client.4144.0:872 2016-10-25 08:51:31.346812 0]
2016-10-25 08:51:31.399420 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] add_log_entry 23'324 (0'0) modify   1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head by client.4144.0:872 2016-10-25 08:51:31.346812 0
2016-10-25 08:51:31.399428 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] append_log: trimming to 23'320 entries 
2016-10-25 08:51:31.399438 7f211246e700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'324, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:31.399673 7f211246e700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139241600 osr(1.1 0x7f21394758a0)
2016-10-25 08:51:31.399679 7f211246e700 10 journal prepare_entry [Transaction(0x7f2147f17880)]
2016-10-25 08:51:31.399941 7f211246e700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:31.400831 7f211246e700 10 journal op_submit_start 2354
2016-10-25 08:51:31.400848 7f211246e700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2354 [Transaction(0x7f2147f17880)]
2016-10-25 08:51:31.400852 7f211246e700 10 journal op_journal_transactions 2354 reqid_t client.4144.0:872
2016-10-25 08:51:31.400855 7f211246e700  5 journal submit_entry seq 2354 len 4202496 (0x7f2143096f30)
2016-10-25 08:51:31.400876 7f211246e700 10 journal op_submit_finish 2354
2016-10-25 08:51:31.400881 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] eval_repop repgather(0x7f21394abb00 23'324 rep_tid=1718 committed?=0 applied?=0) wants=d
2016-10-25 08:51:31.400906 7f211246e700 10 osd.3 23 dequeue_op 0x7f2139756760 finish
2016-10-25 08:51:31.402755 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 672 ==== osd_op(client.4144.0:874 1.283bef5e (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (1247528628 0 24448231) 0x7f2142f37180 con 0x7f2147f52800
2016-10-25 08:51:31.402844 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:31.402870 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f21395087e0 prio 63 cost 4194304 latency 0.002573 osd_op(client.4144.0:874 1.283bef5e (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.403004 7f210fc69700 10 osd.3 23 dequeue_op 0x7f21395087e0 prio 63 cost 4194304 latency 0.002706 osd_op(client.4144.0:874 1.283bef5e (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean]
2016-10-25 08:51:31.403169 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] handle_message: 0x7f21395087e0
2016-10-25 08:51:31.403205 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:31.403250 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] do_op osd_op(client.4144.0:874 1.283bef5e benchmark_data_target167114251239.teuthology_13304_object873 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:31.403318 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] get_object_context: obc NOT found in cache: 1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head
2016-10-25 08:51:31.403333 7f210fc69700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head# '_'
2016-10-25 08:51:31.403479 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.6_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject873__head_283BEF5E__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.403536 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head# '_' = -2
2016-10-25 08:51:31.403542 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] get_object_context: no obc for soid 1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head but can_create
2016-10-25 08:51:31.403563 7f210fc69700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:snapdir# 'snapset'
2016-10-25 08:51:31.403621 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.6_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject873__snapdir_283BEF5E__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.403634 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:snapdir# 'snapset' = -2
2016-10-25 08:51:31.403680 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] create_object_context 0x7f213c576ec0 1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head 
2016-10-25 08:51:31.403694 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] populate_obc_watchers 1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head
2016-10-25 08:51:31.403701 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head
2016-10-25 08:51:31.403710 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] get_object_context: 0x7f213c576ec0 1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head rwstate(none n=0 w=0) oi: 1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f2141375720 snapset: 0=[]:[]
2016-10-25 08:51:31.403734 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] find_object_context 1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head @head oi=1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.403757 7f210fc69700 25 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] do_op oi 1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.403768 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] get_object_context: obc NOT found in cache: 1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:snapdir
2016-10-25 08:51:31.403775 7f210fc69700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:snapdir# '_'
2016-10-25 08:51:31.403809 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.6_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject873__snapdir_283BEF5E__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.403819 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:snapdir# '_' = -2
2016-10-25 08:51:31.403822 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] get_object_context: no obc for soid 1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:snapdir and !can_create
2016-10-25 08:51:31.403859 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] execute_ctx 0x7f2142ed6a00
2016-10-25 08:51:31.403866 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] do_op 1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head [write 0~4194304] ov 0'0 av 23'277 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:31.403881 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] do_osd_op 1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head [write 0~4194304]
2016-10-25 08:51:31.403890 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:31.404243 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] make_writeable 1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head snapset=0x7f2141375758  snapc=0=[]
2016-10-25 08:51:31.404270 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean]  setting DIRTY flag
2016-10-25 08:51:31.404281 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] make_writeable 1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head done, snapset=0=[]:[]+head
2016-10-25 08:51:31.404294 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] finish_ctx 1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head 0x7f2142ed6a00 op modify  
2016-10-25 08:51:31.404329 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean]  set mtime to 2016-10-25 08:51:31.349512
2016-10-25 08:51:31.404349 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean]  final snapset 0=[]:[]+head in 1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head
2016-10-25 08:51:31.404372 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean]  zeroing write result code 0
2016-10-25 08:51:31.404385 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean]  op order client.4144 tid 874 (first)
2016-10-25 08:51:31.404392 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] new_repop rep_tid 1719 on osd_op(client.4144.0:874 1.283bef5e benchmark_data_target167114251239.teuthology_13304_object873 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.404416 7f210fc69700  7 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] issue_repop rep_tid 1719 o 1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head
2016-10-25 08:51:31.404460 7f210fc69700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:31.404468 7f210fc69700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:874 1.6 1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head v 23'277) v1 -- 0x7f2141031200 con 0
2016-10-25 08:51:31.404794 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'276 (0'0,23'276] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] append_log log((0'0,23'276], crt=23'272) [23'277 (0'0) modify   1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head by client.4144.0:874 2016-10-25 08:51:31.349512 0]
2016-10-25 08:51:31.404827 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] add_log_entry 23'277 (0'0) modify   1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head by client.4144.0:874 2016-10-25 08:51:31.349512 0
2016-10-25 08:51:31.404836 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] append_log: trimming to 23'272 entries 
2016-10-25 08:51:31.404848 7f210fc69700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'277, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:31.405093 7f210fc69700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139243700 osr(1.6 0x7f2139474f10)
2016-10-25 08:51:31.405107 7f210fc69700 10 journal prepare_entry [Transaction(0x7f2147f49800)]
2016-10-25 08:51:31.405410 7f210fc69700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:31.405679 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 673 ==== osd_op(client.4144.0:875 1.917ed781 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (190854343 0 2022074828) 0x7f2142e54b00 con 0x7f2147f52800
2016-10-25 08:51:31.405742 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:31.405761 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139880b40 prio 63 cost 4194304 latency 0.002833 osd_op(client.4144.0:875 1.917ed781 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.406470 7f210fc69700 10 journal op_submit_start 2355
2016-10-25 08:51:31.406480 7f210fc69700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2355 [Transaction(0x7f2147f49800)]
2016-10-25 08:51:31.406484 7f210fc69700 10 journal op_journal_transactions 2355 reqid_t client.4144.0:874
2016-10-25 08:51:31.406487 7f210fc69700  5 journal submit_entry seq 2355 len 4202496 (0x7f2140fb50e0)
2016-10-25 08:51:31.406506 7f210fc69700 10 journal op_submit_finish 2355
2016-10-25 08:51:31.406511 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] eval_repop repgather(0x7f21394ad840 23'277 rep_tid=1719 committed?=0 applied?=0) wants=d
2016-10-25 08:51:31.406544 7f210fc69700 10 osd.3 23 dequeue_op 0x7f21395087e0 finish
2016-10-25 08:51:31.406557 7f210fc69700 10 osd.3 23 dequeue_op 0x7f2139880b40 prio 63 cost 4194304 latency 0.003629 osd_op(client.4144.0:875 1.917ed781 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean]
2016-10-25 08:51:31.406579 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] handle_message: 0x7f2139880b40
2016-10-25 08:51:31.406597 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:31.406625 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] do_op osd_op(client.4144.0:875 1.917ed781 benchmark_data_target167114251239.teuthology_13304_object874 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:31.406645 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] get_object_context: obc NOT found in cache: 1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head
2016-10-25 08:51:31.406655 7f210fc69700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head# '_'
2016-10-25 08:51:31.406727 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.1_head/DIR_1/benchmark\udata\utarget167114251239.teuthology\u13304\uobject874__head_917ED781__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.406743 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head# '_' = -2
2016-10-25 08:51:31.406749 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] get_object_context: no obc for soid 1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head but can_create
2016-10-25 08:51:31.406761 7f210fc69700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:snapdir# 'snapset'
2016-10-25 08:51:31.406799 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.1_head/DIR_1/benchmark\udata\utarget167114251239.teuthology\u13304\uobject874__snapdir_917ED781__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.406807 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:snapdir# 'snapset' = -2
2016-10-25 08:51:31.406829 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] create_object_context 0x7f213c576c00 1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head 
2016-10-25 08:51:31.406839 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] populate_obc_watchers 1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head
2016-10-25 08:51:31.406845 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head
2016-10-25 08:51:31.406850 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] get_object_context: 0x7f213c576c00 1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head rwstate(none n=0 w=0) oi: 1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f2141373420 snapset: 0=[]:[]
2016-10-25 08:51:31.406871 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] find_object_context 1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head @head oi=1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.406880 7f210fc69700 25 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] do_op oi 1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.406890 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] get_object_context: obc NOT found in cache: 1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:snapdir
2016-10-25 08:51:31.406896 7f210fc69700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:snapdir# '_'
2016-10-25 08:51:31.406922 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.1_head/DIR_1/benchmark\udata\utarget167114251239.teuthology\u13304\uobject874__snapdir_917ED781__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.406926 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:snapdir# '_' = -2
2016-10-25 08:51:31.406928 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] get_object_context: no obc for soid 1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:snapdir and !can_create
2016-10-25 08:51:31.406965 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] execute_ctx 0x7f2142ed4000
2016-10-25 08:51:31.406973 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] do_op 1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head [write 0~4194304] ov 0'0 av 23'325 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:31.406983 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] do_osd_op 1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head [write 0~4194304]
2016-10-25 08:51:31.406994 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:31.407284 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] make_writeable 1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head snapset=0x7f2141373458  snapc=0=[]
2016-10-25 08:51:31.407395 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean]  setting DIRTY flag
2016-10-25 08:51:31.407425 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] make_writeable 1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head done, snapset=0=[]:[]+head
2016-10-25 08:51:31.407457 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] finish_ctx 1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head 0x7f2142ed4000 op modify  
2016-10-25 08:51:31.407478 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean]  set mtime to 2016-10-25 08:51:31.349683
2016-10-25 08:51:31.407567 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=324 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean]  final snapset 0=[]:[]+head in 1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head
2016-10-25 08:51:31.407614 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean]  zeroing write result code 0
2016-10-25 08:51:31.407637 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean]  op order client.4144 tid 875 (first)
2016-10-25 08:51:31.407662 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] new_repop rep_tid 1720 on osd_op(client.4144.0:875 1.917ed781 benchmark_data_target167114251239.teuthology_13304_object874 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.407702 7f210fc69700  7 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] issue_repop rep_tid 1720 o 1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head
2016-10-25 08:51:31.407770 7f210fc69700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:31.407787 7f210fc69700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:875 1.1 1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head v 23'325) v1 -- 0x7f2141030c00 con 0
2016-10-25 08:51:31.415900 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'324 (0'0,23'324] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] append_log log((0'0,23'324], crt=23'320) [23'325 (0'0) modify   1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head by client.4144.0:875 2016-10-25 08:51:31.349683 0]
2016-10-25 08:51:31.416002 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] add_log_entry 23'325 (0'0) modify   1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head by client.4144.0:875 2016-10-25 08:51:31.349683 0
2016-10-25 08:51:31.416091 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] append_log: trimming to 23'320 entries 
2016-10-25 08:51:31.416175 7f210fc69700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'325, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:31.416563 7f210fc69700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139241600 osr(1.1 0x7f21394758a0)
2016-10-25 08:51:31.416609 7f210fc69700 10 journal prepare_entry [Transaction(0x7f2140fff100)]
2016-10-25 08:51:31.416915 7f210fc69700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:31.426504 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2282 ==== osd_repop(client.4144.0:869 1.5) v1 ==== 1034+0+4195023 (2505805833 0 1347467730) 0x7f2140e67600 con 0x7f2139592800
2016-10-25 08:51:31.426551 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop(client.4144.0:869 1.5) v1 epoch 23
2016-10-25 08:51:31.426558 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:31.426565 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139880000 prio 127 cost 4195023 latency 0.003502 osd_repop(client.4144.0:869 1.5) v1
2016-10-25 08:51:31.426628 7f2112c6f700 10 osd.3 23 dequeue_op 0x7f2139880000 prio 127 cost 4195023 latency 0.003564 osd_repop(client.4144.0:869 1.5) v1 pg pg[1.5( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 crt=23'274 lcod 23'276 active]
2016-10-25 08:51:31.426657 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.5( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 crt=23'274 lcod 23'276 active] handle_message: 0x7f2139880000
2016-10-25 08:51:31.426676 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.5( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 crt=23'274 lcod 23'276 active] sub_op_modify trans 1:ace40720:::benchmark_data_target167114251239.teuthology_13304_object868:head v 23'278 (transaction) 200
2016-10-25 08:51:31.426711 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.5( v 23'277 (0'0,23'277] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 crt=23'274 lcod 23'276 active] append_log log((0'0,23'277], crt=23'274) [23'278 (0'0) modify   1:ace40720:::benchmark_data_target167114251239.teuthology_13304_object868:head by client.4144.0:869 2016-10-25 08:51:31.344192 0]
2016-10-25 08:51:31.426723 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.5( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'277 crt=23'274 lcod 23'276 active] add_log_entry 23'278 (0'0) modify   1:ace40720:::benchmark_data_target167114251239.teuthology_13304_object868:head by client.4144.0:869 2016-10-25 08:51:31.344192 0
2016-10-25 08:51:31.426732 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.5( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'277 crt=23'276 lcod 23'276 active] append_log: trimming to 23'276 entries 23'275 (0'0) modify   1:b5360f4e:::benchmark_data_target167114251239.teuthology_13304_object812:head by client.4144.0:813 2016-10-25 08:51:28.512982 0,23'276 (0'0) modify   1:ae3818e6:::benchmark_data_target167114251239.teuthology_13304_object823:head by client.4144.0:824 2016-10-25 08:51:28.652588 0
2016-10-25 08:51:31.426749 7f2112c6f700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'278, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:31.426947 7f2112c6f700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f21394bbb80 osr(1.5 0x7f2139425410)
2016-10-25 08:51:31.426953 7f2112c6f700 10 journal prepare_entry [Transaction(0x7f213e62c700),Transaction(0x7f213e62c8c0)]
2016-10-25 08:51:31.427456 7f2112c6f700 10 journal  len 4195682 -> 4202496 (head 40 pre_pad 3387 bl 4195682 post_pad 3347 tail 40) (bl alignment 3427)
2016-10-25 08:51:31.428741 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2283 ==== osd_repop(client.4144.0:873 1.5) v1 ==== 1034+0+4195023 (2164620640 0 3757006887) 0x7f2140ff9800 con 0x7f2139592800
2016-10-25 08:51:31.428777 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop(client.4144.0:873 1.5) v1 epoch 23
2016-10-25 08:51:31.428782 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:31.428789 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139882520 prio 127 cost 4195023 latency 0.002188 osd_repop(client.4144.0:873 1.5) v1
2016-10-25 08:51:31.436864 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:31.436882 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:31.436885 7f2123404700 20 journal write_thread_entry aio throttle: aio num 10 bytes 42024960 ... exp 20 min_new 1048576 ... pending 12607488
2016-10-25 08:51:31.436890 7f2123404700 10 journal room 505360383 max_size 1072709632 pos 958439424 header.start 391094272 top 4096
2016-10-25 08:51:31.436892 7f2123404700 10 journal check_for_full at 958439424 : 4202496 < 505360383
2016-10-25 08:51:31.436894 7f2123404700 15 journal prepare_single_write 1 will write 958439424 : seq 2353 len 4195598 -> 4202496
2016-10-25 08:51:31.436920 7f2123404700 10 journal room 501157887 max_size 1072709632 pos 962641920 header.start 391094272 top 4096
2016-10-25 08:51:31.436922 7f2123404700 10 journal check_for_full at 962641920 : 4202496 < 501157887
2016-10-25 08:51:31.436937 7f2123404700 15 journal prepare_single_write 2 will write 962641920 : seq 2354 len 4195598 -> 4202496
2016-10-25 08:51:31.436946 7f2123404700 10 journal room 496955391 max_size 1072709632 pos 966844416 header.start 391094272 top 4096
2016-10-25 08:51:31.436947 7f2123404700 10 journal check_for_full at 966844416 : 4202496 < 496955391
2016-10-25 08:51:31.436948 7f2123404700 15 journal prepare_single_write 3 will write 966844416 : seq 2355 len 4195598 -> 4202496
2016-10-25 08:51:31.436957 7f2123404700 20 journal prepare_multi_write hit max write size 10485760
2016-10-25 08:51:31.436958 7f2123404700 20 journal prepare_multi_write queue_pos now 971046912
2016-10-25 08:51:31.436960 7f2123404700 15 journal do_aio_write writing 958439424~12607488
2016-10-25 08:51:31.436962 7f2123404700 20 journal write_aio_bl 958439424~12607488 seq 2355
2016-10-25 08:51:31.436965 7f2123404700 20 journal write_aio_bl .. 958439424~12607488 in 3
2016-10-25 08:51:31.437878 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:31.437893 7f2123404700  5 journal complete_write finished 3 ops and 12586794 bytes
2016-10-25 08:51:31.437896 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:31.523112 7f2123c05700 15 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) syncfs: doing a full sync (syncfs(2) if possible)
2016-10-25 08:51:31.523140 7f2121400700 10 filestore oid: #1:e0000000::::head# not skipping op, *spos 2332.0.2
2016-10-25 08:51:31.523152 7f2121400700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.523181 7f2121c01700 10 filestore oid: #1:20000000::::head# not skipping op, *spos 2333.0.2
2016-10-25 08:51:31.523185 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.523254 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.4_head/#1:20000000::::head# = 0
2016-10-25 08:51:31.523267 7f2121c01700 10 journal op_apply_finish 2333 open_ops 2 -> 1, max_applied_seq 2331 -> 2333
2016-10-25 08:51:31.523271 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2142f71860 seq 2333 r = 0, finisher 0x7f213ecde520 0x7f214134ef00
2016-10-25 08:51:31.523404 7f2121400700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.7_head/#1:e0000000::::head# = 0
2016-10-25 08:51:31.523413 7f2121400700 10 journal op_apply_finish 2332 open_ops 1 -> 0, max_applied_seq 2333 -> 2333
2016-10-25 08:51:31.523415 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f214133de40 seq 2332 r = 0, finisher 0x7f2140e15460 0x7f214309acc0
2016-10-25 08:51:31.523425 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f214133de40 seq 2332 osr(1.7 0x7f2139474be0)/0x7f2139474be0 lat 1.285384
2016-10-25 08:51:31.523441 7f2121400700 10 journal op_apply_start 2334 open_ops 0 -> 1
2016-10-25 08:51:31.523443 7f2121400700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2143016220 seq 2334 osr(1.1 0x7f21394758a0)/0x7f21394758a0 start
2016-10-25 08:51:31.523445 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147e6e1c0
2016-10-25 08:51:31.523456 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.1_head/#1:8a01b0bd:::benchmark_data_target167114251239.teuthology_13304_object851:head# 0~4194304
2016-10-25 08:51:31.523860 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] op_applied: 1698
2016-10-25 08:51:31.523904 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'264 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] op_applied version 23'265
2016-10-25 08:51:31.523912 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'265 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] repop_all_applied: repop tid 1698 all applied 
2016-10-25 08:51:31.523917 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'265 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] eval_repop repgather(0x7f21396426c0 23'265 rep_tid=1698 committed?=1 applied?=1) wants=d
2016-10-25 08:51:31.523923 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'265 crt=23'263 lcod 23'264 mlcod 23'263 active+clean]  commit: repgather(0x7f21396426c0 23'265 rep_tid=1698 committed?=1 applied?=1)
2016-10-25 08:51:31.523929 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'265 crt=23'263 lcod 23'264 mlcod 23'263 active+clean]  applied: repgather(0x7f21396426c0 23'265 rep_tid=1698 committed?=1 applied?=1) 
2016-10-25 08:51:31.523938 7f211f3fc700 15 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'265 crt=23'263 lcod 23'264 mlcod 23'263 active+clean] publish_stats_to_osd 23:144
2016-10-25 08:51:31.523946 7f211f3fc700 15 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:31.523953 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean]  removing repgather(0x7f21396426c0 23'265 rep_tid=1698 committed?=1 applied?=1)
2016-10-25 08:51:31.523958 7f211f3fc700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean]    q front is repgather(0x7f21396426c0 23'265 rep_tid=1698 committed?=1 applied?=1)
2016-10-25 08:51:31.523963 7f211f3fc700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] remove_repop repgather(0x7f21396426c0 23'265 rep_tid=1698 committed?=1 applied?=1)
2016-10-25 08:51:31.523976 7f210fc69700 10 journal op_submit_start 2356
2016-10-25 08:51:31.523981 7f210fc69700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2356 [Transaction(0x7f2140fff100)]
2016-10-25 08:51:31.523980 7f211f3fc700 15 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean]  requeue_ops 
2016-10-25 08:51:31.523985 7f210fc69700 10 journal op_journal_transactions 2356 reqid_t client.4144.0:875
2016-10-25 08:51:31.523989 7f210fc69700  5 journal submit_entry seq 2356 len 4202496 (0x7f2140fb52f0)
2016-10-25 08:51:31.524018 7f210fc69700 10 journal op_submit_finish 2356
2016-10-25 08:51:31.524023 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] eval_repop repgather(0x7f2139880480 23'325 rep_tid=1720 committed?=0 applied?=0) wants=d
2016-10-25 08:51:31.524044 7f210fc69700 10 osd.3 23 dequeue_op 0x7f2139880b40 finish
2016-10-25 08:51:31.524063 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:31.524066 7f2123404700 20 journal write_thread_entry aio throttle: aio num 11 bytes 54632448 ... exp 22 min_new 4194304 ... pending 4202496
2016-10-25 08:51:31.524070 7f2123404700 10 journal room 492752895 max_size 1072709632 pos 971046912 header.start 391094272 top 4096
2016-10-25 08:51:31.524072 7f2123404700 10 journal check_for_full at 971046912 : 4202496 < 492752895
2016-10-25 08:51:31.524074 7f2123404700 15 journal prepare_single_write 1 will write 971046912 : seq 2356 len 4195598 -> 4202496
2016-10-25 08:51:31.524088 7f2123404700 20 journal prepare_multi_write queue_pos now 975249408
2016-10-25 08:51:31.524091 7f2123404700 15 journal do_aio_write writing 971046912~4202496
2016-10-25 08:51:31.524093 7f2123404700 20 journal write_aio_bl 971046912~4202496 seq 2356
2016-10-25 08:51:31.524097 7f2123404700 20 journal write_aio_bl .. 971046912~4202496 in 1
2016-10-25 08:51:31.524501 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:31.524508 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:31.524511 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:31.524812 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f2142f71860 seq 2333 osr(1.4 0x7f2139475240)/0x7f2139475240 lat 1.286867
2016-10-25 08:51:31.524839 7f2121c01700 10 journal op_apply_start 2335 open_ops 1 -> 2
2016-10-25 08:51:31.524842 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f213ed5df00 seq 2335 osr(1.6 0x7f2139474f10)/0x7f2139474f10 start
2016-10-25 08:51:31.524845 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147e62e00
2016-10-25 08:51:31.524855 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.6_head/#1:71ee6361:::benchmark_data_target167114251239.teuthology_13304_object852:head# 0~4194304
2016-10-25 08:51:31.525134 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'279 crt=23'278 lcod 23'281 mlcod 23'278 active+clean] op_applied: 1697
2016-10-25 08:51:31.525170 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'279 crt=23'278 lcod 23'281 mlcod 23'278 active+clean] op_applied version 23'280
2016-10-25 08:51:31.525182 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'280 crt=23'278 lcod 23'281 mlcod 23'278 active+clean] repop_all_applied: repop tid 1697 all applied 
2016-10-25 08:51:31.525197 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'280 crt=23'278 lcod 23'281 mlcod 23'278 active+clean] eval_repop repgather(0x7f213985c7e0 23'280 rep_tid=1697 committed?=1 applied?=1) wants=d
2016-10-25 08:51:31.525208 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'280 crt=23'278 lcod 23'281 mlcod 23'278 active+clean]  commit: repgather(0x7f213985c7e0 23'280 rep_tid=1697 committed?=1 applied?=1)
2016-10-25 08:51:31.525219 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'280 crt=23'278 lcod 23'281 mlcod 23'278 active+clean]  applied: repgather(0x7f213985c7e0 23'280 rep_tid=1697 committed?=1 applied?=1) 
2016-10-25 08:51:31.525246 7f211ebfb700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'280 crt=23'278 lcod 23'281 mlcod 23'278 active+clean] publish_stats_to_osd 23:144
2016-10-25 08:51:31.525263 7f211ebfb700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'280 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:31.525276 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'280 crt=23'278 lcod 23'281 mlcod 23'281 active+clean]  removing repgather(0x7f213985c7e0 23'280 rep_tid=1697 committed?=1 applied?=1)
2016-10-25 08:51:31.525287 7f211ebfb700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'280 crt=23'278 lcod 23'281 mlcod 23'281 active+clean]    q front is repgather(0x7f213985c7e0 23'280 rep_tid=1697 committed?=1 applied?=1)
2016-10-25 08:51:31.525298 7f211ebfb700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'280 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] remove_repop repgather(0x7f213985c7e0 23'280 rep_tid=1697 committed?=1 applied?=1)
2016-10-25 08:51:31.525315 7f211ebfb700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'280 crt=23'278 lcod 23'281 mlcod 23'281 active+clean]  requeue_ops 
2016-10-25 08:51:31.525378 7f2112c6f700 10 journal op_submit_start 2357
2016-10-25 08:51:31.525381 7f2112c6f700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2357 [Transaction(0x7f213e62c700),Transaction(0x7f213e62c8c0)]
2016-10-25 08:51:31.525387 7f2112c6f700 10 journal op_journal_transactions 2357 reqid_t client.4144.0:869
2016-10-25 08:51:31.525390 7f2112c6f700  5 journal submit_entry seq 2357 len 4202496 (0x7f2140fd9e30)
2016-10-25 08:51:31.525417 7f2112c6f700 10 journal op_submit_finish 2357
2016-10-25 08:51:31.525423 7f2112c6f700 10 osd.3 23 dequeue_op 0x7f2139880000 finish
2016-10-25 08:51:31.525444 7f211046a700 10 osd.3 23 dequeue_op 0x7f2139882520 prio 127 cost 4195023 latency 0.098842 osd_repop(client.4144.0:873 1.5) v1 pg pg[1.5( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'277 crt=23'276 lcod 23'276 active]
2016-10-25 08:51:31.525468 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.5( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'277 crt=23'276 lcod 23'276 active] handle_message: 0x7f2139882520
2016-10-25 08:51:31.525489 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.5( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'277 crt=23'276 lcod 23'276 active] sub_op_modify trans 1:a63a41ae:::benchmark_data_target167114251239.teuthology_13304_object872:head v 23'279 (transaction) 200
2016-10-25 08:51:31.525526 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.5( v 23'278 (0'0,23'278] local-les=22 n=279 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'277 crt=23'276 lcod 23'276 active] append_log log((0'0,23'278], crt=23'276) [23'279 (0'0) modify   1:a63a41ae:::benchmark_data_target167114251239.teuthology_13304_object872:head by client.4144.0:873 2016-10-25 08:51:31.346940 0]
2016-10-25 08:51:31.525543 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.5( v 23'279 (0'0,23'279] local-les=22 n=279 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'277 crt=23'276 lcod 23'276 active] add_log_entry 23'279 (0'0) modify   1:a63a41ae:::benchmark_data_target167114251239.teuthology_13304_object872:head by client.4144.0:873 2016-10-25 08:51:31.346940 0
2016-10-25 08:51:31.525558 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.5( v 23'279 (0'0,23'279] local-les=22 n=279 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'277 crt=23'276 lcod 23'276 active] append_log: trimming to 23'276 entries 
2016-10-25 08:51:31.525580 7f211046a700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'279, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:31.525865 7f211046a700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f21394bbb80 osr(1.5 0x7f2139425410)
2016-10-25 08:51:31.525871 7f211046a700 10 journal prepare_entry [Transaction(0x7f2141371800),Transaction(0x7f21413719c0)]
2016-10-25 08:51:31.527083 7f211046a700 10 journal  len 4195682 -> 4202496 (head 40 pre_pad 3387 bl 4195682 post_pad 3347 tail 40) (bl alignment 3427)
2016-10-25 08:51:31.531490 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:31.531553 7f2123404700 20 journal write_thread_entry aio throttle: aio num 12 bytes 58834944 ... exp 24 min_new 16777216 ... pending 4202496
2016-10-25 08:51:31.531559 7f2123404700 20 journal write_thread_entry deferring until more aios complete: 12 aios with 58834944 bytes needs 16777216 bytes to start a new aio (currently 4202496 pending)
2016-10-25 08:51:31.534103 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.6_head/#1:71ee6361:::benchmark_data_target167114251239.teuthology_13304_object852:head# 0~4194304 = 4194304
2016-10-25 08:51:31.534243 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.6_head/#1:71ee6361:::benchmark_data_target167114251239.teuthology_13304_object852:head#
2016-10-25 08:51:31.534310 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.6_head/#1:71ee6361:::benchmark_data_target167114251239.teuthology_13304_object852:head# = 0
2016-10-25 08:51:31.534335 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.6_head/#1:60000000::::head#
2016-10-25 08:51:31.534357 7f2121c01700 10 filestore oid: #1:60000000::::head# not skipping op, *spos 2335.0.2
2016-10-25 08:51:31.534366 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.534410 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.6_head/#1:60000000::::head# = 0
2016-10-25 08:51:31.534423 7f2121c01700 10 journal op_apply_finish 2335 open_ops 2 -> 1, max_applied_seq 2333 -> 2335
2016-10-25 08:51:31.534433 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f213ed5df00 seq 2335 r = 0, finisher 0x7f2147e83b80 0x7f213edcc340
2016-10-25 08:51:31.534451 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f213ed5df00 seq 2335 osr(1.6 0x7f2139474f10)/0x7f2139474f10 lat 1.285967
2016-10-25 08:51:31.534478 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] op_applied: 1700
2016-10-25 08:51:31.534509 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'273 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] op_applied version 23'274
2016-10-25 08:51:31.534517 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'274 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] repop_all_applied: repop tid 1700 all applied 
2016-10-25 08:51:31.534522 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'274 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] eval_repop repgather(0x7f2139645180 23'274 rep_tid=1700 committed?=1 applied?=1) wants=d
2016-10-25 08:51:31.534528 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'274 crt=23'272 lcod 23'273 mlcod 23'272 active+clean]  commit: repgather(0x7f2139645180 23'274 rep_tid=1700 committed?=1 applied?=1)
2016-10-25 08:51:31.534534 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'274 crt=23'272 lcod 23'273 mlcod 23'272 active+clean]  applied: repgather(0x7f2139645180 23'274 rep_tid=1700 committed?=1 applied?=1) 
2016-10-25 08:51:31.534543 7f211e3fa700 15 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'274 crt=23'272 lcod 23'273 mlcod 23'272 active+clean] publish_stats_to_osd 23:135
2016-10-25 08:51:31.534551 7f211e3fa700 15 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:31.534558 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean]  removing repgather(0x7f2139645180 23'274 rep_tid=1700 committed?=1 applied?=1)
2016-10-25 08:51:31.534563 7f211e3fa700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean]    q front is repgather(0x7f2139645180 23'274 rep_tid=1700 committed?=1 applied?=1)
2016-10-25 08:51:31.534575 7f211e3fa700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] remove_repop repgather(0x7f2139645180 23'274 rep_tid=1700 committed?=1 applied?=1)
2016-10-25 08:51:31.534584 7f211e3fa700 15 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean]  requeue_ops 
2016-10-25 08:51:31.534670 7f2121c01700 10 journal op_apply_start 2336 open_ops 1 -> 2
2016-10-25 08:51:31.534680 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f213ed5cee0 seq 2336 osr(1.4 0x7f2139475240)/0x7f2139475240 start
2016-10-25 08:51:31.534690 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147f68380
2016-10-25 08:51:31.534703 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.4_head/#1:3df0d67a:::benchmark_data_target167114251239.teuthology_13304_object853:head# 0~4194304
2016-10-25 08:51:31.541759 7f211046a700 10 journal op_submit_start 2358
2016-10-25 08:51:31.541789 7f211046a700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2358 [Transaction(0x7f2141371800),Transaction(0x7f21413719c0)]
2016-10-25 08:51:31.541800 7f211046a700 10 journal op_journal_transactions 2358 reqid_t client.4144.0:873
2016-10-25 08:51:31.541807 7f211046a700  5 journal submit_entry seq 2358 len 4202496 (0x7f21414ab320)
2016-10-25 08:51:31.541865 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:31.541867 7f2123404700 20 journal write_thread_entry aio throttle: aio num 12 bytes 58834944 ... exp 24 min_new 16777216 ... pending 8404992
2016-10-25 08:51:31.541871 7f2123404700 20 journal write_thread_entry deferring until more aios complete: 12 aios with 58834944 bytes needs 16777216 bytes to start a new aio (currently 8404992 pending)
2016-10-25 08:51:31.541876 7f211046a700 10 journal op_submit_finish 2358
2016-10-25 08:51:31.541889 7f211046a700 10 osd.3 23 dequeue_op 0x7f2139882520 finish
2016-10-25 08:51:31.544161 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.4_head/#1:3df0d67a:::benchmark_data_target167114251239.teuthology_13304_object853:head# 0~4194304 = 4194304
2016-10-25 08:51:31.544300 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.4_head/#1:3df0d67a:::benchmark_data_target167114251239.teuthology_13304_object853:head#
2016-10-25 08:51:31.544378 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.4_head/#1:3df0d67a:::benchmark_data_target167114251239.teuthology_13304_object853:head# = 0
2016-10-25 08:51:31.544421 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.4_head/#1:20000000::::head#
2016-10-25 08:51:31.544464 7f2121c01700 10 filestore oid: #1:20000000::::head# not skipping op, *spos 2336.0.2
2016-10-25 08:51:31.544487 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.544577 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.4_head/#1:20000000::::head# = 0
2016-10-25 08:51:31.544609 7f2121c01700 10 journal op_apply_finish 2336 open_ops 2 -> 1, max_applied_seq 2335 -> 2336
2016-10-25 08:51:31.544632 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f213ed5cee0 seq 2336 r = 0, finisher 0x7f2148111da0 0x7f213e5fba40
2016-10-25 08:51:31.544668 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f213ed5cee0 seq 2336 osr(1.4 0x7f2139475240)/0x7f2139475240 lat 1.284658
2016-10-25 08:51:31.544701 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'280 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] op_applied: 1703
2016-10-25 08:51:31.544734 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'280 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] op_applied version 23'281
2016-10-25 08:51:31.544741 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'281 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] repop_all_applied: repop tid 1703 all applied 
2016-10-25 08:51:31.544747 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'281 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] eval_repop repgather(0x7f21398aaac0 23'281 rep_tid=1703 committed?=1 applied?=1) wants=d
2016-10-25 08:51:31.544754 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'281 crt=23'278 lcod 23'281 mlcod 23'281 active+clean]  commit: repgather(0x7f21398aaac0 23'281 rep_tid=1703 committed?=1 applied?=1)
2016-10-25 08:51:31.544759 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'281 crt=23'278 lcod 23'281 mlcod 23'281 active+clean]  applied: repgather(0x7f21398aaac0 23'281 rep_tid=1703 committed?=1 applied?=1) 
2016-10-25 08:51:31.544768 7f211ebfb700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'281 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] publish_stats_to_osd 23: no change since 2016-10-25 08:51:31.525231
2016-10-25 08:51:31.544777 7f211ebfb700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'281 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:31.544784 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'281 crt=23'278 lcod 23'281 mlcod 23'281 active+clean]  removing repgather(0x7f21398aaac0 23'281 rep_tid=1703 committed?=1 applied?=1)
2016-10-25 08:51:31.544789 7f211ebfb700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'281 crt=23'278 lcod 23'281 mlcod 23'281 active+clean]    q front is repgather(0x7f21398aaac0 23'281 rep_tid=1703 committed?=1 applied?=1)
2016-10-25 08:51:31.544794 7f211ebfb700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'281 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] remove_repop repgather(0x7f21398aaac0 23'281 rep_tid=1703 committed?=1 applied?=1)
2016-10-25 08:51:31.544803 7f211ebfb700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'281 crt=23'278 lcod 23'281 mlcod 23'281 active+clean]  requeue_ops 
2016-10-25 08:51:31.544938 7f2121c01700 10 journal op_apply_start 2337 open_ops 1 -> 2
2016-10-25 08:51:31.544949 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f214141a960 seq 2337 osr(1.3 0x7f2139475570)/0x7f2139475570 start
2016-10-25 08:51:31.544959 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147ee7340
2016-10-25 08:51:31.544973 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.3_head/#1:db1dc65b:::benchmark_data_target167114251239.teuthology_13304_object854:head# 0~4194304
2016-10-25 08:51:31.546030 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.1_head/#1:8a01b0bd:::benchmark_data_target167114251239.teuthology_13304_object851:head# 0~4194304 = 4194304
2016-10-25 08:51:31.546130 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.1_head/#1:8a01b0bd:::benchmark_data_target167114251239.teuthology_13304_object851:head#
2016-10-25 08:51:31.546202 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.1_head/#1:8a01b0bd:::benchmark_data_target167114251239.teuthology_13304_object851:head# = 0
2016-10-25 08:51:31.546227 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.1_head/#1:80000000::::head#
2016-10-25 08:51:31.546243 7f2121400700 10 filestore oid: #1:80000000::::head# not skipping op, *spos 2334.0.2
2016-10-25 08:51:31.546252 7f2121400700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.546312 7f2121400700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.1_head/#1:80000000::::head# = 0
2016-10-25 08:51:31.546335 7f2121400700 10 journal op_apply_finish 2334 open_ops 2 -> 1, max_applied_seq 2336 -> 2336
2016-10-25 08:51:31.546340 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2143016220 seq 2334 r = 0, finisher 0x7f213a53b0a0 0x7f2142e723c0
2016-10-25 08:51:31.546358 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f2143016220 seq 2334 osr(1.1 0x7f21394758a0)/0x7f21394758a0 lat 1.302427
2016-10-25 08:51:31.546379 7f2121400700 10 journal op_apply_start 2338 open_ops 1 -> 2
2016-10-25 08:51:31.546383 7f2121400700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2139545c20 seq 2338 osr(1.4 0x7f2139475240)/0x7f2139475240 start
2016-10-25 08:51:31.546388 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147f05b80
2016-10-25 08:51:31.546402 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.4_head/#1:3d4d78f7:::benchmark_data_target167114251239.teuthology_13304_object857:head# 0~4194304
2016-10-25 08:51:31.547672 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] op_applied: 1699
2016-10-25 08:51:31.547735 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'321 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] op_applied version 23'322
2016-10-25 08:51:31.547751 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] repop_all_applied: repop tid 1699 all applied 
2016-10-25 08:51:31.547761 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] eval_repop repgather(0x7f21398820a0 23'322 rep_tid=1699 committed?=1 applied?=1) wants=d
2016-10-25 08:51:31.547773 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'320 active+clean]  commit: repgather(0x7f21398820a0 23'322 rep_tid=1699 committed?=1 applied?=1)
2016-10-25 08:51:31.547784 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'320 active+clean]  applied: repgather(0x7f21398820a0 23'322 rep_tid=1699 committed?=1 applied?=1) 
2016-10-25 08:51:31.547800 7f211e3fa700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'320 active+clean] publish_stats_to_osd 23:163
2016-10-25 08:51:31.547814 7f211e3fa700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:31.547829 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean]  removing repgather(0x7f21398820a0 23'322 rep_tid=1699 committed?=1 applied?=1)
2016-10-25 08:51:31.547839 7f211e3fa700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean]    q front is repgather(0x7f21398820a0 23'322 rep_tid=1699 committed?=1 applied?=1)
2016-10-25 08:51:31.547849 7f211e3fa700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] remove_repop repgather(0x7f21398820a0 23'322 rep_tid=1699 committed?=1 applied?=1)
2016-10-25 08:51:31.547866 7f211e3fa700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean]  requeue_ops 
2016-10-25 08:51:31.552196 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.3_head/#1:db1dc65b:::benchmark_data_target167114251239.teuthology_13304_object854:head# 0~4194304 = 4194304
2016-10-25 08:51:31.552349 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.3_head/#1:db1dc65b:::benchmark_data_target167114251239.teuthology_13304_object854:head#
2016-10-25 08:51:31.552464 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.3_head/#1:db1dc65b:::benchmark_data_target167114251239.teuthology_13304_object854:head# = 0
2016-10-25 08:51:31.552519 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.3_head/#1:c0000000::::head#
2016-10-25 08:51:31.552546 7f2121c01700 10 filestore oid: #1:c0000000::::head# not skipping op, *spos 2337.0.2
2016-10-25 08:51:31.552618 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.552697 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.3_head/#1:c0000000::::head# = 0
2016-10-25 08:51:31.552716 7f2121c01700 10 journal op_apply_finish 2337 open_ops 2 -> 1, max_applied_seq 2336 -> 2337
2016-10-25 08:51:31.552726 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f214141a960 seq 2337 r = 0, finisher 0x7f2148112f60 0x7f214124ebc0
2016-10-25 08:51:31.552766 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f214141a960 seq 2337 osr(1.3 0x7f2139475570)/0x7f2139475570 lat 1.282725
2016-10-25 08:51:31.552795 7f2121c01700 10 journal op_apply_start 2339 open_ops 1 -> 2
2016-10-25 08:51:31.552806 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2142f72d60 seq 2339 osr(1.0 0x7f2139475d80)/0x7f2139475d80 start
2016-10-25 08:51:31.552816 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147e48300
2016-10-25 08:51:31.552811 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'280 crt=23'279 lcod 23'280 mlcod 23'279 active+clean] op_applied: 1701
2016-10-25 08:51:31.552847 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.0_head/#1:0bbe1d8e:::benchmark_data_target167114251239.teuthology_13304_object855:head# 0~4194304
2016-10-25 08:51:31.552874 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'280 crt=23'279 lcod 23'280 mlcod 23'279 active+clean] op_applied version 23'281
2016-10-25 08:51:31.552886 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'281 crt=23'279 lcod 23'280 mlcod 23'279 active+clean] repop_all_applied: repop tid 1701 all applied 
2016-10-25 08:51:31.552900 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'281 crt=23'279 lcod 23'280 mlcod 23'279 active+clean] eval_repop repgather(0x7f21395d5a80 23'281 rep_tid=1701 committed?=1 applied?=1) wants=d
2016-10-25 08:51:31.552911 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'281 crt=23'279 lcod 23'280 mlcod 23'279 active+clean]  commit: repgather(0x7f21395d5a80 23'281 rep_tid=1701 committed?=1 applied?=1)
2016-10-25 08:51:31.552920 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'281 crt=23'279 lcod 23'280 mlcod 23'279 active+clean]  applied: repgather(0x7f21395d5a80 23'281 rep_tid=1701 committed?=1 applied?=1) 
2016-10-25 08:51:31.552938 7f211f3fc700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'281 crt=23'279 lcod 23'280 mlcod 23'279 active+clean] publish_stats_to_osd 23:151
2016-10-25 08:51:31.552954 7f211f3fc700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'281 crt=23'279 lcod 23'280 mlcod 23'280 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:31.552968 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'281 crt=23'279 lcod 23'280 mlcod 23'280 active+clean]  removing repgather(0x7f21395d5a80 23'281 rep_tid=1701 committed?=1 applied?=1)
2016-10-25 08:51:31.552980 7f211f3fc700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'281 crt=23'279 lcod 23'280 mlcod 23'280 active+clean]    q front is repgather(0x7f21395d5a80 23'281 rep_tid=1701 committed?=1 applied?=1)
2016-10-25 08:51:31.552989 7f211f3fc700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'281 crt=23'279 lcod 23'280 mlcod 23'280 active+clean] remove_repop repgather(0x7f21395d5a80 23'281 rep_tid=1701 committed?=1 applied?=1)
2016-10-25 08:51:31.553003 7f211f3fc700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'281 crt=23'279 lcod 23'280 mlcod 23'280 active+clean]  requeue_ops 
2016-10-25 08:51:31.553262 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.4_head/#1:3d4d78f7:::benchmark_data_target167114251239.teuthology_13304_object857:head# 0~4194304 = 4194304
2016-10-25 08:51:31.553312 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.4_head/#1:3d4d78f7:::benchmark_data_target167114251239.teuthology_13304_object857:head#
2016-10-25 08:51:31.553359 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.4_head/#1:3d4d78f7:::benchmark_data_target167114251239.teuthology_13304_object857:head# = 0
2016-10-25 08:51:31.553376 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.4_head/#1:20000000::::head#
2016-10-25 08:51:31.553392 7f2121400700 10 filestore oid: #1:20000000::::head# not skipping op, *spos 2338.0.2
2016-10-25 08:51:31.553395 7f2121400700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.553444 7f2121400700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.4_head/#1:20000000::::head# = 0
2016-10-25 08:51:31.553461 7f2121400700 10 journal op_apply_finish 2338 open_ops 2 -> 1, max_applied_seq 2337 -> 2338
2016-10-25 08:51:31.553464 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2139545c20 seq 2338 r = 0, finisher 0x7f213ece1fc0 0x7f2147f328c0
2016-10-25 08:51:31.553493 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f2139545c20 seq 2338 osr(1.4 0x7f2139475240)/0x7f2139475240 lat 1.281638
2016-10-25 08:51:31.553521 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'281 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] op_applied: 1704
2016-10-25 08:51:31.553549 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'281 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] op_applied version 23'282
2016-10-25 08:51:31.553573 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] repop_all_applied: repop tid 1704 all applied 
2016-10-25 08:51:31.553579 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] eval_repop repgather(0x7f213985e400 23'282 rep_tid=1704 committed?=1 applied?=1) wants=d
2016-10-25 08:51:31.553585 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'278 lcod 23'281 mlcod 23'281 active+clean]  commit: repgather(0x7f213985e400 23'282 rep_tid=1704 committed?=1 applied?=1)
2016-10-25 08:51:31.553591 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'278 lcod 23'281 mlcod 23'281 active+clean]  applied: repgather(0x7f213985e400 23'282 rep_tid=1704 committed?=1 applied?=1) 
2016-10-25 08:51:31.553599 7f211ebfb700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] publish_stats_to_osd 23: no change since 2016-10-25 08:51:31.525231
2016-10-25 08:51:31.553613 7f211ebfb700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:31.553619 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'278 lcod 23'281 mlcod 23'281 active+clean]  removing repgather(0x7f213985e400 23'282 rep_tid=1704 committed?=1 applied?=1)
2016-10-25 08:51:31.553624 7f211ebfb700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'278 lcod 23'281 mlcod 23'281 active+clean]    q front is repgather(0x7f213985e400 23'282 rep_tid=1704 committed?=1 applied?=1)
2016-10-25 08:51:31.553630 7f211ebfb700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] remove_repop repgather(0x7f213985e400 23'282 rep_tid=1704 committed?=1 applied?=1)
2016-10-25 08:51:31.553658 7f211ebfb700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'278 lcod 23'281 mlcod 23'281 active+clean]  requeue_ops 
2016-10-25 08:51:31.556113 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.0_head/#1:0bbe1d8e:::benchmark_data_target167114251239.teuthology_13304_object855:head# 0~4194304 = 4194304
2016-10-25 08:51:31.556199 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.0_head/#1:0bbe1d8e:::benchmark_data_target167114251239.teuthology_13304_object855:head#
2016-10-25 08:51:31.556240 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.0_head/#1:0bbe1d8e:::benchmark_data_target167114251239.teuthology_13304_object855:head# = 0
2016-10-25 08:51:31.556258 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.0_head/#1:00000000::::head#
2016-10-25 08:51:31.556277 7f2121c01700 10 filestore oid: #1:00000000::::head# not skipping op, *spos 2339.0.2
2016-10-25 08:51:31.556287 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.556338 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.0_head/#1:00000000::::head# = 0
2016-10-25 08:51:31.556353 7f2121c01700 10 journal op_apply_finish 2339 open_ops 1 -> 0, max_applied_seq 2338 -> 2339
2016-10-25 08:51:31.556362 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2142f72d60 seq 2339 r = 0, finisher 0x7f213ece13c0 0x7f21480635c0
2016-10-25 08:51:31.556393 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f2142f72d60 seq 2339 osr(1.0 0x7f2139475d80)/0x7f2139475d80 lat 1.283269
2016-10-25 08:51:31.556425 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'280 crt=23'279 lcod 23'282 mlcod 23'279 active+clean] op_applied: 1702
2016-10-25 08:51:31.556438 7f2121400700 10 journal op_apply_start 2340 open_ops 0 -> 1
2016-10-25 08:51:31.556447 7f2121400700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f21397fd260 seq 2340 osr(1.0 0x7f2139475d80)/0x7f2139475d80 start
2016-10-25 08:51:31.556452 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147e4b340
2016-10-25 08:51:31.556451 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'280 crt=23'279 lcod 23'282 mlcod 23'279 active+clean] op_applied version 23'281
2016-10-25 08:51:31.556458 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'281 crt=23'279 lcod 23'282 mlcod 23'279 active+clean] repop_all_applied: repop tid 1702 all applied 
2016-10-25 08:51:31.556463 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'281 crt=23'279 lcod 23'282 mlcod 23'279 active+clean] eval_repop repgather(0x7f2139756ac0 23'281 rep_tid=1702 committed?=1 applied?=1) wants=d
2016-10-25 08:51:31.556465 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.0_head/#1:1f23fc2f:::benchmark_data_target167114251239.teuthology_13304_object856:head# 0~4194304
2016-10-25 08:51:31.556470 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'281 crt=23'279 lcod 23'282 mlcod 23'279 active+clean]  commit: repgather(0x7f2139756ac0 23'281 rep_tid=1702 committed?=1 applied?=1)
2016-10-25 08:51:31.556475 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'281 crt=23'279 lcod 23'282 mlcod 23'279 active+clean]  applied: repgather(0x7f2139756ac0 23'281 rep_tid=1702 committed?=1 applied?=1) 
2016-10-25 08:51:31.556484 7f211ebfb700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'281 crt=23'279 lcod 23'282 mlcod 23'279 active+clean] publish_stats_to_osd 23:138
2016-10-25 08:51:31.556493 7f211ebfb700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'281 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:31.556499 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'281 crt=23'279 lcod 23'282 mlcod 23'282 active+clean]  removing repgather(0x7f2139756ac0 23'281 rep_tid=1702 committed?=1 applied?=1)
2016-10-25 08:51:31.556504 7f211ebfb700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'281 crt=23'279 lcod 23'282 mlcod 23'282 active+clean]    q front is repgather(0x7f2139756ac0 23'281 rep_tid=1702 committed?=1 applied?=1)
2016-10-25 08:51:31.556509 7f211ebfb700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'281 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] remove_repop repgather(0x7f2139756ac0 23'281 rep_tid=1702 committed?=1 applied?=1)
2016-10-25 08:51:31.556517 7f211ebfb700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'281 crt=23'279 lcod 23'282 mlcod 23'282 active+clean]  requeue_ops 
2016-10-25 08:51:31.561201 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.0_head/#1:1f23fc2f:::benchmark_data_target167114251239.teuthology_13304_object856:head# 0~4194304 = 4194304
2016-10-25 08:51:31.561315 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.0_head/#1:1f23fc2f:::benchmark_data_target167114251239.teuthology_13304_object856:head#
2016-10-25 08:51:31.561378 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.0_head/#1:1f23fc2f:::benchmark_data_target167114251239.teuthology_13304_object856:head# = 0
2016-10-25 08:51:31.561431 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.0_head/#1:00000000::::head#
2016-10-25 08:51:31.561449 7f2121400700 10 filestore oid: #1:00000000::::head# not skipping op, *spos 2340.0.2
2016-10-25 08:51:31.561452 7f2121400700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.561510 7f2121400700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.0_head/#1:00000000::::head# = 0
2016-10-25 08:51:31.561522 7f2121400700 10 journal op_apply_finish 2340 open_ops 1 -> 0, max_applied_seq 2339 -> 2340
2016-10-25 08:51:31.561525 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f21397fd260 seq 2340 r = 0, finisher 0x7f213ecdf640 0x7f21395fb1c0
2016-10-25 08:51:31.561543 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f21397fd260 seq 2340 osr(1.0 0x7f2139475d80)/0x7f2139475d80 lat 1.286072
2016-10-25 08:51:31.561565 7f2121400700 10 journal op_apply_start 2342 open_ops 0 -> 1
2016-10-25 08:51:31.561570 7f2121400700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f213ed5c220 seq 2342 osr(1.2 0x7f2139474dc0)/0x7f2139474dc0 start
2016-10-25 08:51:31.561574 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147f6f480
2016-10-25 08:51:31.561584 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.2_head/#1:40000000::::head#
2016-10-25 08:51:31.561591 7f2121400700 10 filestore oid: #1:40000000::::head# not skipping op, *spos 2342.0.0
2016-10-25 08:51:31.561593 7f2121400700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.561621 7f2121400700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.2_head/#1:40000000::::head# = 0
2016-10-25 08:51:31.561603 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'281 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] op_applied: 1705
2016-10-25 08:51:31.561625 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147f6f640
2016-10-25 08:51:31.561631 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.2_head/#1:4229aed4:::benchmark_data_target167114251239.teuthology_13304_object848:head# 0~4194304
2016-10-25 08:51:31.561770 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'281 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] op_applied version 23'282
2016-10-25 08:51:31.561786 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'282 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] repop_all_applied: repop tid 1705 all applied 
2016-10-25 08:51:31.561834 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'282 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] eval_repop repgather(0x7f2139756be0 23'282 rep_tid=1705 committed?=1 applied?=1) wants=d
2016-10-25 08:51:31.561843 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'282 crt=23'279 lcod 23'282 mlcod 23'282 active+clean]  commit: repgather(0x7f2139756be0 23'282 rep_tid=1705 committed?=1 applied?=1)
2016-10-25 08:51:31.561849 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'282 crt=23'279 lcod 23'282 mlcod 23'282 active+clean]  applied: repgather(0x7f2139756be0 23'282 rep_tid=1705 committed?=1 applied?=1) 
2016-10-25 08:51:31.561859 7f211ebfb700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'282 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] publish_stats_to_osd 23: no change since 2016-10-25 08:51:31.556482
2016-10-25 08:51:31.561868 7f211ebfb700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'282 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:31.561874 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'282 crt=23'279 lcod 23'282 mlcod 23'282 active+clean]  removing repgather(0x7f2139756be0 23'282 rep_tid=1705 committed?=1 applied?=1)
2016-10-25 08:51:31.561903 7f211ebfb700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'282 crt=23'279 lcod 23'282 mlcod 23'282 active+clean]    q front is repgather(0x7f2139756be0 23'282 rep_tid=1705 committed?=1 applied?=1)
2016-10-25 08:51:31.561909 7f211ebfb700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'282 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] remove_repop repgather(0x7f2139756be0 23'282 rep_tid=1705 committed?=1 applied?=1)
2016-10-25 08:51:31.561918 7f211ebfb700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'282 crt=23'279 lcod 23'282 mlcod 23'282 active+clean]  requeue_ops 
2016-10-25 08:51:31.562115 7f2121c01700 10 journal op_apply_start 2341 open_ops 1 -> 2
2016-10-25 08:51:31.562162 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f21397feb80 seq 2341 osr(1.0 0x7f2139475d80)/0x7f2139475d80 start
2016-10-25 08:51:31.562174 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147e33c00
2016-10-25 08:51:31.562189 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.0_head/#1:1a36f7c7:::benchmark_data_target167114251239.teuthology_13304_object858:head# 0~4194304
2016-10-25 08:51:31.570594 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.0_head/#1:1a36f7c7:::benchmark_data_target167114251239.teuthology_13304_object858:head# 0~4194304 = 4194304
2016-10-25 08:51:31.570795 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.0_head/#1:1a36f7c7:::benchmark_data_target167114251239.teuthology_13304_object858:head#
2016-10-25 08:51:31.570879 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.0_head/#1:1a36f7c7:::benchmark_data_target167114251239.teuthology_13304_object858:head# = 0
2016-10-25 08:51:31.570920 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.0_head/#1:00000000::::head#
2016-10-25 08:51:31.570963 7f2121c01700 10 filestore oid: #1:00000000::::head# not skipping op, *spos 2341.0.2
2016-10-25 08:51:31.570985 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.571071 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.0_head/#1:00000000::::head# = 0
2016-10-25 08:51:31.571099 7f2121c01700 10 journal op_apply_finish 2341 open_ops 2 -> 1, max_applied_seq 2340 -> 2341
2016-10-25 08:51:31.571123 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f21397feb80 seq 2341 r = 0, finisher 0x7f213ece1a40 0x7f214309d180
2016-10-25 08:51:31.571149 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f21397feb80 seq 2341 osr(1.0 0x7f2139475d80)/0x7f2139475d80 lat 1.293557
2016-10-25 08:51:31.571185 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'282 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] op_applied: 1706
2016-10-25 08:51:31.571239 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'282 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] op_applied version 23'283
2016-10-25 08:51:31.571253 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'283 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] repop_all_applied: repop tid 1706 all applied 
2016-10-25 08:51:31.571265 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'283 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] eval_repop repgather(0x7f21396b9c20 23'283 rep_tid=1706 committed?=1 applied?=1) wants=d
2016-10-25 08:51:31.571277 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'283 crt=23'279 lcod 23'282 mlcod 23'282 active+clean]  commit: repgather(0x7f21396b9c20 23'283 rep_tid=1706 committed?=1 applied?=1)
2016-10-25 08:51:31.571289 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'283 crt=23'279 lcod 23'282 mlcod 23'282 active+clean]  applied: repgather(0x7f21396b9c20 23'283 rep_tid=1706 committed?=1 applied?=1) 
2016-10-25 08:51:31.571307 7f211ebfb700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'283 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] publish_stats_to_osd 23: no change since 2016-10-25 08:51:31.556482
2016-10-25 08:51:31.571325 7f211ebfb700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'283 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:31.571339 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'283 crt=23'279 lcod 23'282 mlcod 23'282 active+clean]  removing repgather(0x7f21396b9c20 23'283 rep_tid=1706 committed?=1 applied?=1)
2016-10-25 08:51:31.571350 7f211ebfb700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'283 crt=23'279 lcod 23'282 mlcod 23'282 active+clean]    q front is repgather(0x7f21396b9c20 23'283 rep_tid=1706 committed?=1 applied?=1)
2016-10-25 08:51:31.571362 7f211ebfb700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'283 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] remove_repop repgather(0x7f21396b9c20 23'283 rep_tid=1706 committed?=1 applied?=1)
2016-10-25 08:51:31.571380 7f211ebfb700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'283 crt=23'279 lcod 23'282 mlcod 23'282 active+clean]  requeue_ops 
2016-10-25 08:51:31.574259 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.2_head/#1:4229aed4:::benchmark_data_target167114251239.teuthology_13304_object848:head# 0~4194304 = 4194304
2016-10-25 08:51:31.574361 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.2_head/#1:4229aed4:::benchmark_data_target167114251239.teuthology_13304_object848:head#
2016-10-25 08:51:31.574405 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.2_head/#1:4229aed4:::benchmark_data_target167114251239.teuthology_13304_object848:head# = 0
2016-10-25 08:51:31.574421 7f2121400700 10 journal op_apply_finish 2342 open_ops 1 -> 0, max_applied_seq 2341 -> 2342
2016-10-25 08:51:31.574431 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f213ed5c220 seq 2342 r = 0, finisher 0x7f2139502400 0
2016-10-25 08:51:31.574450 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f213ed5c220 seq 2342 osr(1.2 0x7f2139474dc0)/0x7f2139474dc0 lat 1.276870
2016-10-25 08:51:31.574487 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.2( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'275 crt=23'274 lcod 23'275 active] sub_op_modify_applied on 0x7f2148077818 op osd_repop(client.4144.0:849 1.2 1:4229aed4:::benchmark_data_target167114251239.teuthology_13304_object848:head v 23'276) v1
2016-10-25 08:51:31.574507 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.2( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'275 crt=23'274 lcod 23'275 active] op_applied version 23'276
2016-10-25 08:51:31.577904 7f2122c03700 10 journal write_finish_thread_entry aio 916414464~4202496 done
2016-10-25 08:51:31.577927 7f2122c03700 10 journal write_finish_thread_entry aio 920616960~4202496 done
2016-10-25 08:51:31.577929 7f2122c03700 20 journal check_aio_completion
2016-10-25 08:51:31.577931 7f2122c03700 20 journal check_aio_completion completed seq 2343 916414464~4202496
2016-10-25 08:51:31.577940 7f2122c03700 20 journal check_aio_completion completed seq 2344 920616960~4202496
2016-10-25 08:51:31.577944 7f2122c03700 20 journal check_aio_completion queueing finishers through seq 2344
2016-10-25 08:51:31.577947 7f2122c03700 10 journal queue_completions_thru seq 2344 queueing seq 2343 0x7f214811db90 lat 0.284591
2016-10-25 08:51:31.577981 7f2122c03700 10 journal queue_completions_thru seq 2344 queueing seq 2344 0x7f2142e4e220 lat 0.276924
2016-10-25 08:51:31.577999 7f2122c03700 20 journal write_finish_thread_entry waiting for aio(s)
2016-10-25 08:51:31.578113 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f2143015980 seq 2343 osr(1.0 0x7f2139475d80) [Transaction(0x7f2147ece8c0)]
2016-10-25 08:51:31.578122 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f2143015980 seq 2343 osr(1.0 0x7f2139475d80) 4195584 bytes   (queue has 16 ops and 67129484 bytes)
2016-10-25 08:51:31.578130 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f213a53cd60
2016-10-25 08:51:31.578135 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f21395453e0 seq 2344 osr(1.3 0x7f2139475570) [Transaction(0x7f2147f67480)]
2016-10-25 08:51:31.578138 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f21395453e0 seq 2344 osr(1.3 0x7f2139475570) 4195584 bytes   (queue has 16 ops and 67129484 bytes)
2016-10-25 08:51:31.578143 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f2142e0ae20
2016-10-25 08:51:31.578155 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:31.578156 7f2123404700 20 journal write_thread_entry aio throttle: aio num 10 bytes 50429952 ... exp 20 min_new 1048576 ... pending 8404992
2016-10-25 08:51:31.578162 7f2123404700 10 journal room 488550399 max_size 1072709632 pos 975249408 header.start 391094272 top 4096
2016-10-25 08:51:31.578165 7f2123404700 10 journal check_for_full at 975249408 : 4202496 < 488550399
2016-10-25 08:51:31.578167 7f2123404700 15 journal prepare_single_write 1 will write 975249408 : seq 2357 len 4195682 -> 4202496
2016-10-25 08:51:31.578186 7f2123404700 10 journal room 484347903 max_size 1072709632 pos 979451904 header.start 391094272 top 4096
2016-10-25 08:51:31.578188 7f2123404700 10 journal check_for_full at 979451904 : 4202496 < 484347903
2016-10-25 08:51:31.578189 7f2123404700 15 journal prepare_single_write 2 will write 979451904 : seq 2358 len 4195682 -> 4202496
2016-10-25 08:51:31.578212 7f2123404700 20 journal prepare_multi_write queue_pos now 983654400
2016-10-25 08:51:31.578214 7f2123404700 15 journal do_aio_write writing 975249408~8404992
2016-10-25 08:51:31.578217 7f2123404700 20 journal write_aio_bl 975249408~8404992 seq 2358
2016-10-25 08:51:31.578221 7f2123404700 20 journal write_aio_bl .. 975249408~8404992 in 2
2016-10-25 08:51:31.579226 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:31.579232 7f2123404700  5 journal complete_write finished 2 ops and 8391364 bytes
2016-10-25 08:51:31.579234 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:31.579247 7f21203fe700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'283 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] op_commit: 1707
2016-10-25 08:51:31.579294 7f2120bff700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'281 crt=23'279 lcod 23'280 mlcod 23'280 active+clean] op_commit: 1708
2016-10-25 08:51:31.579431 7f2121c01700 10 journal op_apply_start 2343 open_ops 0 -> 1
2016-10-25 08:51:31.579451 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2143015980 seq 2343 osr(1.0 0x7f2139475d80)/0x7f2139475d80 start
2016-10-25 08:51:31.579468 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147ece8c0
2016-10-25 08:51:31.579494 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.0_head/#1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head# 0~4194304
2016-10-25 08:51:31.583083 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.0_head/#1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head# 0~4194304 = 4194304
2016-10-25 08:51:31.583136 7f2121400700 10 journal op_apply_start 2344 open_ops 1 -> 2
2016-10-25 08:51:31.583206 7f2121400700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f21395453e0 seq 2344 osr(1.3 0x7f2139475570)/0x7f2139475570 start
2016-10-25 08:51:31.583224 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147f67480
2016-10-25 08:51:31.583278 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.3_head/#1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head# 0~4194304
2016-10-25 08:51:31.586570 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.3_head/#1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head# 0~4194304 = 4194304
2016-10-25 08:51:31.586645 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.0_head/#1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head#
2016-10-25 08:51:31.586729 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.0_head/#1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head# = 0
2016-10-25 08:51:31.586756 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.0_head/#1:00000000::::head#
2016-10-25 08:51:31.586783 7f2121c01700 10 filestore oid: #1:00000000::::head# not skipping op, *spos 2343.0.2
2016-10-25 08:51:31.586797 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.586855 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.0_head/#1:00000000::::head# = 0
2016-10-25 08:51:31.586875 7f2121c01700 10 journal op_apply_finish 2343 open_ops 2 -> 1, max_applied_seq 2342 -> 2343
2016-10-25 08:51:31.586889 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2143015980 seq 2343 r = 0, finisher 0x7f213a53a2a0 0x7f21414c4680
2016-10-25 08:51:31.586913 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f2143015980 seq 2343 osr(1.0 0x7f2139475d80)/0x7f2139475d80 lat 0.295233
2016-10-25 08:51:31.586971 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.3_head/#1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head#
2016-10-25 08:51:31.587016 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.3_head/#1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head# = 0
2016-10-25 08:51:31.587039 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.3_head/#1:c0000000::::head#
2016-10-25 08:51:31.587057 7f2121400700 10 filestore oid: #1:c0000000::::head# not skipping op, *spos 2344.0.2
2016-10-25 08:51:31.587071 7f2121400700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.587105 7f2121400700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.3_head/#1:c0000000::::head# = 0
2016-10-25 08:51:31.587122 7f2121400700 10 journal op_apply_finish 2344 open_ops 1 -> 0, max_applied_seq 2343 -> 2344
2016-10-25 08:51:31.587134 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f21395453e0 seq 2344 r = 0, finisher 0x7f2142e097c0 0x7f2147f35a80
2016-10-25 08:51:31.587153 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f21395453e0 seq 2344 osr(1.3 0x7f2139475570)/0x7f2139475570 lat 0.287878
2016-10-25 08:51:31.587179 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'281 crt=23'279 lcod 23'280 mlcod 23'280 active+clean] op_applied: 1708
2016-10-25 08:51:31.587223 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'281 crt=23'279 lcod 23'280 mlcod 23'280 active+clean] op_applied version 23'282
2016-10-25 08:51:31.587268 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'283 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] op_applied: 1707
2016-10-25 08:51:31.587289 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'283 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] op_applied version 23'284
2016-10-25 08:51:31.638516 7f2122c03700 10 journal write_finish_thread_entry aio 924819456~4202496 done
2016-10-25 08:51:31.638528 7f2122c03700 10 journal write_finish_thread_entry aio 929021952~4202496 done
2016-10-25 08:51:31.638529 7f2122c03700 10 journal write_finish_thread_entry aio 933224448~4202496 done
2016-10-25 08:51:31.638531 7f2122c03700 10 journal write_finish_thread_entry aio 937426944~4202496 done
2016-10-25 08:51:31.638532 7f2122c03700 10 journal write_finish_thread_entry aio 941629440~4202496 done
2016-10-25 08:51:31.638533 7f2122c03700 10 journal write_finish_thread_entry aio 945831936~4202496 done
2016-10-25 08:51:31.638534 7f2122c03700 10 journal write_finish_thread_entry aio 950034432~4202496 done
2016-10-25 08:51:31.638534 7f2122c03700 10 journal write_finish_thread_entry aio 954236928~4202496 done
2016-10-25 08:51:31.638535 7f2122c03700 10 journal write_finish_thread_entry aio 958439424~12607488 done
2016-10-25 08:51:31.638536 7f2122c03700 10 journal write_finish_thread_entry aio 971046912~4202496 done
2016-10-25 08:51:31.638538 7f2122c03700 20 journal check_aio_completion
2016-10-25 08:51:31.638539 7f2122c03700 20 journal check_aio_completion completed seq 2345 924819456~4202496
2016-10-25 08:51:31.638547 7f2122c03700 20 journal check_aio_completion completed seq 2346 929021952~4202496
2016-10-25 08:51:31.638551 7f2122c03700 20 journal check_aio_completion completed seq 2347 933224448~4202496
2016-10-25 08:51:31.638554 7f2122c03700 20 journal check_aio_completion completed seq 2348 937426944~4202496
2016-10-25 08:51:31.638556 7f2122c03700 20 journal check_aio_completion completed seq 2349 941629440~4202496
2016-10-25 08:51:31.638559 7f2122c03700 20 journal check_aio_completion completed seq 2350 945831936~4202496
2016-10-25 08:51:31.638561 7f2122c03700 20 journal check_aio_completion completed seq 2351 950034432~4202496
2016-10-25 08:51:31.638563 7f2122c03700 20 journal check_aio_completion completed seq 2352 954236928~4202496
2016-10-25 08:51:31.638565 7f2122c03700 20 journal check_aio_completion completed seq 2355 958439424~12607488
2016-10-25 08:51:31.638580 7f2122c03700 20 journal check_aio_completion completed seq 2356 971046912~4202496
2016-10-25 08:51:31.638583 7f2122c03700 20 journal check_aio_completion queueing finishers through seq 2356
2016-10-25 08:51:31.638587 7f2122c03700 10 journal queue_completions_thru seq 2356 queueing seq 2345 0x7f2147fd2790 lat 0.328438
2016-10-25 08:51:31.638612 7f2122c03700 10 journal queue_completions_thru seq 2356 queueing seq 2346 0x7f2143097bf0 lat 0.319896
2016-10-25 08:51:31.638620 7f2122c03700 10 journal queue_completions_thru seq 2356 queueing seq 2347 0x7f21414ac100 lat 0.307475
2016-10-25 08:51:31.638636 7f2122c03700 10 journal queue_completions_thru seq 2356 queueing seq 2348 0x7f2140fda520 lat 0.300002
2016-10-25 08:51:31.638642 7f2122c03700 10 journal queue_completions_thru seq 2356 queueing seq 2349 0x7f214811dc20 lat 0.279250
2016-10-25 08:51:31.638654 7f2122c03700 10 journal queue_completions_thru seq 2356 queueing seq 2350 0x7f214811cff0 lat 0.273078
2016-10-25 08:51:31.638666 7f2122c03700 10 journal queue_completions_thru seq 2356 queueing seq 2351 0x7f21414aa720 lat 0.256461
2016-10-25 08:51:31.638675 7f2122c03700 10 journal queue_completions_thru seq 2356 queueing seq 2352 0x7f2140fb6970 lat 0.247465
2016-10-25 08:51:31.638683 7f2122c03700 10 journal queue_completions_thru seq 2356 queueing seq 2353 0x7f2140fb4c00 lat 0.242155
2016-10-25 08:51:31.638694 7f2122c03700 10 journal queue_completions_thru seq 2356 queueing seq 2354 0x7f2143096f30 lat 0.237709
2016-10-25 08:51:31.638701 7f2122c03700 10 journal queue_completions_thru seq 2356 queueing seq 2355 0x7f2140fb50e0 lat 0.232080
2016-10-25 08:51:31.638707 7f2122c03700 10 journal queue_completions_thru seq 2356 queueing seq 2356 0x7f2140fb52f0 lat 0.114571
2016-10-25 08:51:31.638715 7f2122c03700 20 journal write_finish_thread_entry waiting for aio(s)
2016-10-25 08:51:31.638954 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f2142e0f780 seq 2345 osr(1.4 0x7f2139475240) [Transaction(0x7f2147f61500)]
2016-10-25 08:51:31.638994 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f2142e0f780 seq 2345 osr(1.4 0x7f2139475240) 4195584 bytes   (queue has 14 ops and 58738316 bytes)
2016-10-25 08:51:31.639001 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f213ece1ea0
2016-10-25 08:51:31.639005 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f2142f1e4c0 seq 2346 osr(1.3 0x7f2139475570) [Transaction(0x7f2147eaba40)]
2016-10-25 08:51:31.639008 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f2142f1e4c0 seq 2346 osr(1.3 0x7f2139475570) 4195584 bytes   (queue has 14 ops and 58738316 bytes)
2016-10-25 08:51:31.639011 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f213eaa3220
2016-10-25 08:51:31.639014 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f2142e0ea60 seq 2347 osr(1.3 0x7f2139475570) [Transaction(0x7f2147e9e000)]
2016-10-25 08:51:31.639016 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f2142e0ea60 seq 2347 osr(1.3 0x7f2139475570) 4195584 bytes   (queue has 14 ops and 58738316 bytes)
2016-10-25 08:51:31.639017 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f2142e08960
2016-10-25 08:51:31.639019 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f21397f6e80 seq 2348 osr(1.0 0x7f2139475d80) [Transaction(0x7f2147e33500)]
2016-10-25 08:51:31.639021 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f21397f6e80 seq 2348 osr(1.0 0x7f2139475d80) 4195584 bytes   (queue has 14 ops and 58738316 bytes)
2016-10-25 08:51:31.639022 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f213ece0280
2016-10-25 08:51:31.639024 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f2146a1ed60 seq 2349 osr(1.7 0x7f2139474be0) [Transaction(0x7f2147ea9d40)]
2016-10-25 08:51:31.639026 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f2146a1ed60 seq 2349 osr(1.7 0x7f2139474be0) 4195584 bytes   (queue has 14 ops and 58738316 bytes)
2016-10-25 08:51:31.639028 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f2148113500
2016-10-25 08:51:31.639029 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f2142e0d020 seq 2350 osr(1.6 0x7f2139474f10) [Transaction(0x7f21481f8d80)]
2016-10-25 08:51:31.639031 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f2142e0d020 seq 2350 osr(1.6 0x7f2139474f10) 4195584 bytes   (queue has 14 ops and 58738316 bytes)
2016-10-25 08:51:31.639033 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f2147e82be0
2016-10-25 08:51:31.639042 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f2142f024c0 seq 2351 osr(1.1 0x7f21394758a0) [Transaction(0x7f2147f70380)]
2016-10-25 08:51:31.639046 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f2142f024c0 seq 2351 osr(1.1 0x7f21394758a0) 4195584 bytes   (queue has 14 ops and 58738316 bytes)
2016-10-25 08:51:31.639048 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f21401ef2a0
2016-10-25 08:51:31.639050 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f213ed5c400 seq 2352 osr(1.7 0x7f2139474be0) [Transaction(0x7f2147f688c0)]
2016-10-25 08:51:31.639051 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f213ed5c400 seq 2352 osr(1.7 0x7f2139474be0) 4195584 bytes   (queue has 14 ops and 58738316 bytes)
2016-10-25 08:51:31.639053 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f213a53d740
2016-10-25 08:51:31.639054 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f214133c0a0 seq 2353 osr(1.6 0x7f2139474f10) [Transaction(0x7f21415d28c0)]
2016-10-25 08:51:31.639055 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f214133c0a0 seq 2353 osr(1.6 0x7f2139474f10) 4195584 bytes   (queue has 14 ops and 58738316 bytes)
2016-10-25 08:51:31.639057 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f2142e09fc0
2016-10-25 08:51:31.639058 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f2142fd7120 seq 2354 osr(1.1 0x7f21394758a0) [Transaction(0x7f2147f17880)]
2016-10-25 08:51:31.639060 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f2142fd7120 seq 2354 osr(1.1 0x7f21394758a0) 4195584 bytes   (queue has 14 ops and 58738316 bytes)
2016-10-25 08:51:31.639062 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f2140e154a0
2016-10-25 08:51:31.639063 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f2139544c00 seq 2355 osr(1.6 0x7f2139474f10) [Transaction(0x7f2147f49800)]
2016-10-25 08:51:31.639065 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f2139544c00 seq 2355 osr(1.6 0x7f2139474f10) 4195584 bytes   (queue has 14 ops and 58738316 bytes)
2016-10-25 08:51:31.639066 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f21401ee680
2016-10-25 08:51:31.639068 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f213c9a00a0 seq 2356 osr(1.1 0x7f21394758a0) [Transaction(0x7f2140fff100)]
2016-10-25 08:51:31.639069 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f213c9a00a0 seq 2356 osr(1.1 0x7f21394758a0) 4195584 bytes   (queue has 14 ops and 58738316 bytes)
2016-10-25 08:51:31.639070 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f21401f0d40
2016-10-25 08:51:31.639078 7f21203fe700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] op_commit: 1709
2016-10-25 08:51:31.639103 7f21203fe700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'284 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] op_commit: 1712
2016-10-25 08:51:31.639152 7f2120bff700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'282 crt=23'279 lcod 23'280 mlcod 23'280 active+clean] op_commit: 1710
2016-10-25 08:51:31.639176 7f2120bff700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'282 crt=23'279 lcod 23'280 mlcod 23'280 active+clean] op_commit: 1711
2016-10-25 08:51:31.639223 7f2120bff700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] op_commit: 1713
2016-10-25 08:51:31.639228 7f211fbfd700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] op_commit: 1714
2016-10-25 08:51:31.639267 7f2120bff700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] op_commit: 1716
2016-10-25 08:51:31.638900 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2284 ==== osd_repop_reply(client.4144.0:861 1.3) v1 ==== 83+0+0 (2547057122 0 0) 0x7f213935a080 con 0x7f2139592800
2016-10-25 08:51:31.639288 7f211fbfd700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] op_commit: 1715
2016-10-25 08:51:31.639298 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:861 1.3) v1 epoch 23
2016-10-25 08:51:31.639305 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:31.639306 7f211fbfd700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] op_commit: 1717
2016-10-25 08:51:31.639315 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139880900 prio 196 cost 0 latency 0.000454 osd_repop_reply(client.4144.0:861 1.3) v1
2016-10-25 08:51:31.639320 7f211fbfd700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] op_commit: 1718
2016-10-25 08:51:31.639333 7f211fbfd700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] op_commit: 1719
2016-10-25 08:51:31.639345 7f211fbfd700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] op_commit: 1720
2016-10-25 08:51:31.639380 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2285 ==== osd_repop_reply(client.4144.0:863 1.3) v1 ==== 83+0+0 (3786985390 0 0) 0x7f214101c280 con 0x7f2139592800
2016-10-25 08:51:31.639390 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:863 1.3) v1 epoch 23
2016-10-25 08:51:31.639392 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:31.639395 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139883de0 prio 196 cost 0 latency 0.000048 osd_repop_reply(client.4144.0:863 1.3) v1
2016-10-25 08:51:31.639419 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2286 ==== osd_repop_reply(client.4144.0:864 1.3) v1 ==== 83+0+0 (2555442803 0 0) 0x7f214101c000 con 0x7f2139592800
2016-10-25 08:51:31.639428 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:864 1.3) v1 epoch 23
2016-10-25 08:51:31.639431 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:31.639433 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139881b00 prio 196 cost 0 latency 0.000022 osd_repop_reply(client.4144.0:864 1.3) v1
2016-10-25 08:51:31.639430 7f210ec67700 10 osd.3 23 dequeue_op 0x7f2139880900 prio 196 cost 0 latency 0.000568 osd_repop_reply(client.4144.0:861 1.3) v1 pg pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'282 crt=23'279 lcod 23'280 mlcod 23'280 active+clean]
2016-10-25 08:51:31.639445 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'282 crt=23'279 lcod 23'280 mlcod 23'280 active+clean] handle_message: 0x7f2139880900
2016-10-25 08:51:31.639457 7f210ec67700  7 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'282 crt=23'279 lcod 23'280 mlcod 23'280 active+clean] sub_op_modify_reply: tid 1708 op  ack_type 4 from 2
2016-10-25 08:51:31.639472 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'282 crt=23'279 lcod 23'280 mlcod 23'280 active+clean] repop_all_applied: repop tid 1708 all applied 
2016-10-25 08:51:31.639478 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'282 crt=23'279 lcod 23'280 mlcod 23'280 active+clean] eval_repop repgather(0x7f213981cea0 23'282 rep_tid=1708 committed?=0 applied?=1) wants=d
2016-10-25 08:51:31.639485 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'282 crt=23'279 lcod 23'280 mlcod 23'280 active+clean]  applied: repgather(0x7f213981cea0 23'282 rep_tid=1708 committed?=0 applied?=1) 
2016-10-25 08:51:31.639493 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'281 lua=23'282 crt=23'279 lcod 23'280 mlcod 23'280 active+clean] repop_all_committed: repop tid 1708 all committed 
2016-10-25 08:51:31.639517 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'279 lcod 23'281 mlcod 23'280 active+clean] eval_repop repgather(0x7f213981cea0 23'282 rep_tid=1708 committed?=1 applied?=1) wants=d
2016-10-25 08:51:31.639524 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'279 lcod 23'281 mlcod 23'280 active+clean]  commit: repgather(0x7f213981cea0 23'282 rep_tid=1708 committed?=1 applied?=1)
2016-10-25 08:51:31.639529 7f2121c01700 10 journal op_apply_start 2345 open_ops 0 -> 1
2016-10-25 08:51:31.639535 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2142e0f780 seq 2345 osr(1.4 0x7f2139475240)/0x7f2139475240 start
2016-10-25 08:51:31.639531 7f210ec67700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'279 lcod 23'281 mlcod 23'280 active+clean] log_op_stats osd_op(client.4144.0:861 1.f3447533 benchmark_data_target167114251239.teuthology_13304_object860 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 0.349916 lat 0.349954
2016-10-25 08:51:31.639539 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147f61500
2016-10-25 08:51:31.639540 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'279 lcod 23'281 mlcod 23'280 active+clean]  sending commit on osd_op(client.4144.0:861 1.f3447533 benchmark_data_target167114251239.teuthology_13304_object860 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f214131e580
2016-10-25 08:51:31.639548 7f210ec67700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(861 benchmark_data_target167114251239.teuthology_13304_object860 [write 0~4194304] v23'282 uv282 ondisk = 0) v7 -- 0x7f214131e580 con 0
2016-10-25 08:51:31.639552 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.4_head/#1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head# 0~4194304
2016-10-25 08:51:31.639567 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'279 lcod 23'281 mlcod 23'280 active+clean]  applied: repgather(0x7f213981cea0 23'282 rep_tid=1708 committed?=1 applied?=1) 
2016-10-25 08:51:31.639576 7f210ec67700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'279 lcod 23'281 mlcod 23'280 active+clean] publish_stats_to_osd 23: no change since 2016-10-25 08:51:31.552934
2016-10-25 08:51:31.639583 7f210ec67700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'279 lcod 23'281 mlcod 23'281 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:31.639589 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'279 lcod 23'281 mlcod 23'281 active+clean]  removing repgather(0x7f213981cea0 23'282 rep_tid=1708 committed?=1 applied?=1)
2016-10-25 08:51:31.639594 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'279 lcod 23'281 mlcod 23'281 active+clean]    q front is repgather(0x7f213981cea0 23'282 rep_tid=1708 committed?=1 applied?=1)
2016-10-25 08:51:31.639600 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'279 lcod 23'281 mlcod 23'281 active+clean] remove_repop repgather(0x7f213981cea0 23'282 rep_tid=1708 committed?=1 applied?=1)
2016-10-25 08:51:31.639615 7f210ec67700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'279 lcod 23'281 mlcod 23'281 active+clean]  requeue_ops 
2016-10-25 08:51:31.639646 7f210ec67700 10 osd.3 23 dequeue_op 0x7f2139880900 finish
2016-10-25 08:51:31.639666 7f210ec67700 10 osd.3 23 dequeue_op 0x7f2139883de0 prio 196 cost 0 latency 0.000314 osd_repop_reply(client.4144.0:863 1.3) v1 pg pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'279 lcod 23'281 mlcod 23'281 active+clean]
2016-10-25 08:51:31.639676 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'279 lcod 23'281 mlcod 23'281 active+clean] handle_message: 0x7f2139883de0
2016-10-25 08:51:31.639686 7f210ec67700  7 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'279 lcod 23'281 mlcod 23'281 active+clean] sub_op_modify_reply: tid 1710 op  ack_type 4 from 2
2016-10-25 08:51:31.639743 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'279 lcod 23'281 mlcod 23'281 active+clean] repop_all_committed: repop tid 1710 all committed 
2016-10-25 08:51:31.639751 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'282 crt=23'279 lcod 23'282 mlcod 23'281 active+clean] eval_repop repgather(0x7f21395d2b40 23'283 rep_tid=1710 committed?=1 applied?=0) wants=d
2016-10-25 08:51:31.639756 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'282 crt=23'279 lcod 23'282 mlcod 23'281 active+clean]  commit: repgather(0x7f21395d2b40 23'283 rep_tid=1710 committed?=1 applied?=0)
2016-10-25 08:51:31.639763 7f210ec67700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'282 crt=23'279 lcod 23'282 mlcod 23'281 active+clean] log_op_stats osd_op(client.4144.0:863 1.85982bc3 benchmark_data_target167114251239.teuthology_13304_object862 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 0.000000 lat 0.329163
2016-10-25 08:51:31.639770 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'282 crt=23'279 lcod 23'282 mlcod 23'281 active+clean]  sending commit on osd_op(client.4144.0:863 1.85982bc3 benchmark_data_target167114251239.teuthology_13304_object862 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f2142efbc80
2016-10-25 08:51:31.639776 7f210ec67700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(863 benchmark_data_target167114251239.teuthology_13304_object862 [write 0~4194304] v23'283 uv283 ondisk = 0) v7 -- 0x7f2142efbc80 con 0
2016-10-25 08:51:31.639788 7f210ec67700 10 osd.3 23 dequeue_op 0x7f2139883de0 finish
2016-10-25 08:51:31.639799 7f211146c700 10 osd.3 23 dequeue_op 0x7f2139881b00 prio 196 cost 0 latency 0.000388 osd_repop_reply(client.4144.0:864 1.3) v1 pg pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'282 crt=23'279 lcod 23'282 mlcod 23'281 active+clean]
2016-10-25 08:51:31.639809 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'282 crt=23'279 lcod 23'282 mlcod 23'281 active+clean] handle_message: 0x7f2139881b00
2016-10-25 08:51:31.639819 7f211146c700  7 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'282 crt=23'279 lcod 23'282 mlcod 23'281 active+clean] sub_op_modify_reply: tid 1711 op  ack_type 4 from 2
2016-10-25 08:51:31.639830 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'282 crt=23'279 lcod 23'282 mlcod 23'281 active+clean] repop_all_committed: repop tid 1711 all committed 
2016-10-25 08:51:31.639835 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] eval_repop repgather(0x7f21397f9e60 23'284 rep_tid=1711 committed?=1 applied?=0) wants=d
2016-10-25 08:51:31.639841 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean]  commit: repgather(0x7f21397f9e60 23'284 rep_tid=1711 committed?=1 applied?=0)
2016-10-25 08:51:31.639847 7f211146c700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] log_op_stats osd_op(client.4144.0:864 1.d38cced3 benchmark_data_target167114251239.teuthology_13304_object863 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 0.000000 lat 0.324539
2016-10-25 08:51:31.639854 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean]  sending commit on osd_op(client.4144.0:864 1.d38cced3 benchmark_data_target167114251239.teuthology_13304_object863 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f214131f8c0
2016-10-25 08:51:31.639861 7f211146c700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(864 benchmark_data_target167114251239.teuthology_13304_object863 [write 0~4194304] v23'284 uv284 ondisk = 0) v7 -- 0x7f214131f8c0 con 0
2016-10-25 08:51:31.639870 7f211146c700 10 osd.3 23 dequeue_op 0x7f2139881b00 finish
2016-10-25 08:51:31.640128 7f2121400700 10 journal op_apply_start 2346 open_ops 1 -> 2
2016-10-25 08:51:31.640177 7f2121400700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2142f1e4c0 seq 2346 osr(1.3 0x7f2139475570)/0x7f2139475570 start
2016-10-25 08:51:31.640190 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147eaba40
2016-10-25 08:51:31.640218 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.3_head/#1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head# 0~4194304
2016-10-25 08:51:31.642984 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.3_head/#1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head# 0~4194304 = 4194304
2016-10-25 08:51:31.643082 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.3_head/#1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head#
2016-10-25 08:51:31.643128 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.3_head/#1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head# = 0
2016-10-25 08:51:31.643149 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.3_head/#1:c0000000::::head#
2016-10-25 08:51:31.643170 7f2121400700 10 filestore oid: #1:c0000000::::head# not skipping op, *spos 2346.0.2
2016-10-25 08:51:31.643190 7f2121400700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.643237 7f2121400700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.3_head/#1:c0000000::::head# = 0
2016-10-25 08:51:31.643252 7f2121400700 10 journal op_apply_finish 2346 open_ops 2 -> 1, max_applied_seq 2344 -> 2346
2016-10-25 08:51:31.643262 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2142f1e4c0 seq 2346 r = 0, finisher 0x7f213eaa4840 0x7f2147f99240
2016-10-25 08:51:31.643278 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f2142f1e4c0 seq 2346 osr(1.3 0x7f2139475570)/0x7f2139475570 lat 0.326186
2016-10-25 08:51:31.643303 7f2121400700 10 journal op_apply_start 2347 open_ops 1 -> 2
2016-10-25 08:51:31.643313 7f2121400700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2142e0ea60 seq 2347 osr(1.3 0x7f2139475570)/0x7f2139475570 start
2016-10-25 08:51:31.643323 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147e9e000
2016-10-25 08:51:31.643372 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.3_head/#1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head# 0~4194304
2016-10-25 08:51:31.646899 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 674 ==== osd_op(client.4144.0:877 1.619240b (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (2298297987 0 3927644603) 0x7f214131f8c0 con 0x7f2147f52800
2016-10-25 08:51:31.646946 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:31.646954 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f21398829a0 prio 63 cost 4194304 latency 0.001886 osd_op(client.4144.0:877 1.619240b (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.647570 7f210ec67700 10 osd.3 23 dequeue_op 0x7f21398829a0 prio 63 cost 4194304 latency 0.002502 osd_op(client.4144.0:877 1.619240b (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean]
2016-10-25 08:51:31.647605 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] handle_message: 0x7f21398829a0
2016-10-25 08:51:31.647623 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:31.647640 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] do_op osd_op(client.4144.0:877 1.619240b benchmark_data_target167114251239.teuthology_13304_object876 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:31.647664 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] get_object_context: obc NOT found in cache: 1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head
2016-10-25 08:51:31.647674 7f210ec67700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head# '_'
2016-10-25 08:51:31.647734 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject876__head_0619240B__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.647742 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head# '_' = -2
2016-10-25 08:51:31.647745 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] get_object_context: no obc for soid 1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head but can_create
2016-10-25 08:51:31.647756 7f210ec67700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:snapdir# 'snapset'
2016-10-25 08:51:31.647806 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject876__snapdir_0619240B__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.647812 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:snapdir# 'snapset' = -2
2016-10-25 08:51:31.647835 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] create_object_context 0x7f2142e11340 1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head 
2016-10-25 08:51:31.647844 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] populate_obc_watchers 1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head
2016-10-25 08:51:31.647854 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head
2016-10-25 08:51:31.647863 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] get_object_context: 0x7f2142e11340 1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head rwstate(none n=0 w=0) oi: 1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f2139585a40 snapset: 0=[]:[]
2016-10-25 08:51:31.647880 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] find_object_context 1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head @head oi=1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.647890 7f210ec67700 25 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] do_op oi 1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.647899 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] get_object_context: obc NOT found in cache: 1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:snapdir
2016-10-25 08:51:31.647906 7f210ec67700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:snapdir# '_'
2016-10-25 08:51:31.647922 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject876__snapdir_0619240B__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.647925 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:snapdir# '_' = -2
2016-10-25 08:51:31.647928 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] get_object_context: no obc for soid 1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:snapdir and !can_create
2016-10-25 08:51:31.647944 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] execute_ctx 0x7f214303d500
2016-10-25 08:51:31.647951 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] do_op 1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head [write 0~4194304] ov 0'0 av 23'285 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:31.647961 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] do_osd_op 1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head [write 0~4194304]
2016-10-25 08:51:31.647967 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:31.648229 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] make_writeable 1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head snapset=0x7f2139585a78  snapc=0=[]
2016-10-25 08:51:31.648236 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean]  setting DIRTY flag
2016-10-25 08:51:31.648241 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] make_writeable 1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head done, snapset=0=[]:[]+head
2016-10-25 08:51:31.648247 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] finish_ctx 1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head 0x7f214303d500 op modify  
2016-10-25 08:51:31.648256 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean]  set mtime to 2016-10-25 08:51:31.642446
2016-10-25 08:51:31.648268 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean]  final snapset 0=[]:[]+head in 1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head
2016-10-25 08:51:31.648282 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean]  zeroing write result code 0
2016-10-25 08:51:31.648290 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean]  op order client.4144 tid 877 (first)
2016-10-25 08:51:31.648296 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] new_repop rep_tid 1721 on osd_op(client.4144.0:877 1.619240b benchmark_data_target167114251239.teuthology_13304_object876 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.648304 7f210ec67700  7 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] issue_repop rep_tid 1721 o 1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head
2016-10-25 08:51:31.648509 7f210ec67700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:31.648516 7f210ec67700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:877 1.3 1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head v 23'285) v1 -- 0x7f2140ff9200 con 0
2016-10-25 08:51:31.648785 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'284 (0'0,23'284] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] append_log log((0'0,23'284], crt=23'279) [23'285 (0'0) modify   1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head by client.4144.0:877 2016-10-25 08:51:31.642446 0]
2016-10-25 08:51:31.648806 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'282 crt=23'279 lcod 23'283 mlcod 23'281 active+clean] add_log_entry 23'285 (0'0) modify   1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head by client.4144.0:877 2016-10-25 08:51:31.642446 0
2016-10-25 08:51:31.648825 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'282 crt=23'281 lcod 23'283 mlcod 23'281 active+clean] append_log: trimming to 23'281 entries 23'280 (0'0) modify   1:de5f9bcb:::benchmark_data_target167114251239.teuthology_13304_object845:head by client.4144.0:846 2016-10-25 08:51:30.150487 0,23'281 (0'0) modify   1:db1dc65b:::benchmark_data_target167114251239.teuthology_13304_object854:head by client.4144.0:855 2016-10-25 08:51:30.215063 0
2016-10-25 08:51:31.648848 7f210ec67700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'285, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:31.649033 7f210ec67700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139241e40 osr(1.3 0x7f2139475570)
2016-10-25 08:51:31.649037 7f210ec67700 10 journal prepare_entry [Transaction(0x7f214159e000)]
2016-10-25 08:51:31.649284 7f210ec67700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:31.650160 7f210ec67700 10 journal op_submit_start 2359
2016-10-25 08:51:31.650164 7f210ec67700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2359 [Transaction(0x7f214159e000)]
2016-10-25 08:51:31.650167 7f210ec67700 10 journal op_journal_transactions 2359 reqid_t client.4144.0:877
2016-10-25 08:51:31.650169 7f210ec67700  5 journal submit_entry seq 2359 len 4202496 (0x7f2140fb4150)
2016-10-25 08:51:31.652381 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:31.652392 7f2123404700 20 journal write_thread_entry aio throttle: aio num 1 bytes 8404992 ... exp 2 min_new 4 ... pending 4202496
2016-10-25 08:51:31.652396 7f2123404700 10 journal room 480145407 max_size 1072709632 pos 983654400 header.start 391094272 top 4096
2016-10-25 08:51:31.652411 7f2123404700 10 journal check_for_full at 983654400 : 4202496 < 480145407
2016-10-25 08:51:31.652413 7f2123404700 15 journal prepare_single_write 1 will write 983654400 : seq 2359 len 4195598 -> 4202496
2016-10-25 08:51:31.652466 7f2123404700 20 journal prepare_multi_write queue_pos now 987856896
2016-10-25 08:51:31.652468 7f2123404700 15 journal do_aio_write writing 983654400~4202496
2016-10-25 08:51:31.652470 7f2123404700 20 journal write_aio_bl 983654400~4202496 seq 2359
2016-10-25 08:51:31.652494 7f2123404700 20 journal write_aio_bl .. 983654400~4202496 in 1
2016-10-25 08:51:31.652873 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:31.652882 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:31.652885 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:31.652909 7f210ec67700 10 journal op_submit_finish 2359
2016-10-25 08:51:31.652928 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'282 crt=23'281 lcod 23'283 mlcod 23'281 active+clean] eval_repop repgather(0x7f2139880900 23'285 rep_tid=1721 committed?=0 applied?=0) wants=d
2016-10-25 08:51:31.652967 7f210ec67700 10 osd.3 23 dequeue_op 0x7f21398829a0 finish
2016-10-25 08:51:31.653736 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 675 ==== osd_op(client.4144.0:878 1.7db9c9f9 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (717434880 0 564370168) 0x7f2142efbc80 con 0x7f2147f52800
2016-10-25 08:51:31.653786 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'282 crt=23'281 lcod 23'283 mlcod 23'281 active+clean] op_applied: 1710
2016-10-25 08:51:31.653827 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'282 crt=23'281 lcod 23'283 mlcod 23'281 active+clean] op_applied version 23'283
2016-10-25 08:51:31.653837 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'283 crt=23'281 lcod 23'283 mlcod 23'281 active+clean] repop_all_applied: repop tid 1710 all applied 
2016-10-25 08:51:31.653844 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'283 crt=23'281 lcod 23'283 mlcod 23'281 active+clean] eval_repop repgather(0x7f21395d2b40 23'283 rep_tid=1710 committed?=1 applied?=1) wants=d
2016-10-25 08:51:31.653850 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'283 crt=23'281 lcod 23'283 mlcod 23'281 active+clean]  commit: repgather(0x7f21395d2b40 23'283 rep_tid=1710 committed?=1 applied?=1)
2016-10-25 08:51:31.653856 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'283 crt=23'281 lcod 23'283 mlcod 23'281 active+clean]  applied: repgather(0x7f21395d2b40 23'283 rep_tid=1710 committed?=1 applied?=1) 
2016-10-25 08:51:31.653866 7f211f3fc700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'283 crt=23'281 lcod 23'283 mlcod 23'281 active+clean] publish_stats_to_osd 23:152
2016-10-25 08:51:31.653874 7f211f3fc700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'283 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:31.653881 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'283 crt=23'281 lcod 23'283 mlcod 23'283 active+clean]  removing repgather(0x7f21395d2b40 23'283 rep_tid=1710 committed?=1 applied?=1)
2016-10-25 08:51:31.653886 7f211f3fc700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'283 crt=23'281 lcod 23'283 mlcod 23'283 active+clean]    q front is repgather(0x7f21395d2b40 23'283 rep_tid=1710 committed?=1 applied?=1)
2016-10-25 08:51:31.653892 7f211f3fc700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'283 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] remove_repop repgather(0x7f21395d2b40 23'283 rep_tid=1710 committed?=1 applied?=1)
2016-10-25 08:51:31.653902 7f211f3fc700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'283 crt=23'281 lcod 23'283 mlcod 23'283 active+clean]  requeue_ops 
2016-10-25 08:51:31.653835 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:31.653972 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f21398817a0 prio 63 cost 4194304 latency 0.006982 osd_op(client.4144.0:878 1.7db9c9f9 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.654052 7f211246e700 10 osd.3 23 dequeue_op 0x7f21398817a0 prio 63 cost 4194304 latency 0.007062 osd_op(client.4144.0:878 1.7db9c9f9 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean]
2016-10-25 08:51:31.654101 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] handle_message: 0x7f21398817a0
2016-10-25 08:51:31.654124 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:31.654148 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.4_head/#1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head# 0~4194304 = 4194304
2016-10-25 08:51:31.654149 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] do_op osd_op(client.4144.0:878 1.7db9c9f9 benchmark_data_target167114251239.teuthology_13304_object877 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:31.654175 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] get_object_context: obc NOT found in cache: 1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head
2016-10-25 08:51:31.654185 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.4_head/#1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head#
2016-10-25 08:51:31.654188 7f211246e700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head# '_'
2016-10-25 08:51:31.654219 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.4_head/#1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head# = 0
2016-10-25 08:51:31.654228 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.4_head/#1:20000000::::head#
2016-10-25 08:51:31.654240 7f2121c01700 10 filestore oid: #1:20000000::::head# not skipping op, *spos 2345.0.2
2016-10-25 08:51:31.654241 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.654242 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.1_head/DIR_9/benchmark\udata\utarget167114251239.teuthology\u13304\uobject877__head_7DB9C9F9__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.654250 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head# '_' = -2
2016-10-25 08:51:31.654252 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] get_object_context: no obc for soid 1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head but can_create
2016-10-25 08:51:31.654261 7f211246e700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:snapdir# 'snapset'
2016-10-25 08:51:31.654276 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.1_head/DIR_9/benchmark\udata\utarget167114251239.teuthology\u13304\uobject877__snapdir_7DB9C9F9__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.654279 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.4_head/#1:20000000::::head# = 0
2016-10-25 08:51:31.654279 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:snapdir# 'snapset' = -2
2016-10-25 08:51:31.654283 7f2121c01700 10 journal op_apply_finish 2345 open_ops 2 -> 1, max_applied_seq 2346 -> 2346
2016-10-25 08:51:31.654285 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2142e0f780 seq 2345 r = 0, finisher 0x7f213ecdfb00 0x7f21480ae640
2016-10-25 08:51:31.654294 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f2142e0f780 seq 2345 osr(1.4 0x7f2139475240)/0x7f2139475240 lat 0.346187
2016-10-25 08:51:31.654295 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] create_object_context 0x7f213ea57b80 1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head 
2016-10-25 08:51:31.654303 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] populate_obc_watchers 1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head
2016-10-25 08:51:31.654312 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] op_applied: 1709
2016-10-25 08:51:31.654321 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head
2016-10-25 08:51:31.654330 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 lua=23'282 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] op_applied version 23'283
2016-10-25 08:51:31.654327 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] get_object_context: 0x7f213ea57b80 1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head rwstate(none n=0 w=0) oi: 1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f21397e5420 snapset: 0=[]:[]
2016-10-25 08:51:31.654340 7f2121c01700 10 journal op_apply_start 2348 open_ops 1 -> 2
2016-10-25 08:51:31.654341 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f21397f6e80 seq 2348 osr(1.0 0x7f2139475d80)/0x7f2139475d80 start
2016-10-25 08:51:31.654344 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147e33500
2016-10-25 08:51:31.654337 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] find_object_context 1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head @head oi=1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.654349 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.0_head/#1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head# 0~4194304
2016-10-25 08:51:31.654354 7f211246e700 25 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] do_op oi 1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.654363 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] get_object_context: obc NOT found in cache: 1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:snapdir
2016-10-25 08:51:31.654369 7f211246e700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:snapdir# '_'
2016-10-25 08:51:31.654383 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.1_head/DIR_9/benchmark\udata\utarget167114251239.teuthology\u13304\uobject877__snapdir_7DB9C9F9__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.654386 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.1_head/#1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:snapdir# '_' = -2
2016-10-25 08:51:31.654388 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] get_object_context: no obc for soid 1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:snapdir and !can_create
2016-10-25 08:51:31.654402 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] execute_ctx 0x7f2148073100
2016-10-25 08:51:31.654410 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] do_op 1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head [write 0~4194304] ov 0'0 av 23'326 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:31.654419 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] do_osd_op 1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head [write 0~4194304]
2016-10-25 08:51:31.654425 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:31.654727 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] make_writeable 1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head snapset=0x7f21397e5458  snapc=0=[]
2016-10-25 08:51:31.654745 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean]  setting DIRTY flag
2016-10-25 08:51:31.654754 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] make_writeable 1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head done, snapset=0=[]:[]+head
2016-10-25 08:51:31.654765 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] finish_ctx 1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head 0x7f2148073100 op modify  
2016-10-25 08:51:31.654780 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean]  set mtime to 2016-10-25 08:51:31.642607
2016-10-25 08:51:31.654798 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=325 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean]  final snapset 0=[]:[]+head in 1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head
2016-10-25 08:51:31.654821 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean]  zeroing write result code 0
2016-10-25 08:51:31.654835 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean]  op order client.4144 tid 878 (first)
2016-10-25 08:51:31.654844 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] new_repop rep_tid 1722 on osd_op(client.4144.0:878 1.7db9c9f9 benchmark_data_target167114251239.teuthology_13304_object877 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.654861 7f211246e700  7 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] issue_repop rep_tid 1722 o 1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head
2016-10-25 08:51:31.654920 7f211246e700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:31.654924 7f211246e700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:878 1.1 1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head v 23'326) v1 -- 0x7f2141054a00 con 0
2016-10-25 08:51:31.655208 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'325 (0'0,23'325] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] append_log log((0'0,23'325], crt=23'320) [23'326 (0'0) modify   1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head by client.4144.0:878 2016-10-25 08:51:31.642607 0]
2016-10-25 08:51:31.655233 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'320 lcod 23'321 mlcod 23'321 active+clean] add_log_entry 23'326 (0'0) modify   1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head by client.4144.0:878 2016-10-25 08:51:31.642607 0
2016-10-25 08:51:31.655251 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'321 lcod 23'321 mlcod 23'321 active+clean] append_log: trimming to 23'321 entries 23'321 (0'0) modify   1:81afd268:::benchmark_data_target167114251239.teuthology_13304_object846:head by client.4144.0:847 2016-10-25 08:51:30.150515 0
2016-10-25 08:51:31.655277 7f211246e700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'326, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:31.655545 7f211246e700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139241600 osr(1.1 0x7f21394758a0)
2016-10-25 08:51:31.655553 7f211246e700 10 journal prepare_entry [Transaction(0x7f2142fe7480)]
2016-10-25 08:51:31.655804 7f211246e700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:31.656682 7f211246e700 10 journal op_submit_start 2360
2016-10-25 08:51:31.656684 7f211246e700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2360 [Transaction(0x7f2142fe7480)]
2016-10-25 08:51:31.656687 7f211246e700 10 journal op_journal_transactions 2360 reqid_t client.4144.0:878
2016-10-25 08:51:31.656688 7f211246e700  5 journal submit_entry seq 2360 len 4202496 (0x7f2139564ff0)
2016-10-25 08:51:31.657032 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:31.657039 7f2123404700 20 journal write_thread_entry aio throttle: aio num 2 bytes 12607488 ... exp 4 min_new 16 ... pending 4202496
2016-10-25 08:51:31.657044 7f2123404700 10 journal room 475942911 max_size 1072709632 pos 987856896 header.start 391094272 top 4096
2016-10-25 08:51:31.657046 7f2123404700 10 journal check_for_full at 987856896 : 4202496 < 475942911
2016-10-25 08:51:31.657048 7f2123404700 15 journal prepare_single_write 1 will write 987856896 : seq 2360 len 4195598 -> 4202496
2016-10-25 08:51:31.657064 7f2123404700 20 journal prepare_multi_write queue_pos now 992059392
2016-10-25 08:51:31.657066 7f2123404700 15 journal do_aio_write writing 987856896~4202496
2016-10-25 08:51:31.657068 7f2123404700 20 journal write_aio_bl 987856896~4202496 seq 2360
2016-10-25 08:51:31.657073 7f2123404700 20 journal write_aio_bl .. 987856896~4202496 in 1
2016-10-25 08:51:31.657343 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:31.657346 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:31.657349 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:31.657471 7f211246e700 10 journal op_submit_finish 2360
2016-10-25 08:51:31.657484 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'321 lcod 23'321 mlcod 23'321 active+clean] eval_repop repgather(0x7f21394ac2e0 23'326 rep_tid=1722 committed?=0 applied?=0) wants=d
2016-10-25 08:51:31.657512 7f211246e700 10 osd.3 23 dequeue_op 0x7f21398817a0 finish
2016-10-25 08:51:31.660570 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.3_head/#1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head# 0~4194304 = 4194304
2016-10-25 08:51:31.660870 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.3_head/#1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head#
2016-10-25 08:51:31.660967 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.3_head/#1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head# = 0
2016-10-25 08:51:31.661012 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.3_head/#1:c0000000::::head#
2016-10-25 08:51:31.661056 7f2121400700 10 filestore oid: #1:c0000000::::head# not skipping op, *spos 2347.0.2
2016-10-25 08:51:31.661097 7f2121400700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.661173 7f2121400700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.3_head/#1:c0000000::::head# = 0
2016-10-25 08:51:31.661210 7f2121400700 10 journal op_apply_finish 2347 open_ops 2 -> 1, max_applied_seq 2346 -> 2347
2016-10-25 08:51:31.661232 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2142e0ea60 seq 2347 r = 0, finisher 0x7f2142e08a40 0x7f2148063200
2016-10-25 08:51:31.661345 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f2142e0ea60 seq 2347 osr(1.3 0x7f2139475570)/0x7f2139475570 lat 0.331530
2016-10-25 08:51:31.661426 7f2121400700 10 journal op_apply_start 2349 open_ops 1 -> 2
2016-10-25 08:51:31.661450 7f2121400700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2146a1ed60 seq 2349 osr(1.7 0x7f2139474be0)/0x7f2139474be0 start
2016-10-25 08:51:31.661471 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147ea9d40
2016-10-25 08:51:31.661500 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.7_head/#1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head# 0~4194304
2016-10-25 08:51:31.662561 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'283 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] op_applied: 1711
2016-10-25 08:51:31.662620 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'283 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] op_applied version 23'284
2016-10-25 08:51:31.662629 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] repop_all_applied: repop tid 1711 all applied 
2016-10-25 08:51:31.662635 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] eval_repop repgather(0x7f21397f9e60 23'284 rep_tid=1711 committed?=1 applied?=1) wants=d
2016-10-25 08:51:31.662642 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean]  commit: repgather(0x7f21397f9e60 23'284 rep_tid=1711 committed?=1 applied?=1)
2016-10-25 08:51:31.662647 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean]  applied: repgather(0x7f21397f9e60 23'284 rep_tid=1711 committed?=1 applied?=1) 
2016-10-25 08:51:31.662657 7f211f3fc700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] publish_stats_to_osd 23: no change since 2016-10-25 08:51:31.653863
2016-10-25 08:51:31.662667 7f211f3fc700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:31.662674 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean]  removing repgather(0x7f21397f9e60 23'284 rep_tid=1711 committed?=1 applied?=1)
2016-10-25 08:51:31.662679 7f211f3fc700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean]    q front is repgather(0x7f21397f9e60 23'284 rep_tid=1711 committed?=1 applied?=1)
2016-10-25 08:51:31.662685 7f211f3fc700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] remove_repop repgather(0x7f21397f9e60 23'284 rep_tid=1711 committed?=1 applied?=1)
2016-10-25 08:51:31.662695 7f211f3fc700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean]  requeue_ops 
2016-10-25 08:51:31.665532 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.0_head/#1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head# 0~4194304 = 4194304
2016-10-25 08:51:31.665654 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.0_head/#1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head#
2016-10-25 08:51:31.665704 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.0_head/#1:15be39b7:::benchmark_data_target167114251239.teuthology_13304_object864:head# = 0
2016-10-25 08:51:31.665726 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.0_head/#1:00000000::::head#
2016-10-25 08:51:31.665749 7f2121c01700 10 filestore oid: #1:00000000::::head# not skipping op, *spos 2348.0.2
2016-10-25 08:51:31.665759 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.665812 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.0_head/#1:00000000::::head# = 0
2016-10-25 08:51:31.665827 7f2121c01700 10 journal op_apply_finish 2348 open_ops 2 -> 1, max_applied_seq 2347 -> 2348
2016-10-25 08:51:31.665837 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f21397f6e80 seq 2348 r = 0, finisher 0x7f213ecde3c0 0x7f21480ad8c0
2016-10-25 08:51:31.665854 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f21397f6e80 seq 2348 osr(1.0 0x7f2139475d80)/0x7f2139475d80 lat 0.329456
2016-10-25 08:51:31.665881 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'284 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] op_applied: 1712
2016-10-25 08:51:31.665912 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'284 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] op_applied version 23'285
2016-10-25 08:51:31.665932 7f2121c01700 10 journal op_apply_start 2350 open_ops 1 -> 2
2016-10-25 08:51:31.665941 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2142e0d020 seq 2350 osr(1.6 0x7f2139474f10)/0x7f2139474f10 start
2016-10-25 08:51:31.665951 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f21481f8d80
2016-10-25 08:51:31.665966 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.6_head/#1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head# 0~4194304
2016-10-25 08:51:31.666348 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.7_head/#1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head# 0~4194304 = 4194304
2016-10-25 08:51:31.666416 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.7_head/#1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head#
2016-10-25 08:51:31.666479 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.7_head/#1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head# = 0
2016-10-25 08:51:31.666502 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.7_head/#1:e0000000::::head#
2016-10-25 08:51:31.666519 7f2121400700 10 filestore oid: #1:e0000000::::head# not skipping op, *spos 2349.0.2
2016-10-25 08:51:31.666527 7f2121400700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.666576 7f2121400700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.7_head/#1:e0000000::::head# = 0
2016-10-25 08:51:31.666591 7f2121400700 10 journal op_apply_finish 2349 open_ops 2 -> 1, max_applied_seq 2348 -> 2349
2016-10-25 08:51:31.666595 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2146a1ed60 seq 2349 r = 0, finisher 0x7f2148112940 0x7f21413b6540
2016-10-25 08:51:31.666609 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f2146a1ed60 seq 2349 osr(1.7 0x7f2139474be0)/0x7f2139474be0 lat 0.308511
2016-10-25 08:51:31.666632 7f2121400700 10 journal op_apply_start 2351 open_ops 1 -> 2
2016-10-25 08:51:31.666640 7f2121400700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2142f024c0 seq 2351 osr(1.1 0x7f21394758a0)/0x7f21394758a0 start
2016-10-25 08:51:31.666643 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147f70380
2016-10-25 08:51:31.666654 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.1_head/#1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head# 0~4194304
2016-10-25 08:51:31.666634 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] op_applied: 1713
2016-10-25 08:51:31.666671 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] op_applied version 23'266
2016-10-25 08:51:31.672775 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.6_head/#1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head# 0~4194304 = 4194304
2016-10-25 08:51:31.672863 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.6_head/#1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head#
2016-10-25 08:51:31.672908 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.6_head/#1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head# = 0
2016-10-25 08:51:31.672929 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.6_head/#1:60000000::::head#
2016-10-25 08:51:31.672949 7f2121c01700 10 filestore oid: #1:60000000::::head# not skipping op, *spos 2350.0.2
2016-10-25 08:51:31.672959 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.673006 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.6_head/#1:60000000::::head# = 0
2016-10-25 08:51:31.673020 7f2121c01700 10 journal op_apply_finish 2350 open_ops 2 -> 1, max_applied_seq 2349 -> 2350
2016-10-25 08:51:31.673030 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2142e0d020 seq 2350 r = 0, finisher 0x7f2147e83960 0x7f214304e780
2016-10-25 08:51:31.673053 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f2142e0d020 seq 2350 osr(1.6 0x7f2139474f10)/0x7f2139474f10 lat 0.308931
2016-10-25 08:51:31.673082 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] op_applied: 1714
2016-10-25 08:51:31.673113 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] op_applied version 23'275
2016-10-25 08:51:31.673134 7f2121c01700 10 journal op_apply_start 2352 open_ops 1 -> 2
2016-10-25 08:51:31.673144 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f213ed5c400 seq 2352 osr(1.7 0x7f2139474be0)/0x7f2139474be0 start
2016-10-25 08:51:31.673153 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147f688c0
2016-10-25 08:51:31.673167 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.7_head/#1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head# 0~4194304
2016-10-25 08:51:31.675207 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.1_head/#1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head# 0~4194304 = 4194304
2016-10-25 08:51:31.675274 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.1_head/#1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head#
2016-10-25 08:51:31.675331 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.1_head/#1:9290c060:::benchmark_data_target167114251239.teuthology_13304_object867:head# = 0
2016-10-25 08:51:31.675349 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.1_head/#1:80000000::::head#
2016-10-25 08:51:31.675367 7f2121400700 10 filestore oid: #1:80000000::::head# not skipping op, *spos 2351.0.2
2016-10-25 08:51:31.675384 7f2121400700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.675430 7f2121400700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.1_head/#1:80000000::::head# = 0
2016-10-25 08:51:31.675438 7f2121400700 10 journal op_apply_finish 2351 open_ops 2 -> 1, max_applied_seq 2350 -> 2351
2016-10-25 08:51:31.675440 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2142f024c0 seq 2351 r = 0, finisher 0x7f21401f0b40 0x7f213e5f9740
2016-10-25 08:51:31.675480 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f2142f024c0 seq 2351 osr(1.1 0x7f21394758a0)/0x7f21394758a0 lat 0.297369
2016-10-25 08:51:31.676798 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.7_head/#1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head# 0~4194304 = 4194304
2016-10-25 08:51:31.676852 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.7_head/#1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head#
2016-10-25 08:51:31.676891 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.7_head/#1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head# = 0
2016-10-25 08:51:31.676911 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.7_head/#1:e0000000::::head#
2016-10-25 08:51:31.676928 7f2121c01700 10 filestore oid: #1:e0000000::::head# not skipping op, *spos 2352.0.2
2016-10-25 08:51:31.676937 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.676979 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.7_head/#1:e0000000::::head# = 0
2016-10-25 08:51:31.676993 7f2121c01700 10 journal op_apply_finish 2352 open_ops 1 -> 0, max_applied_seq 2351 -> 2352
2016-10-25 08:51:31.677002 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f213ed5c400 seq 2352 r = 0, finisher 0x7f213a53b9e0 0x7f21413b9080
2016-10-25 08:51:31.677040 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'321 lcod 23'321 mlcod 23'321 active+clean] op_applied: 1715
2016-10-25 08:51:31.677070 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'322 crt=23'321 lcod 23'321 mlcod 23'321 active+clean] op_applied version 23'323
2016-10-25 08:51:31.677092 7f2121400700 10 journal op_apply_start 2353 open_ops 0 -> 1
2016-10-25 08:51:31.677107 7f2121400700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f214133c0a0 seq 2353 osr(1.6 0x7f2139474f10)/0x7f2139474f10 start
2016-10-25 08:51:31.677096 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f213ed5c400 seq 2352 osr(1.7 0x7f2139474be0)/0x7f2139474be0 lat 0.287266
2016-10-25 08:51:31.677118 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f21415d28c0
2016-10-25 08:51:31.677146 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.6_head/#1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head# 0~4194304
2016-10-25 08:51:31.677155 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'266 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] op_applied: 1716
2016-10-25 08:51:31.677191 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'266 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] op_applied version 23'267
2016-10-25 08:51:31.677165 7f2121c01700 10 journal op_apply_start 2354 open_ops 1 -> 2
2016-10-25 08:51:31.677224 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2142fd7120 seq 2354 osr(1.1 0x7f21394758a0)/0x7f21394758a0 start
2016-10-25 08:51:31.677234 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147f17880
2016-10-25 08:51:31.678634 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.1_head/#1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head# 0~4194304
2016-10-25 08:51:31.682221 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.6_head/#1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head# 0~4194304 = 4194304
2016-10-25 08:51:31.682917 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.6_head/#1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head#
2016-10-25 08:51:31.682962 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.6_head/#1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head# = 0
2016-10-25 08:51:31.682849 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.1_head/#1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head# 0~4194304 = 4194304
2016-10-25 08:51:31.683010 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.1_head/#1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head#
2016-10-25 08:51:31.683038 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.1_head/#1:8dea5602:::benchmark_data_target167114251239.teuthology_13304_object871:head# = 0
2016-10-25 08:51:31.683051 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.6_head/#1:60000000::::head#
2016-10-25 08:51:31.683080 7f2121400700 10 filestore oid: #1:60000000::::head# not skipping op, *spos 2353.0.2
2016-10-25 08:51:31.683088 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.1_head/#1:80000000::::head#
2016-10-25 08:51:31.683111 7f2121c01700 10 filestore oid: #1:80000000::::head# not skipping op, *spos 2354.0.2
2016-10-25 08:51:31.683099 7f2121400700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.683166 7f2121400700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.6_head/#1:60000000::::head# = 0
2016-10-25 08:51:31.683172 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.683205 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.1_head/#1:80000000::::head# = 0
2016-10-25 08:51:31.683217 7f2121c01700 10 journal op_apply_finish 2354 open_ops 2 -> 1, max_applied_seq 2352 -> 2354
2016-10-25 08:51:31.683228 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2142fd7120 seq 2354 r = 0, finisher 0x7f2140e14100 0x7f213e5fbb40
2016-10-25 08:51:31.683282 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f2142fd7120 seq 2354 osr(1.1 0x7f21394758a0)/0x7f21394758a0 lat 0.283603
2016-10-25 08:51:31.683289 7f2121400700 10 journal op_apply_finish 2353 open_ops 1 -> 0, max_applied_seq 2354 -> 2354
2016-10-25 08:51:31.683305 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f214133c0a0 seq 2353 r = 0, finisher 0x7f2142e09320 0x7f21413b8dc0
2016-10-25 08:51:31.683372 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'323 crt=23'321 lcod 23'321 mlcod 23'321 active+clean] op_applied: 1718
2016-10-25 08:51:31.683402 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'323 crt=23'321 lcod 23'321 mlcod 23'321 active+clean] op_applied version 23'324
2016-10-25 08:51:31.683425 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f214133c0a0 seq 2353 osr(1.6 0x7f2139474f10)/0x7f2139474f10 lat 0.288185
2016-10-25 08:51:31.683452 7f2121c01700 10 journal op_apply_start 2355 open_ops 0 -> 1
2016-10-25 08:51:31.683462 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2139544c00 seq 2355 osr(1.6 0x7f2139474f10)/0x7f2139474f10 start
2016-10-25 08:51:31.683471 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147f49800
2016-10-25 08:51:31.683477 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'275 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] op_applied: 1717
2016-10-25 08:51:31.683556 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'275 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] op_applied version 23'276
2016-10-25 08:51:31.683577 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.6_head/#1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head# 0~4194304
2016-10-25 08:51:31.684351 7f2121400700 10 journal op_apply_start 2356 open_ops 1 -> 2
2016-10-25 08:51:31.684373 7f2121400700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f213c9a00a0 seq 2356 osr(1.1 0x7f21394758a0)/0x7f21394758a0 start
2016-10-25 08:51:31.684384 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2140fff100
2016-10-25 08:51:31.684399 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.1_head/#1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head# 0~4194304
2016-10-25 08:51:31.692493 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.1_head/#1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head# 0~4194304 = 4194304
2016-10-25 08:51:31.693286 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.6_head/#1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head# 0~4194304 = 4194304
2016-10-25 08:51:31.693374 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.6_head/#1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head#
2016-10-25 08:51:31.693495 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.6_head/#1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head# = 0
2016-10-25 08:51:31.693392 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.1_head/#1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head#
2016-10-25 08:51:31.693532 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.1_head/#1:81eb7e89:::benchmark_data_target167114251239.teuthology_13304_object874:head# = 0
2016-10-25 08:51:31.693562 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.1_head/#1:80000000::::head#
2016-10-25 08:51:31.693585 7f2121400700 10 filestore oid: #1:80000000::::head# not skipping op, *spos 2356.0.2
2016-10-25 08:51:31.693595 7f2121400700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.693646 7f2121400700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.1_head/#1:80000000::::head# = 0
2016-10-25 08:51:31.693667 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.6_head/#1:60000000::::head#
2016-10-25 08:51:31.693688 7f2121c01700 10 filestore oid: #1:60000000::::head# not skipping op, *spos 2355.0.2
2016-10-25 08:51:31.693697 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:31.693724 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.6_head/#1:60000000::::head# = 0
2016-10-25 08:51:31.693730 7f2121400700 10 journal op_apply_finish 2356 open_ops 2 -> 1, max_applied_seq 2354 -> 2356
2016-10-25 08:51:31.693747 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f213c9a00a0 seq 2356 r = 0, finisher 0x7f21401ee4c0 0x7f2147f99140
2016-10-25 08:51:31.693782 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f213c9a00a0 seq 2356 osr(1.1 0x7f21394758a0)/0x7f21394758a0 lat 0.277173
2016-10-25 08:51:31.693758 7f2121c01700 10 journal op_apply_finish 2355 open_ops 1 -> 0, max_applied_seq 2356 -> 2356
2016-10-25 08:51:31.693810 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2139544c00 seq 2355 r = 0, finisher 0x7f21401efd60 0x7f213ece9540
2016-10-25 08:51:31.693838 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'324 crt=23'321 lcod 23'321 mlcod 23'321 active+clean] op_applied: 1720
2016-10-25 08:51:31.693867 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'324 crt=23'321 lcod 23'321 mlcod 23'321 active+clean] op_applied version 23'325
2016-10-25 08:51:31.693893 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f2139544c00 seq 2355 osr(1.6 0x7f2139474f10)/0x7f2139474f10 lat 0.288786
2016-10-25 08:51:31.693909 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'276 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] op_applied: 1719
2016-10-25 08:51:31.693932 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'276 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] op_applied version 23'277
2016-10-25 08:51:31.694053 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2287 ==== osd_repop_reply(client.4144.0:860 1.0) v1 ==== 83+0+0 (205253255 0 0) 0x7f21412f8580 con 0x7f2139592800
2016-10-25 08:51:31.694072 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:860 1.0) v1 epoch 23
2016-10-25 08:51:31.694077 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:31.694083 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139880240 prio 196 cost 0 latency 0.000068 osd_repop_reply(client.4144.0:860 1.0) v1
2016-10-25 08:51:31.694114 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2288 ==== osd_repop_reply(client.4144.0:862 1.4) v1 ==== 83+0+0 (1649711392 0 0) 0x7f21412f9480 con 0x7f2139592800
2016-10-25 08:51:31.694120 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:862 1.4) v1 epoch 23
2016-10-25 08:51:31.694122 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:31.694125 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f21398807e0 prio 196 cost 0 latency 0.000020 osd_repop_reply(client.4144.0:862 1.4) v1
2016-10-25 08:51:31.694298 7f2110c6b700 10 osd.3 23 dequeue_op 0x7f21398807e0 prio 196 cost 0 latency 0.000192 osd_repop_reply(client.4144.0:862 1.4) v1 pg pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 crt=23'278 lcod 23'281 mlcod 23'281 active+clean]
2016-10-25 08:51:31.694315 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] handle_message: 0x7f21398807e0
2016-10-25 08:51:31.694327 7f2110c6b700  7 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] sub_op_modify_reply: tid 1709 op  ack_type 4 from 2
2016-10-25 08:51:31.694342 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] repop_all_applied: repop tid 1709 all applied 
2016-10-25 08:51:31.694347 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] eval_repop repgather(0x7f213985d320 23'283 rep_tid=1709 committed?=0 applied?=1) wants=d
2016-10-25 08:51:31.694354 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 crt=23'278 lcod 23'281 mlcod 23'281 active+clean]  applied: repgather(0x7f213985d320 23'283 rep_tid=1709 committed?=0 applied?=1) 
2016-10-25 08:51:31.694361 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'282 crt=23'278 lcod 23'281 mlcod 23'281 active+clean] repop_all_committed: repop tid 1709 all committed 
2016-10-25 08:51:31.694366 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'281 active+clean] eval_repop repgather(0x7f213985d320 23'283 rep_tid=1709 committed?=1 applied?=1) wants=d
2016-10-25 08:51:31.694371 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'281 active+clean]  commit: repgather(0x7f213985d320 23'283 rep_tid=1709 committed?=1 applied?=1)
2016-10-25 08:51:31.694378 7f2110c6b700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'281 active+clean] log_op_stats osd_op(client.4144.0:862 1.afa14904 benchmark_data_target167114251239.teuthology_13304_object861 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 0.392740 lat 0.392756
2016-10-25 08:51:31.694385 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'281 active+clean]  sending commit on osd_op(client.4144.0:862 1.afa14904 benchmark_data_target167114251239.teuthology_13304_object861 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f21397a0580
2016-10-25 08:51:31.694392 7f2110c6b700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(862 benchmark_data_target167114251239.teuthology_13304_object861 [write 0~4194304] v23'283 uv283 ondisk = 0) v7 -- 0x7f21397a0580 con 0
2016-10-25 08:51:31.694411 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'281 active+clean]  applied: repgather(0x7f213985d320 23'283 rep_tid=1709 committed?=1 applied?=1) 
2016-10-25 08:51:31.694419 7f2110c6b700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'281 active+clean] publish_stats_to_osd 23: no change since 2016-10-25 08:51:31.525231
2016-10-25 08:51:31.694426 7f2110c6b700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:31.694434 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean]  removing repgather(0x7f213985d320 23'283 rep_tid=1709 committed?=1 applied?=1)
2016-10-25 08:51:31.694439 7f2110c6b700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean]    q front is repgather(0x7f213985d320 23'283 rep_tid=1709 committed?=1 applied?=1)
2016-10-25 08:51:31.694445 7f2110c6b700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] remove_repop repgather(0x7f213985d320 23'283 rep_tid=1709 committed?=1 applied?=1)
2016-10-25 08:51:31.694453 7f2110c6b700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean]  requeue_ops 
2016-10-25 08:51:31.694486 7f2110c6b700 10 osd.3 23 dequeue_op 0x7f21398807e0 finish
2016-10-25 08:51:31.695068 7f2112c6f700 10 osd.3 23 dequeue_op 0x7f2139880240 prio 196 cost 0 latency 0.001053 osd_repop_reply(client.4144.0:860 1.0) v1 pg pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 crt=23'279 lcod 23'282 mlcod 23'282 active+clean]
2016-10-25 08:51:31.695101 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] handle_message: 0x7f2139880240
2016-10-25 08:51:31.695114 7f2112c6f700  7 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] sub_op_modify_reply: tid 1707 op  ack_type 4 from 2
2016-10-25 08:51:31.695130 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] repop_all_applied: repop tid 1707 all applied 
2016-10-25 08:51:31.695136 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] eval_repop repgather(0x7f2139755b00 23'284 rep_tid=1707 committed?=0 applied?=1) wants=d
2016-10-25 08:51:31.695151 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 crt=23'279 lcod 23'282 mlcod 23'282 active+clean]  applied: repgather(0x7f2139755b00 23'284 rep_tid=1707 committed?=0 applied?=1) 
2016-10-25 08:51:31.695158 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 crt=23'279 lcod 23'282 mlcod 23'282 active+clean] repop_all_committed: repop tid 1707 all committed 
2016-10-25 08:51:31.695163 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'279 lcod 23'283 mlcod 23'282 active+clean] eval_repop repgather(0x7f2139755b00 23'284 rep_tid=1707 committed?=1 applied?=1) wants=d
2016-10-25 08:51:31.695168 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'279 lcod 23'283 mlcod 23'282 active+clean]  commit: repgather(0x7f2139755b00 23'284 rep_tid=1707 committed?=1 applied?=1)
2016-10-25 08:51:31.695174 7f2112c6f700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'279 lcod 23'283 mlcod 23'282 active+clean] log_op_stats osd_op(client.4144.0:860 1.24824798 benchmark_data_target167114251239.teuthology_13304_object859 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 0.412651 lat 0.412666
2016-10-25 08:51:31.695182 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'279 lcod 23'283 mlcod 23'282 active+clean]  sending commit on osd_op(client.4144.0:860 1.24824798 benchmark_data_target167114251239.teuthology_13304_object859 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f2142e642c0
2016-10-25 08:51:31.695189 7f2112c6f700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(860 benchmark_data_target167114251239.teuthology_13304_object859 [write 0~4194304] v23'284 uv284 ondisk = 0) v7 -- 0x7f2142e642c0 con 0
2016-10-25 08:51:31.698474 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2289 ==== osd_repop(client.4144.0:876 1.2) v1 ==== 1034+0+4195023 (3333092032 0 4274805080) 0x7f2141054a00 con 0x7f2139592800
2016-10-25 08:51:31.698515 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop(client.4144.0:876 1.2) v1 epoch 23
2016-10-25 08:51:31.698523 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:31.698532 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139883960 prio 127 cost 4195023 latency 0.004397 osd_repop(client.4144.0:876 1.2) v1
2016-10-25 08:51:31.698580 7f2111c6d700 10 osd.3 23 dequeue_op 0x7f2139883960 prio 127 cost 4195023 latency 0.004445 osd_repop(client.4144.0:876 1.2) v1 pg pg[1.2( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 crt=23'274 lcod 23'275 active]
2016-10-25 08:51:31.698609 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.2( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 crt=23'274 lcod 23'275 active] handle_message: 0x7f2139883960
2016-10-25 08:51:31.698625 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.2( v 23'276 (0'0,23'276] local-les=22 n=276 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 crt=23'274 lcod 23'275 active] sub_op_modify trans 1:5ed10711:::benchmark_data_target167114251239.teuthology_13304_object875:head v 23'277 (transaction) 200
2016-10-25 08:51:31.698652 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.2( v 23'276 (0'0,23'276] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 crt=23'274 lcod 23'275 active] append_log log((0'0,23'276], crt=23'274) [23'277 (0'0) modify   1:5ed10711:::benchmark_data_target167114251239.teuthology_13304_object875:head by client.4144.0:876 2016-10-25 08:51:31.640175 0]
2016-10-25 08:51:31.698663 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.2( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'276 crt=23'274 lcod 23'275 active] add_log_entry 23'277 (0'0) modify   1:5ed10711:::benchmark_data_target167114251239.teuthology_13304_object875:head by client.4144.0:876 2016-10-25 08:51:31.640175 0
2016-10-25 08:51:31.698672 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.2( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'276 crt=23'275 lcod 23'275 active] append_log: trimming to 23'275 entries 23'275 (0'0) modify   1:4b422862:::benchmark_data_target167114251239.teuthology_13304_object841:head by client.4144.0:842 2016-10-25 08:51:29.783349 0
2016-10-25 08:51:31.698687 7f2111c6d700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'277, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:31.698880 7f2111c6d700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f21394ba580 osr(1.2 0x7f2139474dc0)
2016-10-25 08:51:31.698892 7f2111c6d700 10 journal prepare_entry [Transaction(0x7f2147f6df80),Transaction(0x7f2147f6e140)]
2016-10-25 08:51:31.699529 7f2111c6d700 10 journal  len 4195682 -> 4202496 (head 40 pre_pad 3387 bl 4195682 post_pad 3347 tail 40) (bl alignment 3427)
2016-10-25 08:51:31.700373 7f2111c6d700 10 journal op_submit_start 2361
2016-10-25 08:51:31.700387 7f2111c6d700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2361 [Transaction(0x7f2147f6df80),Transaction(0x7f2147f6e140)]
2016-10-25 08:51:31.700392 7f2111c6d700 10 journal op_journal_transactions 2361 reqid_t client.4144.0:876
2016-10-25 08:51:31.700396 7f2111c6d700  5 journal submit_entry seq 2361 len 4202496 (0x7f214811ea00)
2016-10-25 08:51:31.700417 7f2111c6d700 10 journal op_submit_finish 2361
2016-10-25 08:51:31.700421 7f2111c6d700 10 osd.3 23 dequeue_op 0x7f2139883960 finish
2016-10-25 08:51:31.700439 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:31.700443 7f2123404700 20 journal write_thread_entry aio throttle: aio num 3 bytes 16809984 ... exp 6 min_new 64 ... pending 4202496
2016-10-25 08:51:31.700446 7f2123404700 10 journal room 471740415 max_size 1072709632 pos 992059392 header.start 391094272 top 4096
2016-10-25 08:51:31.700448 7f2123404700 10 journal check_for_full at 992059392 : 4202496 < 471740415
2016-10-25 08:51:31.700450 7f2123404700 15 journal prepare_single_write 1 will write 992059392 : seq 2361 len 4195682 -> 4202496
2016-10-25 08:51:31.700461 7f2123404700 20 journal prepare_multi_write queue_pos now 996261888
2016-10-25 08:51:31.700463 7f2123404700 15 journal do_aio_write writing 992059392~4202496
2016-10-25 08:51:31.700465 7f2123404700 20 journal write_aio_bl 992059392~4202496 seq 2361
2016-10-25 08:51:31.700468 7f2123404700 20 journal write_aio_bl .. 992059392~4202496 in 1
2016-10-25 08:51:31.700697 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 676 ==== osd_op(client.4144.0:879 1.6ccee076 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (4137899388 0 436065261) 0x7f21397a0580 con 0x7f2147f52800
2016-10-25 08:51:31.700733 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:31.700748 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:31.700787 7f2123404700  5 journal complete_write finished 1 ops and 4195682 bytes
2016-10-25 08:51:31.700790 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:31.700780 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'279 lcod 23'283 mlcod 23'282 active+clean]  applied: repgather(0x7f2139755b00 23'284 rep_tid=1707 committed?=1 applied?=1) 
2016-10-25 08:51:31.700799 7f2112c6f700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'279 lcod 23'283 mlcod 23'282 active+clean] publish_stats_to_osd 23: no change since 2016-10-25 08:51:31.556482
2016-10-25 08:51:31.700807 7f2112c6f700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'279 lcod 23'283 mlcod 23'283 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:31.700814 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'279 lcod 23'283 mlcod 23'283 active+clean]  removing repgather(0x7f2139755b00 23'284 rep_tid=1707 committed?=1 applied?=1)
2016-10-25 08:51:31.700819 7f2112c6f700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'279 lcod 23'283 mlcod 23'283 active+clean]    q front is repgather(0x7f2139755b00 23'284 rep_tid=1707 committed?=1 applied?=1)
2016-10-25 08:51:31.700824 7f2112c6f700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'279 lcod 23'283 mlcod 23'283 active+clean] remove_repop repgather(0x7f2139755b00 23'284 rep_tid=1707 committed?=1 applied?=1)
2016-10-25 08:51:31.700834 7f2112c6f700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'279 lcod 23'283 mlcod 23'283 active+clean]  requeue_ops 
2016-10-25 08:51:31.700861 7f2112c6f700 10 osd.3 23 dequeue_op 0x7f2139880240 finish
2016-10-25 08:51:31.700748 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139882880 prio 63 cost 4194304 latency 0.001965 osd_op(client.4144.0:879 1.6ccee076 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.703562 7f210fc69700 10 osd.3 23 dequeue_op 0x7f2139882880 prio 63 cost 4194304 latency 0.004777 osd_op(client.4144.0:879 1.6ccee076 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean]
2016-10-25 08:51:31.703642 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] handle_message: 0x7f2139882880
2016-10-25 08:51:31.703661 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:31.703681 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] do_op osd_op(client.4144.0:879 1.6ccee076 benchmark_data_target167114251239.teuthology_13304_object878 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:31.703703 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] get_object_context: obc NOT found in cache: 1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head
2016-10-25 08:51:31.703713 7f210fc69700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head# '_'
2016-10-25 08:51:31.703768 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.6_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject878__head_6CCEE076__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.703781 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head# '_' = -2
2016-10-25 08:51:31.703784 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] get_object_context: no obc for soid 1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head but can_create
2016-10-25 08:51:31.703796 7f210fc69700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:snapdir# 'snapset'
2016-10-25 08:51:31.703811 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.6_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject878__snapdir_6CCEE076__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.703816 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:snapdir# 'snapset' = -2
2016-10-25 08:51:31.703834 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] create_object_context 0x7f21431e83c0 1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head 
2016-10-25 08:51:31.703843 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] populate_obc_watchers 1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head
2016-10-25 08:51:31.703850 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head
2016-10-25 08:51:31.703856 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] get_object_context: 0x7f21431e83c0 1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head rwstate(none n=0 w=0) oi: 1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f21396dff00 snapset: 0=[]:[]
2016-10-25 08:51:31.703867 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] find_object_context 1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head @head oi=1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.703877 7f210fc69700 25 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] do_op oi 1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.703886 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] get_object_context: obc NOT found in cache: 1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:snapdir
2016-10-25 08:51:31.703892 7f210fc69700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:snapdir# '_'
2016-10-25 08:51:31.703904 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.6_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject878__snapdir_6CCEE076__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.703910 7f210fc69700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:snapdir# '_' = -2
2016-10-25 08:51:31.703912 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] get_object_context: no obc for soid 1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:snapdir and !can_create
2016-10-25 08:51:31.703929 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] execute_ctx 0x7f2142f0a300
2016-10-25 08:51:31.703937 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] do_op 1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head [write 0~4194304] ov 0'0 av 23'278 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:31.703948 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] do_osd_op 1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head [write 0~4194304]
2016-10-25 08:51:31.703954 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:31.704215 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] make_writeable 1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head snapset=0x7f21396dff38  snapc=0=[]
2016-10-25 08:51:31.704224 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean]  setting DIRTY flag
2016-10-25 08:51:31.704229 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] make_writeable 1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head done, snapset=0=[]:[]+head
2016-10-25 08:51:31.704235 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] finish_ctx 1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head 0x7f2142f0a300 op modify  
2016-10-25 08:51:31.704244 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean]  set mtime to 2016-10-25 08:51:31.694845
2016-10-25 08:51:31.704255 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean]  final snapset 0=[]:[]+head in 1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head
2016-10-25 08:51:31.704272 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean]  zeroing write result code 0
2016-10-25 08:51:31.704283 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean]  op order client.4144 tid 879 (first)
2016-10-25 08:51:31.704290 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] new_repop rep_tid 1723 on osd_op(client.4144.0:879 1.6ccee076 benchmark_data_target167114251239.teuthology_13304_object878 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.704301 7f210fc69700  7 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] issue_repop rep_tid 1723 o 1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head
2016-10-25 08:51:31.704338 7f210fc69700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:31.704343 7f210fc69700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:879 1.6 1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head v 23'278) v1 -- 0x7f2141030600 con 0
2016-10-25 08:51:31.704614 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'277 (0'0,23'277] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] append_log log((0'0,23'277], crt=23'272) [23'278 (0'0) modify   1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head by client.4144.0:879 2016-10-25 08:51:31.694845 0]
2016-10-25 08:51:31.704628 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'277 crt=23'272 lcod 23'273 mlcod 23'273 active+clean] add_log_entry 23'278 (0'0) modify   1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head by client.4144.0:879 2016-10-25 08:51:31.694845 0
2016-10-25 08:51:31.704639 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'277 crt=23'273 lcod 23'273 mlcod 23'273 active+clean] append_log: trimming to 23'273 entries 23'273 (0'0) modify   1:79f6170d:::benchmark_data_target167114251239.teuthology_13304_object829:head by client.4144.0:830 2016-10-25 08:51:29.649888 0
2016-10-25 08:51:31.704652 7f210fc69700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'278, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:31.704833 7f210fc69700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139243700 osr(1.6 0x7f2139474f10)
2016-10-25 08:51:31.704841 7f210fc69700 10 journal prepare_entry [Transaction(0x7f2148080a80)]
2016-10-25 08:51:31.705087 7f210fc69700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:31.705319 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 677 ==== osd_op(client.4144.0:880 1.7d6f66b4 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (3964576173 0 3527757998) 0x7f2142e642c0 con 0x7f2147f52800
2016-10-25 08:51:31.705379 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:31.705395 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139882760 prio 63 cost 4194304 latency 0.002047 osd_op(client.4144.0:880 1.7d6f66b4 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.705437 7f210e466700 10 osd.3 23 dequeue_op 0x7f2139882760 prio 63 cost 4194304 latency 0.002089 osd_op(client.4144.0:880 1.7d6f66b4 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean]
2016-10-25 08:51:31.705457 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] handle_message: 0x7f2139882760
2016-10-25 08:51:31.705472 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:31.705496 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] do_op osd_op(client.4144.0:880 1.7d6f66b4 benchmark_data_target167114251239.teuthology_13304_object879 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:31.705516 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] get_object_context: obc NOT found in cache: 1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head
2016-10-25 08:51:31.705525 7f210e466700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.4_head/#1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head# '_'
2016-10-25 08:51:31.705576 7f210e466700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.4_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject879__head_7D6F66B4__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.705583 7f210e466700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.4_head/#1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head# '_' = -2
2016-10-25 08:51:31.705595 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] get_object_context: no obc for soid 1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head but can_create
2016-10-25 08:51:31.705605 7f210e466700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.4_head/#1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:snapdir# 'snapset'
2016-10-25 08:51:31.705620 7f210e466700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.4_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject879__snapdir_7D6F66B4__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.705623 7f210e466700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.4_head/#1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:snapdir# 'snapset' = -2
2016-10-25 08:51:31.705641 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] create_object_context 0x7f2143046c00 1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head 
2016-10-25 08:51:31.705649 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] populate_obc_watchers 1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head
2016-10-25 08:51:31.705655 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head
2016-10-25 08:51:31.705660 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] get_object_context: 0x7f2143046c00 1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head rwstate(none n=0 w=0) oi: 1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f213987b9c0 snapset: 0=[]:[]
2016-10-25 08:51:31.705669 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] find_object_context 1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head @head oi=1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.705678 7f210e466700 25 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] do_op oi 1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:31.705686 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] get_object_context: obc NOT found in cache: 1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:snapdir
2016-10-25 08:51:31.705692 7f210e466700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.4_head/#1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:snapdir# '_'
2016-10-25 08:51:31.705705 7f210e466700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.4_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject879__snapdir_7D6F66B4__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:31.705708 7f210e466700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.4_head/#1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:snapdir# '_' = -2
2016-10-25 08:51:31.705710 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] get_object_context: no obc for soid 1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:snapdir and !can_create
2016-10-25 08:51:31.705724 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] execute_ctx 0x7f2139423800
2016-10-25 08:51:31.705731 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] do_op 1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head [write 0~4194304] ov 0'0 av 23'284 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:31.705740 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] do_osd_op 1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head [write 0~4194304]
2016-10-25 08:51:31.705749 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:31.705970 7f210fc69700 10 journal op_submit_start 2362
2016-10-25 08:51:31.705974 7f210fc69700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2362 [Transaction(0x7f2148080a80)]
2016-10-25 08:51:31.705976 7f210fc69700 10 journal op_journal_transactions 2362 reqid_t client.4144.0:879
2016-10-25 08:51:31.705979 7f210fc69700  5 journal submit_entry seq 2362 len 4202496 (0x7f21414ada50)
2016-10-25 08:51:31.706005 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:31.706007 7f2123404700 20 journal write_thread_entry aio throttle: aio num 4 bytes 21012480 ... exp 8 min_new 256 ... pending 4202496
2016-10-25 08:51:31.706010 7f2123404700 10 journal room 467537919 max_size 1072709632 pos 996261888 header.start 391094272 top 4096
2016-10-25 08:51:31.706012 7f2123404700 10 journal check_for_full at 996261888 : 4202496 < 467537919
2016-10-25 08:51:31.706013 7f2123404700 15 journal prepare_single_write 1 will write 996261888 : seq 2362 len 4195598 -> 4202496
2016-10-25 08:51:31.706010 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] make_writeable 1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head snapset=0x7f213987b9f8  snapc=0=[]
2016-10-25 08:51:31.706022 7f2123404700 20 journal prepare_multi_write queue_pos now 1000464384
2016-10-25 08:51:31.706019 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean]  setting DIRTY flag
2016-10-25 08:51:31.706024 7f2123404700 15 journal do_aio_write writing 996261888~4202496
2016-10-25 08:51:31.706026 7f2123404700 20 journal write_aio_bl 996261888~4202496 seq 2362
2016-10-25 08:51:31.706028 7f2123404700 20 journal write_aio_bl .. 996261888~4202496 in 1
2016-10-25 08:51:31.706024 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] make_writeable 1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head done, snapset=0=[]:[]+head
2016-10-25 08:51:31.706031 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] finish_ctx 1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head 0x7f2139423800 op modify  
2016-10-25 08:51:31.706038 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean]  set mtime to 2016-10-25 08:51:31.701193
2016-10-25 08:51:31.706049 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=283 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean]  final snapset 0=[]:[]+head in 1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head
2016-10-25 08:51:31.706061 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean]  zeroing write result code 0
2016-10-25 08:51:31.706069 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean]  op order client.4144 tid 880 (first)
2016-10-25 08:51:31.706085 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] new_repop rep_tid 1724 on osd_op(client.4144.0:880 1.7d6f66b4 benchmark_data_target167114251239.teuthology_13304_object879 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:31.706092 7f210e466700  7 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] issue_repop rep_tid 1724 o 1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head
2016-10-25 08:51:31.706119 7f210e466700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:31.706121 7f210e466700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:880 1.4 1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head v 23'284) v1 -- 0x7f2140f90c00 con 0
2016-10-25 08:51:31.706312 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:31.706316 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:31.706318 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:31.706324 7f210fc69700 10 journal op_submit_finish 2362
2016-10-25 08:51:31.706329 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'277 crt=23'273 lcod 23'273 mlcod 23'273 active+clean] eval_repop repgather(0x7f213985cb40 23'278 rep_tid=1723 committed?=0 applied?=0) wants=d
2016-10-25 08:51:31.706348 7f210fc69700 10 osd.3 23 dequeue_op 0x7f2139882880 finish
2016-10-25 08:51:31.706382 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'283 (0'0,23'283] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] append_log log((0'0,23'283], crt=23'278) [23'284 (0'0) modify   1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head by client.4144.0:880 2016-10-25 08:51:31.701193 0]
2016-10-25 08:51:31.706395 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'283 crt=23'278 lcod 23'282 mlcod 23'282 active+clean] add_log_entry 23'284 (0'0) modify   1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head by client.4144.0:880 2016-10-25 08:51:31.701193 0
2016-10-25 08:51:31.706407 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'283 crt=23'282 lcod 23'282 mlcod 23'282 active+clean] append_log: trimming to 23'282 entries 23'279 (0'0) modify   1:308d666e:::benchmark_data_target167114251239.teuthology_13304_object843:head by client.4144.0:844 2016-10-25 08:51:30.150380 0,23'280 (0'0) modify   1:24bae17a:::benchmark_data_target167114251239.teuthology_13304_object849:head by client.4144.0:850 2016-10-25 08:51:30.214576 0,23'281 (0'0) modify   1:3df0d67a:::benchmark_data_target167114251239.teuthology_13304_object853:head by client.4144.0:854 2016-10-25 08:51:30.214971 0,23'282 (0'0) modify   1:3d4d78f7:::benchmark_data_target167114251239.teuthology_13304_object857:head by client.4144.0:858 2016-10-25 08:51:30.215332 0
2016-10-25 08:51:31.706426 7f210e466700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'284, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:31.706608 7f210e466700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139242940 osr(1.4 0x7f2139475240)
2016-10-25 08:51:31.706614 7f210e466700 10 journal prepare_entry [Transaction(0x7f2147f2b880)]
2016-10-25 08:51:31.706861 7f210e466700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:31.708623 7f210e466700 10 journal op_submit_start 2363
2016-10-25 08:51:31.708637 7f210e466700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2363 [Transaction(0x7f2147f2b880)]
2016-10-25 08:51:31.708642 7f210e466700 10 journal op_journal_transactions 2363 reqid_t client.4144.0:880
2016-10-25 08:51:31.708645 7f210e466700  5 journal submit_entry seq 2363 len 4202496 (0x7f2140ed55f0)
2016-10-25 08:51:31.708676 7f210e466700 10 journal op_submit_finish 2363
2016-10-25 08:51:31.708680 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'283 crt=23'282 lcod 23'282 mlcod 23'282 active+clean] eval_repop repgather(0x7f2139770000 23'284 rep_tid=1724 committed?=0 applied?=0) wants=d
2016-10-25 08:51:31.708700 7f210e466700 10 osd.3 23 dequeue_op 0x7f2139882760 finish
2016-10-25 08:51:31.711551 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:31.711568 7f2123404700 20 journal write_thread_entry aio throttle: aio num 5 bytes 25214976 ... exp 10 min_new 1024 ... pending 4202496
2016-10-25 08:51:31.711573 7f2123404700 10 journal room 463335423 max_size 1072709632 pos 1000464384 header.start 391094272 top 4096
2016-10-25 08:51:31.711576 7f2123404700 10 journal check_for_full at 1000464384 : 4202496 < 463335423
2016-10-25 08:51:31.711578 7f2123404700 15 journal prepare_single_write 1 will write 1000464384 : seq 2363 len 4195598 -> 4202496
2016-10-25 08:51:31.711605 7f2123404700 20 journal prepare_multi_write queue_pos now 1004666880
2016-10-25 08:51:31.711607 7f2123404700 15 journal do_aio_write writing 1000464384~4202496
2016-10-25 08:51:31.711609 7f2123404700 20 journal write_aio_bl 1000464384~4202496 seq 2363
2016-10-25 08:51:31.711612 7f2123404700 20 journal write_aio_bl .. 1000464384~4202496 in 1
2016-10-25 08:51:31.711900 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:31.711904 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:31.711906 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:32.022054 7f2125c09700 10 osd.3 23 tick
2016-10-25 08:51:32.022090 7f2125c09700 10 osd.3 23 do_waiters -- start
2016-10-25 08:51:32.022093 7f2125c09700 10 osd.3 23 do_waiters -- finish
2016-10-25 08:51:32.036613 7f2122c03700 10 journal write_finish_thread_entry aio 975249408~8404992 done
2016-10-25 08:51:32.036640 7f2122c03700 20 journal check_aio_completion
2016-10-25 08:51:32.036642 7f2122c03700 20 journal check_aio_completion completed seq 2358 975249408~8404992
2016-10-25 08:51:32.036655 7f2122c03700 20 journal check_aio_completion queueing finishers through seq 2358
2016-10-25 08:51:32.036659 7f2122c03700 10 journal queue_completions_thru seq 2358 queueing seq 2357 0x7f2140fd9e30 lat 0.511252
2016-10-25 08:51:32.036695 7f2122c03700 10 journal queue_completions_thru seq 2358 queueing seq 2358 0x7f21414ab320 lat 0.494799
2016-10-25 08:51:32.036709 7f2122c03700 20 journal write_finish_thread_entry waiting for aio(s)
2016-10-25 08:51:32.036723 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f21397f65e0 seq 2357 osr(1.5 0x7f2139425410) [Transaction(0x7f213e62c700),Transaction(0x7f213e62c8c0)]
2016-10-25 08:51:32.036731 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f21397f65e0 seq 2357 osr(1.5 0x7f2139425410) 4195654 bytes   (queue has 7 ops and 29369298 bytes)
2016-10-25 08:51:32.036739 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f213ece0460
2016-10-25 08:51:32.036747 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f21395461c0 seq 2358 osr(1.5 0x7f2139425410) [Transaction(0x7f2141371800),Transaction(0x7f21413719c0)]
2016-10-25 08:51:32.036750 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f21395461c0 seq 2358 osr(1.5 0x7f2139425410) 4195654 bytes   (queue has 7 ops and 29369298 bytes)
2016-10-25 08:51:32.036755 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f213a53b780
2016-10-25 08:51:32.036799 7f211fbfd700 10 osd.3 pg_epoch: 23 pg[1.5( v 23'279 (0'0,23'279] local-les=22 n=279 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'277 crt=23'276 lcod 23'276 active] sub_op_modify_commit on op osd_repop(client.4144.0:869 1.5 1:ace40720:::benchmark_data_target167114251239.teuthology_13304_object868:head v 23'278) v1, sending commit to osd.2
2016-10-25 08:51:32.036839 7f211fbfd700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:32.036844 7f211fbfd700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop_reply(client.4144.0:869 1.5 ondisk, result = 0) v1 -- 0x7f213ea46c80 con 0
2016-10-25 08:51:32.036909 7f211fbfd700 10 osd.3 pg_epoch: 23 pg[1.5( v 23'279 (0'0,23'279] local-les=22 n=279 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'277 crt=23'276 lcod 23'277 active] sub_op_modify_commit on op osd_repop(client.4144.0:873 1.5 1:a63a41ae:::benchmark_data_target167114251239.teuthology_13304_object872:head v 23'279) v1, sending commit to osd.2
2016-10-25 08:51:32.036926 7f211fbfd700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:32.036928 7f211fbfd700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop_reply(client.4144.0:873 1.5 ondisk, result = 0) v1 -- 0x7f213ea47e00 con 0
2016-10-25 08:51:32.037147 7f2121400700 10 journal op_apply_start 2357 open_ops 0 -> 1
2016-10-25 08:51:32.037172 7f2121400700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f21397f65e0 seq 2357 osr(1.5 0x7f2139425410)/0x7f2139425410 start
2016-10-25 08:51:32.037190 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f213e62c700
2016-10-25 08:51:32.037225 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.5_head/#1:a0000000::::head#
2016-10-25 08:51:32.037258 7f2121400700 10 filestore oid: #1:a0000000::::head# not skipping op, *spos 2357.0.0
2016-10-25 08:51:32.037276 7f2121400700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:32.037357 7f2121400700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.5_head/#1:a0000000::::head# = 0
2016-10-25 08:51:32.037392 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f213e62c8c0
2016-10-25 08:51:32.037427 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.5_head/#1:ace40720:::benchmark_data_target167114251239.teuthology_13304_object868:head# 0~4194304
2016-10-25 08:51:32.048931 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 678 ==== osd_op(client.4144.0:881 1.c392bcdf (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (379492213 0 2110223081) 0x7f214131e580 con 0x7f2147f52800
2016-10-25 08:51:32.048997 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:32.049007 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139883a80 prio 63 cost 4194304 latency 0.003436 osd_op(client.4144.0:881 1.c392bcdf (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:32.049064 7f210f468700 10 osd.3 23 dequeue_op 0x7f2139883a80 prio 63 cost 4194304 latency 0.003491 osd_op(client.4144.0:881 1.c392bcdf (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean]
2016-10-25 08:51:32.049096 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] handle_message: 0x7f2139883a80
2016-10-25 08:51:32.049127 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:32.049272 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] do_op osd_op(client.4144.0:881 1.c392bcdf benchmark_data_target167114251239.teuthology_13304_object880 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:32.049310 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] get_object_context: obc NOT found in cache: 1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head
2016-10-25 08:51:32.049330 7f210f468700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head# '_'
2016-10-25 08:51:32.049421 7f210f468700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.7_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject880__head_C392BCDF__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:32.049557 7f210f468700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head# '_' = -2
2016-10-25 08:51:32.049563 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] get_object_context: no obc for soid 1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head but can_create
2016-10-25 08:51:32.049579 7f210f468700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:snapdir# 'snapset'
2016-10-25 08:51:32.049651 7f210f468700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.7_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject880__snapdir_C392BCDF__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:32.049660 7f210f468700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:snapdir# 'snapset' = -2
2016-10-25 08:51:32.049717 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] create_object_context 0x7f2142e55340 1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head 
2016-10-25 08:51:32.049733 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] populate_obc_watchers 1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head
2016-10-25 08:51:32.049772 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head
2016-10-25 08:51:32.049784 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] get_object_context: 0x7f2142e55340 1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head rwstate(none n=0 w=0) oi: 1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f2139795f80 snapset: 0=[]:[]
2016-10-25 08:51:32.049819 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] find_object_context 1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head @head oi=1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:32.049842 7f210f468700 25 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] do_op oi 1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:32.049858 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] get_object_context: obc NOT found in cache: 1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:snapdir
2016-10-25 08:51:32.049894 7f210f468700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:snapdir# '_'
2016-10-25 08:51:32.049921 7f210f468700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.7_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject880__snapdir_C392BCDF__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:32.049927 7f210f468700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:snapdir# '_' = -2
2016-10-25 08:51:32.049931 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] get_object_context: no obc for soid 1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:snapdir and !can_create
2016-10-25 08:51:32.049977 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] execute_ctx 0x7f214308ee00
2016-10-25 08:51:32.049991 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] do_op 1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head [write 0~4194304] ov 0'0 av 23'268 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:32.050025 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] do_osd_op 1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head [write 0~4194304]
2016-10-25 08:51:32.050038 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:32.050344 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] make_writeable 1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head snapset=0x7f2139795fb8  snapc=0=[]
2016-10-25 08:51:32.050361 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean]  setting DIRTY flag
2016-10-25 08:51:32.050370 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] make_writeable 1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head done, snapset=0=[]:[]+head
2016-10-25 08:51:32.050381 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] finish_ctx 1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head 0x7f214308ee00 op modify  
2016-10-25 08:51:32.050396 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean]  set mtime to 2016-10-25 08:51:32.044402
2016-10-25 08:51:32.050416 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=267 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean]  final snapset 0=[]:[]+head in 1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head
2016-10-25 08:51:32.050470 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean]  zeroing write result code 0
2016-10-25 08:51:32.050485 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean]  op order client.4144 tid 881 (first)
2016-10-25 08:51:32.050498 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] new_repop rep_tid 1725 on osd_op(client.4144.0:881 1.c392bcdf benchmark_data_target167114251239.teuthology_13304_object880 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:32.050514 7f210f468700  7 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] issue_repop rep_tid 1725 o 1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head
2016-10-25 08:51:32.050562 7f210f468700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:32.050566 7f210f468700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:881 1.7 1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head v 23'268) v1 -- 0x7f2141059000 con 0
2016-10-25 08:51:32.050843 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'267 (0'0,23'267] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] append_log log((0'0,23'267], crt=23'263) [23'268 (0'0) modify   1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head by client.4144.0:881 2016-10-25 08:51:32.044402 0]
2016-10-25 08:51:32.050864 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'267 crt=23'263 lcod 23'264 mlcod 23'264 active+clean] add_log_entry 23'268 (0'0) modify   1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head by client.4144.0:881 2016-10-25 08:51:32.044402 0
2016-10-25 08:51:32.050880 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'267 crt=23'264 lcod 23'264 mlcod 23'264 active+clean] append_log: trimming to 23'264 entries 23'264 (0'0) modify   1:f7d16cda:::benchmark_data_target167114251239.teuthology_13304_object820:head by client.4144.0:821 2016-10-25 08:51:28.518817 0
2016-10-25 08:51:32.050902 7f210f468700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'268, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:32.051191 7f210f468700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139243c80 osr(1.7 0x7f2139474be0)
2016-10-25 08:51:32.051200 7f210f468700 10 journal prepare_entry [Transaction(0x7f2147f6ad80)]
2016-10-25 08:51:32.051435 7f210f468700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:32.055027 7f210f468700 10 journal op_submit_start 2364
2016-10-25 08:51:32.055062 7f210f468700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2364 [Transaction(0x7f2147f6ad80)]
2016-10-25 08:51:32.055070 7f210f468700 10 journal op_journal_transactions 2364 reqid_t client.4144.0:881
2016-10-25 08:51:32.055076 7f210f468700  5 journal submit_entry seq 2364 len 4202496 (0x7f2140fb6700)
2016-10-25 08:51:32.055138 7f210f468700 10 journal op_submit_finish 2364
2016-10-25 08:51:32.055146 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'267 crt=23'264 lcod 23'264 mlcod 23'264 active+clean] eval_repop repgather(0x7f21395ba1c0 23'268 rep_tid=1725 committed?=0 applied?=0) wants=d
2016-10-25 08:51:32.055177 7f210f468700 10 osd.3 23 dequeue_op 0x7f2139883a80 finish
2016-10-25 08:51:32.055195 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:32.055209 7f2123404700 20 journal write_thread_entry aio throttle: aio num 5 bytes 21012480 ... exp 10 min_new 1024 ... pending 4202496
2016-10-25 08:51:32.055217 7f2123404700 10 journal room 459132927 max_size 1072709632 pos 1004666880 header.start 391094272 top 4096
2016-10-25 08:51:32.055221 7f2123404700 10 journal check_for_full at 1004666880 : 4202496 < 459132927
2016-10-25 08:51:32.055225 7f2123404700 15 journal prepare_single_write 1 will write 1004666880 : seq 2364 len 4195598 -> 4202496
2016-10-25 08:51:32.055253 7f2123404700 20 journal prepare_multi_write queue_pos now 1008869376
2016-10-25 08:51:32.055255 7f2123404700 15 journal do_aio_write writing 1004666880~4202496
2016-10-25 08:51:32.055258 7f2123404700 20 journal write_aio_bl 1004666880~4202496 seq 2364
2016-10-25 08:51:32.055263 7f2123404700 20 journal write_aio_bl .. 1004666880~4202496 in 1
2016-10-25 08:51:32.057004 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 679 ==== osd_op(client.4144.0:882 1.490e145b (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (2547778642 0 3061838250) 0x7f2141377e40 con 0x7f2147f52800
2016-10-25 08:51:32.057048 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:32.057058 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f21398806c0 prio 63 cost 4194304 latency 0.001728 osd_op(client.4144.0:882 1.490e145b (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:32.057113 7f211146c700 10 osd.3 23 dequeue_op 0x7f21398806c0 prio 63 cost 4194304 latency 0.001783 osd_op(client.4144.0:882 1.490e145b (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean]
2016-10-25 08:51:32.057143 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] handle_message: 0x7f21398806c0
2016-10-25 08:51:32.057150 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:32.057160 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:32.057162 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:32.057167 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:32.057251 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] do_op osd_op(client.4144.0:882 1.490e145b benchmark_data_target167114251239.teuthology_13304_object881 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:32.057282 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] get_object_context: obc NOT found in cache: 1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head
2016-10-25 08:51:32.057298 7f211146c700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head# '_'
2016-10-25 08:51:32.057367 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject881__head_490E145B__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:32.057378 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head# '_' = -2
2016-10-25 08:51:32.057383 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] get_object_context: no obc for soid 1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head but can_create
2016-10-25 08:51:32.057399 7f211146c700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:snapdir# 'snapset'
2016-10-25 08:51:32.057427 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject881__snapdir_490E145B__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:32.057433 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:snapdir# 'snapset' = -2
2016-10-25 08:51:32.057457 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] create_object_context 0x7f214131e000 1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head 
2016-10-25 08:51:32.057470 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] populate_obc_watchers 1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head
2016-10-25 08:51:32.057481 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head
2016-10-25 08:51:32.057491 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] get_object_context: 0x7f214131e000 1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head rwstate(none n=0 w=0) oi: 1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f213946c680 snapset: 0=[]:[]
2016-10-25 08:51:32.057509 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] find_object_context 1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head @head oi=1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:32.057525 7f211146c700 25 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] do_op oi 1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:32.057552 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] get_object_context: obc NOT found in cache: 1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:snapdir
2016-10-25 08:51:32.057569 7f211146c700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:snapdir# '_'
2016-10-25 08:51:32.057593 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject881__snapdir_490E145B__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:32.057603 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:snapdir# '_' = -2
2016-10-25 08:51:32.057607 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] get_object_context: no obc for soid 1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:snapdir and !can_create
2016-10-25 08:51:32.057630 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] execute_ctx 0x7f214303e300
2016-10-25 08:51:32.057643 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] do_op 1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head [write 0~4194304] ov 0'0 av 23'286 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:32.057657 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] do_osd_op 1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head [write 0~4194304]
2016-10-25 08:51:32.057668 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:32.057912 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] make_writeable 1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head snapset=0x7f213946c6b8  snapc=0=[]
2016-10-25 08:51:32.057931 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean]  setting DIRTY flag
2016-10-25 08:51:32.057942 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] make_writeable 1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head done, snapset=0=[]:[]+head
2016-10-25 08:51:32.057952 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] finish_ctx 1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head 0x7f214303e300 op modify  
2016-10-25 08:51:32.057971 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean]  set mtime to 2016-10-25 08:51:32.044478
2016-10-25 08:51:32.057991 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean]  final snapset 0=[]:[]+head in 1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head
2016-10-25 08:51:32.058014 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean]  zeroing write result code 0
2016-10-25 08:51:32.058028 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean]  op order client.4144 tid 882 (first)
2016-10-25 08:51:32.058039 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] new_repop rep_tid 1726 on osd_op(client.4144.0:882 1.490e145b benchmark_data_target167114251239.teuthology_13304_object881 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:32.058059 7f211146c700  7 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] issue_repop rep_tid 1726 o 1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head
2016-10-25 08:51:32.058107 7f211146c700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:32.058115 7f211146c700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:882 1.3 1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head v 23'286) v1 -- 0x7f21480fde00 con 0
2016-10-25 08:51:32.058384 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'285 (0'0,23'285] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] append_log log((0'0,23'285], crt=23'281) [23'286 (0'0) modify   1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head by client.4144.0:882 2016-10-25 08:51:32.044478 0]
2016-10-25 08:51:32.058411 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'281 lcod 23'283 mlcod 23'283 active+clean] add_log_entry 23'286 (0'0) modify   1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head by client.4144.0:882 2016-10-25 08:51:32.044478 0
2016-10-25 08:51:32.058391 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.5_head/#1:ace40720:::benchmark_data_target167114251239.teuthology_13304_object868:head# 0~4194304 = 4194304
2016-10-25 08:51:32.058430 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] append_log: trimming to 23'283 entries 23'282 (0'0) modify   1:ccae22cf:::benchmark_data_target167114251239.teuthology_13304_object860:head by client.4144.0:861 2016-10-25 08:51:31.283863 0,23'283 (0'0) modify   1:c3d419a1:::benchmark_data_target167114251239.teuthology_13304_object862:head by client.4144.0:863 2016-10-25 08:51:31.284296 0
2016-10-25 08:51:32.058497 7f211146c700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'286, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:32.058578 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.5_head/#1:ace40720:::benchmark_data_target167114251239.teuthology_13304_object868:head#
2016-10-25 08:51:32.058719 7f211146c700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139241e40 osr(1.3 0x7f2139475570)
2016-10-25 08:51:32.058732 7f211146c700 10 journal prepare_entry [Transaction(0x7f2140fd2f40)]
2016-10-25 08:51:32.058718 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.5_head/#1:ace40720:::benchmark_data_target167114251239.teuthology_13304_object868:head# = 0
2016-10-25 08:51:32.058803 7f2121400700 10 journal op_apply_finish 2357 open_ops 1 -> 0, max_applied_seq 2356 -> 2357
2016-10-25 08:51:32.058831 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f21397f65e0 seq 2357 r = 0, finisher 0x7f213ecdf4e0 0
2016-10-25 08:51:32.058923 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f21397f65e0 seq 2357 osr(1.5 0x7f2139425410)/0x7f2139425410 lat 0.631969
2016-10-25 08:51:32.059039 7f2121c01700 10 journal op_apply_start 2358 open_ops 0 -> 1
2016-10-25 08:51:32.059044 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f21395461c0 seq 2358 osr(1.5 0x7f2139425410)/0x7f2139425410 start
2016-10-25 08:51:32.059047 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2141371800
2016-10-25 08:51:32.059063 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.5_head/#1:a0000000::::head#
2016-10-25 08:51:32.059081 7f2121c01700 10 filestore oid: #1:a0000000::::head# not skipping op, *spos 2358.0.0
2016-10-25 08:51:32.059084 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:32.058976 7f211146c700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:32.059243 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.5_head/#1:a0000000::::head# = 0
2016-10-25 08:51:32.059252 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f21413719c0
2016-10-25 08:51:32.059258 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.5_head/#1:a63a41ae:::benchmark_data_target167114251239.teuthology_13304_object872:head# 0~4194304
2016-10-25 08:51:32.059953 7f211146c700 10 journal op_submit_start 2365
2016-10-25 08:51:32.059964 7f211146c700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2365 [Transaction(0x7f2140fd2f40)]
2016-10-25 08:51:32.059969 7f211146c700 10 journal op_journal_transactions 2365 reqid_t client.4144.0:882
2016-10-25 08:51:32.059971 7f211146c700  5 journal submit_entry seq 2365 len 4202496 (0x7f21414ac340)
2016-10-25 08:51:32.059989 7f211146c700 10 journal op_submit_finish 2365
2016-10-25 08:51:32.059993 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] eval_repop repgather(0x7f2139883de0 23'286 rep_tid=1726 committed?=0 applied?=0) wants=d
2016-10-25 08:51:32.060012 7f211146c700 10 osd.3 23 dequeue_op 0x7f21398806c0 finish
2016-10-25 08:51:32.063618 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:32.063641 7f2123404700 20 journal write_thread_entry aio throttle: aio num 6 bytes 25214976 ... exp 12 min_new 4096 ... pending 4202496
2016-10-25 08:51:32.063650 7f2123404700 10 journal room 454930431 max_size 1072709632 pos 1008869376 header.start 391094272 top 4096
2016-10-25 08:51:32.063677 7f2123404700 10 journal check_for_full at 1008869376 : 4202496 < 454930431
2016-10-25 08:51:32.063682 7f2123404700 15 journal prepare_single_write 1 will write 1008869376 : seq 2365 len 4195598 -> 4202496
2016-10-25 08:51:32.063720 7f2123404700 20 journal prepare_multi_write queue_pos now 1013071872
2016-10-25 08:51:32.063723 7f2123404700 15 journal do_aio_write writing 1008869376~4202496
2016-10-25 08:51:32.063726 7f2123404700 20 journal write_aio_bl 1008869376~4202496 seq 2365
2016-10-25 08:51:32.063730 7f2123404700 20 journal write_aio_bl .. 1008869376~4202496 in 1
2016-10-25 08:51:32.064457 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:32.064505 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:32.064508 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:32.067028 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.5_head/#1:a63a41ae:::benchmark_data_target167114251239.teuthology_13304_object872:head# 0~4194304 = 4194304
2016-10-25 08:51:32.067222 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.5( v 23'279 (0'0,23'279] local-les=22 n=279 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'277 crt=23'276 lcod 23'278 active] sub_op_modify_applied on 0x7f21396d9c18 op osd_repop(client.4144.0:869 1.5 1:ace40720:::benchmark_data_target167114251239.teuthology_13304_object868:head v 23'278) v1
2016-10-25 08:51:32.067284 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.5( v 23'279 (0'0,23'279] local-les=22 n=279 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'277 crt=23'276 lcod 23'278 active] op_applied version 23'278
2016-10-25 08:51:32.067465 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.5_head/#1:a63a41ae:::benchmark_data_target167114251239.teuthology_13304_object872:head#
2016-10-25 08:51:32.067614 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.5_head/#1:a63a41ae:::benchmark_data_target167114251239.teuthology_13304_object872:head# = 0
2016-10-25 08:51:32.067655 7f2121c01700 10 journal op_apply_finish 2358 open_ops 1 -> 0, max_applied_seq 2357 -> 2358
2016-10-25 08:51:32.067682 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f21395461c0 seq 2358 r = 0, finisher 0x7f2142fe1a40 0
2016-10-25 08:51:32.067712 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f21395461c0 seq 2358 osr(1.5 0x7f2139425410)/0x7f2139425410 lat 0.541841
2016-10-25 08:51:32.067765 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.5( v 23'279 (0'0,23'279] local-les=22 n=279 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'278 crt=23'276 lcod 23'278 active] sub_op_modify_applied on 0x7f21396d9418 op osd_repop(client.4144.0:873 1.5 1:a63a41ae:::benchmark_data_target167114251239.teuthology_13304_object872:head v 23'279) v1
2016-10-25 08:51:32.067814 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.5( v 23'279 (0'0,23'279] local-les=22 n=279 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'278 crt=23'276 lcod 23'278 active] op_applied version 23'279
2016-10-25 08:51:32.083401 7f2122c03700 10 journal write_finish_thread_entry aio 983654400~4202496 done
2016-10-25 08:51:32.083419 7f2122c03700 10 journal write_finish_thread_entry aio 987856896~4202496 done
2016-10-25 08:51:32.083420 7f2122c03700 10 journal write_finish_thread_entry aio 992059392~4202496 done
2016-10-25 08:51:32.083423 7f2122c03700 10 journal write_finish_thread_entry aio 996261888~4202496 done
2016-10-25 08:51:32.083425 7f2122c03700 10 journal write_finish_thread_entry aio 1000464384~4202496 done
2016-10-25 08:51:32.083427 7f2122c03700 20 journal check_aio_completion
2016-10-25 08:51:32.083429 7f2122c03700 20 journal check_aio_completion completed seq 2359 983654400~4202496
2016-10-25 08:51:32.083441 7f2122c03700 20 journal check_aio_completion completed seq 2360 987856896~4202496
2016-10-25 08:51:32.083447 7f2122c03700 20 journal check_aio_completion completed seq 2361 992059392~4202496
2016-10-25 08:51:32.083452 7f2122c03700 20 journal check_aio_completion completed seq 2362 996261888~4202496
2016-10-25 08:51:32.083456 7f2122c03700 20 journal check_aio_completion completed seq 2363 1000464384~4202496
2016-10-25 08:51:32.083461 7f2122c03700 20 journal check_aio_completion queueing finishers through seq 2363
2016-10-25 08:51:32.083465 7f2122c03700 10 journal queue_completions_thru seq 2363 queueing seq 2359 0x7f2140fb4150 lat 0.433281
2016-10-25 08:51:32.083694 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f21397f4720 seq 2359 osr(1.3 0x7f2139475570) [Transaction(0x7f214159e000)]
2016-10-25 08:51:32.083705 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f21397f4720 seq 2359 osr(1.3 0x7f2139475570) 4195584 bytes   (queue has 7 ops and 29369158 bytes)
2016-10-25 08:51:32.083712 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f213eaa23e0
2016-10-25 08:51:32.083724 7f2120bff700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] op_commit: 1721
2016-10-25 08:51:32.083814 7f2122c03700 10 journal queue_completions_thru seq 2363 queueing seq 2360 0x7f2139564ff0 lat 0.426763
2016-10-25 08:51:32.083826 7f2122c03700 10 journal queue_completions_thru seq 2363 queueing seq 2361 0x7f214811ea00 lat 0.383052
2016-10-25 08:51:32.083835 7f2122c03700 10 journal queue_completions_thru seq 2363 queueing seq 2362 0x7f21414ada50 lat 0.377474
2016-10-25 08:51:32.083843 7f2122c03700 10 journal queue_completions_thru seq 2363 queueing seq 2363 0x7f2140ed55f0 lat 0.374796
2016-10-25 08:51:32.083852 7f2122c03700 20 journal write_finish_thread_entry waiting for aio(s)
2016-10-25 08:51:32.083858 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f21480570c0 seq 2360 osr(1.1 0x7f21394758a0) [Transaction(0x7f2142fe7480)]
2016-10-25 08:51:32.083861 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f21480570c0 seq 2360 osr(1.1 0x7f21394758a0) 4195584 bytes   (queue has 7 ops and 29369158 bytes)
2016-10-25 08:51:32.083865 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f2140e14c00
2016-10-25 08:51:32.083870 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f213ed5a1e0 seq 2361 osr(1.2 0x7f2139474dc0) [Transaction(0x7f2147f6df80),Transaction(0x7f2147f6e140)]
2016-10-25 08:51:32.083874 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f213ed5a1e0 seq 2361 osr(1.2 0x7f2139474dc0) 4195654 bytes   (queue has 7 ops and 29369158 bytes)
2016-10-25 08:51:32.083876 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f2148112980
2016-10-25 08:51:32.083880 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f214133ba40 seq 2362 osr(1.6 0x7f2139474f10) [Transaction(0x7f2148080a80)]
2016-10-25 08:51:32.083868 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2290 ==== osd_repop_reply(client.4144.0:865 1.0) v1 ==== 83+0+0 (2764725055 0 0) 0x7f213ea47e00 con 0x7f2139592800
2016-10-25 08:51:32.083882 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f214133ba40 seq 2362 osr(1.6 0x7f2139474f10) 4195584 bytes   (queue has 7 ops and 29369158 bytes)
2016-10-25 08:51:32.083885 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f2141064300
2016-10-25 08:51:32.083887 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f2142f71b60 seq 2363 osr(1.4 0x7f2139475240) [Transaction(0x7f2147f2b880)]
2016-10-25 08:51:32.083889 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f2142f71b60 seq 2363 osr(1.4 0x7f2139475240) 4195584 bytes   (queue has 7 ops and 29369158 bytes)
2016-10-25 08:51:32.083891 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f2139312980
2016-10-25 08:51:32.083892 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:865 1.0) v1 epoch 23
2016-10-25 08:51:32.083899 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:32.083907 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139881440 prio 196 cost 0 latency 0.000078 osd_repop_reply(client.4144.0:865 1.0) v1
2016-10-25 08:51:32.083898 7f211fbfd700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'325 crt=23'321 lcod 23'321 mlcod 23'321 active+clean] op_commit: 1722
2016-10-25 08:51:32.083938 7f211fbfd700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'277 crt=23'273 lcod 23'273 mlcod 23'273 active+clean] op_commit: 1723
2016-10-25 08:51:32.083972 7f21203fe700 10 osd.3 pg_epoch: 23 pg[1.2( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'276 crt=23'275 lcod 23'275 active] sub_op_modify_commit on op osd_repop(client.4144.0:876 1.2 1:5ed10711:::benchmark_data_target167114251239.teuthology_13304_object875:head v 23'277) v1, sending commit to osd.2
2016-10-25 08:51:32.083991 7f21203fe700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:32.083993 7f21203fe700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop_reply(client.4144.0:876 1.2 ondisk, result = 0) v1 -- 0x7f213ec3bb80 con 0
2016-10-25 08:51:32.084004 7f211046a700 10 osd.3 23 dequeue_op 0x7f2139881440 prio 196 cost 0 latency 0.000168 osd_repop_reply(client.4144.0:865 1.0) v1 pg pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'279 lcod 23'283 mlcod 23'283 active+clean]
2016-10-25 08:51:32.084025 7f21203fe700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'283 crt=23'282 lcod 23'282 mlcod 23'282 active+clean] op_commit: 1724
2016-10-25 08:51:32.084035 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'279 lcod 23'283 mlcod 23'283 active+clean] handle_message: 0x7f2139881440
2016-10-25 08:51:32.084055 7f211046a700  7 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'279 lcod 23'283 mlcod 23'283 active+clean] sub_op_modify_reply: tid 1712 op  ack_type 4 from 2
2016-10-25 08:51:32.084079 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'279 lcod 23'283 mlcod 23'283 active+clean] repop_all_applied: repop tid 1712 all applied 
2016-10-25 08:51:32.084090 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'279 lcod 23'283 mlcod 23'283 active+clean] eval_repop repgather(0x7f213981e640 23'285 rep_tid=1712 committed?=0 applied?=1) wants=d
2016-10-25 08:51:32.084100 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'279 lcod 23'283 mlcod 23'283 active+clean]  applied: repgather(0x7f213981e640 23'285 rep_tid=1712 committed?=0 applied?=1) 
2016-10-25 08:51:32.084112 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'279 lcod 23'283 mlcod 23'283 active+clean] repop_all_committed: repop tid 1712 all committed 
2016-10-25 08:51:32.084121 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'283 active+clean] eval_repop repgather(0x7f213981e640 23'285 rep_tid=1712 committed?=1 applied?=1) wants=d
2016-10-25 08:51:32.084130 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'283 active+clean]  commit: repgather(0x7f213981e640 23'285 rep_tid=1712 committed?=1 applied?=1)
2016-10-25 08:51:32.084164 7f211046a700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'283 active+clean] log_op_stats osd_op(client.4144.0:865 1.ed9c7da8 benchmark_data_target167114251239.teuthology_13304_object864 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 0.754659 lat 0.754710
2016-10-25 08:51:32.084178 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'283 active+clean]  sending commit on osd_op(client.4144.0:865 1.ed9c7da8 benchmark_data_target167114251239.teuthology_13304_object864 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f21397a2940
2016-10-25 08:51:32.084190 7f211046a700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(865 benchmark_data_target167114251239.teuthology_13304_object864 [write 0~4194304] v23'285 uv285 ondisk = 0) v7 -- 0x7f21397a2940 con 0
2016-10-25 08:51:32.084210 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'283 active+clean]  applied: repgather(0x7f213981e640 23'285 rep_tid=1712 committed?=1 applied?=1) 
2016-10-25 08:51:32.084224 7f211046a700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'283 active+clean] publish_stats_to_osd 23: no change since 2016-10-25 08:51:31.556482
2016-10-25 08:51:32.084237 7f211046a700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:32.084248 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean]  removing repgather(0x7f213981e640 23'285 rep_tid=1712 committed?=1 applied?=1)
2016-10-25 08:51:32.084257 7f211046a700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean]    q front is repgather(0x7f213981e640 23'285 rep_tid=1712 committed?=1 applied?=1)
2016-10-25 08:51:32.084266 7f211046a700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] remove_repop repgather(0x7f213981e640 23'285 rep_tid=1712 committed?=1 applied?=1)
2016-10-25 08:51:32.084280 7f211046a700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean]  requeue_ops 
2016-10-25 08:51:32.084325 7f211046a700 10 osd.3 23 dequeue_op 0x7f2139881440 finish
2016-10-25 08:51:32.084331 7f2121400700 10 journal op_apply_start 2359 open_ops 0 -> 1
2016-10-25 08:51:32.084367 7f2121400700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f21397f4720 seq 2359 osr(1.3 0x7f2139475570)/0x7f2139475570 start
2016-10-25 08:51:32.084389 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f214159e000
2016-10-25 08:51:32.084412 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.3_head/#1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head# 0~4194304
2016-10-25 08:51:32.084460 7f2121c01700 10 journal op_apply_start 2360 open_ops 1 -> 2
2016-10-25 08:51:32.084469 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f21480570c0 seq 2360 osr(1.1 0x7f21394758a0)/0x7f21394758a0 start
2016-10-25 08:51:32.084473 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2142fe7480
2016-10-25 08:51:32.084483 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.1_head/#1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head# 0~4194304
2016-10-25 08:51:32.090027 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.1_head/#1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head# 0~4194304 = 4194304
2016-10-25 08:51:32.090089 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.1_head/#1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head#
2016-10-25 08:51:32.090142 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.1_head/#1:9f939dbe:::benchmark_data_target167114251239.teuthology_13304_object877:head# = 0
2016-10-25 08:51:32.090161 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.1_head/#1:80000000::::head#
2016-10-25 08:51:32.090180 7f2121c01700 10 filestore oid: #1:80000000::::head# not skipping op, *spos 2360.0.2
2016-10-25 08:51:32.090187 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:32.090238 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.1_head/#1:80000000::::head# = 0
2016-10-25 08:51:32.090250 7f2121c01700 10 journal op_apply_finish 2360 open_ops 2 -> 1, max_applied_seq 2358 -> 2360
2016-10-25 08:51:32.090253 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f21480570c0 seq 2360 r = 0, finisher 0x7f2140e150e0 0x7f2142fcb8c0
2016-10-25 08:51:32.090265 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f21480570c0 seq 2360 osr(1.1 0x7f21394758a0)/0x7f21394758a0 lat 0.434712
2016-10-25 08:51:32.090285 7f2121c01700 10 journal op_apply_start 2361 open_ops 1 -> 2
2016-10-25 08:51:32.090291 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f213ed5a1e0 seq 2361 osr(1.2 0x7f2139474dc0)/0x7f2139474dc0 start
2016-10-25 08:51:32.090294 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147f6df80
2016-10-25 08:51:32.090303 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.2_head/#1:40000000::::head#
2016-10-25 08:51:32.090310 7f2121c01700 10 filestore oid: #1:40000000::::head# not skipping op, *spos 2361.0.0
2016-10-25 08:51:32.090313 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:32.090295 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'325 crt=23'321 lcod 23'321 mlcod 23'321 active+clean] op_applied: 1722
2016-10-25 08:51:32.090340 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.2_head/#1:40000000::::head# = 0
2016-10-25 08:51:32.090336 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 lua=23'325 crt=23'321 lcod 23'321 mlcod 23'321 active+clean] op_applied version 23'326
2016-10-25 08:51:32.090346 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147f6e140
2016-10-25 08:51:32.090351 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.2_head/#1:5ed10711:::benchmark_data_target167114251239.teuthology_13304_object875:head# 0~4194304
2016-10-25 08:51:32.090440 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.3_head/#1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head# 0~4194304 = 4194304
2016-10-25 08:51:32.090500 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.3_head/#1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head#
2016-10-25 08:51:32.090549 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.3_head/#1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head# = 0
2016-10-25 08:51:32.090574 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.3_head/#1:c0000000::::head#
2016-10-25 08:51:32.090595 7f2121400700 10 filestore oid: #1:c0000000::::head# not skipping op, *spos 2359.0.2
2016-10-25 08:51:32.090608 7f2121400700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:32.090648 7f2121400700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.3_head/#1:c0000000::::head# = 0
2016-10-25 08:51:32.090673 7f2121400700 10 journal op_apply_finish 2359 open_ops 2 -> 1, max_applied_seq 2360 -> 2360
2016-10-25 08:51:32.090686 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f21397f4720 seq 2359 r = 0, finisher 0x7f213eaa2860 0x7f214309d680
2016-10-25 08:51:32.090706 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f21397f4720 seq 2359 osr(1.3 0x7f2139475570)/0x7f2139475570 lat 0.441669
2016-10-25 08:51:32.090734 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] op_applied: 1721
2016-10-25 08:51:32.090759 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] op_applied version 23'285
2016-10-25 08:51:32.090797 7f2121400700 10 journal op_apply_start 2362 open_ops 1 -> 2
2016-10-25 08:51:32.090810 7f2121400700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f214133ba40 seq 2362 osr(1.6 0x7f2139474f10)/0x7f2139474f10 start
2016-10-25 08:51:32.090823 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2148080a80
2016-10-25 08:51:32.090841 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.6_head/#1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head# 0~4194304
2016-10-25 08:51:32.093603 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.2_head/#1:5ed10711:::benchmark_data_target167114251239.teuthology_13304_object875:head# 0~4194304 = 4194304
2016-10-25 08:51:32.093650 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.2_head/#1:5ed10711:::benchmark_data_target167114251239.teuthology_13304_object875:head#
2016-10-25 08:51:32.093691 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.2_head/#1:5ed10711:::benchmark_data_target167114251239.teuthology_13304_object875:head# = 0
2016-10-25 08:51:32.093703 7f2121c01700 10 journal op_apply_finish 2361 open_ops 2 -> 1, max_applied_seq 2360 -> 2361
2016-10-25 08:51:32.093705 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f213ed5a1e0 seq 2361 r = 0, finisher 0x7f213ecdf9c0 0
2016-10-25 08:51:32.093745 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f213ed5a1e0 seq 2361 osr(1.2 0x7f2139474dc0)/0x7f2139474dc0 lat 0.394854
2016-10-25 08:51:32.093762 7f2121c01700 10 journal op_apply_start 2363 open_ops 1 -> 2
2016-10-25 08:51:32.093764 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2142f71b60 seq 2363 osr(1.4 0x7f2139475240)/0x7f2139475240 start
2016-10-25 08:51:32.093767 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147f2b880
2016-10-25 08:51:32.093776 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.4_head/#1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head# 0~4194304
2016-10-25 08:51:32.093835 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.2( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'276 crt=23'275 lcod 23'276 active] sub_op_modify_applied on 0x7f213963d018 op osd_repop(client.4144.0:876 1.2 1:5ed10711:::benchmark_data_target167114251239.teuthology_13304_object875:head v 23'277) v1
2016-10-25 08:51:32.093854 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.2( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'276 crt=23'275 lcod 23'276 active] op_applied version 23'277
2016-10-25 08:51:32.093998 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.6_head/#1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head# 0~4194304 = 4194304
2016-10-25 08:51:32.094041 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.6_head/#1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head#
2016-10-25 08:51:32.094085 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.6_head/#1:6e077336:::benchmark_data_target167114251239.teuthology_13304_object878:head# = 0
2016-10-25 08:51:32.094107 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.6_head/#1:60000000::::head#
2016-10-25 08:51:32.094128 7f2121400700 10 filestore oid: #1:60000000::::head# not skipping op, *spos 2362.0.2
2016-10-25 08:51:32.094140 7f2121400700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:32.094185 7f2121400700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.6_head/#1:60000000::::head# = 0
2016-10-25 08:51:32.094203 7f2121400700 10 journal op_apply_finish 2362 open_ops 2 -> 1, max_applied_seq 2361 -> 2362
2016-10-25 08:51:32.094215 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f214133ba40 seq 2362 r = 0, finisher 0x7f21410642c0 0x7f21480636c0
2016-10-25 08:51:32.094233 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f214133ba40 seq 2362 osr(1.6 0x7f2139474f10)/0x7f2139474f10 lat 0.389392
2016-10-25 08:51:32.094258 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'277 crt=23'273 lcod 23'273 mlcod 23'273 active+clean] op_applied: 1723
2016-10-25 08:51:32.094283 7f211e3fa700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 lua=23'277 crt=23'273 lcod 23'273 mlcod 23'273 active+clean] op_applied version 23'278
2016-10-25 08:51:32.096822 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.4_head/#1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head# 0~4194304 = 4194304
2016-10-25 08:51:32.096863 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.4_head/#1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head#
2016-10-25 08:51:32.096897 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.4_head/#1:2d66f6be:::benchmark_data_target167114251239.teuthology_13304_object879:head# = 0
2016-10-25 08:51:32.096908 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.4_head/#1:20000000::::head#
2016-10-25 08:51:32.096919 7f2121c01700 10 filestore oid: #1:20000000::::head# not skipping op, *spos 2363.0.2
2016-10-25 08:51:32.096921 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:32.096970 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.4_head/#1:20000000::::head# = 0
2016-10-25 08:51:32.096975 7f2121c01700 10 journal op_apply_finish 2363 open_ops 1 -> 0, max_applied_seq 2362 -> 2363
2016-10-25 08:51:32.096978 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2142f71b60 seq 2363 r = 0, finisher 0x7f2147e82780 0x7f2142fcaa40
2016-10-25 08:51:32.096989 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f2142f71b60 seq 2363 osr(1.4 0x7f2139475240)/0x7f2139475240 lat 0.390378
2016-10-25 08:51:32.097028 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'283 crt=23'282 lcod 23'282 mlcod 23'282 active+clean] op_applied: 1724
2016-10-25 08:51:32.097063 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 lua=23'283 crt=23'282 lcod 23'282 mlcod 23'282 active+clean] op_applied version 23'284
2016-10-25 08:51:32.147700 7f2122c03700 10 journal write_finish_thread_entry aio 1004666880~4202496 done
2016-10-25 08:51:32.147727 7f2122c03700 20 journal check_aio_completion
2016-10-25 08:51:32.147729 7f2122c03700 20 journal check_aio_completion completed seq 2364 1004666880~4202496
2016-10-25 08:51:32.147738 7f2122c03700 20 journal check_aio_completion queueing finishers through seq 2364
2016-10-25 08:51:32.147741 7f2122c03700 10 journal queue_completions_thru seq 2364 queueing seq 2364 0x7f2140fb6700 lat 0.092610
2016-10-25 08:51:32.147772 7f2122c03700 20 journal write_finish_thread_entry waiting for aio(s)
2016-10-25 08:51:32.147792 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f214133dc60 seq 2364 osr(1.7 0x7f2139474be0) [Transaction(0x7f2147f6ad80)]
2016-10-25 08:51:32.147797 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f214133dc60 seq 2364 osr(1.7 0x7f2139474be0) 4195584 bytes   (queue has 2 ops and 8391168 bytes)
2016-10-25 08:51:32.147803 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f2142e0b2e0
2016-10-25 08:51:32.147814 7f2120bff700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'267 crt=23'264 lcod 23'264 mlcod 23'264 active+clean] op_commit: 1725
2016-10-25 08:51:32.147862 7f2121400700 10 journal op_apply_start 2364 open_ops 0 -> 1
2016-10-25 08:51:32.147891 7f2121400700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f214133dc60 seq 2364 osr(1.7 0x7f2139474be0)/0x7f2139474be0 start
2016-10-25 08:51:32.147896 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147f6ad80
2016-10-25 08:51:32.147877 7f2122c03700 10 journal write_finish_thread_entry aio 1008869376~4202496 done
2016-10-25 08:51:32.147905 7f2122c03700 20 journal check_aio_completion
2016-10-25 08:51:32.147906 7f2122c03700 20 journal check_aio_completion completed seq 2365 1008869376~4202496
2016-10-25 08:51:32.147909 7f2122c03700 20 journal check_aio_completion queueing finishers through seq 2365
2016-10-25 08:51:32.147911 7f2122c03700 10 journal queue_completions_thru seq 2365 queueing seq 2365 0x7f21414ac340 lat 0.087925
2016-10-25 08:51:32.147922 7f2122c03700 20 journal write_finish_thread_entry sleeping
2016-10-25 08:51:32.147925 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f213c99ed80 seq 2365 osr(1.3 0x7f2139475570) [Transaction(0x7f2140fd2f40)]
2016-10-25 08:51:32.147928 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f213c99ed80 seq 2365 osr(1.3 0x7f2139475570) 4195584 bytes   (queue has 2 ops and 8391168 bytes)
2016-10-25 08:51:32.147934 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f2142e0b280
2016-10-25 08:51:32.147939 7f2120bff700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'285 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] op_commit: 1726
2016-10-25 08:51:32.147972 7f2121c01700 10 journal op_apply_start 2365 open_ops 1 -> 2
2016-10-25 08:51:32.147974 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f213c99ed80 seq 2365 osr(1.3 0x7f2139475570)/0x7f2139475570 start
2016-10-25 08:51:32.147977 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2140fd2f40
2016-10-25 08:51:32.147988 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.3_head/#1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head# 0~4194304
2016-10-25 08:51:32.148015 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.7_head/#1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head# 0~4194304
2016-10-25 08:51:32.150826 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.3_head/#1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head# 0~4194304 = 4194304
2016-10-25 08:51:32.150892 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.3_head/#1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head#
2016-10-25 08:51:32.150929 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.3_head/#1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head# = 0
2016-10-25 08:51:32.150943 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.3_head/#1:c0000000::::head#
2016-10-25 08:51:32.150957 7f2121c01700 10 filestore oid: #1:c0000000::::head# not skipping op, *spos 2365.0.2
2016-10-25 08:51:32.150962 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:32.151014 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.3_head/#1:c0000000::::head# = 0
2016-10-25 08:51:32.151025 7f2121c01700 10 journal op_apply_finish 2365 open_ops 2 -> 1, max_applied_seq 2363 -> 2365
2016-10-25 08:51:32.151029 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f213c99ed80 seq 2365 r = 0, finisher 0x7f2142e0b540 0x7f21395fb440
2016-10-25 08:51:32.151039 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f213c99ed80 seq 2365 osr(1.3 0x7f2139475570)/0x7f2139475570 lat 0.092308
2016-10-25 08:51:32.151063 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'285 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] op_applied: 1726
2016-10-25 08:51:32.151099 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'285 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] op_applied version 23'286
2016-10-25 08:51:32.151307 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.7_head/#1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head# 0~4194304 = 4194304
2016-10-25 08:51:32.151359 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.7_head/#1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head#
2016-10-25 08:51:32.151412 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.7_head/#1:fb3d49c3:::benchmark_data_target167114251239.teuthology_13304_object880:head# = 0
2016-10-25 08:51:32.151439 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.7_head/#1:e0000000::::head#
2016-10-25 08:51:32.151462 7f2121400700 10 filestore oid: #1:e0000000::::head# not skipping op, *spos 2364.0.2
2016-10-25 08:51:32.151479 7f2121400700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:32.151568 7f2121400700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.7_head/#1:e0000000::::head# = 0
2016-10-25 08:51:32.151594 7f2121400700 10 journal op_apply_finish 2364 open_ops 1 -> 0, max_applied_seq 2365 -> 2365
2016-10-25 08:51:32.151610 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f214133dc60 seq 2364 r = 0, finisher 0x7f2142e0ade0 0x7f2147f99400
2016-10-25 08:51:32.151634 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f214133dc60 seq 2364 osr(1.7 0x7f2139474be0)/0x7f2139474be0 lat 0.100434
2016-10-25 08:51:32.151673 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'267 crt=23'264 lcod 23'264 mlcod 23'264 active+clean] op_applied: 1725
2016-10-25 08:51:32.151703 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 lua=23'267 crt=23'264 lcod 23'264 mlcod 23'264 active+clean] op_applied version 23'268
2016-10-25 08:51:32.302780 7f2125408700 10 osd.3 23 tick_without_osd_lock
2016-10-25 08:51:32.302804 7f2125408700 25 osd.3 23 heartbeat_check osd.2 first_tx 2016-10-25 08:34:57.103177 last_tx 2016-10-25 08:51:30.940721 last_rx_back 2016-10-25 08:51:30.940721 last_rx_front 2016-10-25 08:51:30.940721
2016-10-25 08:51:32.302815 7f2125408700 20 osd.3 23 scrub_random_backoff lost coin flip, randomly backing off
2016-10-25 08:51:32.302817 7f2125408700 10 osd.3 23 promote_throttle_recalibrate 0 attempts, promoted 0 objects and 0  bytes; target 25 obj/sec or 5120 k bytes/sec
2016-10-25 08:51:32.302828 7f2125408700 20 osd.3 23 promote_throttle_recalibrate  new_prob 1000
2016-10-25 08:51:32.302829 7f2125408700 10 osd.3 23 promote_throttle_recalibrate  actual 0, actual/prob ratio 1, adjusted new_prob 1000, prob 1000 -> 1000
2016-10-25 08:51:32.586121 7f2123c05700 10 filestore(/var/lib/ceph/osd/ceph-3) sync_entry commit took 1.265340, interval was 5.740039
2016-10-25 08:51:32.586162 7f2123c05700 10 journal commit_finish thru 2331
2016-10-25 08:51:32.586175 7f2123c05700  5 journal committed_thru 2331 (last_committed_seq 2216)
2016-10-25 08:51:32.586274 7f2123c05700 10 journal header: block_size 4096 alignment 4096 max_size 1072709632
2016-10-25 08:51:32.586284 7f2123c05700 10 journal header: start 870187008
2016-10-25 08:51:32.586285 7f2123c05700 10 journal  write_pos 1013071872
2016-10-25 08:51:32.586287 7f2123c05700 10 journal committed_thru done
2016-10-25 08:51:32.598038 7f2123c05700 15 filestore(/var/lib/ceph/osd/ceph-3) sync_entry committed to op_seq 2331
2016-10-25 08:51:32.598123 7f2123c05700 20 filestore(/var/lib/ceph/osd/ceph-3) sync_entry waiting for max_interval 5.000000
2016-10-25 08:51:32.641132 7f210cc63700 20 osd.3 23 update_osd_stat osd_stat(9670 MB used, 558 MB avail, 10228 MB total, peers [2]/[] op hist [0,0,0,0,0,0,0,0,0,0,6,8])
2016-10-25 08:51:32.641183 7f210cc63700  5 osd.3 23 heartbeat: osd_stat(9670 MB used, 558 MB avail, 10228 MB total, peers [2]/[] op hist [0,0,0,0,0,0,0,0,0,0,6,8])
2016-10-25 08:51:32.641200 7f210cc63700  1 -- 167.114.251.239:0/12812 --> 167.114.251.239:6802/12736 -- osd_ping(ping e23 stamp 2016-10-25 08:51:32.641193) v2 -- 0x7f213ec79200 con 0
2016-10-25 08:51:32.641268 7f210cc63700  1 -- 167.114.251.239:0/12812 --> 167.114.251.239:6803/12736 -- osd_ping(ping e23 stamp 2016-10-25 08:51:32.641193) v2 -- 0x7f213ec77600 con 0
2016-10-25 08:51:32.641291 7f210cc63700 25 osd.3 23 heartbeat_check osd.2 first_tx 2016-10-25 08:34:57.103177 last_tx 2016-10-25 08:51:32.641193 last_rx_back 2016-10-25 08:51:30.940721 last_rx_front 2016-10-25 08:51:30.940721
2016-10-25 08:51:32.641749 7f2128b4c700  1 -- 167.114.251.239:0/12812 <== osd.2 167.114.251.239:6803/12736 325 ==== osd_ping(ping_reply e23 stamp 2016-10-25 08:51:32.641193) v2 ==== 47+0+0 (2245568904 0 0) 0x7f2139679600 con 0x7f2139591000
2016-10-25 08:51:32.641775 7f2128b4c700 25 osd.3 23 handle_osd_ping got reply from osd.2 first_tx 2016-10-25 08:34:57.103177 last_tx 2016-10-25 08:51:32.641193 last_rx_back 2016-10-25 08:51:30.940721 last_rx_front 2016-10-25 08:51:30.940721 -> 2016-10-25 08:51:32.641193
2016-10-25 08:51:32.641790 7f2128b4c700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:32.641895 7f212934d700  1 -- 167.114.251.239:0/12812 <== osd.2 167.114.251.239:6802/12736 325 ==== osd_ping(ping_reply e23 stamp 2016-10-25 08:51:32.641193) v2 ==== 47+0+0 (2245568904 0 0) 0x7f2139505800 con 0x7f213958e000
2016-10-25 08:51:32.641916 7f212934d700 25 osd.3 23 handle_osd_ping got reply from osd.2 first_tx 2016-10-25 08:34:57.103177 last_tx 2016-10-25 08:51:32.641193 last_rx_back 2016-10-25 08:51:30.940721 -> 2016-10-25 08:51:32.641193 last_rx_front 2016-10-25 08:51:32.641193
2016-10-25 08:51:32.641936 7f212934d700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:32.703270 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2291 ==== osd_repop_reply(client.4144.0:866 1.7) v1 ==== 83+0+0 (754046667 0 0) 0x7f213ec3bb80 con 0x7f2139592800
2016-10-25 08:51:32.703300 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:866 1.7) v1 epoch 23
2016-10-25 08:51:32.703307 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:32.703316 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139881560 prio 196 cost 0 latency 0.000081 osd_repop_reply(client.4144.0:866 1.7) v1
2016-10-25 08:51:32.703371 7f2111c6d700 10 osd.3 23 dequeue_op 0x7f2139881560 prio 196 cost 0 latency 0.000136 osd_repop_reply(client.4144.0:866 1.7) v1 pg pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'264 lcod 23'264 mlcod 23'264 active+clean]
2016-10-25 08:51:32.703390 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'264 lcod 23'264 mlcod 23'264 active+clean] handle_message: 0x7f2139881560
2016-10-25 08:51:32.703403 7f2111c6d700  7 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'264 lcod 23'264 mlcod 23'264 active+clean] sub_op_modify_reply: tid 1713 op  ack_type 4 from 2
2016-10-25 08:51:32.703420 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'264 lcod 23'264 mlcod 23'264 active+clean] repop_all_applied: repop tid 1713 all applied 
2016-10-25 08:51:32.703425 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'264 lcod 23'264 mlcod 23'264 active+clean] eval_repop repgather(0x7f21397ebde0 23'266 rep_tid=1713 committed?=0 applied?=1) wants=d
2016-10-25 08:51:32.703432 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'264 lcod 23'264 mlcod 23'264 active+clean]  applied: repgather(0x7f21397ebde0 23'266 rep_tid=1713 committed?=0 applied?=1) 
2016-10-25 08:51:32.703438 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'265 crt=23'264 lcod 23'264 mlcod 23'264 active+clean] repop_all_committed: repop tid 1713 all committed 
2016-10-25 08:51:32.703443 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'266 crt=23'264 lcod 23'265 mlcod 23'264 active+clean] eval_repop repgather(0x7f21397ebde0 23'266 rep_tid=1713 committed?=1 applied?=1) wants=d
2016-10-25 08:51:32.703451 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'266 crt=23'264 lcod 23'265 mlcod 23'264 active+clean]  commit: repgather(0x7f21397ebde0 23'266 rep_tid=1713 committed?=1 applied?=1)
2016-10-25 08:51:32.703462 7f2111c6d700 15 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'266 crt=23'264 lcod 23'265 mlcod 23'264 active+clean] log_op_stats osd_op(client.4144.0:866 1.3deccf7 benchmark_data_target167114251239.teuthology_13304_object865 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 1.351823 lat 1.351846
2016-10-25 08:51:32.703474 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'266 crt=23'264 lcod 23'265 mlcod 23'264 active+clean]  sending commit on osd_op(client.4144.0:866 1.3deccf7 benchmark_data_target167114251239.teuthology_13304_object865 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f2140fc9e40
2016-10-25 08:51:32.703492 7f2111c6d700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(866 benchmark_data_target167114251239.teuthology_13304_object865 [write 0~4194304] v23'266 uv266 ondisk = 0) v7 -- 0x7f2140fc9e40 con 0
2016-10-25 08:51:32.703535 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'266 crt=23'264 lcod 23'265 mlcod 23'264 active+clean]  applied: repgather(0x7f21397ebde0 23'266 rep_tid=1713 committed?=1 applied?=1) 
2016-10-25 08:51:32.703544 7f2111c6d700 15 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'266 crt=23'264 lcod 23'265 mlcod 23'264 active+clean] publish_stats_to_osd 23:145
2016-10-25 08:51:32.703550 7f2111c6d700 15 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'266 crt=23'264 lcod 23'265 mlcod 23'265 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:32.703556 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'266 crt=23'264 lcod 23'265 mlcod 23'265 active+clean]  removing repgather(0x7f21397ebde0 23'266 rep_tid=1713 committed?=1 applied?=1)
2016-10-25 08:51:32.703561 7f2111c6d700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'266 crt=23'264 lcod 23'265 mlcod 23'265 active+clean]    q front is repgather(0x7f21397ebde0 23'266 rep_tid=1713 committed?=1 applied?=1)
2016-10-25 08:51:32.703566 7f2111c6d700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'266 crt=23'264 lcod 23'265 mlcod 23'265 active+clean] remove_repop repgather(0x7f21397ebde0 23'266 rep_tid=1713 committed?=1 applied?=1)
2016-10-25 08:51:32.703575 7f2111c6d700 15 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'266 crt=23'264 lcod 23'265 mlcod 23'265 active+clean]  requeue_ops 
2016-10-25 08:51:32.703596 7f2111c6d700 10 osd.3 23 dequeue_op 0x7f2139881560 finish
2016-10-25 08:51:32.709716 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 680 ==== osd_op(client.4144.0:884 1.2438ed73 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (3984715103 0 618824157) 0x7f2140fc9e40 con 0x7f2147f52800
2016-10-25 08:51:32.709754 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:32.709761 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f21398810e0 prio 63 cost 4194304 latency 0.003457 osd_op(client.4144.0:884 1.2438ed73 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:32.709833 7f210ec67700 10 osd.3 23 dequeue_op 0x7f21398810e0 prio 63 cost 4194304 latency 0.003529 osd_op(client.4144.0:884 1.2438ed73 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean]
2016-10-25 08:51:32.709856 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] handle_message: 0x7f21398810e0
2016-10-25 08:51:32.709871 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:32.709886 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] do_op osd_op(client.4144.0:884 1.2438ed73 benchmark_data_target167114251239.teuthology_13304_object883 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:32.709908 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] get_object_context: obc NOT found in cache: 1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head
2016-10-25 08:51:32.709917 7f210ec67700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head# '_'
2016-10-25 08:51:32.709965 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject883__head_2438ED73__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:32.709973 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head# '_' = -2
2016-10-25 08:51:32.709976 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] get_object_context: no obc for soid 1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head but can_create
2016-10-25 08:51:32.709986 7f210ec67700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:snapdir# 'snapset'
2016-10-25 08:51:32.709998 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject883__snapdir_2438ED73__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:32.710002 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:snapdir# 'snapset' = -2
2016-10-25 08:51:32.710027 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] create_object_context 0x7f2142e65340 1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head 
2016-10-25 08:51:32.710033 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] populate_obc_watchers 1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head
2016-10-25 08:51:32.710039 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head
2016-10-25 08:51:32.710044 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] get_object_context: 0x7f2142e65340 1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head rwstate(none n=0 w=0) oi: 1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f213964c920 snapset: 0=[]:[]
2016-10-25 08:51:32.710054 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] find_object_context 1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head @head oi=1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:32.710063 7f210ec67700 25 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] do_op oi 1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:32.710072 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] get_object_context: obc NOT found in cache: 1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:snapdir
2016-10-25 08:51:32.710077 7f210ec67700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:snapdir# '_'
2016-10-25 08:51:32.710089 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject883__snapdir_2438ED73__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:32.710091 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:snapdir# '_' = -2
2016-10-25 08:51:32.710094 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] get_object_context: no obc for soid 1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:snapdir and !can_create
2016-10-25 08:51:32.710107 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] execute_ctx 0x7f2140fc5c00
2016-10-25 08:51:32.710113 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] do_op 1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head [write 0~4194304] ov 0'0 av 23'287 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:32.710121 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] do_osd_op 1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head [write 0~4194304]
2016-10-25 08:51:32.710127 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:32.710359 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] make_writeable 1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head snapset=0x7f213964c958  snapc=0=[]
2016-10-25 08:51:32.710365 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean]  setting DIRTY flag
2016-10-25 08:51:32.710377 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] make_writeable 1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head done, snapset=0=[]:[]+head
2016-10-25 08:51:32.710383 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] finish_ctx 1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head 0x7f2140fc5c00 op modify  
2016-10-25 08:51:32.710390 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean]  set mtime to 2016-10-25 08:51:32.704082
2016-10-25 08:51:32.710406 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean]  final snapset 0=[]:[]+head in 1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head
2016-10-25 08:51:32.710427 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean]  zeroing write result code 0
2016-10-25 08:51:32.710439 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean]  op order client.4144 tid 884 (first)
2016-10-25 08:51:32.710449 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] new_repop rep_tid 1727 on osd_op(client.4144.0:884 1.2438ed73 benchmark_data_target167114251239.teuthology_13304_object883 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:32.710458 7f210ec67700  7 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] issue_repop rep_tid 1727 o 1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head
2016-10-25 08:51:32.710489 7f210ec67700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:32.710501 7f210ec67700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:884 1.3 1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head v 23'287) v1 -- 0x7f2140ff8c00 con 0
2016-10-25 08:51:32.710752 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'286 (0'0,23'286] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] append_log log((0'0,23'286], crt=23'283) [23'287 (0'0) modify   1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head by client.4144.0:884 2016-10-25 08:51:32.704082 0]
2016-10-25 08:51:32.710764 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] add_log_entry 23'287 (0'0) modify   1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head by client.4144.0:884 2016-10-25 08:51:32.704082 0
2016-10-25 08:51:32.710772 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] append_log: trimming to 23'283 entries 
2016-10-25 08:51:32.710781 7f210ec67700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'287, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:32.710969 7f210ec67700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139241e40 osr(1.3 0x7f2139475570)
2016-10-25 08:51:32.710976 7f210ec67700 10 journal prepare_entry [Transaction(0x7f2147edea80)]
2016-10-25 08:51:32.711203 7f210ec67700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:32.711951 7f210ec67700 10 journal op_submit_start 2366
2016-10-25 08:51:32.711955 7f210ec67700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2366 [Transaction(0x7f2147edea80)]
2016-10-25 08:51:32.711958 7f210ec67700 10 journal op_journal_transactions 2366 reqid_t client.4144.0:884
2016-10-25 08:51:32.711960 7f210ec67700  5 journal submit_entry seq 2366 len 4202496 (0x7f21430983d0)
2016-10-25 08:51:32.711982 7f210ec67700 10 journal op_submit_finish 2366
2016-10-25 08:51:32.711985 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] eval_repop repgather(0x7f21397be760 23'287 rep_tid=1727 committed?=0 applied?=0) wants=d
2016-10-25 08:51:32.712001 7f210ec67700 10 osd.3 23 dequeue_op 0x7f21398810e0 finish
2016-10-25 08:51:32.712011 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:32.712040 7f2123404700 10 journal room 929820671 max_size 1072709632 pos 1013071872 header.start 870187008 top 4096
2016-10-25 08:51:32.712045 7f2123404700 10 journal check_for_full at 1013071872 : 4202496 < 929820671
2016-10-25 08:51:32.712049 7f2123404700 15 journal prepare_single_write 1 will write 1013071872 : seq 2366 len 4195598 -> 4202496
2016-10-25 08:51:32.712107 7f2123404700 20 journal prepare_multi_write queue_pos now 1017274368
2016-10-25 08:51:32.712132 7f2123404700 15 journal do_aio_write writing 1013071872~4202496 + header
2016-10-25 08:51:32.712136 7f2123404700 20 journal write_aio_bl 0~4096 seq 0
2016-10-25 08:51:32.712153 7f2123404700 20 journal write_aio_bl .. 0~4096 in 1
2016-10-25 08:51:32.712216 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:32.712224 7f2123404700 20 journal write_aio_bl 1013071872~4202496 seq 2366
2016-10-25 08:51:32.712228 7f2123404700 20 journal write_aio_bl .. 1013071872~4202496 in 1
2016-10-25 08:51:32.712604 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:32.712620 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:32.712624 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:32.712638 7f2122c03700 20 journal write_finish_thread_entry waiting for aio(s)
2016-10-25 08:51:32.727156 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2292 ==== osd_repop(client.4144.0:883 1.2) v1 ==== 1034+0+4195023 (617808000 0 3620516913) 0x7f21480fde00 con 0x7f2139592800
2016-10-25 08:51:32.727221 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop(client.4144.0:883 1.2) v1 epoch 23
2016-10-25 08:51:32.727236 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:32.727250 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139881f80 prio 127 cost 4195023 latency 0.012036 osd_repop(client.4144.0:883 1.2) v1
2016-10-25 08:51:32.727290 7f210f468700 10 osd.3 23 dequeue_op 0x7f2139881f80 prio 127 cost 4195023 latency 0.012076 osd_repop(client.4144.0:883 1.2) v1 pg pg[1.2( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 crt=23'275 lcod 23'276 active]
2016-10-25 08:51:32.727310 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.2( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 crt=23'275 lcod 23'276 active] handle_message: 0x7f2139881f80
2016-10-25 08:51:32.727326 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.2( v 23'277 (0'0,23'277] local-les=22 n=277 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 crt=23'275 lcod 23'276 active] sub_op_modify trans 1:577853ca:::benchmark_data_target167114251239.teuthology_13304_object882:head v 23'278 (transaction) 200
2016-10-25 08:51:32.727350 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.2( v 23'277 (0'0,23'277] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 crt=23'275 lcod 23'276 active] append_log log((0'0,23'277], crt=23'275) [23'278 (0'0) modify   1:577853ca:::benchmark_data_target167114251239.teuthology_13304_object882:head by client.4144.0:883 2016-10-25 08:51:32.084747 0]
2016-10-25 08:51:32.727360 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.2( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'277 crt=23'275 lcod 23'276 active] add_log_entry 23'278 (0'0) modify   1:577853ca:::benchmark_data_target167114251239.teuthology_13304_object882:head by client.4144.0:883 2016-10-25 08:51:32.084747 0
2016-10-25 08:51:32.727368 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.2( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'277 crt=23'275 lcod 23'276 active] append_log: trimming to 23'275 entries 
2016-10-25 08:51:32.727378 7f210f468700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'278, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:32.727583 7f210f468700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f21394ba580 osr(1.2 0x7f2139474dc0)
2016-10-25 08:51:32.727589 7f210f468700 10 journal prepare_entry [Transaction(0x7f2147f6ce00),Transaction(0x7f2147f6cfc0)]
2016-10-25 08:51:32.728096 7f210f468700 10 journal  len 4195682 -> 4202496 (head 40 pre_pad 3387 bl 4195682 post_pad 3347 tail 40) (bl alignment 3427)
2016-10-25 08:51:32.728770 7f210f468700 10 journal op_submit_start 2367
2016-10-25 08:51:32.728773 7f210f468700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2367 [Transaction(0x7f2147f6ce00),Transaction(0x7f2147f6cfc0)]
2016-10-25 08:51:32.728777 7f210f468700 10 journal op_journal_transactions 2367 reqid_t client.4144.0:883
2016-10-25 08:51:32.728779 7f210f468700  5 journal submit_entry seq 2367 len 4202496 (0x7f2140fb6640)
2016-10-25 08:51:32.728797 7f210f468700 10 journal op_submit_finish 2367
2016-10-25 08:51:32.728801 7f210f468700 10 osd.3 23 dequeue_op 0x7f2139881f80 finish
2016-10-25 08:51:32.728888 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:32.728891 7f2123404700 20 journal write_thread_entry aio throttle: aio num 2 bytes 4206592 ... exp 4 min_new 16 ... pending 4202496
2016-10-25 08:51:32.728894 7f2123404700 10 journal room 925618175 max_size 1072709632 pos 1017274368 header.start 870187008 top 4096
2016-10-25 08:51:32.728896 7f2123404700 10 journal check_for_full at 1017274368 : 4202496 < 925618175
2016-10-25 08:51:32.728897 7f2123404700 15 journal prepare_single_write 1 will write 1017274368 : seq 2367 len 4195682 -> 4202496
2016-10-25 08:51:32.728906 7f2123404700 20 journal prepare_multi_write queue_pos now 1021476864
2016-10-25 08:51:32.728907 7f2123404700 15 journal do_aio_write writing 1017274368~4202496
2016-10-25 08:51:32.728909 7f2123404700 20 journal write_aio_bl 1017274368~4202496 seq 2367
2016-10-25 08:51:32.728912 7f2123404700 20 journal write_aio_bl .. 1017274368~4202496 in 1
2016-10-25 08:51:32.729169 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:32.729172 7f2123404700  5 journal complete_write finished 1 ops and 4195682 bytes
2016-10-25 08:51:32.729173 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:32.737872 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2293 ==== osd_repop_reply(client.4144.0:867 1.6) v1 ==== 83+0+0 (3724476839 0 0) 0x7f213ea46c80 con 0x7f2139592800
2016-10-25 08:51:32.737924 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:867 1.6) v1 epoch 23
2016-10-25 08:51:32.737938 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:32.737953 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139882d00 prio 196 cost 0 latency 0.000117 osd_repop_reply(client.4144.0:867 1.6) v1
2016-10-25 08:51:32.737987 7f211246e700 10 osd.3 23 dequeue_op 0x7f2139882d00 prio 196 cost 0 latency 0.000152 osd_repop_reply(client.4144.0:867 1.6) v1 pg pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'273 lcod 23'273 mlcod 23'273 active+clean]
2016-10-25 08:51:32.738005 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'273 lcod 23'273 mlcod 23'273 active+clean] handle_message: 0x7f2139882d00
2016-10-25 08:51:32.738016 7f211246e700  7 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'273 lcod 23'273 mlcod 23'273 active+clean] sub_op_modify_reply: tid 1714 op  ack_type 4 from 2
2016-10-25 08:51:32.738035 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'273 lcod 23'273 mlcod 23'273 active+clean] repop_all_applied: repop tid 1714 all applied 
2016-10-25 08:51:32.738040 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'273 lcod 23'273 mlcod 23'273 active+clean] eval_repop repgather(0x7f2139719320 23'275 rep_tid=1714 committed?=0 applied?=1) wants=d
2016-10-25 08:51:32.738047 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'273 lcod 23'273 mlcod 23'273 active+clean]  applied: repgather(0x7f2139719320 23'275 rep_tid=1714 committed?=0 applied?=1) 
2016-10-25 08:51:32.738054 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'274 crt=23'273 lcod 23'273 mlcod 23'273 active+clean] repop_all_committed: repop tid 1714 all committed 
2016-10-25 08:51:32.738059 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'275 crt=23'273 lcod 23'274 mlcod 23'273 active+clean] eval_repop repgather(0x7f2139719320 23'275 rep_tid=1714 committed?=1 applied?=1) wants=d
2016-10-25 08:51:32.738063 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'275 crt=23'273 lcod 23'274 mlcod 23'273 active+clean]  commit: repgather(0x7f2139719320 23'275 rep_tid=1714 committed?=1 applied?=1)
2016-10-25 08:51:32.738070 7f211246e700 15 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'275 crt=23'273 lcod 23'274 mlcod 23'273 active+clean] log_op_stats osd_op(client.4144.0:867 1.7058eed6 benchmark_data_target167114251239.teuthology_13304_object866 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 1.378250 lat 1.378265
2016-10-25 08:51:32.738077 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'275 crt=23'273 lcod 23'274 mlcod 23'273 active+clean]  sending commit on osd_op(client.4144.0:867 1.7058eed6 benchmark_data_target167114251239.teuthology_13304_object866 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f2141376840
2016-10-25 08:51:32.738084 7f211246e700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(867 benchmark_data_target167114251239.teuthology_13304_object866 [write 0~4194304] v23'275 uv275 ondisk = 0) v7 -- 0x7f2141376840 con 0
2016-10-25 08:51:32.738097 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'275 crt=23'273 lcod 23'274 mlcod 23'273 active+clean]  applied: repgather(0x7f2139719320 23'275 rep_tid=1714 committed?=1 applied?=1) 
2016-10-25 08:51:32.738105 7f211246e700 15 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'275 crt=23'273 lcod 23'274 mlcod 23'273 active+clean] publish_stats_to_osd 23:136
2016-10-25 08:51:32.738111 7f211246e700 15 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'275 crt=23'273 lcod 23'274 mlcod 23'274 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:32.738117 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'275 crt=23'273 lcod 23'274 mlcod 23'274 active+clean]  removing repgather(0x7f2139719320 23'275 rep_tid=1714 committed?=1 applied?=1)
2016-10-25 08:51:32.738121 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'275 crt=23'273 lcod 23'274 mlcod 23'274 active+clean]    q front is repgather(0x7f2139719320 23'275 rep_tid=1714 committed?=1 applied?=1)
2016-10-25 08:51:32.738126 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'275 crt=23'273 lcod 23'274 mlcod 23'274 active+clean] remove_repop repgather(0x7f2139719320 23'275 rep_tid=1714 committed?=1 applied?=1)
2016-10-25 08:51:32.738134 7f211246e700 15 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'275 crt=23'273 lcod 23'274 mlcod 23'274 active+clean]  requeue_ops 
2016-10-25 08:51:32.738153 7f211246e700 10 osd.3 23 dequeue_op 0x7f2139882d00 finish
2016-10-25 08:51:32.742434 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 681 ==== osd_op(client.4144.0:885 1.7241a548 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (832611940 0 1561453814) 0x7f2141376840 con 0x7f2147f52800
2016-10-25 08:51:32.742494 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:32.742511 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139882d00 prio 63 cost 4194304 latency 0.002761 osd_op(client.4144.0:885 1.7241a548 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:32.742611 7f2112c6f700 10 osd.3 23 dequeue_op 0x7f2139882d00 prio 63 cost 4194304 latency 0.002861 osd_op(client.4144.0:885 1.7241a548 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean]
2016-10-25 08:51:32.742646 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] handle_message: 0x7f2139882d00
2016-10-25 08:51:32.742665 7f2112c6f700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:32.742683 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] do_op osd_op(client.4144.0:885 1.7241a548 benchmark_data_target167114251239.teuthology_13304_object884 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:32.742704 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] get_object_context: obc NOT found in cache: 1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head
2016-10-25 08:51:32.742715 7f2112c6f700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head# '_'
2016-10-25 08:51:32.742758 7f2112c6f700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.0_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject884__head_7241A548__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:32.742766 7f2112c6f700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head# '_' = -2
2016-10-25 08:51:32.742769 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] get_object_context: no obc for soid 1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head but can_create
2016-10-25 08:51:32.742779 7f2112c6f700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:snapdir# 'snapset'
2016-10-25 08:51:32.742793 7f2112c6f700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.0_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject884__snapdir_7241A548__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:32.742797 7f2112c6f700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:snapdir# 'snapset' = -2
2016-10-25 08:51:32.742825 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] create_object_context 0x7f2140fcaec0 1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head 
2016-10-25 08:51:32.742834 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] populate_obc_watchers 1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head
2016-10-25 08:51:32.742841 7f2112c6f700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head
2016-10-25 08:51:32.742846 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] get_object_context: 0x7f2140fcaec0 1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head rwstate(none n=0 w=0) oi: 1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f213976a220 snapset: 0=[]:[]
2016-10-25 08:51:32.742856 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] find_object_context 1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head @head oi=1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:32.742865 7f2112c6f700 25 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] do_op oi 1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:32.742877 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] get_object_context: obc NOT found in cache: 1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:snapdir
2016-10-25 08:51:32.742884 7f2112c6f700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:snapdir# '_'
2016-10-25 08:51:32.742900 7f2112c6f700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.0_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject884__snapdir_7241A548__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:32.742903 7f2112c6f700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:snapdir# '_' = -2
2016-10-25 08:51:32.742905 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] get_object_context: no obc for soid 1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:snapdir and !can_create
2016-10-25 08:51:32.742921 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] execute_ctx 0x7f2147e26700
2016-10-25 08:51:32.742932 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] do_op 1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head [write 0~4194304] ov 0'0 av 23'286 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:32.742946 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] do_osd_op 1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head [write 0~4194304]
2016-10-25 08:51:32.742956 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:32.743276 7f2112c6f700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] make_writeable 1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head snapset=0x7f213976a258  snapc=0=[]
2016-10-25 08:51:32.743287 7f2112c6f700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean]  setting DIRTY flag
2016-10-25 08:51:32.743295 7f2112c6f700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] make_writeable 1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head done, snapset=0=[]:[]+head
2016-10-25 08:51:32.743306 7f2112c6f700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] finish_ctx 1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head 0x7f2147e26700 op modify  
2016-10-25 08:51:32.743328 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean]  set mtime to 2016-10-25 08:51:32.738522
2016-10-25 08:51:32.743345 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean]  final snapset 0=[]:[]+head in 1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head
2016-10-25 08:51:32.743361 7f2112c6f700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean]  zeroing write result code 0
2016-10-25 08:51:32.743372 7f2112c6f700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean]  op order client.4144 tid 885 (first)
2016-10-25 08:51:32.743378 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] new_repop rep_tid 1728 on osd_op(client.4144.0:885 1.7241a548 benchmark_data_target167114251239.teuthology_13304_object884 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:32.743386 7f2112c6f700  7 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] issue_repop rep_tid 1728 o 1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head
2016-10-25 08:51:32.743439 7f2112c6f700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:32.743442 7f2112c6f700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:885 1.0 1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head v 23'286) v1 -- 0x7f2140ff4000 con 0
2016-10-25 08:51:32.744512 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'285 (0'0,23'285] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] append_log log((0'0,23'285], crt=23'279) [23'286 (0'0) modify   1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head by client.4144.0:885 2016-10-25 08:51:32.738522 0]
2016-10-25 08:51:32.744535 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'279 lcod 23'284 mlcod 23'284 active+clean] add_log_entry 23'286 (0'0) modify   1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head by client.4144.0:885 2016-10-25 08:51:32.738522 0
2016-10-25 08:51:32.744560 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] append_log: trimming to 23'284 entries 23'280 (0'0) modify   1:126b343e:::benchmark_data_target167114251239.teuthology_13304_object847:head by client.4144.0:848 2016-10-25 08:51:30.199586 0,23'281 (0'0) modify   1:0bbe1d8e:::benchmark_data_target167114251239.teuthology_13304_object855:head by client.4144.0:856 2016-10-25 08:51:30.215153 0,23'282 (0'0) modify   1:1f23fc2f:::benchmark_data_target167114251239.teuthology_13304_object856:head by client.4144.0:857 2016-10-25 08:51:30.215243 0,23'283 (0'0) modify   1:1a36f7c7:::benchmark_data_target167114251239.teuthology_13304_object858:head by client.4144.0:859 2016-10-25 08:51:30.215422 0,23'284 (0'0) modify   1:19e24124:::benchmark_data_target167114251239.teuthology_13304_object859:head by client.4144.0:860 2016-10-25 08:51:31.281089 0
2016-10-25 08:51:32.744594 7f2112c6f700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'286, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:32.744876 7f2112c6f700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139241080 osr(1.0 0x7f2139475d80)
2016-10-25 08:51:32.744886 7f2112c6f700 10 journal prepare_entry [Transaction(0x7f2147e38840)]
2016-10-25 08:51:32.745179 7f2112c6f700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:32.746122 7f2112c6f700 10 journal op_submit_start 2368
2016-10-25 08:51:32.746126 7f2112c6f700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2368 [Transaction(0x7f2147e38840)]
2016-10-25 08:51:32.746129 7f2112c6f700 10 journal op_journal_transactions 2368 reqid_t client.4144.0:885
2016-10-25 08:51:32.746132 7f2112c6f700  5 journal submit_entry seq 2368 len 4202496 (0x7f214811f600)
2016-10-25 08:51:32.746160 7f2112c6f700 10 journal op_submit_finish 2368
2016-10-25 08:51:32.746168 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:32.746174 7f2123404700 20 journal write_thread_entry aio throttle: aio num 3 bytes 8409088 ... exp 6 min_new 64 ... pending 4202496
2016-10-25 08:51:32.746164 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] eval_repop repgather(0x7f2139880240 23'286 rep_tid=1728 committed?=0 applied?=0) wants=d
2016-10-25 08:51:32.746178 7f2123404700 10 journal room 921415679 max_size 1072709632 pos 1021476864 header.start 870187008 top 4096
2016-10-25 08:51:32.746180 7f2123404700 10 journal check_for_full at 1021476864 : 4202496 < 921415679
2016-10-25 08:51:32.746182 7f2123404700 15 journal prepare_single_write 1 will write 1021476864 : seq 2368 len 4195598 -> 4202496
2016-10-25 08:51:32.746183 7f2112c6f700 10 osd.3 23 dequeue_op 0x7f2139882d00 finish
2016-10-25 08:51:32.746196 7f2123404700 20 journal prepare_multi_write queue_pos now 1025679360
2016-10-25 08:51:32.746198 7f2123404700 15 journal do_aio_write writing 1021476864~4202496
2016-10-25 08:51:32.746199 7f2123404700 20 journal write_aio_bl 1021476864~4202496 seq 2368
2016-10-25 08:51:32.746202 7f2123404700 20 journal write_aio_bl .. 1021476864~4202496 in 1
2016-10-25 08:51:32.755186 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2294 ==== osd_repop_reply(client.4144.0:868 1.1) v1 ==== 83+0+0 (812022574 0 0) 0x7f213ea46c80 con 0x7f2139592800
2016-10-25 08:51:32.755216 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:868 1.1) v1 epoch 23
2016-10-25 08:51:32.755221 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:32.755227 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139880360 prio 196 cost 0 latency 0.000074 osd_repop_reply(client.4144.0:868 1.1) v1
2016-10-25 08:51:32.755253 7f210fc69700 10 osd.3 23 dequeue_op 0x7f2139880360 prio 196 cost 0 latency 0.000100 osd_repop_reply(client.4144.0:868 1.1) v1 pg pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 crt=23'321 lcod 23'321 mlcod 23'321 active+clean]
2016-10-25 08:51:32.755269 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 crt=23'321 lcod 23'321 mlcod 23'321 active+clean] handle_message: 0x7f2139880360
2016-10-25 08:51:32.755281 7f210fc69700  7 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 crt=23'321 lcod 23'321 mlcod 23'321 active+clean] sub_op_modify_reply: tid 1715 op  ack_type 4 from 2
2016-10-25 08:51:32.755300 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 crt=23'321 lcod 23'321 mlcod 23'321 active+clean] repop_all_applied: repop tid 1715 all applied 
2016-10-25 08:51:32.755306 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 crt=23'321 lcod 23'321 mlcod 23'321 active+clean] eval_repop repgather(0x7f213981cfc0 23'323 rep_tid=1715 committed?=0 applied?=1) wants=d
2016-10-25 08:51:32.755313 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 crt=23'321 lcod 23'321 mlcod 23'321 active+clean]  applied: repgather(0x7f213981cfc0 23'323 rep_tid=1715 committed?=0 applied?=1) 
2016-10-25 08:51:32.755320 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'322 crt=23'321 lcod 23'321 mlcod 23'321 active+clean] repop_all_committed: repop tid 1715 all committed 
2016-10-25 08:51:32.755324 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'323 crt=23'321 lcod 23'322 mlcod 23'321 active+clean] eval_repop repgather(0x7f213981cfc0 23'323 rep_tid=1715 committed?=1 applied?=1) wants=d
2016-10-25 08:51:32.755329 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'323 crt=23'321 lcod 23'322 mlcod 23'321 active+clean]  commit: repgather(0x7f213981cfc0 23'323 rep_tid=1715 committed?=1 applied?=1)
2016-10-25 08:51:32.755336 7f210fc69700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'323 crt=23'321 lcod 23'322 mlcod 23'321 active+clean] log_op_stats osd_op(client.4144.0:868 1.6030949 benchmark_data_target167114251239.teuthology_13304_object867 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 1.385470 lat 1.385486
2016-10-25 08:51:32.755342 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'323 crt=23'321 lcod 23'322 mlcod 23'321 active+clean]  sending commit on osd_op(client.4144.0:868 1.6030949 benchmark_data_target167114251239.teuthology_13304_object867 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f213ea578c0
2016-10-25 08:51:32.755349 7f210fc69700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(868 benchmark_data_target167114251239.teuthology_13304_object867 [write 0~4194304] v23'323 uv323 ondisk = 0) v7 -- 0x7f213ea578c0 con 0
2016-10-25 08:51:32.755360 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'323 crt=23'321 lcod 23'322 mlcod 23'321 active+clean]  applied: repgather(0x7f213981cfc0 23'323 rep_tid=1715 committed?=1 applied?=1) 
2016-10-25 08:51:32.755367 7f210fc69700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'323 crt=23'321 lcod 23'322 mlcod 23'321 active+clean] publish_stats_to_osd 23:164
2016-10-25 08:51:32.755373 7f210fc69700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'323 crt=23'321 lcod 23'322 mlcod 23'322 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:32.755379 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'323 crt=23'321 lcod 23'322 mlcod 23'322 active+clean]  removing repgather(0x7f213981cfc0 23'323 rep_tid=1715 committed?=1 applied?=1)
2016-10-25 08:51:32.755383 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'323 crt=23'321 lcod 23'322 mlcod 23'322 active+clean]    q front is repgather(0x7f213981cfc0 23'323 rep_tid=1715 committed?=1 applied?=1)
2016-10-25 08:51:32.755388 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'323 crt=23'321 lcod 23'322 mlcod 23'322 active+clean] remove_repop repgather(0x7f213981cfc0 23'323 rep_tid=1715 committed?=1 applied?=1)
2016-10-25 08:51:32.755397 7f210fc69700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'323 crt=23'321 lcod 23'322 mlcod 23'322 active+clean]  requeue_ops 
2016-10-25 08:51:32.755418 7f210fc69700 10 osd.3 23 dequeue_op 0x7f2139880360 finish
2016-10-25 08:51:32.758196 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 682 ==== osd_op(client.4144.0:886 1.fcb15813 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (2020739969 0 2527684533) 0x7f213ea578c0 con 0x7f2147f52800
2016-10-25 08:51:32.758250 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:32.758264 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139880360 prio 63 cost 4194304 latency 0.001762 osd_op(client.4144.0:886 1.fcb15813 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:32.758301 7f211146c700 10 osd.3 23 dequeue_op 0x7f2139880360 prio 63 cost 4194304 latency 0.001799 osd_op(client.4144.0:886 1.fcb15813 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean]
2016-10-25 08:51:32.758321 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] handle_message: 0x7f2139880360
2016-10-25 08:51:32.758336 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:32.758353 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] do_op osd_op(client.4144.0:886 1.fcb15813 benchmark_data_target167114251239.teuthology_13304_object885 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:32.758379 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] get_object_context: obc NOT found in cache: 1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head
2016-10-25 08:51:32.758388 7f211146c700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head# '_'
2016-10-25 08:51:32.758435 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject885__head_FCB15813__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:32.758442 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head# '_' = -2
2016-10-25 08:51:32.758445 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] get_object_context: no obc for soid 1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head but can_create
2016-10-25 08:51:32.758455 7f211146c700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:snapdir# 'snapset'
2016-10-25 08:51:32.758469 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject885__snapdir_FCB15813__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:32.758471 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:snapdir# 'snapset' = -2
2016-10-25 08:51:32.758490 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] create_object_context 0x7f2142e12c00 1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head 
2016-10-25 08:51:32.758497 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] populate_obc_watchers 1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head
2016-10-25 08:51:32.758503 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head
2016-10-25 08:51:32.758509 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] get_object_context: 0x7f2142e12c00 1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head rwstate(none n=0 w=0) oi: 1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f2139879f80 snapset: 0=[]:[]
2016-10-25 08:51:32.758518 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] find_object_context 1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head @head oi=1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:32.758526 7f211146c700 25 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] do_op oi 1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:32.758534 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] get_object_context: obc NOT found in cache: 1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:snapdir
2016-10-25 08:51:32.758540 7f211146c700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:snapdir# '_'
2016-10-25 08:51:32.758552 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject885__snapdir_FCB15813__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:32.758555 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:snapdir# '_' = -2
2016-10-25 08:51:32.758557 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] get_object_context: no obc for soid 1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:snapdir and !can_create
2016-10-25 08:51:32.758570 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] execute_ctx 0x7f214303c700
2016-10-25 08:51:32.758577 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] do_op 1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head [write 0~4194304] ov 0'0 av 23'288 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:32.758592 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] do_osd_op 1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head [write 0~4194304]
2016-10-25 08:51:32.758597 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:32.758832 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] make_writeable 1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head snapset=0x7f2139879fb8  snapc=0=[]
2016-10-25 08:51:32.758838 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean]  setting DIRTY flag
2016-10-25 08:51:32.758843 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] make_writeable 1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head done, snapset=0=[]:[]+head
2016-10-25 08:51:32.758848 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] finish_ctx 1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head 0x7f214303c700 op modify  
2016-10-25 08:51:32.758857 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean]  set mtime to 2016-10-25 08:51:32.755751
2016-10-25 08:51:32.758868 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean]  final snapset 0=[]:[]+head in 1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head
2016-10-25 08:51:32.758880 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean]  zeroing write result code 0
2016-10-25 08:51:32.758888 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean]  op order client.4144 tid 886 (first)
2016-10-25 08:51:32.758893 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] new_repop rep_tid 1729 on osd_op(client.4144.0:886 1.fcb15813 benchmark_data_target167114251239.teuthology_13304_object885 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:32.758901 7f211146c700  7 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] issue_repop rep_tid 1729 o 1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head
2016-10-25 08:51:32.758929 7f211146c700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:32.758931 7f211146c700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:886 1.3 1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head v 23'288) v1 -- 0x7f21480fd800 con 0
2016-10-25 08:51:32.759174 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'287 (0'0,23'287] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] append_log log((0'0,23'287], crt=23'283) [23'288 (0'0) modify   1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head by client.4144.0:886 2016-10-25 08:51:32.755751 0]
2016-10-25 08:51:32.759185 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] add_log_entry 23'288 (0'0) modify   1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head by client.4144.0:886 2016-10-25 08:51:32.755751 0
2016-10-25 08:51:32.759192 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] append_log: trimming to 23'283 entries 
2016-10-25 08:51:32.759201 7f211146c700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'288, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:32.759375 7f211146c700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139241e40 osr(1.3 0x7f2139475570)
2016-10-25 08:51:32.759379 7f211146c700 10 journal prepare_entry [Transaction(0x7f2140fd3d40)]
2016-10-25 08:51:32.759631 7f211146c700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:32.760428 7f211146c700 10 journal op_submit_start 2369
2016-10-25 08:51:32.760431 7f211146c700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2369 [Transaction(0x7f2140fd3d40)]
2016-10-25 08:51:32.760434 7f211146c700 10 journal op_journal_transactions 2369 reqid_t client.4144.0:886
2016-10-25 08:51:32.760436 7f211146c700  5 journal submit_entry seq 2369 len 4202496 (0x7f21414acbb0)
2016-10-25 08:51:32.760448 7f211146c700 10 journal op_submit_finish 2369
2016-10-25 08:51:32.760451 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] eval_repop repgather(0x7f21397169a0 23'288 rep_tid=1729 committed?=0 applied?=0) wants=d
2016-10-25 08:51:32.760469 7f211146c700 10 osd.3 23 dequeue_op 0x7f2139880360 finish
2016-10-25 08:51:32.776628 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:32.776644 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:32.776646 7f2123404700 20 journal write_thread_entry aio throttle: aio num 4 bytes 12611584 ... exp 8 min_new 256 ... pending 4202496
2016-10-25 08:51:32.776650 7f2123404700 10 journal room 917213183 max_size 1072709632 pos 1025679360 header.start 870187008 top 4096
2016-10-25 08:51:32.776654 7f2123404700 10 journal check_for_full at 1025679360 : 4202496 < 917213183
2016-10-25 08:51:32.776656 7f2123404700 15 journal prepare_single_write 1 will write 1025679360 : seq 2369 len 4195598 -> 4202496
2016-10-25 08:51:32.776677 7f2123404700 20 journal prepare_multi_write queue_pos now 1029881856
2016-10-25 08:51:32.776678 7f2123404700 15 journal do_aio_write writing 1025679360~4202496
2016-10-25 08:51:32.776680 7f2123404700 20 journal write_aio_bl 1025679360~4202496 seq 2369
2016-10-25 08:51:32.776683 7f2123404700 20 journal write_aio_bl .. 1025679360~4202496 in 1
2016-10-25 08:51:32.776916 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:32.776918 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:32.776919 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:33.022213 7f2125c09700 10 osd.3 23 tick
2016-10-25 08:51:33.022239 7f2125c09700 10 osd.3 23 do_waiters -- start
2016-10-25 08:51:33.022241 7f2125c09700 10 osd.3 23 do_waiters -- finish
2016-10-25 08:51:33.175058 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2295 ==== osd_repop_reply(client.4144.0:877 1.3) v1 ==== 83+0+0 (3676149458 0 0) 0x7f213ea46c80 con 0x7f2139592800
2016-10-25 08:51:33.175106 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:877 1.3) v1 epoch 23
2016-10-25 08:51:33.175113 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:33.175123 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f21398819e0 prio 196 cost 0 latency 0.000116 osd_repop_reply(client.4144.0:877 1.3) v1
2016-10-25 08:51:33.175183 7f210ec67700 10 osd.3 23 dequeue_op 0x7f21398819e0 prio 196 cost 0 latency 0.000176 osd_repop_reply(client.4144.0:877 1.3) v1 pg pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean]
2016-10-25 08:51:33.175221 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] handle_message: 0x7f21398819e0
2016-10-25 08:51:33.175242 7f210ec67700  7 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] sub_op_modify_reply: tid 1721 op  ack_type 4 from 2
2016-10-25 08:51:33.175272 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] repop_all_applied: repop tid 1721 all applied 
2016-10-25 08:51:33.175283 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] eval_repop repgather(0x7f2139880900 23'285 rep_tid=1721 committed?=0 applied?=1) wants=d
2016-10-25 08:51:33.175294 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean]  applied: repgather(0x7f2139880900 23'285 rep_tid=1721 committed?=0 applied?=1) 
2016-10-25 08:51:33.175308 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'286 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] repop_all_committed: repop tid 1721 all committed 
2016-10-25 08:51:33.175317 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'286 crt=23'283 lcod 23'284 mlcod 23'283 active+clean] eval_repop repgather(0x7f2139880900 23'285 rep_tid=1721 committed?=1 applied?=1) wants=d
2016-10-25 08:51:33.175327 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'286 crt=23'283 lcod 23'284 mlcod 23'283 active+clean]  commit: repgather(0x7f2139880900 23'285 rep_tid=1721 committed?=1 applied?=1)
2016-10-25 08:51:33.175340 7f210ec67700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'286 crt=23'283 lcod 23'284 mlcod 23'283 active+clean] log_op_stats osd_op(client.4144.0:877 1.619240b benchmark_data_target167114251239.teuthology_13304_object876 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 1.530239 lat 1.530270
2016-10-25 08:51:33.175354 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'286 crt=23'283 lcod 23'284 mlcod 23'283 active+clean]  sending commit on osd_op(client.4144.0:877 1.619240b benchmark_data_target167114251239.teuthology_13304_object876 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f213c575e40
2016-10-25 08:51:33.175367 7f210ec67700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(877 benchmark_data_target167114251239.teuthology_13304_object876 [write 0~4194304] v23'285 uv285 ondisk = 0) v7 -- 0x7f213c575e40 con 0
2016-10-25 08:51:33.175405 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'286 crt=23'283 lcod 23'284 mlcod 23'283 active+clean]  applied: repgather(0x7f2139880900 23'285 rep_tid=1721 committed?=1 applied?=1) 
2016-10-25 08:51:33.175422 7f210ec67700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'286 crt=23'283 lcod 23'284 mlcod 23'283 active+clean] publish_stats_to_osd 23:153
2016-10-25 08:51:33.175433 7f210ec67700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'286 crt=23'283 lcod 23'284 mlcod 23'284 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:33.175444 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'286 crt=23'283 lcod 23'284 mlcod 23'284 active+clean]  removing repgather(0x7f2139880900 23'285 rep_tid=1721 committed?=1 applied?=1)
2016-10-25 08:51:33.175453 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'286 crt=23'283 lcod 23'284 mlcod 23'284 active+clean]    q front is repgather(0x7f2139880900 23'285 rep_tid=1721 committed?=1 applied?=1)
2016-10-25 08:51:33.175463 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'286 crt=23'283 lcod 23'284 mlcod 23'284 active+clean] remove_repop repgather(0x7f2139880900 23'285 rep_tid=1721 committed?=1 applied?=1)
2016-10-25 08:51:33.175479 7f210ec67700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'286 crt=23'283 lcod 23'284 mlcod 23'284 active+clean]  requeue_ops 
2016-10-25 08:51:33.175544 7f210ec67700 10 osd.3 23 dequeue_op 0x7f21398819e0 finish
2016-10-25 08:51:33.180275 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 683 ==== osd_op(client.4144.0:887 1.aad380f0 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (1154648345 0 3482096769) 0x7f213c575e40 con 0x7f2147f52800
2016-10-25 08:51:33.180328 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:33.180338 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139883180 prio 63 cost 4194304 latency 0.001816 osd_op(client.4144.0:887 1.aad380f0 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:33.180385 7f211046a700 10 osd.3 23 dequeue_op 0x7f2139883180 prio 63 cost 4194304 latency 0.001863 osd_op(client.4144.0:887 1.aad380f0 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean]
2016-10-25 08:51:33.180426 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] handle_message: 0x7f2139883180
2016-10-25 08:51:33.180450 7f211046a700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:33.180475 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] do_op osd_op(client.4144.0:887 1.aad380f0 benchmark_data_target167114251239.teuthology_13304_object886 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:33.180505 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] get_object_context: obc NOT found in cache: 1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head
2016-10-25 08:51:33.180520 7f211046a700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head# '_'
2016-10-25 08:51:33.180580 7f211046a700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.0_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject886__head_AAD380F0__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:33.180590 7f211046a700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head# '_' = -2
2016-10-25 08:51:33.180595 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] get_object_context: no obc for soid 1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head but can_create
2016-10-25 08:51:33.180610 7f211046a700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:snapdir# 'snapset'
2016-10-25 08:51:33.180633 7f211046a700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.0_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject886__snapdir_AAD380F0__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:33.180637 7f211046a700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:snapdir# 'snapset' = -2
2016-10-25 08:51:33.180659 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] create_object_context 0x7f2142efb180 1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head 
2016-10-25 08:51:33.180672 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] populate_obc_watchers 1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head
2016-10-25 08:51:33.180684 7f211046a700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head
2016-10-25 08:51:33.180695 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] get_object_context: 0x7f2142efb180 1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head rwstate(none n=0 w=0) oi: 1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f21394e5b20 snapset: 0=[]:[]
2016-10-25 08:51:33.180712 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] find_object_context 1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head @head oi=1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:33.180728 7f211046a700 25 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] do_op oi 1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:33.180744 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] get_object_context: obc NOT found in cache: 1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:snapdir
2016-10-25 08:51:33.180755 7f211046a700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:snapdir# '_'
2016-10-25 08:51:33.180775 7f211046a700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.0_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject886__snapdir_AAD380F0__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:33.180780 7f211046a700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.0_head/#1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:snapdir# '_' = -2
2016-10-25 08:51:33.180783 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] get_object_context: no obc for soid 1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:snapdir and !can_create
2016-10-25 08:51:33.180806 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] execute_ctx 0x7f213edb0a00
2016-10-25 08:51:33.180818 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] do_op 1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head [write 0~4194304] ov 0'0 av 23'287 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:33.180833 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] do_osd_op 1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head [write 0~4194304]
2016-10-25 08:51:33.180858 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:33.181117 7f211046a700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] make_writeable 1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head snapset=0x7f21394e5b58  snapc=0=[]
2016-10-25 08:51:33.181131 7f211046a700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean]  setting DIRTY flag
2016-10-25 08:51:33.181140 7f211046a700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] make_writeable 1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head done, snapset=0=[]:[]+head
2016-10-25 08:51:33.181152 7f211046a700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] finish_ctx 1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head 0x7f213edb0a00 op modify  
2016-10-25 08:51:33.181166 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean]  set mtime to 2016-10-25 08:51:33.175884
2016-10-25 08:51:33.181185 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=286 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean]  final snapset 0=[]:[]+head in 1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head
2016-10-25 08:51:33.181208 7f211046a700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean]  zeroing write result code 0
2016-10-25 08:51:33.181221 7f211046a700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean]  op order client.4144 tid 887 (first)
2016-10-25 08:51:33.181232 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] new_repop rep_tid 1730 on osd_op(client.4144.0:887 1.aad380f0 benchmark_data_target167114251239.teuthology_13304_object886 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:33.181247 7f211046a700  7 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] issue_repop rep_tid 1730 o 1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head
2016-10-25 08:51:33.181292 7f211046a700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:33.181294 7f211046a700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:887 1.0 1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head v 23'287) v1 -- 0x7f214103ea00 con 0
2016-10-25 08:51:33.181584 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'286 (0'0,23'286] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] append_log log((0'0,23'286], crt=23'284) [23'287 (0'0) modify   1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head by client.4144.0:887 2016-10-25 08:51:33.175884 0]
2016-10-25 08:51:33.181604 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] add_log_entry 23'287 (0'0) modify   1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head by client.4144.0:887 2016-10-25 08:51:33.175884 0
2016-10-25 08:51:33.181618 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] append_log: trimming to 23'284 entries 
2016-10-25 08:51:33.181635 7f211046a700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'287, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:33.181873 7f211046a700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139241080 osr(1.0 0x7f2139475d80)
2016-10-25 08:51:33.181878 7f211046a700 10 journal prepare_entry [Transaction(0x7f2147eb8680)]
2016-10-25 08:51:33.182128 7f211046a700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:33.182894 7f211046a700 10 journal op_submit_start 2370
2016-10-25 08:51:33.182899 7f211046a700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2370 [Transaction(0x7f2147eb8680)]
2016-10-25 08:51:33.182904 7f211046a700 10 journal op_journal_transactions 2370 reqid_t client.4144.0:887
2016-10-25 08:51:33.182908 7f211046a700  5 journal submit_entry seq 2370 len 4202496 (0x7f2140fda0d0)
2016-10-25 08:51:33.183219 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:33.183225 7f2123404700 20 journal write_thread_entry aio throttle: aio num 5 bytes 16814080 ... exp 10 min_new 1024 ... pending 4202496
2016-10-25 08:51:33.183229 7f2123404700 10 journal room 913010687 max_size 1072709632 pos 1029881856 header.start 870187008 top 4096
2016-10-25 08:51:33.183233 7f2123404700 10 journal check_for_full at 1029881856 : 4202496 < 913010687
2016-10-25 08:51:33.183235 7f2123404700 15 journal prepare_single_write 1 will write 1029881856 : seq 2370 len 4195598 -> 4202496
2016-10-25 08:51:33.183265 7f2123404700 20 journal prepare_multi_write queue_pos now 1034084352
2016-10-25 08:51:33.183267 7f2123404700 15 journal do_aio_write writing 1029881856~4202496
2016-10-25 08:51:33.183270 7f2123404700 20 journal write_aio_bl 1029881856~4202496 seq 2370
2016-10-25 08:51:33.183273 7f2123404700 20 journal write_aio_bl .. 1029881856~4202496 in 1
2016-10-25 08:51:33.183730 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:33.183738 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:33.183741 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:33.183765 7f211046a700 10 journal op_submit_finish 2370
2016-10-25 08:51:33.183773 7f211046a700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] eval_repop repgather(0x7f2139881440 23'287 rep_tid=1730 committed?=0 applied?=0) wants=d
2016-10-25 08:51:33.183802 7f211046a700 10 osd.3 23 dequeue_op 0x7f2139883180 finish
2016-10-25 08:51:33.220422 7f2122c03700 10 journal write_finish_thread_entry aio 0~4096 done
2016-10-25 08:51:33.220448 7f2122c03700 10 journal write_finish_thread_entry aio 1013071872~4202496 done
2016-10-25 08:51:33.220450 7f2122c03700 10 journal write_finish_thread_entry aio 1017274368~4202496 done
2016-10-25 08:51:33.220451 7f2122c03700 10 journal write_finish_thread_entry aio 1021476864~4202496 done
2016-10-25 08:51:33.220452 7f2122c03700 10 journal write_finish_thread_entry aio 1025679360~4202496 done
2016-10-25 08:51:33.220454 7f2122c03700 20 journal check_aio_completion
2016-10-25 08:51:33.220456 7f2122c03700 20 journal check_aio_completion completed seq 0 0~4096
2016-10-25 08:51:33.220462 7f2122c03700 20 journal check_aio_completion completed seq 2366 1013071872~4202496
2016-10-25 08:51:33.220467 7f2122c03700 20 journal check_aio_completion completed seq 2367 1017274368~4202496
2016-10-25 08:51:33.220470 7f2122c03700 20 journal check_aio_completion completed seq 2368 1021476864~4202496
2016-10-25 08:51:33.220472 7f2122c03700 20 journal check_aio_completion completed seq 2369 1025679360~4202496
2016-10-25 08:51:33.220475 7f2122c03700 20 journal check_aio_completion queueing finishers through seq 2369
2016-10-25 08:51:33.220478 7f2122c03700 10 journal queue_completions_thru seq 2369 queueing seq 2366 0x7f21430983d0 lat 0.508505
2016-10-25 08:51:33.220510 7f2122c03700 10 journal queue_completions_thru seq 2369 queueing seq 2367 0x7f2140fb6640 lat 0.491686
2016-10-25 08:51:33.220520 7f2122c03700 10 journal queue_completions_thru seq 2369 queueing seq 2368 0x7f214811f600 lat 0.474328
2016-10-25 08:51:33.220528 7f2122c03700 10 journal queue_completions_thru seq 2369 queueing seq 2369 0x7f21414acbb0 lat 0.460029
2016-10-25 08:51:33.220536 7f2122c03700 20 journal write_finish_thread_entry waiting for aio(s)
2016-10-25 08:51:33.220546 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f214141a720 seq 2366 osr(1.3 0x7f2139475570) [Transaction(0x7f2147edea80)]
2016-10-25 08:51:33.220552 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f214141a720 seq 2366 osr(1.3 0x7f2139475570) 4195584 bytes   (queue has 5 ops and 20977990 bytes)
2016-10-25 08:51:33.220561 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f213eaa35e0
2016-10-25 08:51:33.220566 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f2140ec2760 seq 2367 osr(1.2 0x7f2139474dc0) [Transaction(0x7f2147f6ce00),Transaction(0x7f2147f6cfc0)]
2016-10-25 08:51:33.220570 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f2140ec2760 seq 2367 osr(1.2 0x7f2139474dc0) 4195654 bytes   (queue has 5 ops and 20977990 bytes)
2016-10-25 08:51:33.220578 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f2142e08540
2016-10-25 08:51:33.220584 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f2139106f00 seq 2368 osr(1.0 0x7f2139475d80) [Transaction(0x7f2147e38840)]
2016-10-25 08:51:33.220587 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f2139106f00 seq 2368 osr(1.0 0x7f2139475d80) 4195584 bytes   (queue has 5 ops and 20977990 bytes)
2016-10-25 08:51:33.220590 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f21395023a0
2016-10-25 08:51:33.220592 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f214141a660 seq 2369 osr(1.3 0x7f2139475570) [Transaction(0x7f2140fd3d40)]
2016-10-25 08:51:33.220595 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f214141a660 seq 2369 osr(1.3 0x7f2139475570) 4195584 bytes   (queue has 5 ops and 20977990 bytes)
2016-10-25 08:51:33.220597 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f2142e0acc0
2016-10-25 08:51:33.220608 7f2120bff700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'286 crt=23'283 lcod 23'284 mlcod 23'284 active+clean] op_commit: 1727
2016-10-25 08:51:33.220669 7f2120bff700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'286 crt=23'283 lcod 23'284 mlcod 23'284 active+clean] op_commit: 1729
2016-10-25 08:51:33.220697 7f21203fe700 10 osd.3 pg_epoch: 23 pg[1.2( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'277 crt=23'275 lcod 23'276 active] sub_op_modify_commit on op osd_repop(client.4144.0:883 1.2 1:577853ca:::benchmark_data_target167114251239.teuthology_13304_object882:head v 23'278) v1, sending commit to osd.2
2016-10-25 08:51:33.220715 7f21203fe700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:33.220718 7f21203fe700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop_reply(client.4144.0:883 1.2 ondisk, result = 0) v1 -- 0x7f213ec3ac80 con 0
2016-10-25 08:51:33.220739 7f2121400700 10 journal op_apply_start 2366 open_ops 0 -> 1
2016-10-25 08:51:33.220747 7f2121400700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f214141a720 seq 2366 osr(1.3 0x7f2139475570)/0x7f2139475570 start
2016-10-25 08:51:33.220752 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147edea80
2016-10-25 08:51:33.220754 7f21203fe700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] op_commit: 1728
2016-10-25 08:51:33.220765 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.3_head/#1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head# 0~4194304
2016-10-25 08:51:33.220777 7f2121c01700 10 journal op_apply_start 2367 open_ops 1 -> 2
2016-10-25 08:51:33.220779 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2140ec2760 seq 2367 osr(1.2 0x7f2139474dc0)/0x7f2139474dc0 start
2016-10-25 08:51:33.220782 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147f6ce00
2016-10-25 08:51:33.220791 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.2_head/#1:40000000::::head#
2016-10-25 08:51:33.220803 7f2121c01700 10 filestore oid: #1:40000000::::head# not skipping op, *spos 2367.0.0
2016-10-25 08:51:33.220812 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:33.220910 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.2_head/#1:40000000::::head# = 0
2016-10-25 08:51:33.220916 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147f6cfc0
2016-10-25 08:51:33.220921 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.2_head/#1:577853ca:::benchmark_data_target167114251239.teuthology_13304_object882:head# 0~4194304
2016-10-25 08:51:33.223633 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.2_head/#1:577853ca:::benchmark_data_target167114251239.teuthology_13304_object882:head# 0~4194304 = 4194304
2016-10-25 08:51:33.223701 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.2_head/#1:577853ca:::benchmark_data_target167114251239.teuthology_13304_object882:head#
2016-10-25 08:51:33.223752 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.2_head/#1:577853ca:::benchmark_data_target167114251239.teuthology_13304_object882:head# = 0
2016-10-25 08:51:33.223759 7f2121c01700 10 journal op_apply_finish 2367 open_ops 2 -> 1, max_applied_seq 2365 -> 2367
2016-10-25 08:51:33.223762 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2140ec2760 seq 2367 r = 0, finisher 0x7f2142e09220 0
2016-10-25 08:51:33.223774 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f2140ec2760 seq 2367 osr(1.2 0x7f2139474dc0)/0x7f2139474dc0 lat 0.496185
2016-10-25 08:51:33.223787 7f2121c01700 10 journal op_apply_start 2368 open_ops 1 -> 2
2016-10-25 08:51:33.223788 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2139106f00 seq 2368 osr(1.0 0x7f2139475d80)/0x7f2139475d80 start
2016-10-25 08:51:33.223791 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147e38840
2016-10-25 08:51:33.223801 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.0_head/#1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head# 0~4194304
2016-10-25 08:51:33.223818 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.2( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'277 crt=23'275 lcod 23'277 active] sub_op_modify_applied on 0x7f2142f0c018 op osd_repop(client.4144.0:883 1.2 1:577853ca:::benchmark_data_target167114251239.teuthology_13304_object882:head v 23'278) v1
2016-10-25 08:51:33.223846 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.2( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/20) [2,3] r=1 lpr=21 pi=17-20/4 luod=0'0 lua=23'277 crt=23'275 lcod 23'277 active] op_applied version 23'278
2016-10-25 08:51:33.224481 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.3_head/#1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head# 0~4194304 = 4194304
2016-10-25 08:51:33.224542 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.3_head/#1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head#
2016-10-25 08:51:33.224594 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.3_head/#1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head# = 0
2016-10-25 08:51:33.224624 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.3_head/#1:c0000000::::head#
2016-10-25 08:51:33.224652 7f2121400700 10 filestore oid: #1:c0000000::::head# not skipping op, *spos 2366.0.2
2016-10-25 08:51:33.224666 7f2121400700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:33.224725 7f2121400700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.3_head/#1:c0000000::::head# = 0
2016-10-25 08:51:33.224746 7f2121400700 10 journal op_apply_finish 2366 open_ops 2 -> 1, max_applied_seq 2367 -> 2367
2016-10-25 08:51:33.224761 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f214141a720 seq 2366 r = 0, finisher 0x7f213eaa2de0 0x7f2147f99c00
2016-10-25 08:51:33.224796 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f214141a720 seq 2366 osr(1.3 0x7f2139475570)/0x7f2139475570 lat 0.513823
2016-10-25 08:51:33.224825 7f2121400700 10 journal op_apply_start 2369 open_ops 1 -> 2
2016-10-25 08:51:33.224841 7f2121400700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f214141a660 seq 2369 osr(1.3 0x7f2139475570)/0x7f2139475570 start
2016-10-25 08:51:33.224835 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'286 crt=23'283 lcod 23'284 mlcod 23'284 active+clean] op_applied: 1727
2016-10-25 08:51:33.224860 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2140fd3d40
2016-10-25 08:51:33.224875 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'286 crt=23'283 lcod 23'284 mlcod 23'284 active+clean] op_applied version 23'287
2016-10-25 08:51:33.226417 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.0_head/#1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head# 0~4194304 = 4194304
2016-10-25 08:51:33.226447 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.0_head/#1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head#
2016-10-25 08:51:33.226474 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.0_head/#1:12a5824e:::benchmark_data_target167114251239.teuthology_13304_object884:head# = 0
2016-10-25 08:51:33.226482 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.0_head/#1:00000000::::head#
2016-10-25 08:51:33.226490 7f2121c01700 10 filestore oid: #1:00000000::::head# not skipping op, *spos 2368.0.2
2016-10-25 08:51:33.226492 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:33.226520 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.0_head/#1:00000000::::head# = 0
2016-10-25 08:51:33.226523 7f2121c01700 10 journal op_apply_finish 2368 open_ops 2 -> 1, max_applied_seq 2367 -> 2368
2016-10-25 08:51:33.226528 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2139106f00 seq 2368 r = 0, finisher 0x7f2139503d80 0x7f2142edb680
2016-10-25 08:51:33.226535 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f2139106f00 seq 2368 osr(1.0 0x7f2139475d80)/0x7f2139475d80 lat 0.481649
2016-10-25 08:51:33.230118 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] op_applied: 1728
2016-10-25 08:51:33.230173 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'285 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] op_applied version 23'286
2016-10-25 08:51:33.224885 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.3_head/#1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head# 0~4194304
2016-10-25 08:51:33.231605 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2296 ==== osd_repop_reply(client.4144.0:880 1.4) v1 ==== 83+0+0 (920256264 0 0) 0x7f21412f9700 con 0x7f2139592800
2016-10-25 08:51:33.231630 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:880 1.4) v1 epoch 23
2016-10-25 08:51:33.231635 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:33.231641 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139883600 prio 196 cost 0 latency 0.000067 osd_repop_reply(client.4144.0:880 1.4) v1
2016-10-25 08:51:33.231666 7f2110c6b700 10 osd.3 23 dequeue_op 0x7f2139883600 prio 196 cost 0 latency 0.000092 osd_repop_reply(client.4144.0:880 1.4) v1 pg pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 crt=23'282 lcod 23'282 mlcod 23'282 active+clean]
2016-10-25 08:51:33.231685 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 crt=23'282 lcod 23'282 mlcod 23'282 active+clean] handle_message: 0x7f2139883600
2016-10-25 08:51:33.231700 7f2110c6b700  7 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 crt=23'282 lcod 23'282 mlcod 23'282 active+clean] sub_op_modify_reply: tid 1724 op  ack_type 4 from 2
2016-10-25 08:51:33.231716 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 crt=23'282 lcod 23'282 mlcod 23'282 active+clean] repop_all_applied: repop tid 1724 all applied 
2016-10-25 08:51:33.231723 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 crt=23'282 lcod 23'282 mlcod 23'282 active+clean] eval_repop repgather(0x7f2139770000 23'284 rep_tid=1724 committed?=0 applied?=1) wants=d
2016-10-25 08:51:33.231732 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 crt=23'282 lcod 23'282 mlcod 23'282 active+clean]  applied: repgather(0x7f2139770000 23'284 rep_tid=1724 committed?=0 applied?=1) 
2016-10-25 08:51:33.231739 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'283 crt=23'282 lcod 23'282 mlcod 23'282 active+clean] repop_all_committed: repop tid 1724 all committed 
2016-10-25 08:51:33.231744 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'282 active+clean] eval_repop repgather(0x7f2139770000 23'284 rep_tid=1724 committed?=1 applied?=1) wants=d
2016-10-25 08:51:33.231752 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'282 active+clean]  commit: repgather(0x7f2139770000 23'284 rep_tid=1724 committed?=1 applied?=1)
2016-10-25 08:51:33.231759 7f2110c6b700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'282 active+clean] log_op_stats osd_op(client.4144.0:880 1.7d6f66b4 benchmark_data_target167114251239.teuthology_13304_object879 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 1.528391 lat 1.528410
2016-10-25 08:51:33.231770 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'282 active+clean]  sending commit on osd_op(client.4144.0:880 1.7d6f66b4 benchmark_data_target167114251239.teuthology_13304_object879 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f213c577c80
2016-10-25 08:51:33.231786 7f2110c6b700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(880 benchmark_data_target167114251239.teuthology_13304_object879 [write 0~4194304] v23'284 uv284 ondisk = 0) v7 -- 0x7f213c577c80 con 0
2016-10-25 08:51:33.231808 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'282 active+clean]  applied: repgather(0x7f2139770000 23'284 rep_tid=1724 committed?=1 applied?=1) 
2016-10-25 08:51:33.231826 7f2110c6b700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'282 active+clean] publish_stats_to_osd 23:145
2016-10-25 08:51:33.231840 7f2110c6b700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:33.231846 7f2110c6b700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean]  removing repgather(0x7f2139770000 23'284 rep_tid=1724 committed?=1 applied?=1)
2016-10-25 08:51:33.231851 7f2110c6b700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean]    q front is repgather(0x7f2139770000 23'284 rep_tid=1724 committed?=1 applied?=1)
2016-10-25 08:51:33.231857 7f2110c6b700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] remove_repop repgather(0x7f2139770000 23'284 rep_tid=1724 committed?=1 applied?=1)
2016-10-25 08:51:33.231869 7f2110c6b700 15 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean]  requeue_ops 
2016-10-25 08:51:33.231895 7f2110c6b700 10 osd.3 23 dequeue_op 0x7f2139883600 finish
2016-10-25 08:51:33.231932 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2297 ==== osd_repop_reply(client.4144.0:885 1.0) v1 ==== 83+0+0 (1571148041 0 0) 0x7f21412f9700 con 0x7f2139592800
2016-10-25 08:51:33.231944 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:885 1.0) v1 epoch 23
2016-10-25 08:51:33.231947 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:33.231951 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139883600 prio 196 cost 0 latency 0.000031 osd_repop_reply(client.4144.0:885 1.0) v1
2016-10-25 08:51:33.231971 7f2112c6f700 10 osd.3 23 dequeue_op 0x7f2139883600 prio 196 cost 0 latency 0.000051 osd_repop_reply(client.4144.0:885 1.0) v1 pg pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'286 crt=23'284 lcod 23'284 mlcod 23'284 active+clean]
2016-10-25 08:51:33.231985 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'286 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] handle_message: 0x7f2139883600
2016-10-25 08:51:33.231999 7f2112c6f700  7 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'286 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] sub_op_modify_reply: tid 1728 op  ack_type 4 from 2
2016-10-25 08:51:33.232026 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'286 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] repop_all_applied: repop tid 1728 all applied 
2016-10-25 08:51:33.232039 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'286 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] eval_repop repgather(0x7f2139880240 23'286 rep_tid=1728 committed?=0 applied?=1) wants=d
2016-10-25 08:51:33.232045 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'286 crt=23'284 lcod 23'284 mlcod 23'284 active+clean]  applied: repgather(0x7f2139880240 23'286 rep_tid=1728 committed?=0 applied?=1) 
2016-10-25 08:51:33.232052 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'286 crt=23'284 lcod 23'284 mlcod 23'284 active+clean] repop_all_committed: repop tid 1728 all committed 
2016-10-25 08:51:33.232059 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'286 crt=23'284 lcod 23'285 mlcod 23'284 active+clean] eval_repop repgather(0x7f2139880240 23'286 rep_tid=1728 committed?=1 applied?=1) wants=d
2016-10-25 08:51:33.232064 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'286 crt=23'284 lcod 23'285 mlcod 23'284 active+clean]  commit: repgather(0x7f2139880240 23'286 rep_tid=1728 committed?=1 applied?=1)
2016-10-25 08:51:33.232071 7f2112c6f700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'286 crt=23'284 lcod 23'285 mlcod 23'284 active+clean] log_op_stats osd_op(client.4144.0:885 1.7241a548 benchmark_data_target167114251239.teuthology_13304_object884 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 0.492302 lat 0.492321
2016-10-25 08:51:33.232078 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'286 crt=23'284 lcod 23'285 mlcod 23'284 active+clean]  sending commit on osd_op(client.4144.0:885 1.7241a548 benchmark_data_target167114251239.teuthology_13304_object884 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f21397a1e40
2016-10-25 08:51:33.232085 7f2112c6f700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(885 benchmark_data_target167114251239.teuthology_13304_object884 [write 0~4194304] v23'286 uv286 ondisk = 0) v7 -- 0x7f21397a1e40 con 0
2016-10-25 08:51:33.232096 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'286 crt=23'284 lcod 23'285 mlcod 23'284 active+clean]  applied: repgather(0x7f2139880240 23'286 rep_tid=1728 committed?=1 applied?=1) 
2016-10-25 08:51:33.232105 7f2112c6f700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'286 crt=23'284 lcod 23'285 mlcod 23'284 active+clean] publish_stats_to_osd 23:139
2016-10-25 08:51:33.232111 7f2112c6f700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'286 crt=23'284 lcod 23'285 mlcod 23'285 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:33.232117 7f2112c6f700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'286 crt=23'284 lcod 23'285 mlcod 23'285 active+clean]  removing repgather(0x7f2139880240 23'286 rep_tid=1728 committed?=1 applied?=1)
2016-10-25 08:51:33.232124 7f2112c6f700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'286 crt=23'284 lcod 23'285 mlcod 23'285 active+clean]    q front is repgather(0x7f2139880240 23'286 rep_tid=1728 committed?=1 applied?=1)
2016-10-25 08:51:33.232130 7f2112c6f700 20 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'286 crt=23'284 lcod 23'285 mlcod 23'285 active+clean] remove_repop repgather(0x7f2139880240 23'286 rep_tid=1728 committed?=1 applied?=1)
2016-10-25 08:51:33.232149 7f2112c6f700 15 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'286 crt=23'284 lcod 23'285 mlcod 23'285 active+clean]  requeue_ops 
2016-10-25 08:51:33.232169 7f2112c6f700 10 osd.3 23 dequeue_op 0x7f2139883600 finish
2016-10-25 08:51:33.232208 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2298 ==== osd_repop_reply(client.4144.0:870 1.7) v1 ==== 83+0+0 (1733430394 0 0) 0x7f21412f9700 con 0x7f2139592800
2016-10-25 08:51:33.232219 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:870 1.7) v1 epoch 23
2016-10-25 08:51:33.232221 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:33.232225 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139883600 prio 196 cost 0 latency 0.000029 osd_repop_reply(client.4144.0:870 1.7) v1
2016-10-25 08:51:33.232243 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2299 ==== osd_repop_reply(client.4144.0:882 1.3) v1 ==== 83+0+0 (270716641 0 0) 0x7f21412f7900 con 0x7f2139592800
2016-10-25 08:51:33.232252 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:882 1.3) v1 epoch 23
2016-10-25 08:51:33.232254 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:33.232256 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f21398834e0 prio 196 cost 0 latency 0.000020 osd_repop_reply(client.4144.0:882 1.3) v1
2016-10-25 08:51:33.232273 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2300 ==== osd_repop_reply(client.4144.0:881 1.7) v1 ==== 83+0+0 (36616010 0 0) 0x7f21412f7400 con 0x7f2139592800
2016-10-25 08:51:33.232278 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:881 1.7) v1 epoch 23
2016-10-25 08:51:33.232280 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:33.232260 7f2111c6d700 10 osd.3 23 dequeue_op 0x7f2139883600 prio 196 cost 0 latency 0.000063 osd_repop_reply(client.4144.0:870 1.7) v1 pg pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'266 crt=23'264 lcod 23'265 mlcod 23'265 active+clean]
2016-10-25 08:51:33.232283 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139883ba0 prio 196 cost 0 latency 0.000016 osd_repop_reply(client.4144.0:881 1.7) v1
2016-10-25 08:51:33.232299 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2301 ==== osd_repop_reply(client.4144.0:884 1.3) v1 ==== 83+0+0 (3013397076 0 0) 0x7f21412f8800 con 0x7f2139592800
2016-10-25 08:51:33.232293 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'266 crt=23'264 lcod 23'265 mlcod 23'265 active+clean] handle_message: 0x7f2139883600
2016-10-25 08:51:33.232305 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:884 1.3) v1 epoch 23
2016-10-25 08:51:33.232307 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:33.232309 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f21398822e0 prio 196 cost 0 latency 0.000016 osd_repop_reply(client.4144.0:884 1.3) v1
2016-10-25 08:51:33.232312 7f2111c6d700  7 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'266 crt=23'264 lcod 23'265 mlcod 23'265 active+clean] sub_op_modify_reply: tid 1716 op  ack_type 4 from 2
2016-10-25 08:51:33.232325 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2302 ==== osd_repop_reply(client.4144.0:886 1.3) v1 ==== 83+0+0 (777184360 0 0) 0x7f213946f180 con 0x7f2139592800
2016-10-25 08:51:33.232331 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:886 1.3) v1 epoch 23
2016-10-25 08:51:33.232333 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:33.232335 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f21398805a0 prio 196 cost 0 latency 0.000016 osd_repop_reply(client.4144.0:886 1.3) v1
2016-10-25 08:51:33.232339 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'266 crt=23'264 lcod 23'265 mlcod 23'265 active+clean] repop_all_applied: repop tid 1716 all applied 
2016-10-25 08:51:33.232349 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'266 crt=23'264 lcod 23'265 mlcod 23'265 active+clean] eval_repop repgather(0x7f2139643320 23'267 rep_tid=1716 committed?=0 applied?=1) wants=d
2016-10-25 08:51:33.232361 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2303 ==== osd_repop_reply(client.4144.0:871 1.6) v1 ==== 83+0+0 (2113616742 0 0) 0x7f2139471700 con 0x7f2139592800
2016-10-25 08:51:33.232367 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:871 1.6) v1 epoch 23
2016-10-25 08:51:33.232360 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'266 crt=23'264 lcod 23'265 mlcod 23'265 active+clean]  applied: repgather(0x7f2139643320 23'267 rep_tid=1716 committed?=0 applied?=1) 
2016-10-25 08:51:33.232369 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:33.232372 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139882f40 prio 196 cost 0 latency 0.000016 osd_repop_reply(client.4144.0:871 1.6) v1
2016-10-25 08:51:33.232371 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'266 crt=23'264 lcod 23'265 mlcod 23'265 active+clean] repop_all_committed: repop tid 1716 all committed 
2016-10-25 08:51:33.232389 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2304 ==== osd_repop_reply(client.4144.0:872 1.1) v1 ==== 83+0+0 (1042391788 0 0) 0x7f2139470580 con 0x7f2139592800
2016-10-25 08:51:33.232383 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'267 crt=23'264 lcod 23'266 mlcod 23'265 active+clean] eval_repop repgather(0x7f2139643320 23'267 rep_tid=1716 committed?=1 applied?=1) wants=d
2016-10-25 08:51:33.232394 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:872 1.1) v1 epoch 23
2016-10-25 08:51:33.232396 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:33.232399 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f21397707e0 prio 196 cost 0 latency 0.000017 osd_repop_reply(client.4144.0:872 1.1) v1
2016-10-25 08:51:33.232393 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'267 crt=23'264 lcod 23'266 mlcod 23'265 active+clean]  commit: repgather(0x7f2139643320 23'267 rep_tid=1716 committed?=1 applied?=1)
2016-10-25 08:51:33.232411 7f2111c6d700 15 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'267 crt=23'264 lcod 23'266 mlcod 23'265 active+clean] log_op_stats osd_op(client.4144.0:870 1.7c00d607 benchmark_data_target167114251239.teuthology_13304_object869 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 1.849556 lat 1.849593
2016-10-25 08:51:33.232431 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2305 ==== osd_repop_reply(client.4144.0:874 1.6) v1 ==== 83+0+0 (1052138439 0 0) 0x7f2139470300 con 0x7f2139592800
2016-10-25 08:51:33.232425 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'267 crt=23'264 lcod 23'266 mlcod 23'265 active+clean]  sending commit on osd_op(client.4144.0:870 1.7c00d607 benchmark_data_target167114251239.teuthology_13304_object869 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f2142e57c80
2016-10-25 08:51:33.232437 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:874 1.6) v1 epoch 23
2016-10-25 08:51:33.232439 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:33.232442 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139770b40 prio 196 cost 0 latency 0.000032 osd_repop_reply(client.4144.0:874 1.6) v1
2016-10-25 08:51:33.232438 7f2111c6d700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(870 benchmark_data_target167114251239.teuthology_13304_object869 [write 0~4194304] v23'267 uv267 ondisk = 0) v7 -- 0x7f2142e57c80 con 0
2016-10-25 08:51:33.232453 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2306 ==== osd_repop_reply(client.4144.0:875 1.1) v1 ==== 83+0+0 (4007998899 0 0) 0x7f2139470d00 con 0x7f2139592800
2016-10-25 08:51:33.232458 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:875 1.1) v1 epoch 23
2016-10-25 08:51:33.232460 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:33.232462 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139772520 prio 196 cost 0 latency 0.000015 osd_repop_reply(client.4144.0:875 1.1) v1
2016-10-25 08:51:33.232457 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'267 crt=23'264 lcod 23'266 mlcod 23'265 active+clean]  applied: repgather(0x7f2139643320 23'267 rep_tid=1716 committed?=1 applied?=1) 
2016-10-25 08:51:33.232473 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2307 ==== osd_repop_reply(client.4144.0:879 1.6) v1 ==== 83+0+0 (3938225668 0 0) 0x7f2139470080 con 0x7f2139592800
2016-10-25 08:51:33.232469 7f2111c6d700 15 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'267 crt=23'264 lcod 23'266 mlcod 23'265 active+clean] publish_stats_to_osd 23: no change since 2016-10-25 08:51:32.703542
2016-10-25 08:51:33.232479 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:879 1.6) v1 epoch 23
2016-10-25 08:51:33.232481 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:33.232483 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139772e20 prio 196 cost 0 latency 0.000014 osd_repop_reply(client.4144.0:879 1.6) v1
2016-10-25 08:51:33.232481 7f2111c6d700 15 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'267 crt=23'264 lcod 23'266 mlcod 23'266 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:33.232494 7f2129b4e700  1 -- 167.114.251.239:6805/12812 <== osd.2 167.114.251.239:6801/12736 2308 ==== osd_repop_reply(client.4144.0:878 1.1) v1 ==== 83+0+0 (988799088 0 0) 0x7f213946e780 con 0x7f2139592800
2016-10-25 08:51:33.232491 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'267 crt=23'264 lcod 23'266 mlcod 23'266 active+clean]  removing repgather(0x7f2139643320 23'267 rep_tid=1716 committed?=1 applied?=1)
2016-10-25 08:51:33.232500 7f2129b4e700 10 osd.3 23 handle_replica_op osd_repop_reply(client.4144.0:878 1.1) v1 epoch 23
2016-10-25 08:51:33.232501 7f2129b4e700 20 osd.3 23 should_share_map osd.2 167.114.251.239:6801/12736 23
2016-10-25 08:51:33.232504 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f21397710e0 prio 196 cost 0 latency 0.000014 osd_repop_reply(client.4144.0:878 1.1) v1
2016-10-25 08:51:33.232501 7f2111c6d700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'267 crt=23'264 lcod 23'266 mlcod 23'266 active+clean]    q front is repgather(0x7f2139643320 23'267 rep_tid=1716 committed?=1 applied?=1)
2016-10-25 08:51:33.232511 7f2111c6d700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'267 crt=23'264 lcod 23'266 mlcod 23'266 active+clean] remove_repop repgather(0x7f2139643320 23'267 rep_tid=1716 committed?=1 applied?=1)
2016-10-25 08:51:33.232524 7f2111c6d700 15 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'267 crt=23'264 lcod 23'266 mlcod 23'266 active+clean]  requeue_ops 
2016-10-25 08:51:33.232551 7f2111c6d700 10 osd.3 23 dequeue_op 0x7f2139883600 finish
2016-10-25 08:51:33.232566 7f2111c6d700 10 osd.3 23 dequeue_op 0x7f2139883ba0 prio 196 cost 0 latency 0.000299 osd_repop_reply(client.4144.0:881 1.7) v1 pg pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'267 crt=23'264 lcod 23'266 mlcod 23'266 active+clean]
2016-10-25 08:51:33.232580 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'267 crt=23'264 lcod 23'266 mlcod 23'266 active+clean] handle_message: 0x7f2139883ba0
2016-10-25 08:51:33.232594 7f2111c6d700  7 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'267 crt=23'264 lcod 23'266 mlcod 23'266 active+clean] sub_op_modify_reply: tid 1725 op  ack_type 4 from 2
2016-10-25 08:51:33.232613 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'267 crt=23'264 lcod 23'266 mlcod 23'266 active+clean] repop_all_applied: repop tid 1725 all applied 
2016-10-25 08:51:33.232622 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'267 crt=23'264 lcod 23'266 mlcod 23'266 active+clean] eval_repop repgather(0x7f21395ba1c0 23'268 rep_tid=1725 committed?=0 applied?=1) wants=d
2016-10-25 08:51:33.232631 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'267 crt=23'264 lcod 23'266 mlcod 23'266 active+clean]  applied: repgather(0x7f21395ba1c0 23'268 rep_tid=1725 committed?=0 applied?=1) 
2016-10-25 08:51:33.232641 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'267 crt=23'264 lcod 23'266 mlcod 23'266 active+clean] repop_all_committed: repop tid 1725 all committed 
2016-10-25 08:51:33.232650 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'266 active+clean] eval_repop repgather(0x7f21395ba1c0 23'268 rep_tid=1725 committed?=1 applied?=1) wants=d
2016-10-25 08:51:33.232659 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'266 active+clean]  commit: repgather(0x7f21395ba1c0 23'268 rep_tid=1725 committed?=1 applied?=1)
2016-10-25 08:51:33.232668 7f2111c6d700 15 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'266 active+clean] log_op_stats osd_op(client.4144.0:881 1.c392bcdf benchmark_data_target167114251239.teuthology_13304_object880 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 1.187069 lat 1.187096
2016-10-25 08:51:33.232679 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'266 active+clean]  sending commit on osd_op(client.4144.0:881 1.c392bcdf benchmark_data_target167114251239.teuthology_13304_object880 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f2148002100
2016-10-25 08:51:33.232688 7f2111c6d700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(881 benchmark_data_target167114251239.teuthology_13304_object880 [write 0~4194304] v23'268 uv268 ondisk = 0) v7 -- 0x7f2148002100 con 0
2016-10-25 08:51:33.232717 7f211246e700 10 osd.3 23 dequeue_op 0x7f2139882f40 prio 196 cost 0 latency 0.000361 osd_repop_reply(client.4144.0:871 1.6) v1 pg pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'275 crt=23'273 lcod 23'274 mlcod 23'274 active+clean]
2016-10-25 08:51:33.232735 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'275 crt=23'273 lcod 23'274 mlcod 23'274 active+clean] handle_message: 0x7f2139882f40
2016-10-25 08:51:33.232752 7f211246e700  7 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'275 crt=23'273 lcod 23'274 mlcod 23'274 active+clean] sub_op_modify_reply: tid 1717 op  ack_type 4 from 2
2016-10-25 08:51:33.232771 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'275 crt=23'273 lcod 23'274 mlcod 23'274 active+clean] repop_all_applied: repop tid 1717 all applied 
2016-10-25 08:51:33.232781 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'275 crt=23'273 lcod 23'274 mlcod 23'274 active+clean] eval_repop repgather(0x7f21397f8ea0 23'276 rep_tid=1717 committed?=0 applied?=1) wants=d
2016-10-25 08:51:33.232791 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'275 crt=23'273 lcod 23'274 mlcod 23'274 active+clean]  applied: repgather(0x7f21397f8ea0 23'276 rep_tid=1717 committed?=0 applied?=1) 
2016-10-25 08:51:33.232801 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'275 crt=23'273 lcod 23'274 mlcod 23'274 active+clean] repop_all_committed: repop tid 1717 all committed 
2016-10-25 08:51:33.232809 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'276 crt=23'273 lcod 23'275 mlcod 23'274 active+clean] eval_repop repgather(0x7f21397f8ea0 23'276 rep_tid=1717 committed?=1 applied?=1) wants=d
2016-10-25 08:51:33.232818 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'276 crt=23'273 lcod 23'275 mlcod 23'274 active+clean]  commit: repgather(0x7f21397f8ea0 23'276 rep_tid=1717 committed?=1 applied?=1)
2016-10-25 08:51:33.232829 7f211246e700 15 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'276 crt=23'273 lcod 23'275 mlcod 23'274 active+clean] log_op_stats osd_op(client.4144.0:871 1.b7f2f276 benchmark_data_target167114251239.teuthology_13304_object870 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 1.841060 lat 1.841088
2016-10-25 08:51:33.232841 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'276 crt=23'273 lcod 23'275 mlcod 23'274 active+clean]  sending commit on osd_op(client.4144.0:871 1.b7f2f276 benchmark_data_target167114251239.teuthology_13304_object870 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f213c574dc0
2016-10-25 08:51:33.232852 7f211246e700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(871 benchmark_data_target167114251239.teuthology_13304_object870 [write 0~4194304] v23'276 uv276 ondisk = 0) v7 -- 0x7f213c574dc0 con 0
2016-10-25 08:51:33.232869 7f210fc69700 10 osd.3 23 dequeue_op 0x7f21397707e0 prio 196 cost 0 latency 0.000486 osd_repop_reply(client.4144.0:872 1.1) v1 pg pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'323 crt=23'321 lcod 23'322 mlcod 23'322 active+clean]
2016-10-25 08:51:33.232886 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'323 crt=23'321 lcod 23'322 mlcod 23'322 active+clean] handle_message: 0x7f21397707e0
2016-10-25 08:51:33.232902 7f210fc69700  7 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'323 crt=23'321 lcod 23'322 mlcod 23'322 active+clean] sub_op_modify_reply: tid 1718 op  ack_type 4 from 2
2016-10-25 08:51:33.232921 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'323 crt=23'321 lcod 23'322 mlcod 23'322 active+clean] repop_all_applied: repop tid 1718 all applied 
2016-10-25 08:51:33.232930 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'323 crt=23'321 lcod 23'322 mlcod 23'322 active+clean] eval_repop repgather(0x7f21394abb00 23'324 rep_tid=1718 committed?=0 applied?=1) wants=d
2016-10-25 08:51:33.232940 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'323 crt=23'321 lcod 23'322 mlcod 23'322 active+clean]  applied: repgather(0x7f21394abb00 23'324 rep_tid=1718 committed?=0 applied?=1) 
2016-10-25 08:51:33.232951 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'323 crt=23'321 lcod 23'322 mlcod 23'322 active+clean] repop_all_committed: repop tid 1718 all committed 
2016-10-25 08:51:33.232959 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'324 crt=23'321 lcod 23'323 mlcod 23'322 active+clean] eval_repop repgather(0x7f21394abb00 23'324 rep_tid=1718 committed?=1 applied?=1) wants=d
2016-10-25 08:51:33.232968 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'324 crt=23'321 lcod 23'323 mlcod 23'322 active+clean]  commit: repgather(0x7f21394abb00 23'324 rep_tid=1718 committed?=1 applied?=1)
2016-10-25 08:51:33.232979 7f210fc69700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'324 crt=23'321 lcod 23'323 mlcod 23'322 active+clean] log_op_stats osd_op(client.4144.0:872 1.406a57b1 benchmark_data_target167114251239.teuthology_13304_object871 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 1.836228 lat 1.836257
2016-10-25 08:51:33.232992 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'324 crt=23'321 lcod 23'323 mlcod 23'322 active+clean]  sending commit on osd_op(client.4144.0:872 1.406a57b1 benchmark_data_target167114251239.teuthology_13304_object871 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f213ea58680
2016-10-25 08:51:33.233004 7f210fc69700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(872 benchmark_data_target167114251239.teuthology_13304_object871 [write 0~4194304] v23'324 uv324 ondisk = 0) v7 -- 0x7f213ea58680 con 0
2016-10-25 08:51:33.233021 7f210ec67700 10 osd.3 23 dequeue_op 0x7f21398834e0 prio 196 cost 0 latency 0.000785 osd_repop_reply(client.4144.0:882 1.3) v1 pg pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'287 crt=23'283 lcod 23'284 mlcod 23'284 active+clean]
2016-10-25 08:51:33.233038 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'287 crt=23'283 lcod 23'284 mlcod 23'284 active+clean] handle_message: 0x7f21398834e0
2016-10-25 08:51:33.233060 7f210ec67700  7 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'287 crt=23'283 lcod 23'284 mlcod 23'284 active+clean] sub_op_modify_reply: tid 1726 op  ack_type 4 from 2
2016-10-25 08:51:33.233080 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'287 crt=23'283 lcod 23'284 mlcod 23'284 active+clean] repop_all_applied: repop tid 1726 all applied 
2016-10-25 08:51:33.233089 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'287 crt=23'283 lcod 23'284 mlcod 23'284 active+clean] eval_repop repgather(0x7f2139883de0 23'286 rep_tid=1726 committed?=0 applied?=1) wants=d
2016-10-25 08:51:33.233099 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'287 crt=23'283 lcod 23'284 mlcod 23'284 active+clean]  applied: repgather(0x7f2139883de0 23'286 rep_tid=1726 committed?=0 applied?=1) 
2016-10-25 08:51:33.233109 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'285 lua=23'287 crt=23'283 lcod 23'284 mlcod 23'284 active+clean] repop_all_committed: repop tid 1726 all committed 
2016-10-25 08:51:33.233119 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'287 crt=23'283 lcod 23'285 mlcod 23'284 active+clean] eval_repop repgather(0x7f2139883de0 23'286 rep_tid=1726 committed?=1 applied?=1) wants=d
2016-10-25 08:51:33.233128 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'287 crt=23'283 lcod 23'285 mlcod 23'284 active+clean]  commit: repgather(0x7f2139883de0 23'286 rep_tid=1726 committed?=1 applied?=1)
2016-10-25 08:51:33.233139 7f210ec67700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'287 crt=23'283 lcod 23'285 mlcod 23'284 active+clean] log_op_stats osd_op(client.4144.0:882 1.490e145b benchmark_data_target167114251239.teuthology_13304_object881 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 1.177779 lat 1.177809
2016-10-25 08:51:33.233151 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'287 crt=23'283 lcod 23'285 mlcod 23'284 active+clean]  sending commit on osd_op(client.4144.0:882 1.490e145b benchmark_data_target167114251239.teuthology_13304_object881 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f2143044000
2016-10-25 08:51:33.233163 7f210ec67700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(882 benchmark_data_target167114251239.teuthology_13304_object881 [write 0~4194304] v23'286 uv286 ondisk = 0) v7 -- 0x7f2143044000 con 0
2016-10-25 08:51:33.235449 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.3_head/#1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head# 0~4194304 = 4194304
2016-10-25 08:51:33.239080 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'266 active+clean]  applied: repgather(0x7f21395ba1c0 23'268 rep_tid=1725 committed?=1 applied?=1) 
2016-10-25 08:51:33.239109 7f2111c6d700 15 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'266 active+clean] publish_stats_to_osd 23: no change since 2016-10-25 08:51:32.703542
2016-10-25 08:51:33.239123 7f2111c6d700 15 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:33.239134 7f2111c6d700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean]  removing repgather(0x7f21395ba1c0 23'268 rep_tid=1725 committed?=1 applied?=1)
2016-10-25 08:51:33.239142 7f2111c6d700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean]    q front is repgather(0x7f21395ba1c0 23'268 rep_tid=1725 committed?=1 applied?=1)
2016-10-25 08:51:33.239152 7f2111c6d700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] remove_repop repgather(0x7f21395ba1c0 23'268 rep_tid=1725 committed?=1 applied?=1)
2016-10-25 08:51:33.239168 7f2111c6d700 15 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean]  requeue_ops 
2016-10-25 08:51:33.239205 7f2111c6d700 10 osd.3 23 dequeue_op 0x7f2139883ba0 finish
2016-10-25 08:51:33.239236 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'276 crt=23'273 lcod 23'275 mlcod 23'274 active+clean]  applied: repgather(0x7f21397f8ea0 23'276 rep_tid=1717 committed?=1 applied?=1) 
2016-10-25 08:51:33.239251 7f211246e700 15 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'276 crt=23'273 lcod 23'275 mlcod 23'274 active+clean] publish_stats_to_osd 23: no change since 2016-10-25 08:51:32.738102
2016-10-25 08:51:33.239263 7f211246e700 15 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'276 crt=23'273 lcod 23'275 mlcod 23'275 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:33.239273 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'276 crt=23'273 lcod 23'275 mlcod 23'275 active+clean]  removing repgather(0x7f21397f8ea0 23'276 rep_tid=1717 committed?=1 applied?=1)
2016-10-25 08:51:33.239283 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'276 crt=23'273 lcod 23'275 mlcod 23'275 active+clean]    q front is repgather(0x7f21397f8ea0 23'276 rep_tid=1717 committed?=1 applied?=1)
2016-10-25 08:51:33.239292 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'276 crt=23'273 lcod 23'275 mlcod 23'275 active+clean] remove_repop repgather(0x7f21397f8ea0 23'276 rep_tid=1717 committed?=1 applied?=1)
2016-10-25 08:51:33.239305 7f211246e700 15 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'276 crt=23'273 lcod 23'275 mlcod 23'275 active+clean]  requeue_ops 
2016-10-25 08:51:33.239332 7f211246e700 10 osd.3 23 dequeue_op 0x7f2139882f40 finish
2016-10-25 08:51:33.239351 7f211246e700 10 osd.3 23 dequeue_op 0x7f2139770b40 prio 196 cost 0 latency 0.006941 osd_repop_reply(client.4144.0:874 1.6) v1 pg pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'276 crt=23'273 lcod 23'275 mlcod 23'275 active+clean]
2016-10-25 08:51:33.239368 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'276 crt=23'273 lcod 23'275 mlcod 23'275 active+clean] handle_message: 0x7f2139770b40
2016-10-25 08:51:33.239385 7f211246e700  7 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'276 crt=23'273 lcod 23'275 mlcod 23'275 active+clean] sub_op_modify_reply: tid 1719 op  ack_type 4 from 2
2016-10-25 08:51:33.239405 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'276 crt=23'273 lcod 23'275 mlcod 23'275 active+clean] repop_all_applied: repop tid 1719 all applied 
2016-10-25 08:51:33.239414 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'276 crt=23'273 lcod 23'275 mlcod 23'275 active+clean] eval_repop repgather(0x7f21394ad840 23'277 rep_tid=1719 committed?=0 applied?=1) wants=d
2016-10-25 08:51:33.239423 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'276 crt=23'273 lcod 23'275 mlcod 23'275 active+clean]  applied: repgather(0x7f21394ad840 23'277 rep_tid=1719 committed?=0 applied?=1) 
2016-10-25 08:51:33.239434 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'276 crt=23'273 lcod 23'275 mlcod 23'275 active+clean] repop_all_committed: repop tid 1719 all committed 
2016-10-25 08:51:33.239442 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'277 crt=23'273 lcod 23'276 mlcod 23'275 active+clean] eval_repop repgather(0x7f21394ad840 23'277 rep_tid=1719 committed?=1 applied?=1) wants=d
2016-10-25 08:51:33.239450 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'277 crt=23'273 lcod 23'276 mlcod 23'275 active+clean]  commit: repgather(0x7f21394ad840 23'277 rep_tid=1719 committed?=1 applied?=1)
2016-10-25 08:51:33.239462 7f211246e700 15 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'277 crt=23'273 lcod 23'276 mlcod 23'275 active+clean] log_op_stats osd_op(client.4144.0:874 1.283bef5e benchmark_data_target167114251239.teuthology_13304_object873 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 1.839136 lat 1.839163
2016-10-25 08:51:33.239474 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'277 crt=23'273 lcod 23'276 mlcod 23'275 active+clean]  sending commit on osd_op(client.4144.0:874 1.283bef5e benchmark_data_target167114251239.teuthology_13304_object873 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f213c575340
2016-10-25 08:51:33.239485 7f211246e700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(874 benchmark_data_target167114251239.teuthology_13304_object873 [write 0~4194304] v23'277 uv277 ondisk = 0) v7 -- 0x7f213c575340 con 0
2016-10-25 08:51:33.239540 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'324 crt=23'321 lcod 23'323 mlcod 23'322 active+clean]  applied: repgather(0x7f21394abb00 23'324 rep_tid=1718 committed?=1 applied?=1) 
2016-10-25 08:51:33.239556 7f210fc69700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'324 crt=23'321 lcod 23'323 mlcod 23'322 active+clean] publish_stats_to_osd 23: no change since 2016-10-25 08:51:32.755365
2016-10-25 08:51:33.239568 7f210fc69700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'324 crt=23'321 lcod 23'323 mlcod 23'323 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:33.239578 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'324 crt=23'321 lcod 23'323 mlcod 23'323 active+clean]  removing repgather(0x7f21394abb00 23'324 rep_tid=1718 committed?=1 applied?=1)
2016-10-25 08:51:33.239587 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'324 crt=23'321 lcod 23'323 mlcod 23'323 active+clean]    q front is repgather(0x7f21394abb00 23'324 rep_tid=1718 committed?=1 applied?=1)
2016-10-25 08:51:33.239596 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'324 crt=23'321 lcod 23'323 mlcod 23'323 active+clean] remove_repop repgather(0x7f21394abb00 23'324 rep_tid=1718 committed?=1 applied?=1)
2016-10-25 08:51:33.239609 7f210fc69700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'324 crt=23'321 lcod 23'323 mlcod 23'323 active+clean]  requeue_ops 
2016-10-25 08:51:33.239635 7f210fc69700 10 osd.3 23 dequeue_op 0x7f21397707e0 finish
2016-10-25 08:51:33.239657 7f210fc69700 10 osd.3 23 dequeue_op 0x7f2139772520 prio 196 cost 0 latency 0.007202 osd_repop_reply(client.4144.0:875 1.1) v1 pg pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'324 crt=23'321 lcod 23'323 mlcod 23'323 active+clean]
2016-10-25 08:51:33.239672 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'324 crt=23'321 lcod 23'323 mlcod 23'323 active+clean] handle_message: 0x7f2139772520
2016-10-25 08:51:33.239686 7f210fc69700  7 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'324 crt=23'321 lcod 23'323 mlcod 23'323 active+clean] sub_op_modify_reply: tid 1720 op  ack_type 4 from 2
2016-10-25 08:51:33.239704 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'324 crt=23'321 lcod 23'323 mlcod 23'323 active+clean] repop_all_applied: repop tid 1720 all applied 
2016-10-25 08:51:33.239713 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'324 crt=23'321 lcod 23'323 mlcod 23'323 active+clean] eval_repop repgather(0x7f2139880480 23'325 rep_tid=1720 committed?=0 applied?=1) wants=d
2016-10-25 08:51:33.239722 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'324 crt=23'321 lcod 23'323 mlcod 23'323 active+clean]  applied: repgather(0x7f2139880480 23'325 rep_tid=1720 committed?=0 applied?=1) 
2016-10-25 08:51:33.239732 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'324 crt=23'321 lcod 23'323 mlcod 23'323 active+clean] repop_all_committed: repop tid 1720 all committed 
2016-10-25 08:51:33.239741 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'325 crt=23'321 lcod 23'324 mlcod 23'323 active+clean] eval_repop repgather(0x7f2139880480 23'325 rep_tid=1720 committed?=1 applied?=1) wants=d
2016-10-25 08:51:33.239750 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'325 crt=23'321 lcod 23'324 mlcod 23'323 active+clean]  commit: repgather(0x7f2139880480 23'325 rep_tid=1720 committed?=1 applied?=1)
2016-10-25 08:51:33.239760 7f210fc69700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'325 crt=23'321 lcod 23'324 mlcod 23'323 active+clean] log_op_stats osd_op(client.4144.0:875 1.917ed781 benchmark_data_target167114251239.teuthology_13304_object874 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 1.836804 lat 1.836832
2016-10-25 08:51:33.239772 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'325 crt=23'321 lcod 23'324 mlcod 23'323 active+clean]  sending commit on osd_op(client.4144.0:875 1.917ed781 benchmark_data_target167114251239.teuthology_13304_object874 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f21431e7340
2016-10-25 08:51:33.239783 7f210fc69700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(875 benchmark_data_target167114251239.teuthology_13304_object874 [write 0~4194304] v23'325 uv325 ondisk = 0) v7 -- 0x7f21431e7340 con 0
2016-10-25 08:51:33.239799 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'325 crt=23'321 lcod 23'324 mlcod 23'323 active+clean]  applied: repgather(0x7f2139880480 23'325 rep_tid=1720 committed?=1 applied?=1) 
2016-10-25 08:51:33.239809 7f210fc69700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'325 crt=23'321 lcod 23'324 mlcod 23'323 active+clean] publish_stats_to_osd 23: no change since 2016-10-25 08:51:32.755365
2016-10-25 08:51:33.239818 7f210fc69700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'325 crt=23'321 lcod 23'324 mlcod 23'324 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:33.239828 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'325 crt=23'321 lcod 23'324 mlcod 23'324 active+clean]  removing repgather(0x7f2139880480 23'325 rep_tid=1720 committed?=1 applied?=1)
2016-10-25 08:51:33.239837 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'325 crt=23'321 lcod 23'324 mlcod 23'324 active+clean]    q front is repgather(0x7f2139880480 23'325 rep_tid=1720 committed?=1 applied?=1)
2016-10-25 08:51:33.239846 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'325 crt=23'321 lcod 23'324 mlcod 23'324 active+clean] remove_repop repgather(0x7f2139880480 23'325 rep_tid=1720 committed?=1 applied?=1)
2016-10-25 08:51:33.239857 7f210fc69700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'325 crt=23'321 lcod 23'324 mlcod 23'324 active+clean]  requeue_ops 
2016-10-25 08:51:33.239881 7f210fc69700 10 osd.3 23 dequeue_op 0x7f2139772520 finish
2016-10-25 08:51:33.239899 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'287 crt=23'283 lcod 23'285 mlcod 23'284 active+clean]  applied: repgather(0x7f2139883de0 23'286 rep_tid=1726 committed?=1 applied?=1) 
2016-10-25 08:51:33.239916 7f210ec67700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'287 crt=23'283 lcod 23'285 mlcod 23'284 active+clean] publish_stats_to_osd 23: no change since 2016-10-25 08:51:33.175417
2016-10-25 08:51:33.239927 7f210ec67700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'287 crt=23'283 lcod 23'285 mlcod 23'285 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:33.239937 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'287 crt=23'283 lcod 23'285 mlcod 23'285 active+clean]  removing repgather(0x7f2139883de0 23'286 rep_tid=1726 committed?=1 applied?=1)
2016-10-25 08:51:33.239946 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'287 crt=23'283 lcod 23'285 mlcod 23'285 active+clean]    q front is repgather(0x7f2139883de0 23'286 rep_tid=1726 committed?=1 applied?=1)
2016-10-25 08:51:33.239955 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'287 crt=23'283 lcod 23'285 mlcod 23'285 active+clean] remove_repop repgather(0x7f2139883de0 23'286 rep_tid=1726 committed?=1 applied?=1)
2016-10-25 08:51:33.239967 7f210ec67700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'287 crt=23'283 lcod 23'285 mlcod 23'285 active+clean]  requeue_ops 
2016-10-25 08:51:33.239997 7f210ec67700 10 osd.3 23 dequeue_op 0x7f21398834e0 finish
2016-10-25 08:51:33.240039 7f210ec67700 10 osd.3 23 dequeue_op 0x7f21398822e0 prio 196 cost 0 latency 0.007745 osd_repop_reply(client.4144.0:884 1.3) v1 pg pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'287 crt=23'283 lcod 23'285 mlcod 23'285 active+clean]
2016-10-25 08:51:33.240056 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'287 crt=23'283 lcod 23'285 mlcod 23'285 active+clean] handle_message: 0x7f21398822e0
2016-10-25 08:51:33.240071 7f210ec67700  7 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'287 crt=23'283 lcod 23'285 mlcod 23'285 active+clean] sub_op_modify_reply: tid 1727 op  ack_type 4 from 2
2016-10-25 08:51:33.240090 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'287 crt=23'283 lcod 23'285 mlcod 23'285 active+clean] repop_all_applied: repop tid 1727 all applied 
2016-10-25 08:51:33.240099 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'287 crt=23'283 lcod 23'285 mlcod 23'285 active+clean] eval_repop repgather(0x7f21397be760 23'287 rep_tid=1727 committed?=0 applied?=1) wants=d
2016-10-25 08:51:33.240109 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'287 crt=23'283 lcod 23'285 mlcod 23'285 active+clean]  applied: repgather(0x7f21397be760 23'287 rep_tid=1727 committed?=0 applied?=1) 
2016-10-25 08:51:33.240122 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'287 crt=23'283 lcod 23'285 mlcod 23'285 active+clean] repop_all_committed: repop tid 1727 all committed 
2016-10-25 08:51:33.240131 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'287 lua=23'287 crt=23'283 lcod 23'286 mlcod 23'285 active+clean] eval_repop repgather(0x7f21397be760 23'287 rep_tid=1727 committed?=1 applied?=1) wants=d
2016-10-25 08:51:33.240151 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'287 lua=23'287 crt=23'283 lcod 23'286 mlcod 23'285 active+clean]  commit: repgather(0x7f21397be760 23'287 rep_tid=1727 committed?=1 applied?=1)
2016-10-25 08:51:33.240164 7f210ec67700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'287 lua=23'287 crt=23'283 lcod 23'286 mlcod 23'285 active+clean] log_op_stats osd_op(client.4144.0:884 1.2438ed73 benchmark_data_target167114251239.teuthology_13304_object883 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 0.533817 lat 0.533859
2016-10-25 08:51:33.240180 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'287 lua=23'287 crt=23'283 lcod 23'286 mlcod 23'285 active+clean]  sending commit on osd_op(client.4144.0:884 1.2438ed73 benchmark_data_target167114251239.teuthology_13304_object883 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f213c5763c0
2016-10-25 08:51:33.240192 7f210ec67700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(884 benchmark_data_target167114251239.teuthology_13304_object883 [write 0~4194304] v23'287 uv287 ondisk = 0) v7 -- 0x7f213c5763c0 con 0
2016-10-25 08:51:33.240211 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'287 lua=23'287 crt=23'283 lcod 23'286 mlcod 23'285 active+clean]  applied: repgather(0x7f21397be760 23'287 rep_tid=1727 committed?=1 applied?=1) 
2016-10-25 08:51:33.240222 7f210ec67700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'287 lua=23'287 crt=23'283 lcod 23'286 mlcod 23'285 active+clean] publish_stats_to_osd 23: no change since 2016-10-25 08:51:33.175417
2016-10-25 08:51:33.240233 7f210ec67700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'287 lua=23'287 crt=23'283 lcod 23'286 mlcod 23'286 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:33.240243 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'287 lua=23'287 crt=23'283 lcod 23'286 mlcod 23'286 active+clean]  removing repgather(0x7f21397be760 23'287 rep_tid=1727 committed?=1 applied?=1)
2016-10-25 08:51:33.240251 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'287 lua=23'287 crt=23'283 lcod 23'286 mlcod 23'286 active+clean]    q front is repgather(0x7f21397be760 23'287 rep_tid=1727 committed?=1 applied?=1)
2016-10-25 08:51:33.240260 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'287 lua=23'287 crt=23'283 lcod 23'286 mlcod 23'286 active+clean] remove_repop repgather(0x7f21397be760 23'287 rep_tid=1727 committed?=1 applied?=1)
2016-10-25 08:51:33.240272 7f210ec67700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'287 lua=23'287 crt=23'283 lcod 23'286 mlcod 23'286 active+clean]  requeue_ops 
2016-10-25 08:51:33.240302 7f210ec67700 10 osd.3 23 dequeue_op 0x7f21398822e0 finish
2016-10-25 08:51:33.240317 7f211146c700 10 osd.3 23 dequeue_op 0x7f21398805a0 prio 196 cost 0 latency 0.007996 osd_repop_reply(client.4144.0:886 1.3) v1 pg pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'287 lua=23'287 crt=23'283 lcod 23'286 mlcod 23'286 active+clean]
2016-10-25 08:51:33.240334 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'287 lua=23'287 crt=23'283 lcod 23'286 mlcod 23'286 active+clean] handle_message: 0x7f21398805a0
2016-10-25 08:51:33.240349 7f211146c700  7 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'287 lua=23'287 crt=23'283 lcod 23'286 mlcod 23'286 active+clean] sub_op_modify_reply: tid 1729 op  ack_type 4 from 2
2016-10-25 08:51:33.240369 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'287 lua=23'287 crt=23'283 lcod 23'286 mlcod 23'286 active+clean] repop_all_committed: repop tid 1729 all committed 
2016-10-25 08:51:33.240378 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'287 crt=23'283 lcod 23'287 mlcod 23'286 active+clean] eval_repop repgather(0x7f21397169a0 23'288 rep_tid=1729 committed?=1 applied?=0) wants=d
2016-10-25 08:51:33.240388 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'287 crt=23'283 lcod 23'287 mlcod 23'286 active+clean]  commit: repgather(0x7f21397169a0 23'288 rep_tid=1729 committed?=1 applied?=0)
2016-10-25 08:51:33.240399 7f211146c700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'287 crt=23'283 lcod 23'287 mlcod 23'286 active+clean] log_op_stats osd_op(client.4144.0:886 1.fcb15813 benchmark_data_target167114251239.teuthology_13304_object885 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 0.000000 lat 0.483896
2016-10-25 08:51:33.240411 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'287 crt=23'283 lcod 23'287 mlcod 23'286 active+clean]  sending commit on osd_op(client.4144.0:886 1.fcb15813 benchmark_data_target167114251239.teuthology_13304_object885 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f2142ef9600
2016-10-25 08:51:33.240423 7f211146c700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(886 benchmark_data_target167114251239.teuthology_13304_object885 [write 0~4194304] v23'288 uv288 ondisk = 0) v7 -- 0x7f2142ef9600 con 0
2016-10-25 08:51:33.240440 7f211146c700 10 osd.3 23 dequeue_op 0x7f21398805a0 finish
2016-10-25 08:51:33.240478 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'277 crt=23'273 lcod 23'276 mlcod 23'275 active+clean]  applied: repgather(0x7f21394ad840 23'277 rep_tid=1719 committed?=1 applied?=1) 
2016-10-25 08:51:33.240490 7f211246e700 15 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'277 crt=23'273 lcod 23'276 mlcod 23'275 active+clean] publish_stats_to_osd 23: no change since 2016-10-25 08:51:32.738102
2016-10-25 08:51:33.240502 7f211246e700 15 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'277 crt=23'273 lcod 23'276 mlcod 23'276 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:33.240512 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'277 crt=23'273 lcod 23'276 mlcod 23'276 active+clean]  removing repgather(0x7f21394ad840 23'277 rep_tid=1719 committed?=1 applied?=1)
2016-10-25 08:51:33.240522 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'277 crt=23'273 lcod 23'276 mlcod 23'276 active+clean]    q front is repgather(0x7f21394ad840 23'277 rep_tid=1719 committed?=1 applied?=1)
2016-10-25 08:51:33.240531 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'277 crt=23'273 lcod 23'276 mlcod 23'276 active+clean] remove_repop repgather(0x7f21394ad840 23'277 rep_tid=1719 committed?=1 applied?=1)
2016-10-25 08:51:33.240544 7f211246e700 15 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'277 crt=23'273 lcod 23'276 mlcod 23'276 active+clean]  requeue_ops 
2016-10-25 08:51:33.240569 7f211246e700 10 osd.3 23 dequeue_op 0x7f2139770b40 finish
2016-10-25 08:51:33.240589 7f211246e700 10 osd.3 23 dequeue_op 0x7f21397710e0 prio 196 cost 0 latency 0.008099 osd_repop_reply(client.4144.0:878 1.1) v1 pg pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'325 crt=23'321 lcod 23'324 mlcod 23'324 active+clean]
2016-10-25 08:51:33.240606 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'325 crt=23'321 lcod 23'324 mlcod 23'324 active+clean] handle_message: 0x7f21397710e0
2016-10-25 08:51:33.240621 7f211246e700  7 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'325 crt=23'321 lcod 23'324 mlcod 23'324 active+clean] sub_op_modify_reply: tid 1722 op  ack_type 4 from 2
2016-10-25 08:51:33.240640 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'325 crt=23'321 lcod 23'324 mlcod 23'324 active+clean] repop_all_applied: repop tid 1722 all applied 
2016-10-25 08:51:33.240649 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'325 crt=23'321 lcod 23'324 mlcod 23'324 active+clean] eval_repop repgather(0x7f21394ac2e0 23'326 rep_tid=1722 committed?=0 applied?=1) wants=d
2016-10-25 08:51:33.240659 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'325 crt=23'321 lcod 23'324 mlcod 23'324 active+clean]  applied: repgather(0x7f21394ac2e0 23'326 rep_tid=1722 committed?=0 applied?=1) 
2016-10-25 08:51:33.240670 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'325 crt=23'321 lcod 23'324 mlcod 23'324 active+clean] repop_all_committed: repop tid 1722 all committed 
2016-10-25 08:51:33.240679 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'321 lcod 23'325 mlcod 23'324 active+clean] eval_repop repgather(0x7f21394ac2e0 23'326 rep_tid=1722 committed?=1 applied?=1) wants=d
2016-10-25 08:51:33.240688 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'321 lcod 23'325 mlcod 23'324 active+clean]  commit: repgather(0x7f21394ac2e0 23'326 rep_tid=1722 committed?=1 applied?=1)
2016-10-25 08:51:33.240698 7f211246e700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'321 lcod 23'325 mlcod 23'324 active+clean] log_op_stats osd_op(client.4144.0:878 1.7db9c9f9 benchmark_data_target167114251239.teuthology_13304_object877 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 1.593680 lat 1.593708
2016-10-25 08:51:33.240709 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'321 lcod 23'325 mlcod 23'324 active+clean]  sending commit on osd_op(client.4144.0:878 1.7db9c9f9 benchmark_data_target167114251239.teuthology_13304_object877 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f2142e679c0
2016-10-25 08:51:33.240720 7f211246e700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(878 benchmark_data_target167114251239.teuthology_13304_object877 [write 0~4194304] v23'326 uv326 ondisk = 0) v7 -- 0x7f2142e679c0 con 0
2016-10-25 08:51:33.240737 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'321 lcod 23'325 mlcod 23'324 active+clean]  applied: repgather(0x7f21394ac2e0 23'326 rep_tid=1722 committed?=1 applied?=1) 
2016-10-25 08:51:33.240747 7f211246e700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'321 lcod 23'325 mlcod 23'324 active+clean] publish_stats_to_osd 23: no change since 2016-10-25 08:51:32.755365
2016-10-25 08:51:33.240756 7f211246e700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'321 lcod 23'325 mlcod 23'325 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:33.240767 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'321 lcod 23'325 mlcod 23'325 active+clean]  removing repgather(0x7f21394ac2e0 23'326 rep_tid=1722 committed?=1 applied?=1)
2016-10-25 08:51:33.240777 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'321 lcod 23'325 mlcod 23'325 active+clean]    q front is repgather(0x7f21394ac2e0 23'326 rep_tid=1722 committed?=1 applied?=1)
2016-10-25 08:51:33.240788 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'321 lcod 23'325 mlcod 23'325 active+clean] remove_repop repgather(0x7f21394ac2e0 23'326 rep_tid=1722 committed?=1 applied?=1)
2016-10-25 08:51:33.240800 7f211246e700 15 osd.3 pg_epoch: 23 pg[1.1( v 23'326 (0'0,23'326] local-les=22 n=326 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'321 lcod 23'325 mlcod 23'325 active+clean]  requeue_ops 
2016-10-25 08:51:33.240827 7f211246e700 10 osd.3 23 dequeue_op 0x7f21397710e0 finish
2016-10-25 08:51:33.240848 7f210fc69700 10 osd.3 23 dequeue_op 0x7f2139772e20 prio 196 cost 0 latency 0.008379 osd_repop_reply(client.4144.0:879 1.6) v1 pg pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'277 crt=23'273 lcod 23'276 mlcod 23'276 active+clean]
2016-10-25 08:51:33.240863 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'277 crt=23'273 lcod 23'276 mlcod 23'276 active+clean] handle_message: 0x7f2139772e20
2016-10-25 08:51:33.240876 7f210fc69700  7 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'277 crt=23'273 lcod 23'276 mlcod 23'276 active+clean] sub_op_modify_reply: tid 1723 op  ack_type 4 from 2
2016-10-25 08:51:33.240893 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'277 crt=23'273 lcod 23'276 mlcod 23'276 active+clean] repop_all_applied: repop tid 1723 all applied 
2016-10-25 08:51:33.240902 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'277 crt=23'273 lcod 23'276 mlcod 23'276 active+clean] eval_repop repgather(0x7f213985cb40 23'278 rep_tid=1723 committed?=0 applied?=1) wants=d
2016-10-25 08:51:33.240912 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'277 crt=23'273 lcod 23'276 mlcod 23'276 active+clean]  applied: repgather(0x7f213985cb40 23'278 rep_tid=1723 committed?=0 applied?=1) 
2016-10-25 08:51:33.240922 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'277 crt=23'273 lcod 23'276 mlcod 23'276 active+clean] repop_all_committed: repop tid 1723 all committed 
2016-10-25 08:51:33.240930 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'276 active+clean] eval_repop repgather(0x7f213985cb40 23'278 rep_tid=1723 committed?=1 applied?=1) wants=d
2016-10-25 08:51:33.240939 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'276 active+clean]  commit: repgather(0x7f213985cb40 23'278 rep_tid=1723 committed?=1 applied?=1)
2016-10-25 08:51:33.240948 7f210fc69700 15 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'276 active+clean] log_op_stats osd_op(client.4144.0:879 1.6ccee076 benchmark_data_target167114251239.teuthology_13304_object878 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 inb 4194304 outb 0 rlat 1.542138 lat 1.542165
2016-10-25 08:51:33.240960 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'276 active+clean]  sending commit on osd_op(client.4144.0:879 1.6ccee076 benchmark_data_target167114251239.teuthology_13304_object878 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 0x7f21431e6dc0
2016-10-25 08:51:33.240972 7f210fc69700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:0/83112327 -- osd_op_reply(879 benchmark_data_target167114251239.teuthology_13304_object878 [write 0~4194304] v23'278 uv278 ondisk = 0) v7 -- 0x7f21431e6dc0 con 0
2016-10-25 08:51:33.240988 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'276 active+clean]  applied: repgather(0x7f213985cb40 23'278 rep_tid=1723 committed?=1 applied?=1) 
2016-10-25 08:51:33.240999 7f210fc69700 15 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'276 active+clean] publish_stats_to_osd 23: no change since 2016-10-25 08:51:32.738102
2016-10-25 08:51:33.241008 7f210fc69700 15 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:33.241017 7f210fc69700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean]  removing repgather(0x7f213985cb40 23'278 rep_tid=1723 committed?=1 applied?=1)
2016-10-25 08:51:33.241025 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean]    q front is repgather(0x7f213985cb40 23'278 rep_tid=1723 committed?=1 applied?=1)
2016-10-25 08:51:33.241033 7f210fc69700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] remove_repop repgather(0x7f213985cb40 23'278 rep_tid=1723 committed?=1 applied?=1)
2016-10-25 08:51:33.241045 7f210fc69700 15 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean]  requeue_ops 
2016-10-25 08:51:33.241085 7f210fc69700 10 osd.3 23 dequeue_op 0x7f2139772e20 finish
2016-10-25 08:51:33.241954 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.3_head/#1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head#
2016-10-25 08:51:33.242022 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.3_head/#1:c81a8d3f:::benchmark_data_target167114251239.teuthology_13304_object885:head# = 0
2016-10-25 08:51:33.242053 7f2121400700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.3_head/#1:c0000000::::head#
2016-10-25 08:51:33.242079 7f2121400700 10 filestore oid: #1:c0000000::::head# not skipping op, *spos 2369.0.2
2016-10-25 08:51:33.242093 7f2121400700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:33.242144 7f2121400700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.3_head/#1:c0000000::::head# = 0
2016-10-25 08:51:33.242165 7f2121400700 10 journal op_apply_finish 2369 open_ops 1 -> 0, max_applied_seq 2368 -> 2369
2016-10-25 08:51:33.242179 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f214141a660 seq 2369 r = 0, finisher 0x7f2142e0b320 0x7f2147f35280
2016-10-25 08:51:33.242200 7f2121400700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f214141a660 seq 2369 osr(1.3 0x7f2139475570)/0x7f2139475570 lat 0.482821
2016-10-25 08:51:33.243290 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'287 crt=23'283 lcod 23'287 mlcod 23'286 active+clean] op_applied: 1729
2016-10-25 08:51:33.243328 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 lua=23'287 crt=23'283 lcod 23'287 mlcod 23'286 active+clean] op_applied version 23'288
2016-10-25 08:51:33.243335 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'286 active+clean] repop_all_applied: repop tid 1729 all applied 
2016-10-25 08:51:33.243341 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'286 active+clean] eval_repop repgather(0x7f21397169a0 23'288 rep_tid=1729 committed?=1 applied?=1) wants=d
2016-10-25 08:51:33.243347 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'286 active+clean]  commit: repgather(0x7f21397169a0 23'288 rep_tid=1729 committed?=1 applied?=1)
2016-10-25 08:51:33.243352 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'286 active+clean]  applied: repgather(0x7f21397169a0 23'288 rep_tid=1729 committed?=1 applied?=1) 
2016-10-25 08:51:33.243359 7f211f3fc700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'286 active+clean] publish_stats_to_osd 23: no change since 2016-10-25 08:51:33.175417
2016-10-25 08:51:33.243367 7f211f3fc700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] do_osd_op_effects client.4144 con 0x7f2147f52800
2016-10-25 08:51:33.243373 7f211f3fc700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean]  removing repgather(0x7f21397169a0 23'288 rep_tid=1729 committed?=1 applied?=1)
2016-10-25 08:51:33.243378 7f211f3fc700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean]    q front is repgather(0x7f21397169a0 23'288 rep_tid=1729 committed?=1 applied?=1)
2016-10-25 08:51:33.243382 7f211f3fc700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] remove_repop repgather(0x7f21397169a0 23'288 rep_tid=1729 committed?=1 applied?=1)
2016-10-25 08:51:33.243391 7f211f3fc700 15 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean]  requeue_ops 
2016-10-25 08:51:33.243419 7f2122c03700 10 journal write_finish_thread_entry aio 1029881856~4202496 done
2016-10-25 08:51:33.243421 7f2122c03700 20 journal check_aio_completion
2016-10-25 08:51:33.243423 7f2122c03700 20 journal check_aio_completion completed seq 2370 1029881856~4202496
2016-10-25 08:51:33.243430 7f2122c03700 20 journal check_aio_completion queueing finishers through seq 2370
2016-10-25 08:51:33.243431 7f2122c03700 10 journal queue_completions_thru seq 2370 queueing seq 2370 0x7f2140fda0d0 lat 0.060504
2016-10-25 08:51:33.243443 7f2122c03700 20 journal write_finish_thread_entry sleeping
2016-10-25 08:51:33.243448 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) _journaled_ahead 0x7f2146a1da40 seq 2370 osr(1.0 0x7f2139475d80) [Transaction(0x7f2147eb8680)]
2016-10-25 08:51:33.243454 7f2122402700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_op 0x7f2146a1da40 seq 2370 osr(1.0 0x7f2139475d80) 4195584 bytes   (queue has 1 ops and 4195584 bytes)
2016-10-25 08:51:33.243458 7f2122402700 10 filestore(/var/lib/ceph/osd/ceph-3)  queueing ondisk 0x7f2142fe1840
2016-10-25 08:51:33.243475 7f21203fe700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'286 crt=23'284 lcod 23'285 mlcod 23'285 active+clean] op_commit: 1730
2016-10-25 08:51:33.245690 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 684 ==== osd_op(client.4144.0:889 1.54ac3c63 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (1808957430 0 1013604055) 0x7f2142e57c80 con 0x7f2147f52800
2016-10-25 08:51:33.245736 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:33.245743 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f21397734e0 prio 63 cost 4194304 latency 0.006123 osd_op(client.4144.0:889 1.54ac3c63 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:33.245785 7f210ec67700 10 osd.3 23 dequeue_op 0x7f21397734e0 prio 63 cost 4194304 latency 0.006165 osd_op(client.4144.0:889 1.54ac3c63 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean]
2016-10-25 08:51:33.245825 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] handle_message: 0x7f21397734e0
2016-10-25 08:51:33.245850 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:33.245877 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] do_op osd_op(client.4144.0:889 1.54ac3c63 benchmark_data_target167114251239.teuthology_13304_object888 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:33.245906 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] get_object_context: obc NOT found in cache: 1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:head
2016-10-25 08:51:33.245920 7f210ec67700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:head# '_'
2016-10-25 08:51:33.245986 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject888__head_54AC3C63__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:33.245997 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:head# '_' = -2
2016-10-25 08:51:33.246002 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] get_object_context: no obc for soid 1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:head but can_create
2016-10-25 08:51:33.246018 7f210ec67700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:snapdir# 'snapset'
2016-10-25 08:51:33.246044 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject888__snapdir_54AC3C63__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:33.246048 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:snapdir# 'snapset' = -2
2016-10-25 08:51:33.246073 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] create_object_context 0x7f2140fc9e40 1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:head 
2016-10-25 08:51:33.246096 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] populate_obc_watchers 1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:head
2016-10-25 08:51:33.246107 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:head
2016-10-25 08:51:33.246117 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] get_object_context: 0x7f2140fc9e40 1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:head rwstate(none n=0 w=0) oi: 1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f213946d2c0 snapset: 0=[]:[]
2016-10-25 08:51:33.246134 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] find_object_context 1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:head @head oi=1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:33.246149 7f210ec67700 25 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] do_op oi 1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:33.246165 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] get_object_context: obc NOT found in cache: 1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:snapdir
2016-10-25 08:51:33.246177 7f210ec67700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:snapdir# '_'
2016-10-25 08:51:33.246200 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject888__snapdir_54AC3C63__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:33.246204 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:snapdir# '_' = -2
2016-10-25 08:51:33.246208 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] get_object_context: no obc for soid 1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:snapdir and !can_create
2016-10-25 08:51:33.246232 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] execute_ctx 0x7f2140fc5c00
2016-10-25 08:51:33.246244 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] do_op 1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:head [write 0~4194304] ov 0'0 av 23'289 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:33.246258 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] do_osd_op 1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:head [write 0~4194304]
2016-10-25 08:51:33.246269 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:33.246529 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] make_writeable 1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:head snapset=0x7f213946d2f8  snapc=0=[]
2016-10-25 08:51:33.246542 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean]  setting DIRTY flag
2016-10-25 08:51:33.246550 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] make_writeable 1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:head done, snapset=0=[]:[]+head
2016-10-25 08:51:33.246561 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] finish_ctx 1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:head 0x7f2140fc5c00 op modify  
2016-10-25 08:51:33.246574 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean]  set mtime to 2016-10-25 08:51:33.232730
2016-10-25 08:51:33.246593 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=288 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean]  final snapset 0=[]:[]+head in 1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:head
2016-10-25 08:51:33.246617 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean]  zeroing write result code 0
2016-10-25 08:51:33.246629 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean]  op order client.4144 tid 889 (first)
2016-10-25 08:51:33.246640 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] new_repop rep_tid 1731 on osd_op(client.4144.0:889 1.54ac3c63 benchmark_data_target167114251239.teuthology_13304_object888 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:33.246654 7f210ec67700  7 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] issue_repop rep_tid 1731 o 1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:head
2016-10-25 08:51:33.246702 7f210ec67700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:33.246705 7f210ec67700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:889 1.3 1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:head v 23'289) v1 -- 0x7f2140ff8600 con 0
2016-10-25 08:51:33.246993 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'288 (0'0,23'288] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] append_log log((0'0,23'288], crt=23'283) [23'289 (0'0) modify   1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:head by client.4144.0:889 2016-10-25 08:51:33.232730 0]
2016-10-25 08:51:33.247020 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'283 lcod 23'287 mlcod 23'287 active+clean] add_log_entry 23'289 (0'0) modify   1:c63c352a:::benchmark_data_target167114251239.teuthology_13304_object888:head by client.4144.0:889 2016-10-25 08:51:33.232730 0
2016-10-25 08:51:33.247038 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] append_log: trimming to 23'287 entries 23'284 (0'0) modify   1:cb7331cb:::benchmark_data_target167114251239.teuthology_13304_object863:head by client.4144.0:864 2016-10-25 08:51:31.284487 0,23'285 (0'0) modify   1:d0249860:::benchmark_data_target167114251239.teuthology_13304_object876:head by client.4144.0:877 2016-10-25 08:51:31.642446 0,23'286 (0'0) modify   1:da287092:::benchmark_data_target167114251239.teuthology_13304_object881:head by client.4144.0:882 2016-10-25 08:51:32.044478 0,23'287 (0'0) modify   1:ceb71c24:::benchmark_data_target167114251239.teuthology_13304_object883:head by client.4144.0:884 2016-10-25 08:51:32.704082 0
2016-10-25 08:51:33.247069 7f210ec67700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'289, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:33.247311 7f210ec67700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139241e40 osr(1.3 0x7f2139475570)
2016-10-25 08:51:33.247319 7f210ec67700 10 journal prepare_entry [Transaction(0x7f21481f8140)]
2016-10-25 08:51:33.247680 7f210ec67700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:33.248625 7f210ec67700 10 journal op_submit_start 2371
2016-10-25 08:51:33.248632 7f210ec67700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2371 [Transaction(0x7f21481f8140)]
2016-10-25 08:51:33.248637 7f210ec67700 10 journal op_journal_transactions 2371 reqid_t client.4144.0:889
2016-10-25 08:51:33.248641 7f210ec67700  5 journal submit_entry seq 2371 len 4202496 (0x7f2143096690)
2016-10-25 08:51:33.248991 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:33.249001 7f2123404700 10 journal room 908808191 max_size 1072709632 pos 1034084352 header.start 870187008 top 4096
2016-10-25 08:51:33.249004 7f2123404700 10 journal check_for_full at 1034084352 : 4202496 < 908808191
2016-10-25 08:51:33.249006 7f2123404700 15 journal prepare_single_write 1 will write 1034084352 : seq 2371 len 4195598 -> 4202496
2016-10-25 08:51:33.249023 7f2123404700 20 journal prepare_multi_write queue_pos now 1038286848
2016-10-25 08:51:33.249025 7f2123404700 15 journal do_aio_write writing 1034084352~4202496
2016-10-25 08:51:33.249028 7f2123404700 20 journal write_aio_bl 1034084352~4202496 seq 2371
2016-10-25 08:51:33.249031 7f2123404700 20 journal write_aio_bl .. 1034084352~4202496 in 1
2016-10-25 08:51:33.249389 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:33.249404 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:33.249407 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:33.249477 7f210ec67700 10 journal op_submit_finish 2371
2016-10-25 08:51:33.249488 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] eval_repop repgather(0x7f21398810e0 23'289 rep_tid=1731 committed?=0 applied?=0) wants=d
2016-10-25 08:51:33.249516 7f210ec67700 10 osd.3 23 dequeue_op 0x7f21397734e0 finish
2016-10-25 08:51:33.249654 7f2122c03700 20 journal write_finish_thread_entry waiting for aio(s)
2016-10-25 08:51:33.249678 7f2121c01700 10 journal op_apply_start 2370 open_ops 0 -> 1
2016-10-25 08:51:33.249680 7f2121c01700  5 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2146a1da40 seq 2370 osr(1.0 0x7f2139475d80)/0x7f2139475d80 start
2016-10-25 08:51:33.249685 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x7f2147eb8680
2016-10-25 08:51:33.249696 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) write 1.0_head/#1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head# 0~4194304
2016-10-25 08:51:33.252016 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 685 ==== osd_op(client.4144.0:890 1.16e6b296 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (229913409 0 4157791636) 0x7f21397a1e40 con 0x7f2147f52800
2016-10-25 08:51:33.252103 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:33.252127 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139773de0 prio 63 cost 4194304 latency 0.006360 osd_op(client.4144.0:890 1.16e6b296 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:33.252213 7f211246e700 10 osd.3 23 dequeue_op 0x7f2139773de0 prio 63 cost 4194304 latency 0.006446 osd_op(client.4144.0:890 1.16e6b296 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean]
2016-10-25 08:51:33.252247 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] handle_message: 0x7f2139773de0
2016-10-25 08:51:33.252269 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:33.252295 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] do_op osd_op(client.4144.0:890 1.16e6b296 benchmark_data_target167114251239.teuthology_13304_object889 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:33.252324 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] get_object_context: obc NOT found in cache: 1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:head
2016-10-25 08:51:33.252339 7f211246e700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:head# '_'
2016-10-25 08:51:33.252403 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.6_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject889__head_16E6B296__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:33.252415 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:head# '_' = -2
2016-10-25 08:51:33.252419 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] get_object_context: no obc for soid 1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:head but can_create
2016-10-25 08:51:33.252435 7f211246e700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:snapdir# 'snapset'
2016-10-25 08:51:33.252462 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.6_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject889__snapdir_16E6B296__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:33.252466 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:snapdir# 'snapset' = -2
2016-10-25 08:51:33.252491 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] create_object_context 0x7f2142efbc80 1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:head 
2016-10-25 08:51:33.252504 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] populate_obc_watchers 1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:head
2016-10-25 08:51:33.252514 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:head
2016-10-25 08:51:33.252524 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] get_object_context: 0x7f2142efbc80 1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:head rwstate(none n=0 w=0) oi: 1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f2139797480 snapset: 0=[]:[]
2016-10-25 08:51:33.252540 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] find_object_context 1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:head @head oi=1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:33.252556 7f211246e700 25 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] do_op oi 1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:33.252571 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] get_object_context: obc NOT found in cache: 1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:snapdir
2016-10-25 08:51:33.252581 7f211246e700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:snapdir# '_'
2016-10-25 08:51:33.252606 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.6_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject889__snapdir_16E6B296__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:33.252611 7f211246e700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.6_head/#1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:snapdir# '_' = -2
2016-10-25 08:51:33.252614 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] get_object_context: no obc for soid 1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:snapdir and !can_create
2016-10-25 08:51:33.252638 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] execute_ctx 0x7f2148073100
2016-10-25 08:51:33.252650 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] do_op 1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:head [write 0~4194304] ov 0'0 av 23'279 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:33.252665 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] do_osd_op 1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:head [write 0~4194304]
2016-10-25 08:51:33.252675 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:33.252937 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] make_writeable 1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:head snapset=0x7f21397974b8  snapc=0=[]
2016-10-25 08:51:33.252952 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean]  setting DIRTY flag
2016-10-25 08:51:33.252960 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] make_writeable 1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:head done, snapset=0=[]:[]+head
2016-10-25 08:51:33.252966 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) write 1.0_head/#1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head# 0~4194304 = 4194304
2016-10-25 08:51:33.252971 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] finish_ctx 1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:head 0x7f2148073100 op modify  
2016-10-25 08:51:33.252985 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean]  set mtime to 2016-10-25 08:51:33.232849
2016-10-25 08:51:33.253003 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.0_head/#1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head#
2016-10-25 08:51:33.253004 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=278 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean]  final snapset 0=[]:[]+head in 1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:head
2016-10-25 08:51:33.253027 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=279 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean]  zeroing write result code 0
2016-10-25 08:51:33.253033 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 1.0_head/#1:0f01cb55:::benchmark_data_target167114251239.teuthology_13304_object886:head# = 0
2016-10-25 08:51:33.253042 7f2121c01700 15 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.0_head/#1:00000000::::head#
2016-10-25 08:51:33.253041 7f211246e700 20 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=279 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean]  op order client.4144 tid 890 (first)
2016-10-25 08:51:33.253052 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=279 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] new_repop rep_tid 1732 on osd_op(client.4144.0:890 1.16e6b296 benchmark_data_target167114251239.teuthology_13304_object889 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:33.253069 7f2121c01700 10 filestore oid: #1:00000000::::head# not skipping op, *spos 2370.0.2
2016-10-25 08:51:33.253071 7f2121c01700 10 filestore  > header.spos 0.0.0
2016-10-25 08:51:33.253067 7f211246e700  7 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=279 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] issue_repop rep_tid 1732 o 1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:head
2016-10-25 08:51:33.253115 7f211246e700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:33.253118 7f2121c01700 20 filestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 1.0_head/#1:00000000::::head# = 0
2016-10-25 08:51:33.253122 7f2121c01700 10 journal op_apply_finish 2370 open_ops 1 -> 0, max_applied_seq 2369 -> 2370
2016-10-25 08:51:33.253124 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _do_op 0x7f2146a1da40 seq 2370 r = 0, finisher 0x7f2142fe2460 0x7f2148062340
2016-10-25 08:51:33.253118 7f211246e700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:890 1.6 1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:head v 23'279) v1 -- 0x7f2141054400 con 0
2016-10-25 08:51:33.253137 7f2121c01700 10 filestore(/var/lib/ceph/osd/ceph-3) _finish_op 0x7f2146a1da40 seq 2370 osr(1.0 0x7f2139475d80)/0x7f2139475d80 lat 0.071259
2016-10-25 08:51:33.253159 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'286 crt=23'284 lcod 23'285 mlcod 23'285 active+clean] op_applied: 1730
2016-10-25 08:51:33.253181 7f211ebfb700 10 osd.3 pg_epoch: 23 pg[1.0( v 23'287 (0'0,23'287] local-les=22 n=287 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'286 lua=23'286 crt=23'284 lcod 23'285 mlcod 23'285 active+clean] op_applied version 23'287
2016-10-25 08:51:33.253404 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'278 (0'0,23'278] local-les=22 n=279 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] append_log log((0'0,23'278], crt=23'273) [23'279 (0'0) modify   1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:head by client.4144.0:890 2016-10-25 08:51:33.232849 0]
2016-10-25 08:51:33.253426 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'279 (0'0,23'279] local-les=22 n=279 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'278 lua=23'278 crt=23'273 lcod 23'277 mlcod 23'277 active+clean] add_log_entry 23'279 (0'0) modify   1:694d6768:::benchmark_data_target167114251239.teuthology_13304_object889:head by client.4144.0:890 2016-10-25 08:51:33.232849 0
2016-10-25 08:51:33.253443 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'279 (0'0,23'279] local-les=22 n=279 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'278 lua=23'278 crt=23'277 lcod 23'277 mlcod 23'277 active+clean] append_log: trimming to 23'277 entries 23'274 (0'0) modify   1:71ee6361:::benchmark_data_target167114251239.teuthology_13304_object852:head by client.4144.0:853 2016-10-25 08:51:30.214879 0,23'275 (0'0) modify   1:6b771a0e:::benchmark_data_target167114251239.teuthology_13304_object866:head by client.4144.0:867 2016-10-25 08:51:31.343878 0,23'276 (0'0) modify   1:6e4f4fed:::benchmark_data_target167114251239.teuthology_13304_object870:head by client.4144.0:871 2016-10-25 08:51:31.346684 0,23'277 (0'0) modify   1:7af7dc14:::benchmark_data_target167114251239.teuthology_13304_object873:head by client.4144.0:874 2016-10-25 08:51:31.349512 0
2016-10-25 08:51:33.253475 7f211246e700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'279, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:33.253732 7f211246e700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139243700 osr(1.6 0x7f2139474f10)
2016-10-25 08:51:33.253739 7f211246e700 10 journal prepare_entry [Transaction(0x7f2147f0ec40)]
2016-10-25 08:51:33.253992 7f211246e700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:33.255030 7f211246e700 10 journal op_submit_start 2372
2016-10-25 08:51:33.255039 7f211246e700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2372 [Transaction(0x7f2147f0ec40)]
2016-10-25 08:51:33.255044 7f211246e700 10 journal op_journal_transactions 2372 reqid_t client.4144.0:890
2016-10-25 08:51:33.255048 7f211246e700  5 journal submit_entry seq 2372 len 4202496 (0x7f21430986d0)
2016-10-25 08:51:33.259898 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:33.259911 7f2123404700 20 journal write_thread_entry aio throttle: aio num 1 bytes 4202496 ... exp 2 min_new 4 ... pending 4202496
2016-10-25 08:51:33.259917 7f2123404700 10 journal room 904605695 max_size 1072709632 pos 1038286848 header.start 870187008 top 4096
2016-10-25 08:51:33.259920 7f2123404700 10 journal check_for_full at 1038286848 : 4202496 < 904605695
2016-10-25 08:51:33.259922 7f2123404700 15 journal prepare_single_write 1 will write 1038286848 : seq 2372 len 4195598 -> 4202496
2016-10-25 08:51:33.259947 7f2123404700 20 journal prepare_multi_write queue_pos now 1042489344
2016-10-25 08:51:33.259950 7f2123404700 15 journal do_aio_write writing 1038286848~4202496
2016-10-25 08:51:33.259952 7f2123404700 20 journal write_aio_bl 1038286848~4202496 seq 2372
2016-10-25 08:51:33.259957 7f2123404700 20 journal write_aio_bl .. 1038286848~4202496 in 1
2016-10-25 08:51:33.260343 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:33.260348 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:33.260351 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:33.260465 7f211246e700 10 journal op_submit_finish 2372
2016-10-25 08:51:33.260488 7f211246e700 10 osd.3 pg_epoch: 23 pg[1.6( v 23'279 (0'0,23'279] local-les=22 n=279 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'278 lua=23'278 crt=23'277 lcod 23'277 mlcod 23'277 active+clean] eval_repop repgather(0x7f21397710e0 23'279 rep_tid=1732 committed?=0 applied?=0) wants=d
2016-10-25 08:51:33.260518 7f211246e700 10 osd.3 23 dequeue_op 0x7f2139773de0 finish
2016-10-25 08:51:33.262792 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 686 ==== osd_op(client.4144.0:891 1.d387f46b (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (2135274234 0 791094306) 0x7f213c577c80 con 0x7f2147f52800
2016-10-25 08:51:33.262832 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:33.262840 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139771f80 prio 63 cost 4194304 latency 0.007273 osd_op(client.4144.0:891 1.d387f46b (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:33.262874 7f211146c700 10 osd.3 23 dequeue_op 0x7f2139771f80 prio 63 cost 4194304 latency 0.007307 osd_op(client.4144.0:891 1.d387f46b (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean]
2016-10-25 08:51:33.262896 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] handle_message: 0x7f2139771f80
2016-10-25 08:51:33.262912 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:33.262929 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] do_op osd_op(client.4144.0:891 1.d387f46b benchmark_data_target167114251239.teuthology_13304_object890 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:33.262949 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] get_object_context: obc NOT found in cache: 1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:head
2016-10-25 08:51:33.262958 7f211146c700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:head# '_'
2016-10-25 08:51:33.263122 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject890__head_D387F46B__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:33.263131 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:head# '_' = -2
2016-10-25 08:51:33.263134 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] get_object_context: no obc for soid 1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:head but can_create
2016-10-25 08:51:33.263145 7f211146c700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:snapdir# 'snapset'
2016-10-25 08:51:33.263164 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject890__snapdir_D387F46B__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:33.263167 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:snapdir# 'snapset' = -2
2016-10-25 08:51:33.263184 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] create_object_context 0x7f2142e81340 1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:head 
2016-10-25 08:51:33.263192 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] populate_obc_watchers 1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:head
2016-10-25 08:51:33.263197 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:head
2016-10-25 08:51:33.263203 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] get_object_context: 0x7f2142e81340 1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:head rwstate(none n=0 w=0) oi: 1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f213985bc60 snapset: 0=[]:[]
2016-10-25 08:51:33.263213 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] find_object_context 1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:head @head oi=1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:33.263223 7f211146c700 25 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] do_op oi 1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:33.263232 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] get_object_context: obc NOT found in cache: 1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:snapdir
2016-10-25 08:51:33.263238 7f211146c700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:snapdir# '_'
2016-10-25 08:51:33.263252 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject890__snapdir_D387F46B__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:33.263254 7f211146c700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:snapdir# '_' = -2
2016-10-25 08:51:33.263257 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] get_object_context: no obc for soid 1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:snapdir and !can_create
2016-10-25 08:51:33.263272 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] execute_ctx 0x7f21412fd100
2016-10-25 08:51:33.263282 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] do_op 1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:head [write 0~4194304] ov 0'0 av 23'290 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:33.263291 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] do_osd_op 1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:head [write 0~4194304]
2016-10-25 08:51:33.263297 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:33.263586 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] make_writeable 1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:head snapset=0x7f213985bc98  snapc=0=[]
2016-10-25 08:51:33.263597 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean]  setting DIRTY flag
2016-10-25 08:51:33.263602 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] make_writeable 1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:head done, snapset=0=[]:[]+head
2016-10-25 08:51:33.263609 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] finish_ctx 1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:head 0x7f21412fd100 op modify  
2016-10-25 08:51:33.263618 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean]  set mtime to 2016-10-25 08:51:33.232934
2016-10-25 08:51:33.263637 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=289 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean]  final snapset 0=[]:[]+head in 1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:head
2016-10-25 08:51:33.263651 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=290 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean]  zeroing write result code 0
2016-10-25 08:51:33.263660 7f211146c700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=290 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean]  op order client.4144 tid 891 (first)
2016-10-25 08:51:33.263665 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=290 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] new_repop rep_tid 1733 on osd_op(client.4144.0:891 1.d387f46b benchmark_data_target167114251239.teuthology_13304_object890 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:33.263674 7f211146c700  7 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=290 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] issue_repop rep_tid 1733 o 1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:head
2016-10-25 08:51:33.263705 7f211146c700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:33.263707 7f211146c700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:891 1.3 1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:head v 23'290) v1 -- 0x7f214812c000 con 0
2016-10-25 08:51:33.263991 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'289 (0'0,23'289] local-les=22 n=290 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] append_log log((0'0,23'289], crt=23'287) [23'290 (0'0) modify   1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:head by client.4144.0:891 2016-10-25 08:51:33.232934 0]
2016-10-25 08:51:33.264009 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'290 (0'0,23'290] local-les=22 n=290 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] add_log_entry 23'290 (0'0) modify   1:d62fe1cb:::benchmark_data_target167114251239.teuthology_13304_object890:head by client.4144.0:891 2016-10-25 08:51:33.232934 0
2016-10-25 08:51:33.264017 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'290 (0'0,23'290] local-les=22 n=290 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] append_log: trimming to 23'287 entries 
2016-10-25 08:51:33.264027 7f211146c700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'290, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:33.264218 7f211146c700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139241e40 osr(1.3 0x7f2139475570)
2016-10-25 08:51:33.264221 7f211146c700 10 journal prepare_entry [Transaction(0x7f2147ee0140)]
2016-10-25 08:51:33.264467 7f211146c700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:33.265301 7f211146c700 10 journal op_submit_start 2373
2016-10-25 08:51:33.265305 7f211146c700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2373 [Transaction(0x7f2147ee0140)]
2016-10-25 08:51:33.265308 7f211146c700 10 journal op_journal_transactions 2373 reqid_t client.4144.0:891
2016-10-25 08:51:33.265310 7f211146c700  5 journal submit_entry seq 2373 len 4202496 (0x7f21413483a0)
2016-10-25 08:51:33.265332 7f211146c700 10 journal op_submit_finish 2373
2016-10-25 08:51:33.265344 7f211146c700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'290 (0'0,23'290] local-les=22 n=290 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] eval_repop repgather(0x7f21398822e0 23'290 rep_tid=1733 committed?=0 applied?=0) wants=d
2016-10-25 08:51:33.265360 7f211146c700 10 osd.3 23 dequeue_op 0x7f2139771f80 finish
2016-10-25 08:51:33.265375 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:33.265377 7f2123404700 20 journal write_thread_entry aio throttle: aio num 2 bytes 8404992 ... exp 4 min_new 16 ... pending 4202496
2016-10-25 08:51:33.265380 7f2123404700 10 journal room 900403199 max_size 1072709632 pos 1042489344 header.start 870187008 top 4096
2016-10-25 08:51:33.265382 7f2123404700 10 journal check_for_full at 1042489344 : 4202496 < 900403199
2016-10-25 08:51:33.265384 7f2123404700 15 journal prepare_single_write 1 will write 1042489344 : seq 2373 len 4195598 -> 4202496
2016-10-25 08:51:33.265393 7f2123404700 20 journal prepare_multi_write queue_pos now 1046691840
2016-10-25 08:51:33.265394 7f2123404700 15 journal do_aio_write writing 1042489344~4202496
2016-10-25 08:51:33.265396 7f2123404700 20 journal write_aio_bl 1042489344~4202496 seq 2373
2016-10-25 08:51:33.265399 7f2123404700 20 journal write_aio_bl .. 1042489344~4202496 in 1
2016-10-25 08:51:33.265687 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:33.265690 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:33.265692 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:33.267092 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 687 ==== osd_op(client.4144.0:892 1.3879f2cf (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (567311647 0 3835668321) 0x7f21397a2940 con 0x7f2147f52800
2016-10-25 08:51:33.267118 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:33.267126 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139773720 prio 63 cost 4194304 latency 0.001418 osd_op(client.4144.0:892 1.3879f2cf (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:33.267186 7f210f468700 10 osd.3 23 dequeue_op 0x7f2139773720 prio 63 cost 4194304 latency 0.001478 osd_op(client.4144.0:892 1.3879f2cf (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean]
2016-10-25 08:51:33.267225 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] handle_message: 0x7f2139773720
2016-10-25 08:51:33.267251 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:33.267279 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] do_op osd_op(client.4144.0:892 1.3879f2cf benchmark_data_target167114251239.teuthology_13304_object891 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:33.267309 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] get_object_context: obc NOT found in cache: 1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:head
2016-10-25 08:51:33.267325 7f210f468700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:head# '_'
2016-10-25 08:51:33.267396 7f210f468700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.7_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject891__head_3879F2CF__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:33.267407 7f210f468700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:head# '_' = -2
2016-10-25 08:51:33.267412 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] get_object_context: no obc for soid 1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:head but can_create
2016-10-25 08:51:33.267433 7f210f468700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:snapdir# 'snapset'
2016-10-25 08:51:33.267460 7f210f468700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.7_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject891__snapdir_3879F2CF__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:33.267466 7f210f468700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:snapdir# 'snapset' = -2
2016-10-25 08:51:33.267491 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] create_object_context 0x7f2142f61b80 1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:head 
2016-10-25 08:51:33.267536 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] populate_obc_watchers 1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:head
2016-10-25 08:51:33.267546 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:head
2016-10-25 08:51:33.267556 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] get_object_context: 0x7f2142f61b80 1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:head rwstate(none n=0 w=0) oi: 1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f2139796e60 snapset: 0=[]:[]
2016-10-25 08:51:33.267572 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] find_object_context 1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:head @head oi=1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:33.267586 7f210f468700 25 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] do_op oi 1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:33.267604 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] get_object_context: obc NOT found in cache: 1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:snapdir
2016-10-25 08:51:33.267615 7f210f468700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:snapdir# '_'
2016-10-25 08:51:33.267641 7f210f468700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.7_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject891__snapdir_3879F2CF__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:33.267647 7f210f468700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.7_head/#1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:snapdir# '_' = -2
2016-10-25 08:51:33.267651 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] get_object_context: no obc for soid 1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:snapdir and !can_create
2016-10-25 08:51:33.267674 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] execute_ctx 0x7f2141587500
2016-10-25 08:51:33.267686 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] do_op 1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:head [write 0~4194304] ov 0'0 av 23'269 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:33.267699 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] do_osd_op 1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:head [write 0~4194304]
2016-10-25 08:51:33.267709 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:33.267970 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] make_writeable 1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:head snapset=0x7f2139796e98  snapc=0=[]
2016-10-25 08:51:33.267984 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean]  setting DIRTY flag
2016-10-25 08:51:33.267992 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] make_writeable 1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:head done, snapset=0=[]:[]+head
2016-10-25 08:51:33.268002 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] finish_ctx 1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:head 0x7f2141587500 op modify  
2016-10-25 08:51:33.268016 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean]  set mtime to 2016-10-25 08:51:33.238024
2016-10-25 08:51:33.268035 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=268 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean]  final snapset 0=[]:[]+head in 1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:head
2016-10-25 08:51:33.268057 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=269 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean]  zeroing write result code 0
2016-10-25 08:51:33.268069 7f210f468700 20 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=269 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean]  op order client.4144 tid 892 (first)
2016-10-25 08:51:33.268079 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=269 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] new_repop rep_tid 1734 on osd_op(client.4144.0:892 1.3879f2cf benchmark_data_target167114251239.teuthology_13304_object891 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:33.268093 7f210f468700  7 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=269 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] issue_repop rep_tid 1734 o 1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:head
2016-10-25 08:51:33.268164 7f210f468700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:33.268168 7f210f468700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:892 1.7 1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:head v 23'269) v1 -- 0x7f2141058a00 con 0
2016-10-25 08:51:33.268449 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'268 (0'0,23'268] local-les=22 n=269 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] append_log log((0'0,23'268], crt=23'264) [23'269 (0'0) modify   1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:head by client.4144.0:892 2016-10-25 08:51:33.238024 0]
2016-10-25 08:51:33.268472 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'269 (0'0,23'269] local-les=22 n=269 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'268 lua=23'268 crt=23'264 lcod 23'267 mlcod 23'267 active+clean] add_log_entry 23'269 (0'0) modify   1:f34f9e1c:::benchmark_data_target167114251239.teuthology_13304_object891:head by client.4144.0:892 2016-10-25 08:51:33.238024 0
2016-10-25 08:51:33.268489 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'269 (0'0,23'269] local-les=22 n=269 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'268 lua=23'268 crt=23'267 lcod 23'267 mlcod 23'267 active+clean] append_log: trimming to 23'267 entries 23'265 (0'0) modify   1:ffe055e2:::benchmark_data_target167114251239.teuthology_13304_object850:head by client.4144.0:851 2016-10-25 08:51:30.214691 0,23'266 (0'0) modify   1:ef337bc0:::benchmark_data_target167114251239.teuthology_13304_object865:head by client.4144.0:866 2016-10-25 08:51:31.343707 0,23'267 (0'0) modify   1:e06b003e:::benchmark_data_target167114251239.teuthology_13304_object869:head by client.4144.0:870 2016-10-25 08:51:31.346527 0
2016-10-25 08:51:33.268518 7f210f468700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'269, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:33.268753 7f210f468700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139243c80 osr(1.7 0x7f2139474be0)
2016-10-25 08:51:33.268760 7f210f468700 10 journal prepare_entry [Transaction(0x7f213eada380)]
2016-10-25 08:51:33.269012 7f210f468700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:33.269927 7f210f468700 10 journal op_submit_start 2374
2016-10-25 08:51:33.269933 7f210f468700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2374 [Transaction(0x7f213eada380)]
2016-10-25 08:51:33.269939 7f210f468700 10 journal op_journal_transactions 2374 reqid_t client.4144.0:892
2016-10-25 08:51:33.269943 7f210f468700  5 journal submit_entry seq 2374 len 4202496 (0x7f214811e460)
2016-10-25 08:51:33.270472 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:33.270479 7f2123404700 20 journal write_thread_entry aio throttle: aio num 3 bytes 12607488 ... exp 6 min_new 64 ... pending 4202496
2016-10-25 08:51:33.270483 7f2123404700 10 journal room 896200703 max_size 1072709632 pos 1046691840 header.start 870187008 top 4096
2016-10-25 08:51:33.270486 7f2123404700 10 journal check_for_full at 1046691840 : 4202496 < 896200703
2016-10-25 08:51:33.270488 7f2123404700 15 journal prepare_single_write 1 will write 1046691840 : seq 2374 len 4195598 -> 4202496
2016-10-25 08:51:33.270505 7f2123404700 20 journal prepare_multi_write queue_pos now 1050894336
2016-10-25 08:51:33.270507 7f2123404700 15 journal do_aio_write writing 1046691840~4202496
2016-10-25 08:51:33.270509 7f2123404700 20 journal write_aio_bl 1046691840~4202496 seq 2374
2016-10-25 08:51:33.270514 7f2123404700 20 journal write_aio_bl .. 1046691840~4202496 in 1
2016-10-25 08:51:33.271026 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:33.271032 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:33.271035 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:33.271108 7f210f468700 10 journal op_submit_finish 2374
2016-10-25 08:51:33.271118 7f210f468700 10 osd.3 pg_epoch: 23 pg[1.7( v 23'269 (0'0,23'269] local-les=22 n=269 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'268 lua=23'268 crt=23'267 lcod 23'267 mlcod 23'267 active+clean] eval_repop repgather(0x7f21395bb600 23'269 rep_tid=1734 committed?=0 applied?=0) wants=d
2016-10-25 08:51:33.271148 7f210f468700 10 osd.3 23 dequeue_op 0x7f2139773720 finish
2016-10-25 08:51:33.291046 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 688 ==== osd_op(client.4144.0:894 1.cde940d4 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (3994067688 0 1979897622) 0x7f21431e6dc0 con 0x7f2147f52800
2016-10-25 08:51:33.291099 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:33.291110 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139772640 prio 63 cost 4194304 latency 0.008813 osd_op(client.4144.0:894 1.cde940d4 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:33.291147 7f210e466700 10 osd.3 23 dequeue_op 0x7f2139772640 prio 63 cost 4194304 latency 0.008850 osd_op(client.4144.0:894 1.cde940d4 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean]
2016-10-25 08:51:33.291169 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] handle_message: 0x7f2139772640
2016-10-25 08:51:33.291185 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:33.291201 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] do_op osd_op(client.4144.0:894 1.cde940d4 benchmark_data_target167114251239.teuthology_13304_object893 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:33.291222 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] get_object_context: obc NOT found in cache: 1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:head
2016-10-25 08:51:33.291231 7f210e466700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.4_head/#1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:head# '_'
2016-10-25 08:51:33.291290 7f210e466700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.4_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject893__head_CDE940D4__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:33.291298 7f210e466700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.4_head/#1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:head# '_' = -2
2016-10-25 08:51:33.291301 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] get_object_context: no obc for soid 1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:head but can_create
2016-10-25 08:51:33.291311 7f210e466700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.4_head/#1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:snapdir# 'snapset'
2016-10-25 08:51:33.291327 7f210e466700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.4_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject893__snapdir_CDE940D4__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:33.291330 7f210e466700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.4_head/#1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:snapdir# 'snapset' = -2
2016-10-25 08:51:33.291348 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] create_object_context 0x7f213ed7c100 1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:head 
2016-10-25 08:51:33.291356 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] populate_obc_watchers 1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:head
2016-10-25 08:51:33.291362 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:head
2016-10-25 08:51:33.291367 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] get_object_context: 0x7f213ed7c100 1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:head rwstate(none n=0 w=0) oi: 1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0]) ssc: 0x7f213987b8e0 snapset: 0=[]:[]
2016-10-25 08:51:33.291376 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] find_object_context 1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:head @head oi=1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:33.291386 7f210e466700 25 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] do_op oi 1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:head(0'0 unknown.0.0:0 s 0 uv 0 alloc_hint [0 0 0])
2016-10-25 08:51:33.291395 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] get_object_context: obc NOT found in cache: 1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:snapdir
2016-10-25 08:51:33.291401 7f210e466700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.4_head/#1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:snapdir# '_'
2016-10-25 08:51:33.291414 7f210e466700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.4_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject893__snapdir_CDE940D4__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:33.291416 7f210e466700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.4_head/#1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:snapdir# '_' = -2
2016-10-25 08:51:33.291419 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] get_object_context: no obc for soid 1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:snapdir and !can_create
2016-10-25 08:51:33.291433 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] execute_ctx 0x7f2139422300
2016-10-25 08:51:33.291443 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] do_op 1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:head [write 0~4194304] ov 0'0 av 23'285 snapc 0=[] snapset 0=[]:[]
2016-10-25 08:51:33.291452 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] do_osd_op 1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:head [write 0~4194304]
2016-10-25 08:51:33.291458 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] do_osd_op  write 0~4194304
2016-10-25 08:51:33.296914 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] make_writeable 1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:head snapset=0x7f213987b918  snapc=0=[]
2016-10-25 08:51:33.296954 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean]  setting DIRTY flag
2016-10-25 08:51:33.296961 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] make_writeable 1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:head done, snapset=0=[]:[]+head
2016-10-25 08:51:33.296968 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] finish_ctx 1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:head 0x7f2139422300 op modify  
2016-10-25 08:51:33.296979 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean]  set mtime to 2016-10-25 08:51:33.269307
2016-10-25 08:51:33.296992 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=284 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean]  final snapset 0=[]:[]+head in 1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:head
2016-10-25 08:51:33.297010 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean]  zeroing write result code 0
2016-10-25 08:51:33.297019 7f210e466700 20 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean]  op order client.4144 tid 894 (first)
2016-10-25 08:51:33.297025 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] new_repop rep_tid 1735 on osd_op(client.4144.0:894 1.cde940d4 benchmark_data_target167114251239.teuthology_13304_object893 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:33.297035 7f210e466700  7 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] issue_repop rep_tid 1735 o 1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:head
2016-10-25 08:51:33.297066 7f210e466700 20 osd.3 23 share_map_peer 0x7f2139592800 already has epoch 23
2016-10-25 08:51:33.297069 7f210e466700  1 -- 167.114.251.239:6805/12812 --> 167.114.251.239:6801/12736 -- osd_repop(client.4144.0:894 1.4 1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:head v 23'285) v1 -- 0x7f2140f93000 con 0
2016-10-25 08:51:33.297336 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'284 (0'0,23'284] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] append_log log((0'0,23'284], crt=23'282) [23'285 (0'0) modify   1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:head by client.4144.0:894 2016-10-25 08:51:33.269307 0]
2016-10-25 08:51:33.297349 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'282 lcod 23'283 mlcod 23'283 active+clean] add_log_entry 23'285 (0'0) modify   1:2b0297b3:::benchmark_data_target167114251239.teuthology_13304_object893:head by client.4144.0:894 2016-10-25 08:51:33.269307 0
2016-10-25 08:51:33.297358 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] append_log: trimming to 23'283 entries 23'283 (0'0) modify   1:209285f5:::benchmark_data_target167114251239.teuthology_13304_object861:head by client.4144.0:862 2016-10-25 08:51:31.284082 0
2016-10-25 08:51:33.297371 7f210e466700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 23'285, trimmed: , clear_divergent_priors: 0
2016-10-25 08:51:33.297569 7f210e466700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions existing 0x7f2139242940 osr(1.4 0x7f2139475240)
2016-10-25 08:51:33.297574 7f210e466700 10 journal prepare_entry [Transaction(0x7f2147f79340)]
2016-10-25 08:51:33.297827 7f210e466700 10 journal  len 4195598 -> 4202496 (head 40 pre_pad 4046 bl 4195598 post_pad 2772 tail 40) (bl alignment 4086)
2016-10-25 08:51:33.298699 7f210e466700 10 journal op_submit_start 2375
2016-10-25 08:51:33.298711 7f210e466700  5 filestore(/var/lib/ceph/osd/ceph-3) queue_transactions (writeahead) 2375 [Transaction(0x7f2147f79340)]
2016-10-25 08:51:33.298714 7f210e466700 10 journal op_journal_transactions 2375 reqid_t client.4144.0:894
2016-10-25 08:51:33.298716 7f210e466700  5 journal submit_entry seq 2375 len 4202496 (0x7f2140ed6af0)
2016-10-25 08:51:33.298732 7f210e466700 10 journal op_submit_finish 2375
2016-10-25 08:51:33.298739 7f2123404700 20 journal write_thread_entry woke up
2016-10-25 08:51:33.298735 7f210e466700 10 osd.3 pg_epoch: 23 pg[1.4( v 23'285 (0'0,23'285] local-les=22 n=285 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'284 lua=23'284 crt=23'283 lcod 23'283 mlcod 23'283 active+clean] eval_repop repgather(0x7f213985dc20 23'285 rep_tid=1735 committed?=0 applied?=0) wants=d
2016-10-25 08:51:33.298749 7f210e466700 10 osd.3 23 dequeue_op 0x7f2139772640 finish
2016-10-25 08:51:33.298748 7f2123404700 20 journal write_thread_entry aio throttle: aio num 4 bytes 16809984 ... exp 8 min_new 256 ... pending 4202496
2016-10-25 08:51:33.298753 7f2123404700 10 journal room 891998207 max_size 1072709632 pos 1050894336 header.start 870187008 top 4096
2016-10-25 08:51:33.298756 7f2123404700 10 journal check_for_full at 1050894336 : 4202496 < 891998207
2016-10-25 08:51:33.298758 7f2123404700 15 journal prepare_single_write 1 will write 1050894336 : seq 2375 len 4195598 -> 4202496
2016-10-25 08:51:33.298779 7f2123404700 20 journal prepare_multi_write queue_pos now 1055096832
2016-10-25 08:51:33.298781 7f2123404700 15 journal do_aio_write writing 1050894336~4202496
2016-10-25 08:51:33.298783 7f2123404700 20 journal write_aio_bl 1050894336~4202496 seq 2375
2016-10-25 08:51:33.298787 7f2123404700 20 journal write_aio_bl .. 1050894336~4202496 in 1
2016-10-25 08:51:33.299161 7f2123404700 20 journal write_aio_bl io_submit return value: 1
2016-10-25 08:51:33.299167 7f2123404700  5 journal complete_write finished 1 ops and 4195598 bytes
2016-10-25 08:51:33.299170 7f2123404700 20 journal write_thread_entry going to sleep
2016-10-25 08:51:33.303541 7f2125408700 10 osd.3 23 tick_without_osd_lock
2016-10-25 08:51:33.303562 7f2125408700 25 osd.3 23 heartbeat_check osd.2 first_tx 2016-10-25 08:34:57.103177 last_tx 2016-10-25 08:51:32.641193 last_rx_back 2016-10-25 08:51:32.641193 last_rx_front 2016-10-25 08:51:32.641193
2016-10-25 08:51:33.303574 7f2125408700 20 osd.3 23 tick_without_osd_lock stats backoff 1 adjusted_min 5 - sending report
2016-10-25 08:51:33.303589 7f2125408700 20 osd.3 23 send_pg_stats
2016-10-25 08:51:33.303597 7f2125408700 10 osd.3 23 send_pg_stats - 6 pgs updated
2016-10-25 08:51:33.303603 7f2125408700 25 osd.3 23  sending 1.1 23:164
2016-10-25 08:51:33.303608 7f2125408700 25 osd.3 23  sending 1.0 23:139
2016-10-25 08:51:33.303611 7f2125408700 25 osd.3 23  sending 1.7 23:145
2016-10-25 08:51:33.303618 7f2125408700 25 osd.3 23  sending 1.6 23:136
2016-10-25 08:51:33.303622 7f2125408700 25 osd.3 23  sending 1.4 23:145
2016-10-25 08:51:33.303625 7f2125408700 25 osd.3 23  sending 1.3 23:153
2016-10-25 08:51:33.303628 7f2125408700 20 osd.3 23 send_pg_stats  updates pending: 200
2016-10-25 08:51:33.303638 7f2125408700  1 -- 167.114.251.239:6804/12812 --> 167.114.251.239:6789/0 -- pg_stats(6 pgs tid 200 v 0) v1 -- 0x7f2141379700 con 0
2016-10-25 08:51:33.303653 7f2125408700 20 osd.3 23 can_inc_scrubs_pending0 -> 1 (max 1, active 0)
2016-10-25 08:51:33.303657 7f2125408700 20 osd.3 23 scrub_time_permit should run between 0 - 24 now 8 = yes
2016-10-25 08:51:33.303653 7f2129b4e700  1 -- 167.114.251.239:6804/12812 <== client.4144 167.114.251.239:0/83112327 689 ==== osd_op(client.4144.0:895 1.e8dec853 (undecoded) ondisk+write+known_if_redirected e23) v7 ==== 246+0+4194304 (1329456558 0 267498045) 0x7f2142e679c0 con 0x7f2147f52800
2016-10-25 08:51:33.303674 7f2129b4e700 20 osd.3 23 should_share_map client.4144 167.114.251.239:0/83112327 23
2016-10-25 08:51:33.303681 7f2129b4e700 15 osd.3 23 enqueue_op 0x7f2139770a20 prio 63 cost 4194304 latency 0.004908 osd_op(client.4144.0:895 1.e8dec853 (undecoded) ondisk+write+known_if_redirected e23) v7
2016-10-25 08:51:33.303710 7f2125408700 20 osd.3 23 scrub_load_below_threshold loadavg 13.28 >= max 5 and ( >= daily_loadavg 0.504907 or >= 15m avg 2.15) = no
2016-10-25 08:51:33.303721 7f2125408700 20 osd.3 23 sched_scrub load_is_low=0
2016-10-25 08:51:33.303724 7f2125408700 10 osd.3 23 sched_scrub 1.4 scheduled at 2016-10-26 08:38:34.010902 > 2016-10-25 08:51:33.303656
2016-10-25 08:51:33.303728 7f2125408700 20 osd.3 23 sched_scrub done
2016-10-25 08:51:33.303731 7f2125408700 10 osd.3 23 promote_throttle_recalibrate 0 attempts, promoted 0 objects and 0  bytes; target 25 obj/sec or 5120 k bytes/sec
2016-10-25 08:51:33.303734 7f2125408700 20 osd.3 23 promote_throttle_recalibrate  new_prob 1000
2016-10-25 08:51:33.303747 7f2125408700 10 osd.3 23 promote_throttle_recalibrate  actual 0, actual/prob ratio 1, adjusted new_prob 1000, prob 1000 -> 1000
2016-10-25 08:51:33.303781 7f210ec67700 10 osd.3 23 dequeue_op 0x7f2139770a20 prio 63 cost 4194304 latency 0.005009 osd_op(client.4144.0:895 1.e8dec853 (undecoded) ondisk+write+known_if_redirected e23) v7 pg pg[1.3( v 23'290 (0'0,23'290] local-les=22 n=290 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean]
2016-10-25 08:51:33.303813 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'290 (0'0,23'290] local-les=22 n=290 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] handle_message: 0x7f2139770a20
2016-10-25 08:51:33.303835 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'290 (0'0,23'290] local-les=22 n=290 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] op_has_sufficient_caps pool=1 (rbd ) owner=0 need_read_cap=0 need_write_cap=1 classes=[] -> yes
2016-10-25 08:51:33.303858 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'290 (0'0,23'290] local-les=22 n=290 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] do_op osd_op(client.4144.0:895 1.e8dec853 benchmark_data_target167114251239.teuthology_13304_object894 [write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e23) v7 may_write -> write-ordered flags ondisk+write+known_if_redirected
2016-10-25 08:51:33.303883 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'290 (0'0,23'290] local-les=22 n=290 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] get_object_context: obc NOT found in cache: 1:ca137b17:::benchmark_data_target167114251239.teuthology_13304_object894:head
2016-10-25 08:51:33.303898 7f210ec67700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:ca137b17:::benchmark_data_target167114251239.teuthology_13304_object894:head# '_'
2016-10-25 08:51:33.303948 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject894__head_E8DEC853__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:33.303956 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:ca137b17:::benchmark_data_target167114251239.teuthology_13304_object894:head# '_' = -2
2016-10-25 08:51:33.303961 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'290 (0'0,23'290] local-les=22 n=290 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] get_object_context: no obc for soid 1:ca137b17:::benchmark_data_target167114251239.teuthology_13304_object894:head but can_create
2016-10-25 08:51:33.303975 7f210ec67700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:ca137b17:::benchmark_data_target167114251239.teuthology_13304_object894:snapdir# 'snapset'
2016-10-25 08:51:33.304003 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/1.3_head/benchmark\udata\utarget167114251239.teuthology\u13304\uobject894__snapdir_E8DEC853__1 with flags=2: (2) No such file or directory
2016-10-25 08:51:33.304008 7f210ec67700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 1.3_head/#1:ca137b17:::benchmark_data_target167114251239.teuthology_13304_object894:snapdir# 'snapset' = -2
2016-10-25 08:51:33.304032 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'290 (0'0,23'290] local-les=22 n=290 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] create_object_context 0x7f2143046100 1:ca137b17:::benchmark_data_target167114251239.teuthology_13304_object894:head 
2016-10-25 08:51:33.304044 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'290 (0'0,23'290] local-les=22 n=290 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] populate_obc_watchers 1:ca137b17:::benchmark_data_target167114251239.teuthology_13304_object894:head
2016-10-25 08:51:33.304054 7f210ec67700 20 osd.3 pg_epoch: 23 pg[1.3( v 23'290 (0'0,23'290] local-les=22 n=290 ec=17 les/c/f 22/22/0 21/21/21) [3,2] r=0 lpr=21 luod=23'288 lua=23'288 crt=23'287 lcod 23'287 mlcod 23'287 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 1:ca137b17:::benchmark_data_target167114251239.teuthology_13304_object894:head
2016-10-25 08:51:33.304064 7f210ec67700 10 osd.3 pg_epoch: 23 pg[1.3( v 23'290 (0'0,23'290] local-les=