ceph_slow_requests.txt

2015-11-11 09:25:20.360179 osd.3 [WRN] 22 slow requests, 22 included below; oldest blocked for > 30.079355 secs
2015-11-11 09:25:20.360190 osd.3 [WRN] slow request 30.025693 seconds old, received at 2015-11-11 09:24:50.334029: osd_op(mds.0.752:890064 10000b64dfc.00000000 [create 0~0,setxattr parent (
236)] 26.5dac1d0b ondisk+write+known_if_redirected e5941) currently no flag points reached
2015-11-11 09:25:20.360209 osd.3 [WRN] slow request 30.077728 seconds old, received at 2015-11-11 09:24:50.281994: osd_op(mds.0.752:889424 1000059b15a.00000000 [delete] 26.116d4830 snapc 1=
[] ondisk+write+known_if_redirected e5941) currently waiting for subops from 5,12,13
2015-11-11 09:25:20.360212 osd.3 [WRN] slow request 30.023940 seconds old, received at 2015-11-11 09:24:50.335782: osd_op(mds.0.752:890102 10000b64e0f.00000000 [create 0~0,setxattr parent (
234)] 26.8a48943a ondisk+write+known_if_redirected e5941) currently waiting for subops from 10,12,14
2015-11-11 09:25:20.360216 osd.3 [WRN] slow request 30.077226 seconds old, received at 2015-11-11 09:24:50.282495: osd_op(mds.0.752:889448 1000059b14e.00000000 [delete] 26.775d4cbb snapc 1=
[] ondisk+write+known_if_redirected e5941) currently commit_sent
2015-11-11 09:25:20.360219 osd.3 [WRN] slow request 30.072944 seconds old, received at 2015-11-11 09:24:50.286778: osd_op(mds.0.752:889512 1000059b12e.00000000 [delete] 26.99d48fcb snapc 1=
[] ondisk+write+known_if_redirected e5941) currently no flag points reached
2015-11-11 09:25:21.360365 osd.3 [WRN] 20 slow requests, 16 included below; oldest blocked for > 31.079941 secs
2015-11-11 09:25:21.360369 osd.3 [WRN] slow request 31.072182 seconds old, received at 2015-11-11 09:24:50.288125: osd_op(mds.0.752:889542 1000059b11f.00000000 [delete] 26.dda0134b snapc 1=
[] ondisk+write+known_if_redirected e5941) currently no flag points reached
2015-11-11 09:25:21.360376 osd.3 [WRN] slow request 31.071815 seconds old, received at 2015-11-11 09:24:50.288493: osd_op(mds.0.752:889548 1000059b11c.00000000 [delete] 26.5d2cc330 snapc 1=
[] ondisk+write+known_if_redirected e5941) currently waiting for subops from 5,12,13
2015-11-11 09:25:21.360379 osd.3 [WRN] slow request 31.067253 seconds old, received at 2015-11-11 09:24:50.293054: osd_op(mds.0.752:889674 1000059ac76.00000000 [delete] 26.be4f3d0b snapc 1=
[] ondisk+write+known_if_redirected e5941) currently no flag points reached
2015-11-11 09:25:21.360383 osd.3 [WRN] slow request 31.066589 seconds old, received at 2015-11-11 09:24:50.293719: osd_op(mds.0.752:889688 1000059ac41.00000000 [delete] 26.1f03933a snapc 1=
[] ondisk+write+known_if_redirected e5941) currently waiting for subops from 10,12,14
2015-11-11 09:25:21.360387 osd.3 [WRN] slow request 31.065099 seconds old, received at 2015-11-11 09:24:50.295208: osd_op(mds.0.752:889716 1000059abd7.00000000 [delete] 26.306b66ed snapc 1=
[] ondisk+write+known_if_redirected e5941) currently waiting for subops from 0,1,11
2015-11-11 09:25:22.360652 osd.3 [WRN] 23 slow requests, 14 included below; oldest blocked for > 32.080192 secs
2015-11-11 09:25:22.360661 osd.3 [WRN] slow request 30.170783 seconds old, received at 2015-11-11 09:24:52.189775: osd_op(client.9786104.0:379862 10000b659ea.00000000 [write 0~23930] 26.aee
953b0 snapc 1=[] ondisk+write+known_if_redirected e5941) currently waiting for subops from 5,12,13
2015-11-11 09:25:22.360677 osd.3 [WRN] slow request 30.046605 seconds old, received at 2015-11-11 09:24:52.313952: osd_op(client.9786104.0:379872 10000b659f4.00000000 [write 0~42553] 26.b3c
2e5b0 snapc 1=[] ondisk+write+known_if_redirected e5941) currently waiting for subops from 5,12,13
2015-11-11 09:25:22.360681 osd.3 [WRN] slow request 31.380938 seconds old, received at 2015-11-11 09:24:50.979620: osd_op(client.9786104.0:379760 10000b65984.00000000 [write 0~34682] 26.579
3e6cb snapc 1=[] ondisk+write+known_if_redirected e5941) currently no flag points reached
2015-11-11 09:25:22.360686 osd.3 [WRN] slow request 32.063867 seconds old, received at 2015-11-11 09:24:50.296691: osd_op(mds.0.752:889750 1000059ab43.00000000 [delete] 26.ab890e4b snapc 1=
[] ondisk+write+known_if_redirected e5941) currently no flag points reached
2015-11-11 09:25:22.360690 osd.3 [WRN] slow request 31.912902 seconds old, received at 2015-11-11 09:24:50.447656: osd_op(client.9786104.0:379710 10000b65952.00000000 [write 0~5714] 26.68a6
0cb0 snapc 1=[] ondisk+write+known_if_redirected e5941) currently waiting for subops from 5,12,13
2015-11-11 09:25:23.360851 osd.3 [WRN] 24 slow requests, 11 included below; oldest blocked for > 33.080416 secs
2015-11-11 09:25:23.360856 osd.3 [WRN] slow request 30.386750 seconds old, received at 2015-11-11 09:24:52.974033: osd_op(client.9786104.0:379936 10000b65a34.00000000 [write 0~24719] 26.455
813f0 snapc 1=[] ondisk+write+known_if_redirected e5941) currently waiting for subops from 5,12,13
2015-11-11 09:25:23.360863 osd.3 [WRN] slow request 30.092910 seconds old, received at 2015-11-11 09:24:53.267873: osd_op(client.9786104.0:379965 10000b65a51.00000000 [write 0~25556] 26.447
ee6cb snapc 1=[] ondisk+write+known_if_redirected e5941) currently no flag points reached
2015-11-11 09:25:23.360866 osd.3 [WRN] slow request 33.034594 seconds old, received at 2015-11-11 09:24:50.326188: osd_op(mds.0.752:889872 10000b64d9c.00000000 [create 0~0,setxattr parent (
236)] 26.d615d830 ondisk+write+known_if_redirected e5941) currently waiting for subops from 5,12,13
2015-11-11 09:25:23.360871 osd.3 [WRN] slow request 33.034087 seconds old, received at 2015-11-11 09:24:50.326696: osd_op(mds.0.752:889886 10000b64da3.00000000 [create 0~0,setxattr parent (
236)] 26.c3106aba ondisk+write+known_if_redirected e5941) currently waiting for subops from 10,12,14
2015-11-11 09:25:23.360874 osd.3 [WRN] slow request 32.997665 seconds old, received at 2015-11-11 09:24:50.363118: osd_op(client.9786104.0:379702 10000b6594a.00000000 [write 0~29521] 26.26e
82c4b snapc 1=[] ondisk+write+known_if_redirected e5941) currently waiting for subops from 4,11,14
2015-11-11 09:25:24.361039 osd.3 [WRN] 18 slow requests, 5 included below; oldest blocked for > 34.080608 secs
2015-11-11 09:25:24.361047 osd.3 [WRN] slow request 33.836580 seconds old, received at 2015-11-11 09:24:50.524394: osd_op(client.9786104.0:379716 10000b65958.00000000 [write 0~39439] 26.872
08d70 snapc 1=[] ondisk+write+known_if_redirected e5941) currently waiting for subops from 5,12,13
2015-11-11 09:25:24.361070 osd.3 [WRN] slow request 34.080608 seconds old, received at 2015-11-11 09:24:50.280366: osd_op(mds.0.752:889348 1000059b180.00000000 [delete] 26.2c78f2b0 snapc 1=
[] ondisk+write+known_if_redirected e5941) currently waiting for subops from 5,12,13
2015-11-11 09:25:24.361074 osd.3 [WRN] slow request 33.883248 seconds old, received at 2015-11-11 09:24:50.477727: osd_op(client.9786104.0:379713 10000b65955.00000000 [write 0~32644] 26.471
665f0 snapc 1=[] ondisk+write+known_if_redirected e5941) currently waiting for subops from 5,12,13
2015-11-11 09:25:24.361081 osd.3 [WRN] slow request 32.283744 seconds old, received at 2015-11-11 09:24:52.077230: osd_op(client.9786104.0:379852 10000b659e0.00000000 [write 0~33204] 26.a79
2954b snapc 1=[] ondisk+write+known_if_redirected e5941) currently commit_sent
2015-11-11 09:25:24.361084 osd.3 [WRN] slow request 30.240650 seconds old, received at 2015-11-11 09:24:54.120324: osd_op(client.9786104.0:380045 10000b65aa1.00000000 [write 0~40029] 26.955
d237a snapc 1=[] ondisk+write+known_if_redirected e5941) currently waiting for subops from 10,12,14
2015-11-11 09:25:25.361258 osd.3 [WRN] 16 slow requests, 5 included below; oldest blocked for > 35.080815 secs
2015-11-11 09:25:25.361265 osd.3 [WRN] slow request 30.079251 seconds old, received at 2015-11-11 09:24:55.281930: osd_op(mds.0.752:892215 1000059b42c.00000000 [delete] 26.576c4af0 snapc 1=
[] ondisk+write+known_if_redirected e5941) currently waiting for subops from 5,12,13
2015-11-11 09:25:25.361272 osd.3 [WRN] slow request 30.403463 seconds old, received at 2015-11-11 09:24:54.957718: osd_op(client.9786104.0:380122 10000b65aee.00000000 [write 0~38620] 26.d78
56ab0 snapc 1=[] ondisk+write+known_if_redirected e5941) currently waiting for subops from 5,12,13
2015-11-11 09:25:25.361275 osd.3 [WRN] slow request 30.185416 seconds old, received at 2015-11-11 09:24:55.175765: osd_op(client.9786104.0:380144 10000b65b04.00000000 [write 0~35792] 26.3c9
c1b3a snapc 1=[] ondisk+write+known_if_redirected e5941) currently waiting for subops from 10,12,14
2015-11-11 09:25:25.361279 osd.3 [WRN] slow request 30.167413 seconds old, received at 2015-11-11 09:24:55.193768: osd_op(client.9786104.0:380145 10000b65b05.00000000 [write 0~62702] 26.f2a
4ec3a snapc 1=[] ondisk+write+known_if_redirected e5941) currently waiting for subops from 10,12,14
2015-11-11 09:25:25.361282 osd.3 [WRN] slow request 30.121541 seconds old, received at 2015-11-11 09:24:55.239640: osd_op(client.9786104.0:380150 10000b65b0a.00000000 [write 0~32516] 26.5d0
1eccb snapc 1=[] ondisk+write+known_if_redirected e5941) currently commit_sent