Skip to content

Instantly share code, notes, and snippets.

@nuriel77

nuriel77/with_noout_flag

Last active December 8, 2016 15:53
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save nuriel77/92898cb8026c23499fbbda349733748b to your computer and use it in GitHub Desktop.
Save nuriel77/92898cb8026c23499fbbda349733748b to your computer and use it in GitHub Desktop.
Download ZIP
Turning off a node with x2 OSDs for more than 300 seconds (5 min), without and with "noout" flag set on ceph
Raw
with_noout_flag
# Shutting down a node with two OSDs
# "noout" flag is set, meaning no count down starts (default 300 sec)
# There will be no data re-placement.
########################################################################
[root@overcloud-controller-0 ~]# ceph -w
cluster ce7cc1f4-bccc-11e6-a544-525400ac16ed
health HEALTH_WARN
80 pgs stale
1 requests are blocked > 32 sec
2/6 in osds are down
noout,sortbitwise flag(s) set
monmap e1: 1 mons at {overcloud-controller-0=172.16.1.8:6789/0}
election epoch 4, quorum 0 overcloud-controller-0
osdmap e191: 6 osds: 4 up, 6 in; 224 remapped pgs
flags noout,sortbitwise
pgmap v3361: 224 pgs, 6 pools, 3148 MB data, 2173 objects
9685 MB used, 206 GB / 215 GB avail
144 active+clean
80 stale+active+clean
2016-12-08 15:34:31.740861 mon.0 [INF] pgmap v3361: 224 pgs: 80 stale+active+clean, 144 active+clean; 3148 MB data, 9685 MB used, 206 GB / 215 GB avail
2016-12-08 15:34:34.919130 mon.0 [INF] pgmap v3362: 224 pgs: 103 active+undersized+degraded, 46 stale+active+clean, 75 active+clean; 3148 MB data, 9686 MB used, 206 GB / 215 GB avail; 244 B/s rd, 3 op/s; 808/6519 objects degraded (12.395%); 61276 kB/s, 0 objects/s recovering
2016-12-08 15:34:35.925762 mon.0 [INF] pgmap v3363: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9687 MB used, 206 GB / 215 GB avail; 244 B/s rd, 1 op/s; 2173/6519 objects degraded (33.333%)
2016-12-08 15:34:39.068785 mon.0 [INF] pgmap v3364: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9677 MB used, 206 GB / 215 GB avail; 2173/6519 objects degraded (33.333%); 0 B/s, 7 objects/s recovering
2016-12-08 15:34:41.076447 mon.0 [INF] pgmap v3365: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9677 MB used, 206 GB / 215 GB avail; 2173/6519 objects degraded (33.333%)
2016-12-08 15:35:05.239242 mon.0 [INF] pgmap v3366: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9677 MB used, 206 GB / 215 GB avail; 821 B/s wr, 1 op/s; 2215/6582 objects degraded (33.652%)
2016-12-08 15:35:09.074965 mon.0 [INF] pgmap v3367: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9678 MB used, 206 GB / 215 GB avail; 1133 B/s wr, 2 op/s; 2235/6612 objects degraded (33.802%)
2016-12-08 15:35:11.081511 mon.0 [INF] pgmap v3368: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9678 MB used, 206 GB / 215 GB avail; 2103 B/s wr, 2 op/s; 2239/6618 objects degraded (33.832%)
2016-12-08 15:35:14.076549 mon.0 [INF] pgmap v3369: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9678 MB used, 206 GB / 215 GB avail; 409 B/s wr, 1 op/s; 2239/6618 objects degraded (33.832%)
2016-12-08 15:35:16.084020 mon.0 [INF] pgmap v3370: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9678 MB used, 206 GB / 215 GB avail; 2239/6618 objects degraded (33.832%)
2016-12-08 15:36:00.252167 mon.0 [INF] pgmap v3371: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9678 MB used, 206 GB / 215 GB avail; 23 B/s wr, 0 op/s; 2241/6621 objects degraded (33.847%)
2016-12-08 15:36:04.086274 mon.0 [INF] pgmap v3372: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9678 MB used, 206 GB / 215 GB avail; 5098 B/s rd, 2474 B/s wr, 7 op/s; 2235/6612 objects degraded (33.802%)
2016-12-08 15:36:06.092273 mon.0 [INF] pgmap v3373: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9678 MB used, 206 GB / 215 GB avail; 169 kB/s rd, 88513 B/s wr, 270 op/s; 2211/6576 objects degraded (33.622%)
2016-12-08 15:36:09.086174 mon.0 [INF] pgmap v3374: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9680 MB used, 206 GB / 215 GB avail; 198 kB/s rd, 102 kB/s wr, 317 op/s; 2203/6564 objects degraded (33.562%)
2016-12-08 15:36:11.093371 mon.0 [INF] pgmap v3375: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9681 MB used, 206 GB / 215 GB avail; 113 kB/s rd, 54684 B/s wr, 176 op/s; 2185/6537 objects degraded (33.425%)
2016-12-08 15:36:14.088035 mon.0 [INF] pgmap v3376: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9681 MB used, 206 GB / 215 GB avail; 67162 B/s rd, 29690 B/s wr, 101 op/s; 2185/6537 objects degraded (33.425%)
2016-12-08 15:36:16.094828 mon.0 [INF] pgmap v3377: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9681 MB used, 206 GB / 215 GB avail; 2185/6537 objects degraded (33.425%)
2016-12-08 15:37:04.097292 mon.0 [INF] pgmap v3378: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9682 MB used, 206 GB / 215 GB avail; 106 B/s rd, 1002 B/s wr, 1 op/s; 2179/6528 objects degraded (33.379%)
2016-12-08 15:37:06.104664 mon.0 [INF] pgmap v3379: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9682 MB used, 206 GB / 215 GB avail; 5938 B/s rd, 2948 B/s wr, 9 op/s; 2173/6519 objects degraded (33.333%)
2016-12-08 15:37:09.097804 mon.0 [INF] pgmap v3380: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9682 MB used, 206 GB / 215 GB avail; 58357 B/s rd, 19861 B/s wr, 79 op/s; 2173/6519 objects degraded (33.333%)
2016-12-08 15:37:11.105459 mon.0 [INF] pgmap v3381: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9682 MB used, 206 GB / 215 GB avail; 2173/6519 objects degraded (33.333%)
^[[C^[[D^H^H^H^H^C[root@overcloud-controller-0 ~]# date
Thu Dec 8 15:40:18 UTC 2016
[root@overcloud-controller-0 ~]#
### More than 5 minutes passed since node/OSDs are down. We see no notification of OSDs down for more than ... seconds
### No data replaced.
### Node being booted back up:
[root@overcloud-controller-0 ~]# ceph -w
cluster ce7cc1f4-bccc-11e6-a544-525400ac16ed
health HEALTH_WARN
224 pgs degraded
224 pgs stuck unclean
224 pgs undersized
recovery 2173/6519 objects degraded (33.333%)
2/6 in osds are down
noout,sortbitwise flag(s) set
monmap e1: 1 mons at {overcloud-controller-0=172.16.1.8:6789/0}
election epoch 4, quorum 0 overcloud-controller-0
osdmap e191: 6 osds: 4 up, 6 in; 224 remapped pgs
flags noout,sortbitwise
pgmap v3381: 224 pgs, 6 pools, 3148 MB data, 2173 objects
9682 MB used, 206 GB / 215 GB avail
2173/6519 objects degraded (33.333%)
224 active+undersized+degraded
2016-12-08 15:37:11.105459 mon.0 [INF] pgmap v3381: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9682 MB used, 206 GB / 215 GB avail; 2173/6519 objects degraded (33.333%)
2016-12-08 15:40:53.231917 mon.0 [INF] from='client.? 172.16.1.15:0/390530742' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "args": ["host=overcloud-novacompute-0", "root=default"], "id": 0, "weight": 0.0351}]: dispatch
2016-12-08 15:40:56.535274 mon.0 [INF] osd.0 172.16.1.15:6800/3012 boot
2016-12-08 15:40:56.536873 mon.0 [INF] osdmap e192: 6 osds: 5 up, 6 in
2016-12-08 15:40:56.538102 mon.0 [INF] pgmap v3382: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9682 MB used, 206 GB / 215 GB avail; 2173/6519 objects degraded (33.333%)
2016-12-08 15:40:57.543953 mon.0 [INF] osdmap e193: 6 osds: 5 up, 6 in
2016-12-08 15:40:57.545006 mon.0 [INF] pgmap v3383: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9682 MB used, 206 GB / 215 GB avail; 2173/6519 objects degraded (33.333%)
2016-12-08 15:40:58.550637 mon.0 [INF] osdmap e194: 6 osds: 5 up, 6 in
2016-12-08 15:40:58.551735 mon.0 [INF] pgmap v3384: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9682 MB used, 206 GB / 215 GB avail; 2173/6519 objects degraded (33.333%)
2016-12-08 15:40:58.777156 mon.0 [INF] from='client.? 172.16.1.15:0/2512575288' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "args": ["host=overcloud-novacompute-0", "root=default"], "id": 3, "weight": 0.0351}]: dispatch
2016-12-08 15:40:59.555232 mon.0 [INF] pgmap v3385: 224 pgs: 5 active+degraded, 34 active+clean, 185 active+undersized+degraded; 3148 MB data, 9682 MB used, 206 GB / 215 GB avail; 1930/6519 objects degraded (29.606%)
2016-12-08 15:41:00.564012 mon.0 [INF] osd.3 172.16.1.15:6802/3273 boot
2016-12-08 15:41:00.564112 mon.0 [INF] osdmap e195: 6 osds: 6 up, 6 in
2016-12-08 15:41:00.565502 mon.0 [INF] pgmap v3386: 224 pgs: 1 active+recovering+degraded, 8 active+degraded, 69 active+clean, 146 active+undersized+degraded; 3148 MB data, 9683 MB used, 206 GB / 215 GB avail; 1711/6519 objects degraded (26.246%); 625 kB/s, 40 objects/s recovering
2016-12-08 15:41:00.566662 mon.0 [INF] pgmap v3387: 224 pgs: 1 active+recovering+degraded, 8 active+degraded, 69 active+clean, 146 active+undersized+degraded; 3148 MB data, 9683 MB used, 206 GB / 215 GB avail; 1711/6519 objects degraded (26.246%); 1240 kB/s, 80 objects/s recovering
2016-12-08 15:41:01.569206 mon.0 [INF] pgmap v3388: 224 pgs: 2 active+recovering+degraded, 11 active+degraded, 101 active+clean, 110 active+undersized+degraded; 3148 MB data, 9684 MB used, 206 GB / 215 GB avail; 1389/6519 objects degraded (21.307%)
2016-12-08 15:41:01.572968 mon.0 [INF] osdmap e196: 6 osds: 6 up, 6 in
2016-12-08 15:41:01.574148 mon.0 [INF] pgmap v3389: 224 pgs: 2 active+recovering+degraded, 11 active+degraded, 101 active+clean, 110 active+undersized+degraded; 3148 MB data, 9684 MB used, 206 GB / 215 GB avail; 1389/6519 objects degraded (21.307%)
2016-12-08 15:41:02.579834 mon.0 [INF] osdmap e197: 6 osds: 6 up, 6 in
2016-12-08 15:41:02.580834 mon.0 [INF] pgmap v3390: 224 pgs: 2 active+recovering+degraded, 11 active+degraded, 101 active+clean, 110 active+undersized+degraded; 3148 MB data, 9684 MB used, 206 GB / 215 GB avail; 1389/6519 objects degraded (21.307%)
2016-12-08 15:41:03.585029 mon.0 [INF] osdmap e198: 6 osds: 6 up, 6 in
2016-12-08 15:41:03.586140 mon.0 [INF] pgmap v3391: 224 pgs: 2 active+recovering+degraded, 11 active+degraded, 101 active+clean, 110 active+undersized+degraded; 3148 MB data, 9684 MB used, 206 GB / 215 GB avail; 1389/6519 objects degraded (21.307%)
2016-12-08 15:41:04.590017 mon.0 [INF] pgmap v3392: 224 pgs: 2 active+recovering+degraded, 7 active+degraded, 135 active+clean, 80 active+undersized+degraded; 3148 MB data, 9685 MB used, 206 GB / 215 GB avail; 1143/6519 objects degraded (17.533%); 2254 kB/s, 143 objects/s recovering
2016-12-08 15:41:05.595390 mon.0 [INF] pgmap v3393: 224 pgs: 2 active+recovering+degraded, 12 active+degraded, 210 active+clean; 3148 MB data, 9688 MB used, 206 GB / 215 GB avail; 370/6519 objects degraded (5.676%); 0 B/s, 354 objects/s recovering
2016-12-08 15:41:06.601513 mon.0 [INF] pgmap v3394: 224 pgs: 1 active+recovering+degraded, 9 active+degraded, 214 active+clean; 3148 MB data, 9688 MB used, 206 GB / 215 GB avail; 278/6519 objects degraded (4.264%); 0 B/s, 209 objects/s recovering
2016-12-08 15:41:09.139962 mon.0 [INF] pgmap v3395: 224 pgs: 1 active+recovering+degraded, 8 active+degraded, 215 active+clean; 3148 MB data, 9688 MB used, 206 GB / 215 GB avail; 258/6519 objects degraded (3.958%); 167 kB/s, 11 objects/s recovering
2016-12-08 15:41:10.147203 mon.0 [INF] pgmap v3396: 224 pgs: 1 active+recovering+degraded, 2 active+degraded, 221 active+clean; 3148 MB data, 9688 MB used, 206 GB / 215 GB avail; 83/6519 objects degraded (1.273%); 167 kB/s, 11 objects/s recovering
2016-12-08 15:41:11.151054 mon.0 [INF] pgmap v3397: 224 pgs: 224 active+clean; 3148 MB data, 9688 MB used, 206 GB / 215 GB avail; 1349 kB/s, 82 objects/s recovering
2016-12-08 15:41:14.140755 mon.0 [INF] pgmap v3398: 224 pgs: 224 active+clean; 3148 MB data, 9688 MB used, 206 GB / 215 GB avail; 679 kB/s, 41 objects/s recovering
2016-12-08 15:41:15.144234 mon.0 [INF] pgmap v3399: 224 pgs: 224 active+clean; 3148 MB data, 9688 MB used, 206 GB / 215 GB avail
2016-12-08 15:41:16.150121 mon.0 [INF] pgmap v3400: 224 pgs: 224 active+clean; 3148 MB data, 9688 MB used, 206 GB / 215 GB avail
Raw
without_noout_flag
# Shutting down a node with two OSDs
# No "noout" flag set, meaning a count down starts (default 300 sec)
# When count down finishes, data will be replaced.
# Watch for timestamp: 2016-12-08 15:28:50.846398
########################################################################
[root@overcloud-controller-0 ~]# ceph -w
cluster ce7cc1f4-bccc-11e6-a544-525400ac16ed
health HEALTH_OK
monmap e1: 1 mons at {overcloud-controller-0=172.16.1.8:6789/0}
election epoch 4, quorum 0 overcloud-controller-0
osdmap e176: 6 osds: 6 up, 6 in
flags sortbitwise
pgmap v3258: 224 pgs, 6 pools, 3148 MB data, 2173 objects
9673 MB used, 206 GB / 215 GB avail
224 active+clean
client io 679 B/s rd, 53 op/s rd, 0 op/s wr
2016-12-08 15:23:26.945456 mon.0 [INF] pgmap v3258: 224 pgs: 224 active+clean; 3148 MB data, 9673 MB used, 206 GB / 215 GB avail; 679 B/s rd, 53 op/s
2016-12-08 15:23:28.951811 mon.0 [INF] pgmap v3259: 224 pgs: 224 active+clean; 3148 MB data, 9673 MB used, 206 GB / 215 GB avail; 679 B/s rd, 28 op/s
2016-12-08 15:23:33.934039 mon.0 [INF] pgmap v3260: 224 pgs: 224 active+clean; 3148 MB data, 9673 MB used, 206 GB / 215 GB avail
2016-12-08 15:23:38.936221 mon.0 [INF] pgmap v3261: 224 pgs: 224 active+clean; 3148 MB data, 9673 MB used, 206 GB / 215 GB avail
2016-12-08 15:23:43.936126 mon.0 [INF] pgmap v3262: 224 pgs: 224 active+clean; 3148 MB data, 9673 MB used, 206 GB / 215 GB avail
2016-12-08 15:23:47.687187 mon.0 [INF] osd.0 marked itself down
2016-12-08 15:23:47.690254 mon.0 [INF] osd.3 marked itself down
2016-12-08 15:23:47.747592 mon.0 [INF] osdmap e177: 6 osds: 4 up, 6 in
2016-12-08 15:23:47.749449 mon.0 [INF] pgmap v3263: 224 pgs: 224 active+clean; 3148 MB data, 9673 MB used, 206 GB / 215 GB avail
2016-12-08 15:23:48.759116 mon.0 [INF] osdmap e178: 6 osds: 4 up, 6 in
2016-12-08 15:23:48.761244 mon.0 [INF] pgmap v3264: 224 pgs: 80 stale+active+clean, 144 active+clean; 3148 MB data, 9673 MB used, 206 GB / 215 GB avail
2016-12-08 15:23:49.765431 mon.0 [INF] pgmap v3265: 224 pgs: 103 active+undersized+degraded, 46 stale+active+clean, 75 active+clean; 3148 MB data, 9673 MB used, 206 GB / 215 GB avail; 808/6519 objects degraded (12.395%)
2016-12-08 15:23:50.771890 mon.0 [INF] pgmap v3266: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9673 MB used, 206 GB / 215 GB avail; 2173/6519 objects degraded (33.333%)
2016-12-08 15:23:53.938271 mon.0 [INF] pgmap v3267: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9674 MB used, 206 GB / 215 GB avail; 2173/6519 objects degraded (33.333%)
2016-12-08 15:23:55.946025 mon.0 [INF] pgmap v3268: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9675 MB used, 206 GB / 215 GB avail; 2173/6519 objects degraded (33.333%)
2016-12-08 15:23:58.938504 mon.0 [INF] pgmap v3269: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9675 MB used, 206 GB / 215 GB avail; 2173/6519 objects degraded (33.333%)
2016-12-08 15:25:05.128955 mon.0 [INF] pgmap v3270: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9675 MB used, 206 GB / 215 GB avail; 30 B/s rd, 293 B/s wr, 0 op/s; 2211/6576 objects degraded (33.622%)
2016-12-08 15:25:08.951473 mon.0 [INF] pgmap v3271: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9676 MB used, 206 GB / 215 GB avail; 29 B/s rd, 438 B/s wr, 0 op/s; 2233/6609 objects degraded (33.787%)
2016-12-08 15:25:10.957401 mon.0 [INF] pgmap v3272: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9676 MB used, 206 GB / 215 GB avail; 2459 B/s wr, 3 op/s; 2239/6618 objects degraded (33.832%)
2016-12-08 15:25:13.952514 mon.0 [INF] pgmap v3273: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9676 MB used, 206 GB / 215 GB avail; 614 B/s wr, 1 op/s; 2239/6618 objects degraded (33.832%)
2016-12-08 15:25:15.961280 mon.0 [INF] pgmap v3274: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9675 MB used, 206 GB / 215 GB avail; 2239/6618 objects degraded (33.832%)
2016-12-08 15:26:00.137742 mon.0 [INF] pgmap v3275: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9675 MB used, 206 GB / 215 GB avail; 23 B/s wr, 0 op/s; 2241/6621 objects degraded (33.847%)
2016-12-08 15:26:03.961375 mon.0 [INF] pgmap v3276: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9675 MB used, 206 GB / 215 GB avail; 5077 B/s rd, 2431 B/s wr, 7 op/s; 2239/6618 objects degraded (33.832%)
2016-12-08 15:26:05.969391 mon.0 [INF] pgmap v3277: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9676 MB used, 206 GB / 215 GB avail; 167 kB/s rd, 87470 B/s wr, 267 op/s; 2211/6576 objects degraded (33.622%)
2016-12-08 15:26:08.961265 mon.0 [INF] pgmap v3278: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9671 MB used, 206 GB / 215 GB avail; 195 kB/s rd, 101 kB/s wr, 316 op/s; 2195/6552 objects degraded (33.501%)
2016-12-08 15:26:10.968150 mon.0 [INF] pgmap v3279: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9673 MB used, 206 GB / 215 GB avail; 116 kB/s rd, 56122 B/s wr, 181 op/s; 2185/6537 objects degraded (33.425%)
2016-12-08 15:26:13.963530 mon.0 [INF] pgmap v3280: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9673 MB used, 206 GB / 215 GB avail; 70029 B/s rd, 30714 B/s wr, 103 op/s; 2185/6537 objects degraded (33.425%)
2016-12-08 15:26:15.969861 mon.0 [INF] pgmap v3281: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9673 MB used, 206 GB / 215 GB avail; 2185/6537 objects degraded (33.425%)
2016-12-08 15:27:03.969978 mon.0 [INF] pgmap v3282: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9674 MB used, 206 GB / 215 GB avail; 85 B/s rd, 1002 B/s wr, 1 op/s; 2181/6531 objects degraded (33.395%)
2016-12-08 15:27:05.976354 mon.0 [INF] pgmap v3283: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9674 MB used, 206 GB / 215 GB avail; 5999 B/s rd, 2948 B/s wr, 9 op/s; 2173/6519 objects degraded (33.333%)
2016-12-08 15:27:08.971596 mon.0 [INF] pgmap v3284: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9674 MB used, 206 GB / 215 GB avail; 59181 B/s rd, 19863 B/s wr, 80 op/s; 2173/6519 objects degraded (33.333%)
2016-12-08 15:27:10.981302 mon.0 [INF] pgmap v3285: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 9674 MB used, 206 GB / 215 GB avail; 2173/6519 objects degraded (33.333%)
...
###### Note these lines, here the counter reaches 300 seconds OSDs down and data replacment starts:
2016-12-08 15:28:50.846398 mon.0 [INF] osd.0 out (down for 303.101847)
2016-12-08 15:28:50.846441 mon.0 [INF] osd.3 out (down for 303.101847)
2016-12-08 15:28:50.855987 mon.0 [INF] osdmap e179: 6 osds: 4 up, 4 in
2016-12-08 15:28:50.857469 mon.0 [INF] pgmap v3286: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 6453 MB used, 137 GB / 143 GB avail; 2173/6519 objects degraded (33.333%)
2016-12-08 15:28:51.861826 mon.0 [INF] osdmap e180: 6 osds: 4 up, 4 in
2016-12-08 15:28:51.862959 mon.0 [INF] pgmap v3287: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 6453 MB used, 137 GB / 143 GB avail; 2173/6519 objects degraded (33.333%)
2016-12-08 15:28:52.867407 mon.0 [INF] osdmap e181: 6 osds: 4 up, 4 in
2016-12-08 15:28:52.869470 mon.0 [INF] pgmap v3288: 224 pgs: 224 active+undersized+degraded; 3148 MB data, 6453 MB used, 137 GB / 143 GB avail; 2173/6519 objects degraded (33.333%)
2016-12-08 15:28:54.868761 mon.0 [INF] pgmap v3289: 224 pgs: 8 active+degraded, 22 active+degraded+remapped, 12 active+remapped, 9 active, 173 active+undersized+degraded; 3148 MB data, 6454 MB used, 137 GB / 143 GB avail; 2616/6519 objects degraded (40.129%); 411/6519 objects misplaced (6.305%); 11443 kB/s, 36 objects/s recovering
2016-12-08 15:28:55.876588 mon.0 [INF] pgmap v3290: 224 pgs: 21 active+degraded, 44 active+degraded+remapped, 26 active+remapped, 21 active, 1 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 6466 MB used, 137 GB / 143 GB avail; 3246/6519 objects degraded (49.793%); 1119/6519 objects misplaced (17.165%); 55074 kB/s, 47 objects/s recovering
2016-12-08 15:28:58.991067 mon.0 [INF] pgmap v3291: 224 pgs: 20 active+degraded, 41 active+degraded+remapped, 28 active+remapped, 22 active, 2 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 6534 MB used, 137 GB / 143 GB avail; 3122/6519 objects degraded (47.891%); 1119/6519 objects misplaced (17.165%); 64005 kB/s, 41 objects/s recovering
2016-12-08 15:29:00.998867 mon.0 [INF] pgmap v3292: 224 pgs: 19 active+degraded, 37 active+degraded+remapped, 32 active+remapped, 23 active, 2 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 6537 MB used, 137 GB / 143 GB avail; 3089/6519 objects degraded (47.385%); 1119/6519 objects misplaced (17.165%); 75471 kB/s, 38 objects/s recovering
2016-12-08 15:29:03.991501 mon.0 [INF] pgmap v3293: 224 pgs: 17 active+degraded, 35 active+degraded+remapped, 34 active+remapped, 25 active, 2 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 6678 MB used, 137 GB / 143 GB avail; 3064/6519 objects degraded (47.001%); 1119/6519 objects misplaced (17.165%); 93371 kB/s, 0 keys/s, 18 objects/s recovering
2016-12-08 15:29:05.998354 mon.0 [INF] pgmap v3294: 224 pgs: 1 active+recovery_wait+degraded+remapped, 17 active+degraded, 32 active+degraded+remapped, 36 active+remapped, 25 active, 2 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 6808 MB used, 137 GB / 143 GB avail; 204 B/s rd, 0 op/s; 3049/6519 objects degraded (46.771%); 1119/6519 objects misplaced (17.165%); 72092 kB/s, 0 keys/s, 12 objects/s recovering
2016-12-08 15:29:08.995443 mon.0 [INF] pgmap v3295: 224 pgs: 1 active+recovery_wait+degraded+remapped, 17 active+degraded, 29 active+degraded+remapped, 40 active+remapped, 25 active, 1 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 7006 MB used, 137 GB / 143 GB avail; 204 B/s rd, 0 op/s; 3023/6519 objects degraded (46.372%); 1119/6519 objects misplaced (17.165%); 86779 kB/s, 16 objects/s recovering
2016-12-08 15:29:11.003061 mon.0 [INF] pgmap v3296: 224 pgs: 17 active+degraded, 22 active+degraded+remapped, 49 active+remapped, 25 active, 111 active+undersized+degraded; 3148 MB data, 7170 MB used, 136 GB / 143 GB avail; 2831/6519 objects degraded (43.427%); 1119/6519 objects misplaced (17.165%); 108 MB/s, 87 objects/s recovering
2016-12-08 15:29:13.996324 mon.0 [INF] pgmap v3297: 224 pgs: 14 active+degraded, 18 active+degraded+remapped, 53 active+remapped, 28 active, 111 active+undersized+degraded; 3148 MB data, 7308 MB used, 136 GB / 143 GB avail; 2675/6519 objects degraded (41.034%); 1119/6519 objects misplaced (17.165%); 94794 kB/s, 117 objects/s recovering
2016-12-08 15:29:16.004148 mon.0 [INF] pgmap v3298: 224 pgs: 12 active+degraded, 16 active+degraded+remapped, 55 active+remapped, 30 active, 111 active+undersized+degraded; 3148 MB data, 7400 MB used, 136 GB / 143 GB avail; 2544/6519 objects degraded (39.024%); 1119/6519 objects misplaced (17.165%); 56386 kB/s, 65 objects/s recovering
2016-12-08 15:29:18.996644 mon.0 [INF] pgmap v3299: 224 pgs: 12 active+degraded, 14 active+degraded+remapped, 57 active+remapped, 30 active, 111 active+undersized+degraded; 3148 MB data, 7410 MB used, 136 GB / 143 GB avail; 2531/6519 objects degraded (38.825%); 1119/6519 objects misplaced (17.165%); 42922 kB/s, 29 objects/s recovering
2016-12-08 15:29:21.003846 mon.0 [INF] pgmap v3300: 224 pgs: 11 active+degraded, 13 active+degraded+remapped, 57 active+remapped, 31 active, 1 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 7573 MB used, 136 GB / 143 GB avail; 2505/6519 objects degraded (38.426%); 1119/6519 objects misplaced (17.165%); 50783 kB/s, 0 keys/s, 11 objects/s recovering
2016-12-08 15:29:24.013272 mon.0 [INF] pgmap v3301: 224 pgs: 1 active+recovery_wait+degraded, 3 active+recovery_wait+degraded+remapped, 10 active+degraded, 10 active+degraded+remapped, 57 active+remapped, 31 active, 1 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 7580 MB used, 136 GB / 143 GB avail; 0 B/s rd, 5 op/s; 2491/6519 objects degraded (38.211%); 1119/6519 objects misplaced (17.165%); 34291 kB/s, 0 keys/s, 8 objects/s recovering
2016-12-08 15:29:26.021705 mon.0 [INF] pgmap v3302: 224 pgs: 2 active+recovery_wait+degraded, 3 active+recovery_wait+degraded+remapped, 9 active+degraded, 10 active+degraded+remapped, 57 active+remapped, 31 active, 1 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 7620 MB used, 136 GB / 143 GB avail; 0 B/s rd, 7 op/s; 2472/6519 objects degraded (37.920%); 1119/6519 objects misplaced (17.165%); 29391 kB/s, 7 objects/s recovering
2016-12-08 15:29:28.998275 mon.0 [INF] pgmap v3303: 224 pgs: 2 active+recovery_wait+degraded, 3 active+recovery_wait+degraded+remapped, 9 active+degraded, 10 active+degraded+remapped, 57 active+remapped, 31 active, 1 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 7696 MB used, 136 GB / 143 GB avail; 0 B/s rd, 1 op/s; 2472/6519 objects degraded (37.920%); 1119/6519 objects misplaced (17.165%); 18077 kB/s, 4 objects/s recovering
2016-12-08 15:29:31.004534 mon.0 [INF] pgmap v3304: 224 pgs: 2 active+recovery_wait+degraded, 3 active+recovery_wait+degraded+remapped, 9 active+degraded, 10 active+degraded+remapped, 57 active+remapped, 31 active, 1 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 7716 MB used, 136 GB / 143 GB avail; 2470/6519 objects degraded (37.889%); 1119/6519 objects misplaced (17.165%); 3287 kB/s, 0 objects/s recovering
2016-12-08 15:29:33.999049 mon.0 [INF] pgmap v3305: 224 pgs: 2 active+recovery_wait+degraded, 3 active+recovery_wait+degraded+remapped, 9 active+degraded, 10 active+degraded+remapped, 57 active+remapped, 31 active, 1 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 7732 MB used, 136 GB / 143 GB avail; 2470/6519 objects degraded (37.889%); 1119/6519 objects misplaced (17.165%); 3276 kB/s, 0 objects/s recovering
2016-12-08 15:29:36.006236 mon.0 [INF] pgmap v3306: 224 pgs: 2 active+recovery_wait+degraded, 3 active+recovery_wait+degraded+remapped, 9 active+degraded, 10 active+degraded+remapped, 57 active+remapped, 31 active, 1 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 7731 MB used, 136 GB / 143 GB avail; 2467/6519 objects degraded (37.843%); 1119/6519 objects misplaced (17.165%); 4913 kB/s, 1 objects/s recovering
2016-12-08 15:29:39.001801 mon.0 [INF] pgmap v3307: 224 pgs: 2 active+recovery_wait+degraded, 2 active+recovery_wait+degraded+remapped, 8 active+degraded, 8 active+degraded+remapped, 60 active+remapped, 32 active, 1 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 7744 MB used, 136 GB / 143 GB avail; 2406/6519 objects degraded (36.908%); 1119/6519 objects misplaced (17.165%); 29707 kB/s, 21 objects/s recovering
2016-12-08 15:29:41.012650 mon.0 [INF] pgmap v3308: 224 pgs: 1 active+recovery_wait+degraded, 2 active+recovery_wait+degraded+remapped, 7 active+degraded, 6 active+degraded+remapped, 63 active+remapped, 34 active, 111 active+undersized+degraded; 3148 MB data, 7842 MB used, 136 GB / 143 GB avail; 2334/6519 objects degraded (35.803%); 1119/6519 objects misplaced (17.165%); 57776 kB/s, 44 objects/s recovering
2016-12-08 15:29:44.001454 mon.0 [INF] pgmap v3309: 224 pgs: 1 active+recovery_wait+degraded, 2 active+recovery_wait+degraded+remapped, 7 active+degraded, 5 active+degraded+remapped, 63 active+remapped, 34 active, 1 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 7874 MB used, 136 GB / 143 GB avail; 2332/6519 objects degraded (35.772%); 1119/6519 objects misplaced (17.165%); 36261 kB/s, 24 objects/s recovering
2016-12-08 15:29:46.007635 mon.0 [INF] pgmap v3310: 224 pgs: 1 active+recovery_wait+degraded, 2 active+recovery_wait+degraded+remapped, 7 active+degraded, 4 active+degraded+remapped, 63 active+remapped, 34 active, 2 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 7893 MB used, 136 GB / 143 GB avail; 2331/6519 objects degraded (35.757%); 1119/6519 objects misplaced (17.165%); 6554 kB/s, 1 objects/s recovering
2016-12-08 15:29:49.005404 mon.0 [INF] pgmap v3311: 224 pgs: 1 active+recovery_wait+degraded, 2 active+recovery_wait+degraded+remapped, 7 active+degraded, 3 active+degraded+remapped, 64 active+remapped, 34 active, 2 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 7909 MB used, 136 GB / 143 GB avail; 2315/6519 objects degraded (35.512%); 1119/6519 objects misplaced (17.165%); 29484 kB/s, 6 objects/s recovering
2016-12-08 15:29:51.012937 mon.0 [INF] pgmap v3312: 224 pgs: 1 active+recovery_wait+degraded, 2 active+recovery_wait+degraded+remapped, 6 active+degraded, 1 active+degraded+remapped, 67 active+remapped, 35 active, 1 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 8034 MB used, 136 GB / 143 GB avail; 2215/6519 objects degraded (33.978%); 1119/6519 objects misplaced (17.165%); 33321 kB/s, 45 objects/s recovering
2016-12-08 15:29:54.004620 mon.0 [INF] pgmap v3313: 224 pgs: 6 active+degraded, 71 active+remapped, 36 active, 111 active+undersized+degraded; 3148 MB data, 8036 MB used, 136 GB / 143 GB avail; 2186/6519 objects degraded (33.533%); 1119/6519 objects misplaced (17.165%); 57917 kB/s, 1 keys/s, 50 objects/s recovering
2016-12-08 15:29:56.012196 mon.0 [INF] pgmap v3314: 224 pgs: 71 active+remapped, 42 active, 111 active+undersized+degraded; 3148 MB data, 8055 MB used, 136 GB / 143 GB avail; 2001/6519 objects degraded (30.695%); 1119/6519 objects misplaced (17.165%); 51136 kB/s, 1 keys/s, 34 objects/s recovering
2016-12-08 15:29:59.005266 mon.0 [INF] pgmap v3315: 224 pgs: 71 active+remapped, 42 active, 111 active+undersized+degraded; 3148 MB data, 8057 MB used, 136 GB / 143 GB avail; 2001/6519 objects degraded (30.695%); 1119/6519 objects misplaced (17.165%); 338 kB/s, 22 objects/s recovering
2016-12-08 15:30:01.012263 mon.0 [INF] pgmap v3316: 224 pgs: 71 active+remapped, 42 active, 111 active+undersized+degraded; 3148 MB data, 8192 MB used, 135 GB / 143 GB avail; 2001/6519 objects degraded (30.695%); 1119/6519 objects misplaced (17.165%)
2016-12-08 15:30:04.005813 mon.0 [INF] pgmap v3317: 224 pgs: 71 active+remapped, 42 active, 111 active+undersized+degraded; 3148 MB data, 8191 MB used, 135 GB / 143 GB avail; 2001/6519 objects degraded (30.695%); 1119/6519 objects misplaced (17.165%)
2016-12-08 15:30:06.011658 mon.0 [INF] pgmap v3318: 224 pgs: 71 active+remapped, 42 active, 111 active+undersized+degraded; 3148 MB data, 8191 MB used, 135 GB / 143 GB avail; 2001/6519 objects degraded (30.695%); 1119/6519 objects misplaced (17.165%)
2 active+recovery_wait+degraded, 3 active+recovery_wait+degraded+remapped, 9 active+degraded, 10 active+degraded+remapped, 57 active+remapped, 31 active, 1 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 7732 MB used, 136 GB / 143 GB avail; 2470/6519 objects degraded (37.889%); 1119/6519 objects misplaced (17.165%); 3276 kB/s, 0 objects/s recovering
2016-12-08 15:29:36.006236 mon.0 [INF] pgmap v3306: 224 pgs: 2 active+recovery_wait+degraded, 3 active+recovery_wait+degraded+remapped, 9 active+degraded, 10 active+degraded+remapped, 57 active+remapped, 31 active, 1 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 7731 MB used, 136 GB / 143 GB avail; 2467/6519 objects degraded (37.843%); 1119/6519 objects misplaced (17.165%); 4913 kB/s, 1 objects/s recovering
2016-12-08 15:29:39.001801 mon.0 [INF] pgmap v3307: 224 pgs: 2 active+recovery_wait+degraded, 2 active+recovery_wait+degraded+remapped, 8 active+degraded, 8 active+degraded+remapped, 60 active+remapped, 32 active, 1 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 7744 MB used, 136 GB / 143 GB avail; 2406/6519 objects degraded (36.908%); 1119/6519 objects misplaced (17.165%); 29707 kB/s, 21 objects/s recovering
2016-12-08 15:29:41.012650 mon.0 [INF] pgmap v3308: 224 pgs: 1 active+recovery_wait+degraded, 2 active+recovery_wait+degraded+remapped, 7 active+degraded, 6 active+degraded+remapped, 63 active+remapped, 34 active, 111 active+undersized+degraded; 3148 MB data, 7842 MB used, 136 GB / 143 GB avail; 2334/6519 objects degraded (35.803%); 1119/6519 objects misplaced (17.165%); 57776 kB/s, 44 objects/s recovering
2016-12-08 15:29:44.001454 mon.0 [INF] pgmap v3309: 224 pgs: 1 active+recovery_wait+degraded, 2 active+recovery_wait+degraded+remapped, 7 active+degraded, 5 active+degraded+remapped, 63 active+remapped, 34 active, 1 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 7874 MB used, 136 GB / 143 GB avail; 2332/6519 objects degraded (35.772%); 1119/6519 objects misplaced (17.165%); 36261 kB/s, 24 objects/s recovering
2016-12-08 15:29:46.007635 mon.0 [INF] pgmap v3310: 224 pgs: 1 active+recovery_wait+degraded, 2 active+recovery_wait+degraded+remapped, 7 active+degraded, 4 active+degraded+remapped, 63 active+remapped, 34 active, 2 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 7893 MB used, 136 GB / 143 GB avail; 2331/6519 objects degraded (35.757%); 1119/6519 objects misplaced (17.165%); 6554 kB/s, 1 objects/s recovering
2016-12-08 15:29:49.005404 mon.0 [INF] pgmap v3311: 224 pgs: 1 active+recovery_wait+degraded, 2 active+recovery_wait+degraded+remapped, 7 active+degraded, 3 active+degraded+remapped, 64 active+remapped, 34 active, 2 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 7909 MB used, 136 GB / 143 GB avail; 2315/6519 objects degraded (35.512%); 1119/6519 objects misplaced (17.165%); 29484 kB/s, 6 objects/s recovering
2016-12-08 15:29:51.012937 mon.0 [INF] pgmap v3312: 224 pgs: 1 active+recovery_wait+degraded, 2 active+recovery_wait+degraded+remapped, 6 active+degraded, 1 active+degraded+remapped, 67 active+remapped, 35 active, 1 active+recovering+degraded+remapped, 111 active+undersized+degraded; 3148 MB data, 8034 MB used, 136 GB / 143 GB avail; 2215/6519 objects degraded (33.978%); 1119/6519 objects misplaced (17.165%); 33321 kB/s, 45 objects/s recovering
2016-12-08 15:29:54.004620 mon.0 [INF] pgmap v3313: 224 pgs: 6 active+degraded, 71 active+remapped, 36 active, 111 active+undersized+degraded; 3148 MB data, 8036 MB used, 136 GB / 143 GB avail; 2186/6519 objects degraded (33.533%); 1119/6519 objects misplaced (17.165%); 57917 kB/s, 1 keys/s, 50 objects/s recovering
....
### After some time booted the node back up
[root@overcloud-controller-0 ~]# ceph -w
cluster ce7cc1f4-bccc-11e6-a544-525400ac16ed
health HEALTH_WARN
111 pgs degraded
224 pgs stuck unclean
111 pgs undersized
recovery 2001/6519 objects degraded (30.695%)
recovery 1119/6519 objects misplaced (17.165%)
monmap e1: 1 mons at {overcloud-controller-0=172.16.1.8:6789/0}
election epoch 4, quorum 0 overcloud-controller-0
osdmap e181: 6 osds: 4 up, 4 in; 224 remapped pgs
flags sortbitwise
pgmap v3323: 224 pgs, 6 pools, 3148 MB data, 2173 objects
8191 MB used, 135 GB / 143 GB avail
2001/6519 objects degraded (30.695%)
1119/6519 objects misplaced (17.165%)
111 active+undersized+degraded
71 active+remapped
42 active
client io 0 B/s rd, 131 op/s rd, 0 op/s wr
2016-12-08 15:32:29.037917 mon.0 [INF] pgmap v3323: 224 pgs: 71 active+remapped, 42 active, 111 active+undersized+degraded; 3148 MB data, 8191 MB used, 135 GB / 143 GB avail; 0 B/s rd, 131 op/s; 2001/6519 objects degraded (30.695%); 1119/6519 objects misplaced (17.165%)
2016-12-08 15:32:34.040095 mon.0 [INF] pgmap v3324: 224 pgs: 71 active+remapped, 42 active, 111 active+undersized+degraded; 3148 MB data, 8191 MB used, 135 GB / 143 GB avail; 2001/6519 objects degraded (30.695%); 1119/6519 objects misplaced (17.165%)
2016-12-08 15:32:39.040081 mon.0 [INF] pgmap v3325: 224 pgs: 71 active+remapped, 42 active, 111 active+undersized+degraded; 3148 MB data, 8191 MB used, 135 GB / 143 GB avail; 2001/6519 objects degraded (30.695%); 1119/6519 objects misplaced (17.165%)
2016-12-08 15:32:44.042231 mon.0 [INF] pgmap v3326: 224 pgs: 71 active+remapped, 42 active, 111 active+undersized+degraded; 3148 MB data, 8191 MB used, 135 GB / 143 GB avail; 2001/6519 objects degraded (30.695%); 1119/6519 objects misplaced (17.165%)
2016-12-08 15:32:45.017049 mon.0 [INF] from='client.? 172.16.1.15:0/3661551027' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "args": ["host=overcloud-novacompute-0", "root=default"], "id": 3, "weight": 0.0351}]: dispatch
2016-12-08 15:32:47.871210 mon.0 [INF] osd.3 172.16.1.15:6800/3008 boot
2016-12-08 15:32:47.872535 mon.0 [INF] osdmap e182: 6 osds: 5 up, 5 in
2016-12-08 15:32:47.873510 mon.0 [INF] pgmap v3327: 224 pgs: 71 active+remapped, 42 active, 111 active+undersized+degraded; 3148 MB data, 8191 MB used, 135 GB / 143 GB avail; 2001/6519 objects degraded (30.695%); 1119/6519 objects misplaced (17.165%)
2016-12-08 15:32:48.880204 mon.0 [INF] osdmap e183: 6 osds: 5 up, 5 in
2016-12-08 15:32:48.880825 mon.0 [INF] pgmap v3328: 224 pgs: 71 active+remapped, 42 active, 111 active+undersized+degraded; 3148 MB data, 8191 MB used, 135 GB / 143 GB avail; 2001/6519 objects degraded (30.695%); 1119/6519 objects misplaced (17.165%)
2016-12-08 15:32:49.750693 mon.0 [INF] from='client.? 172.16.1.15:0/750932074' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "args": ["host=overcloud-novacompute-0", "root=default"], "id": 0, "weight": 0.0351}]: dispatch
2016-12-08 15:32:49.888141 mon.0 [INF] pgmap v3329: 224 pgs: 83 peering, 44 active+remapped, 34 active, 63 active+undersized+degraded; 3148 MB data, 8191 MB used, 135 GB / 143 GB avail; 1335/6519 objects degraded (20.479%); 774/6519 objects misplaced (11.873%)
2016-12-08 15:32:49.893123 mon.0 [INF] osdmap e184: 6 osds: 5 up, 5 in
2016-12-08 15:32:49.894477 mon.0 [INF] pgmap v3330: 224 pgs: 83 peering, 44 active+remapped, 34 active, 63 active+undersized+degraded; 3148 MB data, 8191 MB used, 135 GB / 143 GB avail; 1335/6519 objects degraded (20.479%); 774/6519 objects misplaced (11.873%)
2016-12-08 15:32:50.901809 mon.0 [INF] pgmap v3331: 224 pgs: 34 activating+degraded, 83 peering, 59 activating, 13 active+remapped, 22 active, 13 active+undersized+degraded; 3148 MB data, 8191 MB used, 135 GB / 143 GB avail; 473/6519 objects degraded (7.256%); 349/6519 objects misplaced (5.354%)
2016-12-08 15:32:51.905066 mon.0 [INF] osd.0 172.16.1.15:6802/3276 boot
2016-12-08 15:32:51.906730 mon.0 [INF] osdmap e185: 6 osds: 6 up, 6 in
2016-12-08 15:32:51.908654 mon.0 [INF] pgmap v3332: 224 pgs: 34 activating+degraded, 83 peering, 59 activating, 13 active+remapped, 22 active, 13 active+undersized+degraded; 3148 MB data, 8191 MB used, 135 GB / 143 GB avail; 473/6519 objects degraded (7.256%); 349/6519 objects misplaced (5.354%)
2016-12-08 15:32:52.915873 mon.0 [INF] osdmap e186: 6 osds: 6 up, 6 in
2016-12-08 15:32:52.917715 mon.0 [INF] pgmap v3333: 224 pgs: 34 activating+degraded, 83 peering, 59 activating, 13 active+remapped, 22 active, 13 active+undersized+degraded; 3148 MB data, 8191 MB used, 135 GB / 143 GB avail; 473/6519 objects degraded (7.256%); 349/6519 objects misplaced (5.354%)
2016-12-08 15:32:53.922346 mon.0 [INF] osdmap e187: 6 osds: 6 up, 6 in
2016-12-08 15:32:53.930632 mon.0 [INF] pgmap v3335: 224 pgs: 49 activating+degraded, 83 peering, 92 activating; 3148 MB data, 9829 MB used, 170 GB / 179 GB avail
2016-12-08 15:32:54.935250 mon.0 [INF] pgmap v3336: 224 pgs: 1 active+recovery_wait+degraded, 29 active+clean, 49 activating+degraded, 53 peering, 92 activating; 3148 MB data, 9654 MB used, 170 GB / 179 GB avail; 20/6519 objects degraded (0.307%)
2016-12-08 15:32:54.940292 mon.0 [INF] osdmap e188: 6 osds: 6 up, 6 in
2016-12-08 15:32:54.941521 mon.0 [INF] pgmap v3337: 224 pgs: 1 active+recovery_wait+degraded, 29 active+clean, 49 activating+degraded, 53 peering, 92 activating; 3148 MB data, 9654 MB used, 170 GB / 179 GB avail; 20/6519 objects degraded (0.307%)
2016-12-08 15:32:55.946123 mon.0 [INF] pgmap v3338: 224 pgs: 1 active+recovering+degraded, 5 active+degraded, 5 active+recovery_wait+degraded, 94 active+clean, 25 activating+degraded, 53 peering, 41 activating; 3148 MB data, 9483 MB used, 170 GB / 179 GB avail; 321/6519 objects degraded (4.924%); 1919 kB/s, 128 objects/s recovering
2016-12-08 15:32:57.953542 mon.0 [INF] pgmap v3339: 224 pgs: 2 active+recovering+degraded, 10 active+degraded, 11 active+recovery_wait+degraded, 162 active+clean, 39 peering; 3148 MB data, 11070 MB used, 205 GB / 215 GB avail; 828/6519 objects degraded (12.701%); 22987 kB/s, 0 keys/s, 83 objects/s recovering
2016-12-08 15:32:59.960862 mon.0 [INF] pgmap v3340: 224 pgs: 2 active+recovering+degraded, 10 active+degraded, 10 active+recovery_wait+degraded, 202 active+clean; 3148 MB data, 10593 MB used, 205 GB / 215 GB avail; 808/6519 objects degraded (12.395%); 18021 kB/s, 0 keys/s, 112 objects/s recovering
2016-12-08 15:33:00.964346 mon.0 [INF] pgmap v3341: 224 pgs: 1 active+recovering+degraded, 5 active+degraded, 6 active+recovery_wait+degraded, 212 active+clean; 3148 MB data, 9776 MB used, 206 GB / 215 GB avail; 507/6519 objects degraded (7.777%); 4681 kB/s, 308 objects/s recovering
2016-12-08 15:33:01.976172 mon.0 [INF] pgmap v3342: 224 pgs: 224 active+clean; 3148 MB data, 9782 MB used, 206 GB / 215 GB avail; 128 MB/s, 422 objects/s recovering
2016-12-08 15:33:04.047616 mon.0 [INF] pgmap v3343: 224 pgs: 224 active+clean; 3148 MB data, 9720 MB used, 206 GB / 215 GB avail; 83272 kB/s, 81 objects/s recovering
2016-12-08 15:33:06.054830 mon.0 [INF] pgmap v3344: 224 pgs: 224 active+clean; 3148 MB data, 9685 MB used, 206 GB / 215 GB avail
2016-12-08 15:33:07.061120 mon.0 [INF] pgmap v3345: 224 pgs: 224 active+clean; 3148 MB data, 9689 MB used, 206 GB / 215 GB avail
2016-12-08 15:33:09.068450 mon.0 [INF] pgmap v3346: 224 pgs: 224 active+clean; 3148 MB data, 9689 MB used, 206 GB / 215 GB avail
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment