vitastor #1 OSDs

bmhv3 osd4

[OSD 4] avg latency for subop 15 (ping): 256 us
[OSD 4] avg latency for subop 15 (ping): 254 us
[OSD 4] avg latency for subop 15 (ping): 212 us
[OSD 4] avg latency for subop 15 (ping): 240 us
[OSD 4] avg latency for subop 15 (ping): 213 us
[OSD 4] avg latency for subop 15 (ping): 272 us
[OSD 4] avg latency for subop 15 (ping): 275 us
[OSD 4] avg latency for subop 15 (ping): 241 us
[OSD 4] avg latency for subop 15 (ping): 249 us
[OSD 4] avg latency for subop 15 (ping): 230 us
[OSD 4] avg latency for subop 15 (ping): 223 us
[OSD 4] avg latency for subop 15 (ping): 251 us
[OSD 4] avg latency for subop 15 (ping): 228 us
[OSD 4] avg latency for subop 15 (ping): 236 us
[OSD 4] avg latency for subop 15 (ping): 207 us
[OSD 4] avg latency for subop 15 (ping): 216 us
[OSD 4] avg latency for subop 15 (ping): 217 us
[OSD 4] avg latency for subop 15 (ping): 233 us
[OSD 4] avg latency for subop 15 (ping): 223 us
[OSD 4] avg latency for subop 15 (ping): 219 us
[OSD 4] avg latency for subop 15 (ping): 222 us
[OSD 4] avg latency for subop 15 (ping): 264 us
[OSD 4] new client 16: connection from 10.10.0.171:52594
[OSD 4] avg latency for op 1 (read): 10 us, B/W: 134.67 KB/s
[OSD 4] avg latency for op 3 (write_stable): 4181 us, B/W: 277.33 KB/s
[OSD 4] avg latency for op 4 (sync): 8949 us
[OSD 4] avg latency for op 10 (show_config): 21 us
[OSD 4] avg latency for op 11 (primary_read): 12 us, B/W: 134.67 KB/s
[OSD 4] avg latency for op 12 (primary_write): 5198 us, B/W: 137.33 KB/s
[OSD 4] avg latency for subop 3 (write_stable): 5186 us
[OSD 4] avg latency for op 1 (read): 6367 us, B/W: 1.33 KB/s
[OSD 4] avg latency for op 4 (sync): 12 us
[OSD 4] avg latency for op 11 (primary_read): 6375 us, B/W: 1.33 KB/s
[OSD 4] avg latency for subop 4 (sync): 183 us
[OSD 4] avg latency for subop 15 (ping): 233 us
[OSD 4] avg latency for op 1 (read): 2 us, B/W: 0.00 KB/s
[OSD 4] avg latency for op 3 (write_stable): 3297 us, B/W: 2.67 MB/s
[OSD 4] avg latency for op 4 (sync): 613 us
[OSD 4] avg latency for op 12 (primary_write): 4356 us, B/W: 2.25 MB/s
[OSD 4] avg latency for subop 3 (write_stable): 4315 us
[OSD 4] avg latency for op 1 (read): 3 us, B/W: 0.00 KB/s
[OSD 4] avg latency for op 3 (write_stable): 1235 us, B/W: 3.60 MB/s
[OSD 4] avg latency for op 4 (sync): 1180 us
[OSD 4] avg latency for op 12 (primary_write): 1702 us, B/W: 2.83 MB/s
[OSD 4] avg latency for subop 3 (write_stable): 1678 us
[OSD 4] avg latency for subop 4 (sync): 1845 us
[OSD 4] avg latency for subop 15 (ping): 218 us
[OSD 4] avg latency for op 1 (read): 4 us, B/W: 0.00 KB/s
[OSD 4] avg latency for op 3 (write_stable): 392 us, B/W: 6.17 MB/s
[OSD 4] avg latency for op 4 (sync): 869 us
[OSD 4] avg latency for op 12 (primary_write): 1799 us, B/W: 5.28 MB/s
[OSD 4] avg latency for subop 3 (write_stable): 760 us
[OSD 4] avg latency for subop 4 (sync): 1207 us
[OSD 4] avg latency for subop 15 (ping): 232 us
[OSD 4] avg latency for op 1 (read): 5 us, B/W: 0.00 KB/s
[OSD 4] avg latency for op 3 (write_stable): 81 us, B/W: 512.00 KB/s
[OSD 4] avg latency for op 4 (sync): 8943 us
[OSD 4] avg latency for op 12 (primary_write): 532 us, B/W: 213.33 KB/s
[OSD 4] avg latency for subop 3 (write_stable): 522 us
[OSD 4] avg latency for op 1 (read): 4 us, B/W: 0.00 KB/s
[OSD 4] avg latency for op 3 (write_stable): 330 us, B/W: 1.23 MB/s
[OSD 4] avg latency for op 4 (sync): 742 us
[OSD 4] avg latency for op 12 (primary_write): 1900 us, B/W: 1.01 MB/s
[OSD 4] avg latency for subop 3 (write_stable): 584 us
[OSD 4] avg latency for subop 4 (sync): 765 us
[OSD 4] avg latency for subop 15 (ping): 245 us
[OSD 4] avg latency for op 1 (read): 4 us, B/W: 0.00 KB/s
[OSD 4] avg latency for op 3 (write_stable): 316 us, B/W: 4.45 MB/s
[OSD 4] avg latency for op 4 (sync): 897 us
[OSD 4] avg latency for op 12 (primary_write): 2018 us, B/W: 3.62 MB/s
[OSD 4] avg latency for subop 3 (write_stable): 587 us
[OSD 4] avg latency for subop 4 (sync): 933 us
[OSD 4] avg latency for op 1 (read): 3 us, B/W: 0.00 KB/s
[OSD 4] avg latency for op 3 (write_stable): 284 us, B/W: 5.01 MB/s
[OSD 4] avg latency for op 4 (sync): 641 us
[OSD 4] avg latency for op 12 (primary_write): 2169 us, B/W: 4.04 MB/s
[OSD 4] avg latency for subop 3 (write_stable): 580 us
[OSD 4] avg latency for subop 4 (sync): 973 us
[OSD 4] avg latency for subop 15 (ping): 170 us
[OSD 4] avg latency for op 1 (read): 4 us, B/W: 0.00 KB/s
[OSD 4] avg latency for op 3 (write_stable): 270 us, B/W: 5.43 MB/s
[OSD 4] avg latency for op 4 (sync): 655 us
[OSD 4] avg latency for op 12 (primary_write): 2126 us, B/W: 4.37 MB/s
[OSD 4] avg latency for subop 3 (write_stable): 537 us
[OSD 4] avg latency for subop 4 (sync): 1037 us
[OSD 4] avg latency for subop 15 (ping): 272 us
[OSD 4] avg latency for op 1 (read): 4 us, B/W: 0.00 KB/s
[OSD 4] avg latency for op 3 (write_stable): 282 us, B/W: 5.46 MB/s
[OSD 4] avg latency for op 4 (sync): 704 us
[OSD 4] avg latency for op 12 (primary_write): 2273 us, B/W: 4.54 MB/s
[OSD 4] avg latency for subop 3 (write_stable): 535 us
[OSD 4] avg latency for subop 4 (sync): 872 us
[OSD 4] avg latency for op 1 (read): 4 us, B/W: 0.00 KB/s
[OSD 4] avg latency for op 3 (write_stable): 279 us, B/W: 5.97 MB/s
[OSD 4] avg latency for op 4 (sync): 675 us
[OSD 4] avg latency for op 12 (primary_write): 2274 us, B/W: 4.76 MB/s
[OSD 4] avg latency for subop 3 (write_stable): 526 us
[OSD 4] avg latency for subop 4 (sync): 870 us
[OSD 4] avg latency for subop 15 (ping): 347 us
[OSD 4] avg latency for op 1 (read): 4 us, B/W: 0.00 KB/s
[OSD 4] avg latency for op 3 (write_stable): 269 us, B/W: 5.93 MB/s
[OSD 4] avg latency for op 4 (sync): 622 us
[OSD 4] avg latency for op 12 (primary_write): 2282 us, B/W: 4.71 MB/s
[OSD 4] avg latency for subop 3 (write_stable): 525 us
[OSD 4] avg latency for subop 4 (sync): 806 us
[OSD 4] avg latency for op 1 (read): 3 us, B/W: 0.00 KB/s
[OSD 4] avg latency for op 3 (write_stable): 272 us, B/W: 9.03 MB/s
[OSD 4] avg latency for op 4 (sync): 586 us
[OSD 4] avg latency for op 12 (primary_write): 2417 us, B/W: 7.34 MB/s
[OSD 4] avg latency for subop 3 (write_stable): 537 us
[OSD 4] avg latency for subop 4 (sync): 842 us
[OSD 4] avg latency for subop 15 (ping): 240 us
[OSD 4] avg latency for op 1 (read): 4 us, B/W: 0.00 KB/s
[OSD 4] avg latency for op 3 (write_stable): 275 us, B/W: 10.73 MB/s
[OSD 4] avg latency for op 4 (sync): 774 us
[OSD 4] avg latency for op 12 (primary_write): 2614 us, B/W: 8.66 MB/s
[OSD 4] avg latency for subop 3 (write_stable): 577 us
[OSD 4] avg latency for subop 4 (sync): 920 us
[OSD 4] avg latency for subop 15 (ping): 243 us
[OSD 4] avg latency for op 4 (sync): 4 us
Ping timed out for OSD 2 (client 12), disconnecting peer
[OSD 4] Stopping client 12 (OSD peer 2)
[PG 2/4] Repeer because of OSD 2
[PG 2/4] is peering (0 objects)
[PG 2/6] Repeer because of OSD 2
[PG 2/6] is peering (0 objects)
[PG 2/8] Repeer because of OSD 2
[PG 2/8] is peering (0 objects)
[PG 2/9] Repeer because of OSD 2
[PG 2/9] is peering (0 objects)
[PG 2/21] Repeer because of OSD 2
[PG 2/21] is peering (0 objects)
[PG 2/22] Repeer because of OSD 2
[PG 2/22] is peering (0 objects)
[PG 2/34] Repeer because of OSD 2
[PG 2/34] is peering (0 objects)
[PG 2/39] Repeer because of OSD 2
[PG 2/39] is peering (0 objects)
[PG 2/43] Repeer because of OSD 2
[PG 2/43] is peering (0 objects)
[PG 2/50] Repeer because of OSD 2
[PG 2/50] is peering (0 objects)
[PG 2/58] Repeer because of OSD 2
[PG 2/58] is peering (0 objects)
[PG 2/65] Repeer because of OSD 2
[PG 2/65] is peering (0 objects)
[PG 2/72] Repeer because of OSD 2
[PG 2/72] is peering (0 objects)
[PG 2/81] Repeer because of OSD 2
[PG 2/81] is peering (0 objects)
[PG 2/84] Repeer because of OSD 2
[PG 2/84] is peering (0 objects)
[PG 2/92] Repeer because of OSD 2
[PG 2/92] is peering (0 objects)
[PG 2/93] Repeer because of OSD 2
[PG 2/93] is peering (0 objects)
[PG 2/97] Repeer because of OSD 2
[PG 2/97] is peering (0 objects)
[PG 2/101] Repeer because of OSD 2
[PG 2/101] is peering (0 objects)
[PG 2/102] Repeer because of OSD 2
[PG 2/102] is peering (0 objects)
[PG 2/107] Repeer because of OSD 2
[PG 2/107] is peering (0 objects)
[PG 2/111] Repeer because of OSD 2
[PG 2/111] is peering (0 objects)
[PG 2/114] Repeer because of OSD 2
[PG 2/114] is peering (0 objects)
[PG 2/117] Repeer because of OSD 2
[PG 2/117] is peering (0 objects)
[PG 2/118] Repeer because of OSD 2
[PG 2/118] is peering (0 objects)
[PG 2/119] Repeer because of OSD 2
[PG 2/119] is peering (0 objects)
[PG 2/126] Repeer because of OSD 2
[PG 2/126] is peering (0 objects)
[PG 2/128] Repeer because of OSD 2
[PG 2/128] is peering (0 objects)
[PG 2/129] Repeer because of OSD 2
[PG 2/129] is peering (0 objects)
[PG 2/131] Repeer because of OSD 2
[PG 2/131] is peering (0 objects)
[PG 2/132] Repeer because of OSD 2
[PG 2/132] is peering (0 objects)
[PG 2/134] Repeer because of OSD 2
[PG 2/134] is peering (0 objects)
[PG 2/138] Repeer because of OSD 2

bmhv4 osd3

[OSD 3] avg latency for subop 15 (ping): 296 us
[OSD 3] avg latency for subop 15 (ping): 312 us
[OSD 3] avg latency for subop 15 (ping): 288 us
[OSD 3] avg latency for subop 15 (ping): 285 us
[OSD 3] avg latency for subop 15 (ping): 284 us
[OSD 3] avg latency for subop 15 (ping): 276 us
[OSD 3] avg latency for subop 15 (ping): 276 us
[OSD 3] avg latency for subop 15 (ping): 275 us
[OSD 3] avg latency for subop 15 (ping): 290 us
[OSD 3] avg latency for subop 15 (ping): 272 us
[OSD 3] avg latency for subop 15 (ping): 282 us
[OSD 3] avg latency for subop 15 (ping): 276 us
[OSD 3] avg latency for subop 15 (ping): 267 us
[OSD 3] avg latency for subop 15 (ping): 281 us
[OSD 3] avg latency for subop 15 (ping): 288 us
[OSD 3] avg latency for subop 15 (ping): 277 us
[OSD 3] avg latency for subop 15 (ping): 283 us
p[OSD 3] avg latency for subop 15 (ping): 268 us
[OSD 3] avg latency for subop 15 (ping): 274 us
[OSD 3] avg latency for subop 15 (ping): 288 us
[OSD 3] avg latency for subop 15 (ping): 288 us
[OSD 3] avg latency for subop 15 (ping): 287 us
[OSD 3] avg latency for subop 15 (ping): 281 us
[OSD 3] avg latency for subop 15 (ping): 287 us
[OSD 3] avg latency for subop 15 (ping): 283 us
[OSD 3] avg latency for subop 15 (ping): 281 us
[OSD 3] avg latency for subop 15 (ping): 292 us
[OSD 3] new client 16: connection from 10.10.0.171:54290
[OSD 3] avg latency for op 1 (read): 28 us, B/W: 384.00 KB/s
[OSD 3] avg latency for op 3 (write_stable): 4013 us, B/W: 137.33 KB/s
[OSD 3] avg latency for op 4 (sync): 8898 us
[OSD 3] avg latency for op 10 (show_config): 21 us
[OSD 3] avg latency for op 11 (primary_read): 35 us, B/W: 384.00 KB/s
[OSD 3] avg latency for op 12 (primary_write): 4119 us, B/W: 277.33 KB/s
[OSD 3] avg latency for subop 3 (write_stable): 4111 us
[OSD 3] avg latency for subop 4 (sync): 9221 us
[OSD 3] avg latency for op 4 (sync): 4 us
[OSD 3] avg latency for subop 15 (ping): 286 us
[OSD 3] avg latency for op 1 (read): 4 us, B/W: 0.00 KB/s
[OSD 3] avg latency for op 3 (write_stable): 3205 us, B/W: 2.67 MB/s
[OSD 3] avg latency for op 4 (sync): 616 us
[OSD 3] avg latency for op 12 (primary_write): 3303 us, B/W: 3.14 MB/s
[OSD 3] avg latency for subop 3 (write_stable): 3094 us
[OSD 3] avg latency for subop 4 (sync): 855 us
[OSD 3] avg latency for op 1 (read): 3 us, B/W: 0.00 KB/s
[OSD 3] avg latency for op 3 (write_stable): 1492 us, B/W: 2.78 MB/s
[OSD 3] avg latency for op 4 (sync): 1708 us
[OSD 3] avg latency for op 12 (primary_write): 2207 us, B/W: 3.49 MB/s
[OSD 3] avg latency for subop 3 (write_stable): 1756 us
[OSD 3] avg latency for subop 4 (sync): 927 us
[OSD 3] avg latency for subop 15 (ping): 235 us
[OSD 3] avg latency for op 1 (read): 5 us, B/W: 0.00 KB/s
[OSD 3] avg latency for op 3 (write_stable): 456 us, B/W: 5.13 MB/s
[OSD 3] avg latency for op 4 (sync): 865 us
[OSD 3] avg latency for op 12 (primary_write): 1649 us, B/W: 6.30 MB/s
[OSD 3] avg latency for subop 3 (write_stable): 665 us
[OSD 3] avg latency for subop 4 (sync): 876 us
[OSD 3] avg latency for subop 15 (ping): 222 us
[OSD 3] avg latency for op 4 (sync): 9192 us
[OSD 3] avg latency for subop 4 (sync): 9216 us
[OSD 3] avg latency for op 1 (read): 4 us, B/W: 0.00 KB/s
[OSD 3] avg latency for op 3 (write_stable): 337 us, B/W: 1.40 MB/s
[OSD 3] avg latency for op 4 (sync): 664 us
[OSD 3] avg latency for op 12 (primary_write): 1915 us, B/W: 1.65 MB/s
[OSD 3] avg latency for subop 3 (write_stable): 578 us
[OSD 3] avg latency for subop 4 (sync): 1177 us
[OSD 3] avg latency for subop 15 (ping): 260 us
[OSD 3] avg latency for op 1 (read): 4 us, B/W: 0.00 KB/s
[OSD 3] avg latency for op 3 (write_stable): 312 us, B/W: 3.60 MB/s
[OSD 3] avg latency for op 4 (sync): 789 us
[OSD 3] avg latency for op 12 (primary_write): 2034 us, B/W: 4.50 MB/s
[OSD 3] avg latency for subop 3 (write_stable): 591 us
[OSD 3] avg latency for subop 4 (sync): 1187 us
[OSD 3] avg latency for op 1 (read): 5 us, B/W: 0.00 KB/s
[OSD 3] avg latency for op 3 (write_stable): 310 us, B/W: 4.11 MB/s
[OSD 3] avg latency for op 4 (sync): 718 us
[OSD 3] avg latency for op 12 (primary_write): 2083 us, B/W: 5.19 MB/s
[OSD 3] avg latency for subop 3 (write_stable): 560 us
[OSD 3] avg latency for subop 4 (sync): 749 us
[OSD 3] avg latency for subop 15 (ping): 262 us
[OSD 3] avg latency for op 1 (read): 4 us, B/W: 0.00 KB/s
[OSD 3] avg latency for op 3 (write_stable): 278 us, B/W: 4.45 MB/s
[OSD 3] avg latency for op 4 (sync): 703 us
[OSD 3] avg latency for op 12 (primary_write): 2094 us, B/W: 5.35 MB/s
[OSD 3] avg latency for subop 3 (write_stable): 528 us
[OSD 3] avg latency for subop 4 (sync): 998 us
[OSD 3] avg latency for subop 15 (ping): 287 us
[OSD 3] avg latency for op 1 (read): 4 us, B/W: 0.00 KB/s
[OSD 3] avg latency for op 3 (write_stable): 265 us, B/W: 4.61 MB/s
[OSD 3] avg latency for op 4 (sync): 630 us
[OSD 3] avg latency for op 12 (primary_write): 2257 us, B/W: 5.64 MB/s
[OSD 3] avg latency for subop 3 (write_stable): 551 us
[OSD 3] avg latency for subop 4 (sync): 920 us
[OSD 3] avg latency for op 1 (read): 4 us, B/W: 0.00 KB/s
[OSD 3] avg latency for op 3 (write_stable): 248 us, B/W: 4.78 MB/s
[OSD 3] avg latency for op 4 (sync): 719 us
[OSD 3] avg latency for op 12 (primary_write): 2347 us, B/W: 5.90 MB/s
[OSD 3] avg latency for subop 3 (write_stable): 559 us
[OSD 3] avg latency for subop 4 (sync): 913 us
[OSD 3] avg latency for subop 15 (ping): 239 us
[OSD 3] avg latency for op 1 (read): 4 us, B/W: 0.00 KB/s
[OSD 3] avg latency for op 3 (write_stable): 268 us, B/W: 4.61 MB/s
[OSD 3] avg latency for op 4 (sync): 605 us
[OSD 3] avg latency for op 12 (primary_write): 2195 us, B/W: 5.74 MB/s
[OSD 3] avg latency for subop 3 (write_stable): 511 us
[OSD 3] avg latency for subop 4 (sync): 761 us
[OSD 3] avg latency for op 1 (read): 4 us, B/W: 0.00 KB/s
[OSD 3] avg latency for op 3 (write_stable): 282 us, B/W: 8.55 MB/s
[OSD 3] avg latency for op 4 (sync): 731 us
[OSD 3] avg latency for op 12 (primary_write): 2408 us, B/W: 10.61 MB/s
[OSD 3] avg latency for subop 3 (write_stable): 539 us
[OSD 3] avg latency for subop 4 (sync): 904 us
[OSD 3] avg latency for subop 15 (ping): 230 us
[OSD 3] avg latency for op 1 (read): 4 us, B/W: 0.00 KB/s
[OSD 3] avg latency for op 3 (write_stable): 276 us, B/W: 6.96 MB/s
[OSD 3] avg latency for op 4 (sync): 710 us
[OSD 3] avg latency for op 12 (primary_write): 2524 us, B/W: 8.65 MB/s
[OSD 3] avg latency for subop 3 (write_stable): 545 us
[OSD 3] avg latency for subop 4 (sync): 1036 us
[OSD 3] avg latency for subop 15 (ping): 281 us
[OSD 3] avg latency for op 4 (sync): 50 us
[OSD 3] avg latency for subop 4 (sync): 301 us
Ping timed out for OSD 2 (client 13), disconnecting peer
[OSD 3] Stopping client 13 (OSD peer 2)
[PG 2/12] Repeer because of OSD 2
[PG 2/12] is peering (0 objects)
[PG 2/13] Repeer because of OSD 2
[PG 2/13] is peering (0 objects)
[PG 2/15] Repeer because of OSD 2
[PG 2/15] is peering (0 objects)
[PG 2/18] Repeer because of OSD 2
[PG 2/18] is peering (0 objects)
[PG 2/27] Repeer because of OSD 2
[PG 2/27] is peering (0 objects)
[PG 2/30] Repeer because of OSD 2
[PG 2/30] is peering (0 objects)
[PG 2/32] Repeer because of OSD 2
[PG 2/32] is peering (0 objects)
[PG 2/33] Repeer because of OSD 2
[PG 2/33] is peering (0 objects)
[PG 2/38] Repeer because of OSD 2
[PG 2/38] is peering (0 objects)
[PG 2/45] Repeer because of OSD 2
[PG 2/45] is peering (0 objects)
[PG 2/53] Repeer because of OSD 2
[PG 2/53] is peering (0 objects)
[PG 2/61] Repeer because of OSD 2
[PG 2/61] is peering (0 objects)
[PG 2/63] Repeer because of OSD 2
[PG 2/63] is peering (0 objects)
[PG 2/68] Repeer because of OSD 2
[PG 2/68] is peering (0 objects)
[PG 2/69] Repeer because of OSD 2
[PG 2/69] is peering (0 objects)
[PG 2/71] Repeer because of OSD 2
[PG 2/71] is peering (0 objects)
[PG 2/75] Repeer because of OSD 2
[PG 2/75] is peering (0 objects)
[PG 2/76] Repeer because of OSD 2
[PG 2/76] is peering (0 objects)
[PG 2/78] Repeer because of OSD 2
[PG 2/78] is peering (0 objects)
[PG 2/82] Repeer because of OSD 2
[PG 2/82] is peering (0 objects)
[PG 2/95] Repeer because of OSD 2
[PG 2/95] is peering (0 objects)
[PG 2/98] Repeer because of OSD 2
[PG 2/98] is peering (0 objects)
[PG 2/100] Repeer because of OSD 2
[PG 2/100] is peering (0 objects)
[PG 2/108] Repeer because of OSD 2
[PG 2/108] is peering (0 objects)
[PG 2/112] Repeer because of OSD 2
[PG 2/112] is peering (0 objects)
[PG 2/115] Repeer because of OSD 2
[PG 2/115] is peering (0 objects)
[PG 2/122] Repeer because of OSD 2
[PG 2/122] is peering (0 objects)
[PG 2/124] Repeer because of OSD 2
[PG 2/124] is peering (0 objects)
[PG 2/133] Repeer because of OSD 2
[PG 2/133] is peering (0 objects)
[PG 2/135] Repeer because of OSD 2
[PG 2/135] is peering (0 objects)
[PG 2/137] Repeer because of OSD 2
[PG 2/137] is peering (0 objects)
[PG 2/139] Repeer because of OSD 2
[PG 2/139] is peering (0 objects)
[PG 2/148] Repeer because of OSD 2
[PG 2/148] is peering (0 objects)
[PG 2/152] Repeer because of OSD 2
[PG 2/152] is peering (0 objects)
[PG 2/153] Repeer because of OSD 2
[PG 2/153] is peering (0 objects)
[PG 2/157] Repeer because of OSD 2
[PG 2/157] is peering (0 objects)
[PG 2/158] Repeer because of OSD 2
[PG 2/158] is peering (0 objects)
[PG 2/164] Repeer because of OSD 2
[PG 2/164] is peering (0 objects)
[PG 2/168] Repeer because of OSD 2
[PG 2/168] is peering (0 objects)
[PG 2/169] Repeer because of OSD 2
[PG 2/169] is peering (0 objects)

bmhv7 osd1

[OSD 1] avg latency for subop 15 (ping): 281 us
[OSD 1] avg latency for subop 15 (ping): 294 us
[OSD 1] avg latency for subop 15 (ping): 289 us
[OSD 1] avg latency for subop 15 (ping): 290 us
[OSD 1] avg latency for subop 15 (ping): 278 us
[OSD 1] avg latency for subop 15 (ping): 299 us
[OSD 1] avg latency for subop 15 (ping): 297 us
[OSD 1] avg latency for subop 15 (ping): 303 us
[OSD 1] avg latency for subop 15 (ping): 291 us
[OSD 1] avg latency for subop 15 (ping): 295 us
[OSD 1] avg latency for subop 15 (ping): 299 us
[OSD 1] avg latency for subop 15 (ping): 304 us
[OSD 1] avg latency for subop 15 (ping): 287 us
[OSD 1] avg latency for subop 15 (ping): 289 us
[OSD 1] avg latency for subop 15 (ping): 296 us
[OSD 1] avg latency for subop 15 (ping): 282 us
[OSD 1] avg latency for subop 15 (ping): 304 us
[OSD 1] avg latency for subop 15 (ping): 287 us
[OSD 1] avg latency for subop 15 (ping): 277 us
[OSD 1] avg latency for subop 15 (ping): 295 us
[OSD 1] new client 17: connection from 10.10.0.171:34898
[OSD 1] avg latency for op 1 (read): 28 us, B/W: 538.67 KB/s
[OSD 1] avg latency for op 3 (write_stable): 3084 us, B/W: 298.67 KB/s
[OSD 1] avg latency for op 10 (show_config): 24 us
[OSD 1] avg latency for op 11 (primary_read): 36 us, B/W: 538.67 KB/s
[OSD 1] avg latency for op 12 (primary_write): 3693 us, B/W: 736.00 KB/s
[OSD 1] avg latency for subop 3 (write_stable): 3240 us
[OSD 1] avg latency for op 1 (read): 67 us, B/W: 6.67 KB/s
[OSD 1] avg latency for op 4 (sync): 6040 us
[OSD 1] avg latency for op 11 (primary_read): 70 us, B/W: 6.67 KB/s
[OSD 1] avg latency for subop 4 (sync): 298 us
[OSD 1] avg latency for subop 15 (ping): 267 us
[OSD 1] avg latency for op 1 (read): 107 us, B/W: 46.67 KB/s
[OSD 1] avg latency for op 3 (write_stable): 5175 us, B/W: 4.29 MB/s
[OSD 1] avg latency for op 4 (sync): 4936 us
[OSD 1] avg latency for op 11 (primary_read): 3271 us, B/W: 46.67 KB/s
[OSD 1] avg latency for op 12 (primary_write): 7541 us, B/W: 5.00 MB/s
[OSD 1] avg latency for subop 3 (write_stable): 4525 us
[OSD 1] avg latency for subop 4 (sync): 864 us
[OSD 1] avg latency for op 1 (read): 6 us, B/W: 0.00 KB/s
[OSD 1] avg latency for op 3 (write_stable): 1056 us, B/W: 6.54 MB/s
[OSD 1] avg latency for op 4 (sync): 10181 us
[OSD 1] avg latency for op 12 (primary_write): 2624 us, B/W: 6.88 MB/s
[OSD 1] avg latency for subop 3 (write_stable): 2299 us
[OSD 1] avg latency for subop 4 (sync): 1011 us
[OSD 1] avg latency for subop 15 (ping): 224 us
[OSD 1] avg latency for op 1 (read): 12 us, B/W: 0.00 KB/s
[OSD 1] avg latency for op 3 (write_stable): 3536 us, B/W: 11.25 MB/s
[OSD 1] avg latency for op 4 (sync): 13617 us
[OSD 1] avg latency for op 12 (primary_write): 8282 us, B/W: 12.09 MB/s
[OSD 1] avg latency for subop 3 (write_stable): 798 us
[OSD 1] avg latency for subop 4 (sync): 902 us
[OSD 1] avg latency for subop 15 (ping): 226 us
[OSD 1] avg latency for op 1 (read): 6 us, B/W: 0.00 KB/s
[OSD 1] avg latency for op 3 (write_stable): 218 us, B/W: 597.33 KB/s
[OSD 1] avg latency for op 4 (sync): 3927 us
[OSD 1] avg latency for op 12 (primary_write): 809 us, B/W: 725.33 KB/s
[OSD 1] avg latency for subop 3 (write_stable): 794 us
[OSD 1] avg latency for subop 4 (sync): 955 us
[OSD 1] avg latency for op 1 (read): 11 us, B/W: 0.00 KB/s
[OSD 1] avg latency for op 3 (write_stable): 2284 us, B/W: 2.02 MB/s
[OSD 1] avg latency for op 4 (sync): 9631 us
[OSD 1] avg latency for op 12 (primary_write): 6920 us, B/W: 2.22 MB/s
[OSD 1] avg latency for subop 3 (write_stable): 571 us
[OSD 1] avg latency for subop 4 (sync): 1336 us
[OSD 1] avg latency for subop 15 (ping): 300 us
[OSD 1] avg latency for op 1 (read): 9 us, B/W: 0.00 KB/s
[OSD 1] avg latency for op 3 (write_stable): 2810 us, B/W: 7.57 MB/s
[OSD 1] avg latency for op 4 (sync): 12417 us
[OSD 1] avg latency for op 12 (primary_write): 9162 us, B/W: 8.20 MB/s
[OSD 1] avg latency for subop 3 (write_stable): 636 us
[OSD 1] avg latency for subop 4 (sync): 828 us
[OSD 1] avg latency for op 1 (read): 9 us, B/W: 0.00 KB/s
[OSD 1] avg latency for op 3 (write_stable): 2356 us, B/W: 8.85 MB/s
[OSD 1] avg latency for op 4 (sync): 12155 us
[OSD 1] avg latency for op 12 (primary_write): 9006 us, B/W: 9.54 MB/s
[OSD 1] avg latency for subop 3 (write_stable): 621 us
[OSD 1] avg latency for subop 4 (sync): 801 us
[OSD 1] avg latency for subop 15 (ping): 240 us
[OSD 1] avg latency for op 1 (read): 8 us, B/W: 0.00 KB/s
[OSD 1] avg latency for op 3 (write_stable): 2170 us, B/W: 9.56 MB/s
[OSD 1] avg latency for op 4 (sync): 10599 us
[OSD 1] avg latency for op 12 (primary_write): 8743 us, B/W: 10.46 MB/s
[OSD 1] avg latency for subop 3 (write_stable): 588 us
[OSD 1] avg latency for subop 4 (sync): 923 us
[OSD 1] avg latency for subop 15 (ping): 192 us
[OSD 1] avg latency for op 1 (read): 8 us, B/W: 0.00 KB/s
[OSD 1] avg latency for op 3 (write_stable): 2228 us, B/W: 9.69 MB/s
[OSD 1] avg latency for op 4 (sync): 12782 us
[OSD 1] avg latency for op 12 (primary_write): 9077 us, B/W: 10.66 MB/s
[OSD 1] avg latency for subop 3 (write_stable): 572 us
[OSD 1] avg latency for subop 4 (sync): 712 us
[OSD 1] avg latency for op 1 (read): 8 us, B/W: 0.00 KB/s
[OSD 1] avg latency for op 3 (write_stable): 2241 us, B/W: 10.48 MB/s
[OSD 1] avg latency for op 4 (sync): 10713 us
[OSD 1] avg latency for op 12 (primary_write): 9276 us, B/W: 11.32 MB/s
[OSD 1] avg latency for subop 3 (write_stable): 588 us
[OSD 1] avg latency for subop 4 (sync): 654 us
[OSD 1] avg latency for subop 15 (ping): 293 us
[OSD 1] avg latency for op 1 (read): 8 us, B/W: 0.00 KB/s
[OSD 1] avg latency for op 3 (write_stable): 2301 us, B/W: 10.09 MB/s
[OSD 1] avg latency for op 4 (sync): 10222 us
[OSD 1] avg latency for op 12 (primary_write): 10434 us, B/W: 10.82 MB/s
[OSD 1] avg latency for subop 3 (write_stable): 622 us
[OSD 1] avg latency for subop 4 (sync): 772 us
[OSD 1] avg latency for op 1 (read): 8 us, B/W: 0.00 KB/s
[OSD 1] avg latency for op 3 (write_stable): 1464 us, B/W: 15.79 MB/s
[OSD 1] avg latency for op 4 (sync): 7250 us
[OSD 1] avg latency for op 12 (primary_write): 7043 us, B/W: 17.33 MB/s
[OSD 1] avg latency for subop 3 (write_stable): 636 us
[OSD 1] avg latency for subop 4 (sync): 982 us
[OSD 1] avg latency for subop 15 (ping): 226 us
[OSD 1] avg latency for op 1 (read): 9 us, B/W: 0.00 KB/s
[OSD 1] avg latency for op 3 (write_stable): 357 us, B/W: 19.35 MB/s
[OSD 1] avg latency for op 4 (sync): 910 us
[OSD 1] avg latency for op 12 (primary_write): 3157 us, B/W: 20.61 MB/s
[OSD 1] avg latency for subop 3 (write_stable): 701 us
[OSD 1] avg latency for subop 4 (sync): 1179 us
[OSD 1] avg latency for subop 15 (ping): 289 us
[OSD 1] avg latency for op 4 (sync): 8579 us
Ping timed out for OSD 2 (client 14), disconnecting peer
[OSD 1] Stopping client 14 (OSD peer 2)
[PG 1/1] Repeer because of OSD 2
[PG 1/1] is peering (110 objects)
[OSD 1] Stopping client 17 (regular client)
[PG 1/4] Repeer because of OSD 2
[PG 1/4] is peering (98 objects)
[PG 1/7] Repeer because of OSD 2
[PG 1/7] is peering (99 objects)
[PG 1/11] Repeer because of OSD 2
[PG 1/11] is peering (99 objects)
[PG 1/12] Repeer because of OSD 2
[PG 1/12] is peering (99 objects)
[PG 1/13] Repeer because of OSD 2
[PG 1/13] is peering (99 objects)
[PG 1/16] Repeer because of OSD 2
[PG 1/16] is peering (99 objects)
[PG 1/17] Repeer because of OSD 2
[PG 1/17] is peering (98 objects)
[PG 1/18] Repeer because of OSD 2
[PG 1/18] is peering (98 objects)
[PG 1/20] Repeer because of OSD 2
[PG 1/20] is peering (99 objects)
[PG 1/23] Repeer because of OSD 2
[PG 1/23] is peering (98 objects)
[PG 1/24] Repeer because of OSD 2
[PG 1/24] is peering (98 objects)
[PG 1/25] Repeer because of OSD 2
[PG 1/25] is peering (98 objects)
[PG 1/27] Repeer because of OSD 2
[PG 1/27] is peering (97 objects)
[PG 1/28] Repeer because of OSD 2
[PG 1/28] is peering (98 objects)
[PG 1/29] Repeer because of OSD 2
[PG 1/29] is peering (97 objects)
[PG 1/33] Repeer because of OSD 2
[PG 1/33] is peering (99 objects)
[PG 1/36] Repeer because of OSD 2
[PG 1/36] is peering (97 objects)
[PG 1/50] Repeer because of OSD 2
[PG 1/50] is peering (98 objects)
[PG 1/51] Repeer because of OSD 2
[PG 1/51] is peering (98 objects)
[PG 1/52] Repeer because of OSD 2
[PG 1/52] is peering (98 objects)
[PG 1/56] Repeer because of OSD 2
[PG 1/56] is peering (99 objects)
[PG 1/60] Repeer because of OSD 2
[PG 1/60] is peering (98 objects)
[PG 1/63] Repeer because of OSD 2
[PG 1/63] is peering (99 objects)
[PG 1/67] Repeer because of OSD 2
[PG 1/67] is peering (103 objects)
[PG 1/68] Repeer because of OSD 2
[PG 1/68] is peering (104 objects)
[PG 1/69] Repeer because of OSD 2
[PG 1/69] is peering (101 objects)
[PG 1/70] Repeer because of OSD 2
[PG 1/70] is peering (104 objects)
[PG 1/71] Repeer because of OSD 2
[PG 1/71] is peering (103 objects)
[PG 1/74] Repeer because of OSD 2
[PG 1/74] is peering (104 objects)
[PG 1/77] Repeer because of OSD 2
[PG 1/77] is peering (102 objects)
[PG 1/79] Repeer because of OSD 2
[PG 1/79] is peering (104 objects)
[PG 1/82] Repeer because of OSD 2
[PG 1/82] is peering (104 objects)
[PG 1/83] Repeer because of OSD 2
[PG 1/83] is peering (104 objects)
[PG 1/86] Repeer because of OSD 2
[PG 1/86] is peering (104 objects)

bmhv9 osd2

[OSD 2] avg latency for subop 15 (ping): 228 us
[OSD 2] avg latency for subop 15 (ping): 227 us
[OSD 2] avg latency for subop 15 (ping): 226 us
[OSD 2] avg latency for subop 15 (ping): 238 us
[OSD 2] avg latency for subop 15 (ping): 224 us
[OSD 2] avg latency for subop 15 (ping): 220 us
[OSD 2] avg latency for subop 15 (ping): 250 us
[OSD 2] avg latency for subop 15 (ping): 242 us
[OSD 2] avg latency for subop 15 (ping): 223 us
[OSD 2] avg latency for subop 15 (ping): 228 us
[OSD 2] avg latency for subop 15 (ping): 240 us
[OSD 2] avg latency for subop 15 (ping): 240 us
[OSD 2] avg latency for subop 15 (ping): 236 us
[OSD 2] avg latency for subop 15 (ping): 229 us
[OSD 2] avg latency for subop 15 (ping): 254 us
[OSD 2] avg latency for subop 15 (ping): 227 us
[OSD 2] avg latency for subop 15 (ping): 259 us
[OSD 2] avg latency for subop 15 (ping): 263 us
[OSD 2] avg latency for subop 15 (ping): 251 us
[OSD 2] avg latency for subop 15 (ping): 256 us
[OSD 2] avg latency for subop 15 (ping): 241 us
[OSD 2] avg latency for subop 15 (ping): 258 us
[OSD 2] avg latency for subop 15 (ping): 253 us
[OSD 2] avg latency for subop 15 (ping): 234 us
[OSD 2] avg latency for subop 15 (ping): 235 us
[OSD 2] avg latency for subop 15 (ping): 235 us
[OSD 2] avg latency for subop 15 (ping): 238 us
[OSD 2] avg latency for subop 15 (ping): 234 us
[OSD 2] avg latency for subop 15 (ping): 231 us
[OSD 2] new client 16: connection from 10.10.0.171:48820
[OSD 2] avg latency for op 1 (read): 18 us, B/W: 1.33 KB/s
[OSD 2] avg latency for op 10 (show_config): 23 us
[OSD 2] avg latency for op 11 (primary_read): 19 us, B/W: 1.33 KB/s
[OSD 2] avg latency for subop 15 (ping): 239 us
[OSD 2] avg latency for op 1 (read): 18 us, B/W: 2.67 KB/s
[OSD 2] avg latency for op 3 (write_stable): 2994 us, B/W: 736.00 KB/s
[OSD 2] avg latency for op 4 (sync): 4109 us
[OSD 2] avg latency for op 11 (primary_read): 62 us, B/W: 2.67 KB/s
[OSD 2] avg latency for op 12 (primary_write): 4411 us, B/W: 298.67 KB/s
[OSD 2] avg latency for subop 3 (write_stable): 4333 us
[OSD 2] avg latency for subop 4 (sync): 12277 us
[OSD 2] avg latency for subop 15 (ping): 243 us
[OSD 2] avg latency for op 1 (read): 4 us, B/W: 0.00 KB/s
[OSD 2] avg latency for op 3 (write_stable): 4654 us, B/W: 2.46 MB/s
[OSD 2] avg latency for op 12 (primary_write): 7738 us, B/W: 2.71 MB/s
[OSD 2] avg latency for subop 3 (write_stable): 6550 us
[OSD 2] avg latency for op 1 (read): 9 us, B/W: 0.00 KB/s
[OSD 2] avg latency for op 3 (write_stable): 1957 us, B/W: 7.12 MB/s
[OSD 2] avg latency for op 4 (sync): 998 us
[OSD 2] avg latency for op 12 (primary_write): 3323 us, B/W: 5.75 MB/s
[OSD 2] avg latency for subop 3 (write_stable): 2273 us
[OSD 2] avg latency for subop 4 (sync): 8057 us
[OSD 2] avg latency for subop 15 (ping): 232 us
[OSD 2] avg latency for op 1 (read): 12 us, B/W: 0.00 KB/s
[OSD 2] avg latency for op 3 (write_stable): 346 us, B/W: 10.34 MB/s
[OSD 2] avg latency for op 4 (sync): 796 us
[OSD 2] avg latency for op 12 (primary_write): 10169 us, B/W: 9.66 MB/s
[OSD 2] avg latency for subop 3 (write_stable): 4463 us
[OSD 2] avg latency for subop 4 (sync): 15477 us
[OSD 2] avg latency for op 1 (read): 13 us, B/W: 0.00 KB/s
[OSD 2] avg latency for op 3 (write_stable): 285 us, B/W: 4.75 MB/s
[OSD 2] avg latency for op 4 (sync): 522 us
[OSD 2] avg latency for op 12 (primary_write): 10702 us, B/W: 4.55 MB/s
[OSD 2] avg latency for subop 3 (write_stable): 4297 us
[OSD 2] avg latency for subop 4 (sync): 13231 us
[OSD 2] avg latency for subop 15 (ping): 316 us
[OSD 2] avg latency for op 3 (write_stable): 153 us, B/W: 30.67 KB/s
[OSD 2] avg latency for op 4 (sync): 8965 us
[OSD 2] avg latency for subop 4 (sync): 12276 us
[OSD 2] avg latency for subop 15 (ping): 250 us
[OSD 2] avg latency for op 1 (read): 9 us, B/W: 0.00 KB/s
[OSD 2] avg latency for op 3 (write_stable): 266 us, B/W: 7.62 MB/s
[OSD 2] avg latency for op 4 (sync): 601 us
[OSD 2] avg latency for op 12 (primary_write): 11266 us, B/W: 7.15 MB/s
[OSD 2] avg latency for subop 3 (write_stable): 3358 us
[OSD 2] avg latency for subop 4 (sync): 13912 us
[OSD 2] avg latency for op 1 (read): 9 us, B/W: 0.00 KB/s
[OSD 2] avg latency for op 3 (write_stable): 255 us, B/W: 9.02 MB/s
[OSD 2] avg latency for op 4 (sync): 555 us
[OSD 2] avg latency for op 12 (primary_write): 11147 us, B/W: 8.19 MB/s
[OSD 2] avg latency for subop 3 (write_stable): 3161 us
[OSD 2] avg latency for subop 4 (sync): 12241 us
[OSD 2] avg latency for subop 15 (ping): 237 us
[OSD 2] avg latency for op 1 (read): 10 us, B/W: 0.00 KB/s
[OSD 2] avg latency for op 3 (write_stable): 248 us, B/W: 10.23 MB/s
[OSD 2] avg latency for op 4 (sync): 694 us
[OSD 2] avg latency for op 12 (primary_write): 10515 us, B/W: 9.33 MB/s
[OSD 2] avg latency for subop 3 (write_stable): 2825 us
[OSD 2] avg latency for subop 4 (sync): 12504 us
[OSD 2] avg latency for op 1 (read): 9 us, B/W: 0.00 KB/s
[OSD 2] avg latency for op 3 (write_stable): 225 us, B/W: 10.68 MB/s
[OSD 2] avg latency for op 4 (sync): 461 us
[OSD 2] avg latency for op 12 (primary_write): 11171 us, B/W: 9.84 MB/s
[OSD 2] avg latency for subop 3 (write_stable): 2813 us
[OSD 2] avg latency for subop 4 (sync): 10933 us
[OSD 2] avg latency for subop 15 (ping): 246 us
[OSD 2] avg latency for op 1 (read): 9 us, B/W: 0.00 KB/s
[OSD 2] avg latency for op 3 (write_stable): 228 us, B/W: 11.12 MB/s
[OSD 2] avg latency for op 4 (sync): 465 us
[OSD 2] avg latency for op 12 (primary_write): 12162 us, B/W: 10.13 MB/s
[OSD 2] avg latency for subop 3 (write_stable): 2911 us
[OSD 2] avg latency for subop 4 (sync): 12870 us
[OSD 2] avg latency for subop 15 (ping): 260 us
[OSD 2] avg latency for op 1 (read): 11 us, B/W: 0.00 KB/s
[OSD 2] avg latency for op 3 (write_stable): 234 us, B/W: 11.31 MB/s
[OSD 2] avg latency for op 4 (sync): 536 us
[OSD 2] avg latency for op 12 (primary_write): 11983 us, B/W: 10.41 MB/s
[OSD 2] avg latency for subop 3 (write_stable): 2815 us
[OSD 2] avg latency for subop 4 (sync): 9240 us
[OSD 2] avg latency for op 1 (read): 8 us, B/W: 0.00 KB/s
[OSD 2] avg latency for op 3 (write_stable): 209 us, B/W: 11.01 MB/s
[OSD 2] avg latency for op 4 (sync): 599 us
[OSD 2] avg latency for op 12 (primary_write): 13712 us, B/W: 10.49 MB/s
[OSD 2] avg latency for subop 3 (write_stable): 3192 us
[OSD 2] avg latency for subop 4 (sync): 12764 us
[OSD 2] avg latency for subop 15 (ping): 326 us

Message from syslogd@bmhv9 at Mar 21 15:47:00 ...
 kernel:[73646.799077] NMI watchdog: Watchdog detected hard LOCKUP on cpu 26

Message from syslogd@bmhv9 at Mar 21 15:47:00 ...
 kernel:[73647.598937] NMI watchdog: Watchdog detected hard LOCKUP on cpu 27

Message from syslogd@bmhv9 at Mar 21 15:47:00 ...
 kernel:[73659.545002] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [kworker/10:3:1797192]

Message from syslogd@bmhv9 at Mar 21 15:47:00 ...
 kernel:[73659.557003] watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [migration/14:103]

Message from syslogd@bmhv9 at Mar 21 15:47:00 ...
 kernel:[73659.561002] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [runc:1799309]

Message from syslogd@bmhv9 at Mar 21 15:47:16 ...
 kernel:[73675.560922] watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [lvm:1799455]

Message from syslogd@bmhv9 at Mar 21 15:47:24 ...
 kernel:[73683.536883] watchdog: BUG: soft lockup - CPU#6 stuck for 26s! [migration/6:53]

Message from syslogd@bmhv9 at Mar 21 15:47:28 ...
 kernel:[73687.544859] watchdog: BUG: soft lockup - CPU#10 stuck for 48s! [kworker/10:3:1797192]

Message from syslogd@bmhv9 at Mar 21 15:47:28 ...
 kernel:[73687.552859] watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [migration/13:97]

Message from syslogd@bmhv9 at Mar 21 15:47:28 ...
 kernel:[73687.556859] watchdog: BUG: soft lockup - CPU#14 stuck for 48s! [migration/14:103]

Message from syslogd@bmhv9 at Mar 21 15:47:28 ...
 kernel:[73687.560859] watchdog: BUG: soft lockup - CPU#15 stuck for 48s! [runc:1799309]

Message from syslogd@bmhv9 at Mar 21 15:47:28 ...
 kernel:[73687.568858] watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [mongo:1799488]

Message from syslogd@bmhv9 at Mar 21 15:47:36 ...
 kernel:[73695.564818] watchdog: BUG: soft lockup - CPU#17 stuck for 23s! [sshd:1799504]

Message from syslogd@bmhv9 at Mar 21 15:47:40 ...
 kernel:[73699.584796] watchdog: BUG: soft lockup - CPU#25 stuck for 23s! [migration/25:169]

Message from syslogd@bmhv9 at Mar 21 15:47:44 ...
 kernel:[73703.560776] watchdog: BUG: soft lockup - CPU#16 stuck for 48s! [lvm:1799455]

Message from syslogd@bmhv9 at Mar 21 15:47:52 ...
 kernel:[73711.536737] watchdog: BUG: soft lockup - CPU#6 stuck for 52s! [migration/6:53]

Message from syslogd@bmhv9 at Mar 21 15:47:56 ...
 kernel:[73715.544713] watchdog: BUG: soft lockup - CPU#10 stuck for 74s! [kworker/10:3:1797192]

Message from syslogd@bmhv9 at Mar 21 15:47:56 ...
 kernel:[73715.552714] watchdog: BUG: soft lockup - CPU#13 stuck for 48s! [migration/13:97]

Message from syslogd@bmhv9 at Mar 21 15:47:56 ...
 kernel:[73715.556714] watchdog: BUG: soft lockup - CPU#14 stuck for 74s! [migration/14:103]

Message from syslogd@bmhv9 at Mar 21 15:47:56 ...
 kernel:[73715.560713] watchdog: BUG: soft lockup - CPU#15 stuck for 74s! [runc:1799309]

Message from syslogd@bmhv9 at Mar 21 15:47:56 ...
 kernel:[73715.568714] watchdog: BUG: soft lockup - CPU#18 stuck for 48s! [mongo:1799488]

Message from syslogd@bmhv9 at Mar 21 15:48:04 ...
 kernel:[73723.564672] watchdog: BUG: soft lockup - CPU#17 stuck for 49s! [sshd:1799504]

Message from syslogd@bmhv9 at Mar 21 15:48:08 ...
 kernel:[73727.584651] watchdog: BUG: soft lockup - CPU#25 stuck for 49s! [migration/25:169]