Hung MD on 3.13.0-rc5-00034-gb257bab when scheduling a check of disks

gistfile1.txt

[363692.014637] md: data-check of RAID array md2
[363692.014641] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[363692.014642] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
[363692.014645] md: using 128k window, over a total of 131008k.
[363692.873302] md: delaying data-check of md3 until md2 has finished (they share one or more physical units)
[363693.718181] md: delaying data-check of md4 until md2 has finished (they share one or more physical units)
[363694.599760] md: md2: data-check done.
[363694.608141] md: data-check of RAID array md4
[363694.608146] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[363694.608147] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
[363694.608151] md: using 128k window, over a total of 487987008k.
[363694.608153] md: delaying data-check of md3 until md4 has finished (they share one or more physical units)
[363694.626078] md: delaying data-check of md3 until md4 has finished (they share one or more physical units)
[363959.474576] INFO: task kjournald:13993 blocked for more than 120 seconds.
[363959.474580]       Not tainted 3.13.0-rc5-00034-gb257bab #53
[363959.474582] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[363959.474583] kjournald       D 0000000000000000     0 13993      2 0x00000000
[363959.474588]  ffff88081f46d9e8 0000000000000002 ffff88081f46dfd8 ffff88081c0c4900
[363959.474592]  00000000001d39c0 00000000001d39c0 ffff8808224e4900 ffff88081c0c4900
[363959.474595]  ffff88081c0c50b0 0000000000000001 ffff88081f46d938 ffffffff81069de7
[363959.474599] Call Trace:
[363959.474605]  [<ffffffff81069de7>] ? local_clock+0x41/0x5a
[363959.474609]  [<ffffffff8107bff9>] ? lock_release_holdtime.part.5+0xc2/0xc7
[363959.474612]  [<ffffffff81075f53>] ? prepare_to_wait_event+0xbb/0xf9
[363959.474616]  [<ffffffff814a1da8>] schedule+0x6f/0x71
[363959.474631]  [<ffffffffa03457f3>] md_write_start+0x11e/0x136 [md_mod]
[363959.474634]  [<ffffffff8107604b>] ? __init_waitqueue_head+0x50/0x50
[363959.474639]  [<ffffffffa03bf5df>] make_request+0x49/0xc29 [raid1]
[363959.474642]  [<ffffffff8107d52e>] ? __lock_acquire.isra.23+0x599/0xbfb
[363959.474647]  [<ffffffff81008f83>] ? native_sched_clock+0x29/0x6f
[363959.474649]  [<ffffffff81008fd2>] ? sched_clock+0x9/0xd
[363959.474652]  [<ffffffff81069d1d>] ? sched_clock_cpu+0x48/0xd1
[363959.474655]  [<ffffffff8107bff9>] ? lock_release_holdtime.part.5+0xc2/0xc7
[363959.474662]  [<ffffffffa03444d1>] ? md_make_request+0x139/0x2a4 [md_mod]
[363959.474668]  [<ffffffffa0344503>] md_make_request+0x16b/0x2a4 [md_mod]
[363959.474674]  [<ffffffffa0344402>] ? md_make_request+0x6a/0x2a4 [md_mod]
[363959.474682]  [<ffffffffa0302c8f>] ? dm_put_live_table+0x35/0x3a [dm_mod]
[363959.474689]  [<ffffffffa0302f7c>] ? dm_request+0x1d4/0x1e1 [dm_mod]
[363959.474695]  [<ffffffffa0302de2>] ? dm_request+0x3a/0x1e1 [dm_mod]
[363959.474698]  [<ffffffff81213860>] generic_make_request+0x9f/0xe0
[363959.474700]  [<ffffffff812139ac>] submit_bio+0x10b/0x129
[363959.474704]  [<ffffffff81186cbb>] _submit_bh+0x1bb/0x1dc
[363959.474707]  [<ffffffff811e0ede>] journal_commit_transaction+0xa8e/0x1174
[363959.474711]  [<ffffffff81047a72>] ? try_to_del_timer_sync+0x55/0x61
[363959.474714]  [<ffffffff81047a83>] ? del_timer_sync+0x5/0xce
[363959.474717]  [<ffffffff811e4713>] kjournald+0xed/0x27e
[363959.474719]  [<ffffffff8107604b>] ? __init_waitqueue_head+0x50/0x50
[363959.474722]  [<ffffffff811e4626>] ? commit_timeout+0x10/0x10
[363959.474725]  [<ffffffff8105a7ae>] kthread+0xed/0xf5
[363959.474729]  [<ffffffff8105a6c1>] ? __init_kthread_worker+0x5a/0x5a
[363959.474732]  [<ffffffff814adffc>] ret_from_fork+0x7c/0xb0
[363959.474735]  [<ffffffff8105a6c1>] ? __init_kthread_worker+0x5a/0x5a
[363959.474737] no locks held by kjournald/13993.
[363959.474744] INFO: task syslog-ng:9788 blocked for more than 120 seconds.
[363959.474745]       Not tainted 3.13.0-rc5-00034-gb257bab #53
[363959.474746] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[363959.474748] syslog-ng       D 0000000000000000     0  9788  14284 0x00000000
[363959.474751]  ffff88014e899a08 0000000000000002 ffff88014e899fd8 ffff8808227ea480
[363959.474754]  00000000001d39c0 00000000001d39c0 ffffffff81911480 ffff8808227ea480
[363959.474757]  ffff8808227eacd8 0000000000000001 ffff88014e899958 ffffffff81069de7
[363959.474761] Call Trace:
[363959.474764]  [<ffffffff81069de7>] ? local_clock+0x41/0x5a
[363959.474767]  [<ffffffff8107bff9>] ? lock_release_holdtime.part.5+0xc2/0xc7
[363959.474769]  [<ffffffff81075cd1>] ? prepare_to_wait+0x72/0x7d
[363959.474772]  [<ffffffff814a1da8>] schedule+0x6f/0x71
[363959.474774]  [<ffffffff811dfc71>] do_get_write_access+0x228/0x41b
[363959.474777]  [<ffffffff810760af>] ? wake_atomic_t_function+0x2d/0x2d
[363959.474780]  [<ffffffff811cd94a>] ? ext3_dirty_inode+0x6c/0x83
[363959.474782]  [<ffffffff811dffc2>] journal_get_write_access+0x2b/0x40
[363959.474786]  [<ffffffff811dbb90>] __ext3_journal_get_write_access+0x25/0x51
[363959.474788]  [<ffffffff811cd94a>] ? ext3_dirty_inode+0x6c/0x83
[363959.474791]  [<ffffffff811cafcf>] ext3_reserve_inode_write+0x41/0x87
[363959.474793]  [<ffffffff811cb064>] ext3_mark_inode_dirty+0x4f/0x7f
[363959.474795]  [<ffffffff811cd94a>] ext3_dirty_inode+0x6c/0x83
[363959.474799]  [<ffffffff8118155f>] __mark_inode_dirty+0x59/0x25e
[363959.474802]  [<ffffffff811730cc>] update_time+0xa4/0xad
[363959.474805]  [<ffffffff81176e2f>] ? mnt_clone_write+0x12/0x2a
[363959.474808]  [<ffffffff81173172>] file_update_time+0x9d/0xbe
[363959.474812]  [<ffffffff810fcf0a>] ? generic_file_aio_write+0x49/0x95
[363959.474815]  [<ffffffff810fcda9>] __generic_file_aio_write+0x1ca/0x2e2
[363959.474817]  [<ffffffff810fcf1d>] generic_file_aio_write+0x5c/0x95
[363959.474821]  [<ffffffff8115b785>] do_sync_readv_writev+0x4b/0x6e
[363959.474824]  [<ffffffff8115caa8>] do_readv_writev+0xf5/0x1db
[363959.474826]  [<ffffffff810fcec1>] ? __generic_file_aio_write+0x2e2/0x2e2
[363959.474829]  [<ffffffff8115b7a8>] ? do_sync_readv_writev+0x6e/0x6e
[363959.474832]  [<ffffffff8107bff9>] ? lock_release_holdtime.part.5+0xc2/0xc7
[363959.474835]  [<ffffffff811752ae>] ? fget_light+0xd6/0x108
[363959.474838]  [<ffffffff8115cbc8>] vfs_writev+0x3a/0x4e
[363959.474840]  [<ffffffff8115cd1d>] SyS_writev+0x4c/0xaa
[363959.474842]  [<ffffffff814ae2d0>] tracesys+0xdd/0xe2
[363959.474844] 3 locks held by syslog-ng/9788:
[363959.474845]  #0:  (sb_writers#12){......}, at: [<ffffffff8115ca79>] do_readv_writev+0xc6/0x1db
[363959.474851]  #1:  (&sb->s_type->i_mutex_key#11){......}, at: [<ffffffff810fcf0a>] generic_file_aio_write+0x49/0x95
[363959.474856]  #2:  (jbd_handle){......}, at: [<ffffffff811de7a8>] start_this_handle+0x319/0x398
[363959.474909] INFO: task kworker/u16:4:9492 blocked for more than 120 seconds.
[363959.474910]       Not tainted 3.13.0-rc5-00034-gb257bab #53
[363959.474911] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[363959.474913] kworker/u16:4   D 0000000000000001     0  9492      2 0x00000000
[363959.474918] Workqueue: writeback bdi_writeback_workfn (flush-254:1)
[363959.474920]  ffff8801483a5598 0000000000000002 ffff8801483a5fd8 ffff88061c2d4900
[363959.474923]  00000000001d39c0 00000000001d39c0 ffff8808225cc900 ffff88061c2d4900
[363959.474926]  ffff88061c2d5158 0000000000000001 ffff8801483a54e8 ffffffff81069de7
[363959.474930] Call Trace:
[363959.474932]  [<ffffffff81069de7>] ? local_clock+0x41/0x5a
[363959.474935]  [<ffffffff8107bff9>] ? lock_release_holdtime.part.5+0xc2/0xc7
[363959.474938]  [<ffffffff81075f53>] ? prepare_to_wait_event+0xbb/0xf9
[363959.474940]  [<ffffffff814a1da8>] schedule+0x6f/0x71
[363959.474947]  [<ffffffffa03457f3>] md_write_start+0x11e/0x136 [md_mod]
[363959.474950]  [<ffffffff8107604b>] ? __init_waitqueue_head+0x50/0x50
[363959.474955]  [<ffffffffa03bf5df>] make_request+0x49/0xc29 [raid1]
[363959.474958]  [<ffffffff8107d52e>] ? __lock_acquire.isra.23+0x599/0xbfb
[363959.474961]  [<ffffffff81008f83>] ? native_sched_clock+0x29/0x6f
[363959.474964]  [<ffffffff81008fd2>] ? sched_clock+0x9/0xd
[363959.474967]  [<ffffffff81069d1d>] ? sched_clock_cpu+0x48/0xd1
[363959.474970]  [<ffffffff8107bff9>] ? lock_release_holdtime.part.5+0xc2/0xc7
[363959.474976]  [<ffffffffa03444d1>] ? md_make_request+0x139/0x2a4 [md_mod]
[363959.474982]  [<ffffffffa0344503>] md_make_request+0x16b/0x2a4 [md_mod]
[363959.474988]  [<ffffffffa0344402>] ? md_make_request+0x6a/0x2a4 [md_mod]
[363959.474995]  [<ffffffffa0302c8f>] ? dm_put_live_table+0x35/0x3a [dm_mod]
[363959.475001]  [<ffffffffa0302f7c>] ? dm_request+0x1d4/0x1e1 [dm_mod]
[363959.475007]  [<ffffffffa0302de2>] ? dm_request+0x3a/0x1e1 [dm_mod]
[363959.475010]  [<ffffffff81213860>] generic_make_request+0x9f/0xe0
[363959.475012]  [<ffffffff812139ac>] submit_bio+0x10b/0x129
[363959.475015]  [<ffffffff81186cbb>] _submit_bh+0x1bb/0x1dc
[363959.475017]  [<ffffffff81186cec>] submit_bh+0x10/0x12
[363959.475020]  [<ffffffff811891a6>] __block_write_full_page+0x1ad/0x2b3
[363959.475022]  [<ffffffff811879a0>] ? end_buffer_async_read+0x16c/0x16c
[363959.475025]  [<ffffffff8118935c>] block_write_full_page_endio+0xb0/0xb5
[363959.475027]  [<ffffffff81189376>] block_write_full_page+0x15/0x17
[363959.475031]  [<ffffffff811c9e58>] ext3_writeback_writepage+0xdb/0x179
[363959.475034]  [<ffffffff811049fd>] __writepage+0x18/0x37
[363959.475037]  [<ffffffff81104ecc>] write_cache_pages+0x25d/0x37c
[363959.475040]  [<ffffffff811049e5>] ? global_dirtyable_memory+0x3c/0x3c
[363959.475043]  [<ffffffff8107d52e>] ? __lock_acquire.isra.23+0x599/0xbfb
[363959.475045]  [<ffffffff81069d1d>] ? sched_clock_cpu+0x48/0xd1
[363959.475048]  [<ffffffff81069de7>] ? local_clock+0x41/0x5a
[363959.475051]  [<ffffffff8110502c>] generic_writepages+0x41/0x5b
[363959.475054]  [<ffffffff811064e5>] do_writepages+0x28/0x2f
[363959.475057]  [<ffffffff81180d5a>] __writeback_single_inode+0x6e/0x1d8
[363959.475060]  [<ffffffff81181e13>] writeback_sb_inodes+0x1f1/0x329
[363959.475063]  [<ffffffff81181fbf>] __writeback_inodes_wb+0x74/0xb9
[363959.475065]  [<ffffffff811821d3>] wb_writeback+0x13b/0x280
[363959.475068]  [<ffffffff8117371e>] ? get_nr_dirty_inodes+0x4a/0x6f
[363959.475071]  [<ffffffff811825d5>] bdi_writeback_workfn+0x1a0/0x39d
[363959.475074]  [<ffffffff8105451a>] process_one_work+0x24d/0x42d
[363959.475077]  [<ffffffff8105448f>] ? process_one_work+0x1c2/0x42d
[363959.475080]  [<ffffffff8107bff9>] ? lock_release_holdtime.part.5+0xc2/0xc7
[363959.475083]  [<ffffffff81054e9a>] worker_thread+0x1d6/0x2d1
[363959.475087]  [<ffffffff81054cc4>] ? manage_workers.isra.19+0x1cb/0x1cb
[363959.475089]  [<ffffffff8105a7ae>] kthread+0xed/0xf5
[363959.475093]  [<ffffffff8105a6c1>] ? __init_kthread_worker+0x5a/0x5a
[363959.475095]  [<ffffffff814adffc>] ret_from_fork+0x7c/0xb0
[363959.475098]  [<ffffffff8105a6c1>] ? __init_kthread_worker+0x5a/0x5a
[363959.475100] 3 locks held by kworker/u16:4/9492:
[363959.475101]  #0:  (writeback){......}, at: [<ffffffff8105448f>] process_one_work+0x1c2/0x42d
[363959.475106]  #1:  ((&(&wb->dwork)->work)){......}, at: [<ffffffff8105448f>] process_one_work+0x1c2/0x42d
[363959.475111]  #2:  (&type->s_umount_key#28){......}, at: [<ffffffff8115ed12>] grab_super_passive+0x51/0x7b
[363959.475117] INFO: task md4_resync:11683 blocked for more than 120 seconds.
[363959.475119]       Not tainted 3.13.0-rc5-00034-gb257bab #53
[363959.475120] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[363959.475121] md4_resync      D 0000000000000001     0 11683      2 0x00000000
[363959.475124]  ffff88019270db18 0000000000000002 ffff88019270dfd8 ffff880820544900
[363959.475127]  00000000001d39c0 00000000001d39c0 ffff8808225cc900 ffff880820544900
[363959.475130]  ffff88019270da48 ffffffff81008f83 ffff88019270da58 ffffffff81008fd2
[363959.475134] Call Trace:
[363959.475136]  [<ffffffff81008f83>] ? native_sched_clock+0x29/0x6f
[363959.475139]  [<ffffffff81008fd2>] ? sched_clock+0x9/0xd
[363959.475142]  [<ffffffff81069d1d>] ? sched_clock_cpu+0x48/0xd1
[363959.475144]  [<ffffffff81069de7>] ? local_clock+0x41/0x5a
[363959.475147]  [<ffffffff8107bff9>] ? lock_release_holdtime.part.5+0xc2/0xc7
[363959.475152]  [<ffffffffa03bce5a>] ? sync_request+0x2d6/0x944 [raid1]
[363959.475155]  [<ffffffff814a1da8>] schedule+0x6f/0x71
[363959.475160]  [<ffffffffa03bce5f>] sync_request+0x2db/0x944 [raid1]
[363959.475164]  [<ffffffffa03bce74>] ? sync_request+0x2f0/0x944 [raid1]
[363959.475167]  [<ffffffff8107bff9>] ? lock_release_holdtime.part.5+0xc2/0xc7
[363959.475170]  [<ffffffff8107604b>] ? __init_waitqueue_head+0x50/0x50
[363959.475178]  [<ffffffffa0347449>] md_do_sync+0x88b/0xcf2 [md_mod]
[363959.475184]  [<ffffffffa0346cfd>] ? md_do_sync+0x13f/0xcf2 [md_mod]
[363959.475187]  [<ffffffff81008fd2>] ? sched_clock+0x9/0xd
[363959.475194]  [<ffffffffa0344a23>] md_thread+0x117/0x135 [md_mod]
[363959.475200]  [<ffffffffa034490c>] ? md_register_thread+0xd5/0xd5 [md_mod]
[363959.475203]  [<ffffffff8105a7ae>] kthread+0xed/0xf5
[363959.475206]  [<ffffffff814a2576>] ? wait_for_common+0x30/0x152
[363959.475209]  [<ffffffff8105a6c1>] ? __init_kthread_worker+0x5a/0x5a
[363959.475211]  [<ffffffff814adffc>] ret_from_fork+0x7c/0xb0
[363959.475214]  [<ffffffff8105a6c1>] ? __init_kthread_worker+0x5a/0x5a
[363959.475216] no locks held by md4_resync/11683.
[363959.475217] INFO: task checkarray:11741 blocked for more than 120 seconds.
[363959.475219]       Not tainted 3.13.0-rc5-00034-gb257bab #53
[363959.475220] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[363959.475221] checkarray      D 0000000000000005     0 11741  26934 0x00000004
[363959.475224]  ffff88076c2adc58 0000000000000002 ffff88076c2adfd8 ffff880504bd4900
[363959.475227]  00000000001d39c0 00000000001d39c0 ffff8808225e8000 ffff880504bd4900
[363959.475230]  ffffffff81947660 ffff880504bd4900 ffff88076c2adba8 ffffffff81069de7
[363959.475234] Call Trace:
[363959.475236]  [<ffffffff81069de7>] ? local_clock+0x41/0x5a
[363959.475239]  [<ffffffff81008f83>] ? native_sched_clock+0x29/0x6f
[363959.475242]  [<ffffffff81008fd2>] ? sched_clock+0x9/0xd
[363959.475244]  [<ffffffff81069d1d>] ? sched_clock_cpu+0x48/0xd1
[363959.475247]  [<ffffffff81069de7>] ? local_clock+0x41/0x5a
[363959.475250]  [<ffffffff8107d52e>] ? __lock_acquire.isra.23+0x599/0xbfb
[363959.475252]  [<ffffffff81069de7>] ? local_clock+0x41/0x5a
[363959.475254]  [<ffffffff814a1da8>] schedule+0x6f/0x71
[363959.475257]  [<ffffffff814a0ecc>] schedule_timeout+0x2c/0x1ff
[363959.475259]  [<ffffffff81069de7>] ? local_clock+0x41/0x5a
[363959.475262]  [<ffffffff8107bff9>] ? lock_release_holdtime.part.5+0xc2/0xc7
[363959.475264]  [<ffffffff814a2658>] ? wait_for_common+0x112/0x152
[363959.475267]  [<ffffffff814a2660>] wait_for_common+0x11a/0x152
[363959.475269]  [<ffffffff81067373>] ? try_to_wake_up+0x26a/0x26a
[363959.475272]  [<ffffffff814a26b5>] wait_for_completion+0x1d/0x1f
[363959.475275]  [<ffffffff8105aa89>] kthread_stop+0x61/0xb8
[363959.475281]  [<ffffffffa034482a>] md_unregister_thread+0x68/0x75 [md_mod]
[363959.475288]  [<ffffffffa0349f21>] md_reap_sync_thread+0x1b/0x13d [md_mod]
[363959.475295]  [<ffffffffa034a0e0>] action_store+0x9d/0x1df [md_mod]
[363959.475302]  [<ffffffffa0346a2b>] md_attr_store+0xbe/0xf3 [md_mod]
[363959.475305]  [<ffffffff811c1ab4>] sysfs_write_file+0x138/0x18b
[363959.475308]  [<ffffffff8115c350>] vfs_write+0xc3/0x18d
[363959.475310]  [<ffffffff8115c751>] SyS_write+0x4c/0x85
[363959.475313]  [<ffffffff814ae2d0>] tracesys+0xdd/0xe2
[363959.475314] 4 locks held by checkarray/11741:
[363959.475315]  #0:  (sb_writers#5){......}, at: [<ffffffff8115c335>] vfs_write+0xa8/0x18d
[363959.475320]  #1:  (&of->mutex){......}, at: [<ffffffff811c1a4a>] sysfs_write_file+0xce/0x18b
[363959.475325]  #2:  (s_active#369){......}, at: [<ffffffff811c1a52>] sysfs_write_file+0xd6/0x18b
[363959.475330]  #3:  (&mddev->reconfig_mutex){......}, at: [<ffffffffa0346a16>] md_attr_store+0xa9/0xf3 [md_mod]
[363959.475338] INFO: task emerge:11761 blocked for more than 120 seconds.
[363959.475339]       Not tainted 3.13.0-rc5-00034-gb257bab #53
[363959.475341] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[363959.475342] emerge          D 0000000000000001     0 11761  15489 0x00000004
[363959.475344]  ffff8801938bd6f8 0000000000000002 ffff8801938bdfd8 ffff880812a74900
[363959.475348]  00000000001d39c0 00000000001d39c0 ffff8808225cc900 ffff880812a74900
[363959.475351]  ffff880812a75190 0000000000000001 ffff8801938bd648 ffffffff81069de7
[363959.475354] Call Trace:
[363959.475357]  [<ffffffff81069de7>] ? local_clock+0x41/0x5a
[363959.475360]  [<ffffffff8107bff9>] ? lock_release_holdtime.part.5+0xc2/0xc7
[363959.475362]  [<ffffffff81075f53>] ? prepare_to_wait_event+0xbb/0xf9
[363959.475365]  [<ffffffff814a1da8>] schedule+0x6f/0x71
[363959.475371]  [<ffffffffa03457f3>] md_write_start+0x11e/0x136 [md_mod]
[363959.475374]  [<ffffffff8107604b>] ? __init_waitqueue_head+0x50/0x50
[363959.475379]  [<ffffffffa03bf5df>] make_request+0x49/0xc29 [raid1]
[363959.475382]  [<ffffffff8107d52e>] ? __lock_acquire.isra.23+0x599/0xbfb
[363959.475385]  [<ffffffff81008f83>] ? native_sched_clock+0x29/0x6f
[363959.475387]  [<ffffffff81008fd2>] ? sched_clock+0x9/0xd
[363959.475390]  [<ffffffff81069d1d>] ? sched_clock_cpu+0x48/0xd1
[363959.475393]  [<ffffffff8107bff9>] ? lock_release_holdtime.part.5+0xc2/0xc7
[363959.475399]  [<ffffffffa03444d1>] ? md_make_request+0x139/0x2a4 [md_mod]
[363959.475405]  [<ffffffffa0344503>] md_make_request+0x16b/0x2a4 [md_mod]
[363959.475411]  [<ffffffffa0344402>] ? md_make_request+0x6a/0x2a4 [md_mod]
[363959.475418]  [<ffffffffa0302c8f>] ? dm_put_live_table+0x35/0x3a [dm_mod]
[363959.475424]  [<ffffffffa0302f7c>] ? dm_request+0x1d4/0x1e1 [dm_mod]
[363959.475431]  [<ffffffffa0302de2>] ? dm_request+0x3a/0x1e1 [dm_mod]
[363959.475433]  [<ffffffff81213860>] generic_make_request+0x9f/0xe0
[363959.475435]  [<ffffffff812139ac>] submit_bio+0x10b/0x129
[363959.475438]  [<ffffffff81186cbb>] _submit_bh+0x1bb/0x1dc
[363959.475440]  [<ffffffff81186cec>] submit_bh+0x10/0x12
[363959.475443]  [<ffffffff811891a6>] __block_write_full_page+0x1ad/0x2b3
[363959.475445]  [<ffffffff811879a0>] ? end_buffer_async_read+0x16c/0x16c
[363959.475448]  [<ffffffff8118935c>] block_write_full_page_endio+0xb0/0xb5
[363959.475450]  [<ffffffff81189376>] block_write_full_page+0x15/0x17
[363959.475453]  [<ffffffff811c9e58>] ext3_writeback_writepage+0xdb/0x179
[363959.475456]  [<ffffffff811049fd>] __writepage+0x18/0x37
[363959.475459]  [<ffffffff81104ecc>] write_cache_pages+0x25d/0x37c
[363959.475462]  [<ffffffff811049e5>] ? global_dirtyable_memory+0x3c/0x3c
[363959.475465]  [<ffffffff81008f83>] ? native_sched_clock+0x29/0x6f
[363959.475467]  [<ffffffff81008fd2>] ? sched_clock+0x9/0xd
[363959.475470]  [<ffffffff81069d1d>] ? sched_clock_cpu+0x48/0xd1
[363959.475473]  [<ffffffff8107cf71>] ? debug_check_no_locks_freed+0x135/0x159
[363959.475476]  [<ffffffff8110502c>] generic_writepages+0x41/0x5b
[363959.475479]  [<ffffffff811064e5>] do_writepages+0x28/0x2f
[363959.475481]  [<ffffffff810fc9b3>] __filemap_fdatawrite_range+0x55/0x57
[363959.475503]  [<ffffffff810fca35>] filemap_flush+0x1c/0x1e
[363959.475511]  [<ffffffff811d127f>] ext3_rename+0x54f/0x586
[363959.475519]  [<ffffffff81168a73>] vfs_rename+0x326/0x512
[363959.475529]  [<ffffffff8116ac23>] SyS_renameat+0x21c/0x309
[363959.475532]  [<ffffffff8100d203>] ? syscall_trace_enter+0x1bb/0x1f5
[363959.475535]  [<ffffffff8116ad2b>] SyS_rename+0x1b/0x20
[363959.475537]  [<ffffffff814ae2d0>] tracesys+0xdd/0xe2
[363959.475539] 4 locks held by emerge/11761:
[363959.475539]  #0:  (sb_writers#12){......}, at: [<ffffffff81177ce6>] mnt_want_write+0x24/0x4b
[363959.475544]  #1:  (&type->i_mutex_dir_key#3/1){......}, at: [<ffffffff811657dc>] lock_rename+0x33/0xbb
[363959.475549]  #2:  (&sb->s_type->i_mutex_key#11){......}, at: [<ffffffff81172a90>] lock_two_nondirectories+0x96/0xc5
[363959.475554]  #3:  (&sb->s_type->i_mutex_key#11/4){......}, at: [<ffffffff81172aba>] lock_two_nondirectories+0xc0/0xc5
[364079.395867] INFO: task kjournald:13993 blocked for more than 120 seconds.
[364079.395871]       Not tainted 3.13.0-rc5-00034-gb257bab #53
[364079.395872] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[364079.395874] kjournald       D 0000000000000000     0 13993      2 0x00000000
[364079.395877]  ffff88081f46d9e8 0000000000000002 ffff88081f46dfd8 ffff88081c0c4900
[364079.395881]  00000000001d39c0 00000000001d39c0 ffff8808224e4900 ffff88081c0c4900
[364079.395883]  ffff88081c0c50b0 0000000000000001 ffff88081f46d938 ffffffff81069de7
[364079.395886] Call Trace:
[364079.395892]  [<ffffffff81069de7>] ? local_clock+0x41/0x5a
[364079.395896]  [<ffffffff8107bff9>] ? lock_release_holdtime.part.5+0xc2/0xc7
[364079.395898]  [<ffffffff81075f53>] ? prepare_to_wait_event+0xbb/0xf9
[364079.395901]  [<ffffffff814a1da8>] schedule+0x6f/0x71
[364079.395911]  [<ffffffffa03457f3>] md_write_start+0x11e/0x136 [md_mod]
[364079.395913]  [<ffffffff8107604b>] ? __init_waitqueue_head+0x50/0x50
[364079.395918]  [<ffffffffa03bf5df>] make_request+0x49/0xc29 [raid1]
[364079.395920]  [<ffffffff8107d52e>] ? __lock_acquire.isra.23+0x599/0xbfb
[364079.395924]  [<ffffffff81008f83>] ? native_sched_clock+0x29/0x6f
[364079.395926]  [<ffffffff81008fd2>] ? sched_clock+0x9/0xd
[364079.395928]  [<ffffffff81069d1d>] ? sched_clock_cpu+0x48/0xd1
[364079.395930]  [<ffffffff8107bff9>] ? lock_release_holdtime.part.5+0xc2/0xc7
[364079.395936]  [<ffffffffa03444d1>] ? md_make_request+0x139/0x2a4 [md_mod]
[364079.395941]  [<ffffffffa0344503>] md_make_request+0x16b/0x2a4 [md_mod]
[364079.395946]  [<ffffffffa0344402>] ? md_make_request+0x6a/0x2a4 [md_mod]
[364079.395952]  [<ffffffffa0302c8f>] ? dm_put_live_table+0x35/0x3a [dm_mod]
[364079.395957]  [<ffffffffa0302f7c>] ? dm_request+0x1d4/0x1e1 [dm_mod]
[364079.395962]  [<ffffffffa0302de2>] ? dm_request+0x3a/0x1e1 [dm_mod]
[364079.395964]  [<ffffffff81213860>] generic_make_request+0x9f/0xe0
[364079.395966]  [<ffffffff812139ac>] submit_bio+0x10b/0x129
[364079.395969]  [<ffffffff81186cbb>] _submit_bh+0x1bb/0x1dc
[364079.395971]  [<ffffffff811e0ede>] journal_commit_transaction+0xa8e/0x1174
[364079.395975]  [<ffffffff81047a72>] ? try_to_del_timer_sync+0x55/0x61
[364079.395977]  [<ffffffff81047a83>] ? del_timer_sync+0x5/0xce
[364079.395979]  [<ffffffff811e4713>] kjournald+0xed/0x27e
[364079.395981]  [<ffffffff8107604b>] ? __init_waitqueue_head+0x50/0x50
[364079.395983]  [<ffffffff811e4626>] ? commit_timeout+0x10/0x10
[364079.395986]  [<ffffffff8105a7ae>] kthread+0xed/0xf5
[364079.395989]  [<ffffffff8105a6c1>] ? __init_kthread_worker+0x5a/0x5a
[364079.395991]  [<ffffffff814adffc>] ret_from_fork+0x7c/0xb0
[364079.395993]  [<ffffffff8105a6c1>] ? __init_kthread_worker+0x5a/0x5a
[364079.395995] no locks held by kjournald/13993.
[364079.396002] INFO: task syslog-ng:9788 blocked for more than 120 seconds.
[364079.396003]       Not tainted 3.13.0-rc5-00034-gb257bab #53
[364079.396004] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[364079.396005] syslog-ng       D 0000000000000000     0  9788  14284 0x00000000
[364079.396007]  ffff88014e899a08 0000000000000002 ffff88014e899fd8 ffff8808227ea480
[364079.396010]  00000000001d39c0 00000000001d39c0 ffffffff81911480 ffff8808227ea480
[364079.396013]  ffff8808227eacd8 0000000000000001 ffff88014e899958 ffffffff81069de7
[364079.396015] Call Trace:
[364079.396017]  [<ffffffff81069de7>] ? local_clock+0x41/0x5a
[364079.396020]  [<ffffffff8107bff9>] ? lock_release_holdtime.part.5+0xc2/0xc7
[364079.396022]  [<ffffffff81075cd1>] ? prepare_to_wait+0x72/0x7d
[364079.396024]  [<ffffffff814a1da8>] schedule+0x6f/0x71
[364079.396026]  [<ffffffff811dfc71>] do_get_write_access+0x228/0x41b
[364079.396028]  [<ffffffff810760af>] ? wake_atomic_t_function+0x2d/0x2d
[364079.396030]  [<ffffffff811cd94a>] ? ext3_dirty_inode+0x6c/0x83
[364079.396032]  [<ffffffff811dffc2>] journal_get_write_access+0x2b/0x40
[364079.396035]  [<ffffffff811dbb90>] __ext3_journal_get_write_access+0x25/0x51
[364079.396037]  [<ffffffff811cd94a>] ? ext3_dirty_inode+0x6c/0x83
[364079.396039]  [<ffffffff811cafcf>] ext3_reserve_inode_write+0x41/0x87
[364079.396041]  [<ffffffff811cb064>] ext3_mark_inode_dirty+0x4f/0x7f
[364079.396043]  [<ffffffff811cd94a>] ext3_dirty_inode+0x6c/0x83
[364079.396045]  [<ffffffff8118155f>] __mark_inode_dirty+0x59/0x25e
[364079.396048]  [<ffffffff811730cc>] update_time+0xa4/0xad
[364079.396051]  [<ffffffff81176e2f>] ? mnt_clone_write+0x12/0x2a
[364079.396053]  [<ffffffff81173172>] file_update_time+0x9d/0xbe
[364079.396056]  [<ffffffff810fcf0a>] ? generic_file_aio_write+0x49/0x95
[364079.396058]  [<ffffffff810fcda9>] __generic_file_aio_write+0x1ca/0x2e2
[364079.396060]  [<ffffffff810fcf1d>] generic_file_aio_write+0x5c/0x95
[364079.396063]  [<ffffffff8115b785>] do_sync_readv_writev+0x4b/0x6e
[364079.396065]  [<ffffffff8115caa8>] do_readv_writev+0xf5/0x1db
[364079.396067]  [<ffffffff810fcec1>] ? __generic_file_aio_write+0x2e2/0x2e2
[364079.396070]  [<ffffffff8115b7a8>] ? do_sync_readv_writev+0x6e/0x6e
[364079.396072]  [<ffffffff8107bff9>] ? lock_release_holdtime.part.5+0xc2/0xc7
[364079.396074]  [<ffffffff811752ae>] ? fget_light+0xd6/0x108
[364079.396076]  [<ffffffff8115cbc8>] vfs_writev+0x3a/0x4e
[364079.396078]  [<ffffffff8115cd1d>] SyS_writev+0x4c/0xaa
[364079.396080]  [<ffffffff814ae2d0>] tracesys+0xdd/0xe2
[364079.396082] 3 locks held by syslog-ng/9788:
[364079.396083]  #0:  (sb_writers#12){......}, at: [<ffffffff8115ca79>] do_readv_writev+0xc6/0x1db
[364079.396087]  #1:  (&sb->s_type->i_mutex_key#11){......}, at: [<ffffffff810fcf0a>] generic_file_aio_write+0x49/0x95
[364079.396091]  #2:  (jbd_handle){......}, at: [<ffffffff811de7a8>] start_this_handle+0x319/0x398
[364079.396139] INFO: task kworker/u16:4:9492 blocked for more than 120 seconds.
[364079.396140]       Not tainted 3.13.0-rc5-00034-gb257bab #53
[364079.396141] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[364079.396142] kworker/u16:4   D 0000000000000001     0  9492      2 0x00000000
[364079.396146] Workqueue: writeback bdi_writeback_workfn (flush-254:1)
[364079.396148]  ffff8801483a5598 0000000000000002 ffff8801483a5fd8 ffff88061c2d4900
[364079.396150]  00000000001d39c0 00000000001d39c0 ffff8808225cc900 ffff88061c2d4900
[364079.396153]  ffff88061c2d5158 0000000000000001 ffff8801483a54e8 ffffffff81069de7
[364079.396155] Call Trace:
[364079.396158]  [<ffffffff81069de7>] ? local_clock+0x41/0x5a
[364079.396160]  [<ffffffff8107bff9>] ? lock_release_holdtime.part.5+0xc2/0xc7
[364079.396162]  [<ffffffff81075f53>] ? prepare_to_wait_event+0xbb/0xf9
[364079.396164]  [<ffffffff814a1da8>] schedule+0x6f/0x71
[364079.396170]  [<ffffffffa03457f3>] md_write_start+0x11e/0x136 [md_mod]
[364079.396172]  [<ffffffff8107604b>] ? __init_waitqueue_head+0x50/0x50
[364079.396176]  [<ffffffffa03bf5df>] make_request+0x49/0xc29 [raid1]
[364079.396178]  [<ffffffff8107d52e>] ? __lock_acquire.isra.23+0x599/0xbfb
[364079.396181]  [<ffffffff81008f83>] ? native_sched_clock+0x29/0x6f
[364079.396183]  [<ffffffff81008fd2>] ? sched_clock+0x9/0xd
[364079.396185]  [<ffffffff81069d1d>] ? sched_clock_cpu+0x48/0xd1
[364079.396187]  [<ffffffff8107bff9>] ? lock_release_holdtime.part.5+0xc2/0xc7
[364079.396192]  [<ffffffffa03444d1>] ? md_make_request+0x139/0x2a4 [md_mod]
[364079.396197]  [<ffffffffa0344503>] md_make_request+0x16b/0x2a4 [md_mod]
[364079.396202]  [<ffffffffa0344402>] ? md_make_request+0x6a/0x2a4 [md_mod]
[364079.396207]  [<ffffffffa0302c8f>] ? dm_put_live_table+0x35/0x3a [dm_mod]
[364079.396212]  [<ffffffffa0302f7c>] ? dm_request+0x1d4/0x1e1 [dm_mod]
[364079.396217]  [<ffffffffa0302de2>] ? dm_request+0x3a/0x1e1 [dm_mod]
[364079.396219]  [<ffffffff81213860>] generic_make_request+0x9f/0xe0
[364079.396220]  [<ffffffff812139ac>] submit_bio+0x10b/0x129
[364079.396222]  [<ffffffff81186cbb>] _submit_bh+0x1bb/0x1dc
[364079.396224]  [<ffffffff81186cec>] submit_bh+0x10/0x12
[364079.396226]  [<ffffffff811891a6>] __block_write_full_page+0x1ad/0x2b3
[364079.396228]  [<ffffffff811879a0>] ? end_buffer_async_read+0x16c/0x16c
[364079.396230]  [<ffffffff8118935c>] block_write_full_page_endio+0xb0/0xb5
[364079.396232]  [<ffffffff81189376>] block_write_full_page+0x15/0x17
[364079.396235]  [<ffffffff811c9e58>] ext3_writeback_writepage+0xdb/0x179
[364079.396237]  [<ffffffff811049fd>] __writepage+0x18/0x37
[364079.396240]  [<ffffffff81104ecc>] write_cache_pages+0x25d/0x37c
[364079.396242]  [<ffffffff811049e5>] ? global_dirtyable_memory+0x3c/0x3c
[364079.396245]  [<ffffffff8107d52e>] ? __lock_acquire.isra.23+0x599/0xbfb
[364079.396247]  [<ffffffff81069d1d>] ? sched_clock_cpu+0x48/0xd1
[364079.396249]  [<ffffffff81069de7>] ? local_clock+0x41/0x5a
[364079.396251]  [<ffffffff8110502c>] generic_writepages+0x41/0x5b
[364079.396253]  [<ffffffff811064e5>] do_writepages+0x28/0x2f
[364079.396256]  [<ffffffff81180d5a>] __writeback_single_inode+0x6e/0x1d8
[364079.396258]  [<ffffffff81181e13>] writeback_sb_inodes+0x1f1/0x329
[364079.396260]  [<ffffffff81181fbf>] __writeback_inodes_wb+0x74/0xb9
[364079.396262]  [<ffffffff811821d3>] wb_writeback+0x13b/0x280
[364079.396264]  [<ffffffff8117371e>] ? get_nr_dirty_inodes+0x4a/0x6f
[364079.396267]  [<ffffffff811825d5>] bdi_writeback_workfn+0x1a0/0x39d
[364079.396269]  [<ffffffff8105451a>] process_one_work+0x24d/0x42d
[364079.396272]  [<ffffffff8105448f>] ? process_one_work+0x1c2/0x42d
[364079.396274]  [<ffffffff8107bff9>] ? lock_release_holdtime.part.5+0xc2/0xc7
[364079.396277]  [<ffffffff81054e9a>] worker_thread+0x1d6/0x2d1
[364079.396279]  [<ffffffff81054cc4>] ? manage_workers.isra.19+0x1cb/0x1cb
[364079.396281]  [<ffffffff8105a7ae>] kthread+0xed/0xf5
[364079.396284]  [<ffffffff8105a6c1>] ? __init_kthread_worker+0x5a/0x5a
[364079.396286]  [<ffffffff814adffc>] ret_from_fork+0x7c/0xb0
[364079.396288]  [<ffffffff8105a6c1>] ? __init_kthread_worker+0x5a/0x5a
[364079.396289] 3 locks held by kworker/u16:4/9492:
[364079.396290]  #0:  (writeback){......}, at: [<ffffffff8105448f>] process_one_work+0x1c2/0x42d
[364079.396294]  #1:  ((&(&wb->dwork)->work)){......}, at: [<ffffffff8105448f>] process_one_work+0x1c2/0x42d
[364079.396298]  #2:  (&type->s_umount_key#28){......}, at: [<ffffffff8115ed12>] grab_super_passive+0x51/0x7b
[364079.396303] INFO: task checkarray:11741 blocked for more than 120 seconds.
[364079.396304]       Not tainted 3.13.0-rc5-00034-gb257bab #53
[364079.396305] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[364079.396306] checkarray      D 0000000000000005     0 11741  26934 0x00000004
[364079.396309]  ffff88076c2adc58 0000000000000002 ffff88076c2adfd8 ffff880504bd4900
[364079.396311]  00000000001d39c0 00000000001d39c0 ffff8808225e8000 ffff880504bd4900
[364079.396314]  ffffffff81947660 ffff880504bd4900 ffff88076c2adba8 ffffffff81069de7
[364079.396316] Call Trace:
[364079.396318]  [<ffffffff81069de7>] ? local_clock+0x41/0x5a
[364079.396320]  [<ffffffff81008f83>] ? native_sched_clock+0x29/0x6f
[364079.396322]  [<ffffffff81008fd2>] ? sched_clock+0x9/0xd
[364079.396324]  [<ffffffff81069d1d>] ? sched_clock_cpu+0x48/0xd1
[364079.396326]  [<ffffffff81069de7>] ? local_clock+0x41/0x5a
[364079.396329]  [<ffffffff8107d52e>] ? __lock_acquire.isra.23+0x599/0xbfb
[364079.396331]  [<ffffffff81069de7>] ? local_clock+0x41/0x5a
[364079.396332]  [<ffffffff814a1da8>] schedule+0x6f/0x71
[364079.396334]  [<ffffffff814a0ecc>] schedule_timeout+0x2c/0x1ff
[364079.396336]  [<ffffffff81069de7>] ? local_clock+0x41/0x5a
[364079.396338]  [<ffffffff8107bff9>] ? lock_release_holdtime.part.5+0xc2/0xc7
[364079.396340]  [<ffffffff814a2658>] ? wait_for_common+0x112/0x152
[364079.396342]  [<ffffffff814a2660>] wait_for_common+0x11a/0x152
[364079.396344]  [<ffffffff81067373>] ? try_to_wake_up+0x26a/0x26a
[364079.396346]  [<ffffffff814a26b5>] wait_for_completion+0x1d/0x1f
[364079.396348]  [<ffffffff8105aa89>] kthread_stop+0x61/0xb8
[364079.396354]  [<ffffffffa034482a>] md_unregister_thread+0x68/0x75 [md_mod]
[364079.396359]  [<ffffffffa0349f21>] md_reap_sync_thread+0x1b/0x13d [md_mod]
[364079.396365]  [<ffffffffa034a0e0>] action_store+0x9d/0x1df [md_mod]
[364079.396371]  [<ffffffffa0346a2b>] md_attr_store+0xbe/0xf3 [md_mod]
[364079.396373]  [<ffffffff811c1ab4>] sysfs_write_file+0x138/0x18b
[364079.396375]  [<ffffffff8115c350>] vfs_write+0xc3/0x18d
[364079.396377]  [<ffffffff8115c751>] SyS_write+0x4c/0x85
[364079.396379]  [<ffffffff814ae2d0>] tracesys+0xdd/0xe2
[364079.396380] 4 locks held by checkarray/11741:
[364079.396381]  #0:  (sb_writers#5){......}, at: [<ffffffff8115c335>] vfs_write+0xa8/0x18d
[364079.396385]  #1:  (&of->mutex){......}, at: [<ffffffff811c1a4a>] sysfs_write_file+0xce/0x18b
[364079.396388]  #2:  (s_active#369){......}, at: [<ffffffff811c1a52>] sysfs_write_file+0xd6/0x18b
[364079.396392]  #3:  (&mddev->reconfig_mutex){......}, at: [<ffffffffa0346a16>] md_attr_store+0xa9/0xf3 [md_mod]