gistfile1.txt

Aug 21 02:26:59 avalon kernel: [94687.742338] INFO: task rsync:6740 blocked for more than 120 seconds.
Aug 21 02:26:59 avalon kernel: [94687.742343] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 21 02:26:59 avalon kernel: [94687.742346] rsync         D 0000000000000002     0  6740   4892 0x00000000
Aug 21 02:26:59 avalon kernel: [94687.742352]  ffff88010e70b308 0000000000000082 0000000000015bc0 0000000000015bc0
Aug 21 02:26:59 avalon kernel: [94687.742358]  ffff880121ebb1a0 ffff88010e70bfd8 0000000000015bc0 ffff880121ebade0
Aug 21 02:26:59 avalon kernel: [94687.742362]  0000000000015bc0 ffff88010e70bfd8 0000000000015bc0 ffff880121ebb1a0
Aug 21 02:26:59 avalon kernel: [94687.742367] Call Trace:
Aug 21 02:26:59 avalon kernel: [94687.742378]  [<ffffffff810f4980>] ? sync_page+0x0/0x50
Aug 21 02:26:59 avalon kernel: [94687.742385]  [<ffffffff815420c7>] io_schedule+0x47/0x70
Aug 21 02:26:59 avalon kernel: [94687.742389]  [<ffffffff810f49bd>] sync_page+0x3d/0x50
Aug 21 02:26:59 avalon kernel: [94687.742393]  [<ffffffff8154292f>] __wait_on_bit+0x5f/0x90
Aug 21 02:26:59 avalon kernel: [94687.742397]  [<ffffffff810f4b73>] wait_on_page_bit+0x73/0x80
Aug 21 02:26:59 avalon kernel: [94687.742403]  [<ffffffff81085470>] ? wake_bit_function+0x0/0x40
Aug 21 02:26:59 avalon kernel: [94687.742408]  [<ffffffff81102576>] shrink_page_list+0x176/0x600
Aug 21 02:26:59 avalon kernel: [94687.742415]  [<ffffffff810397a9>] ? default_spin_lock_flags+0x9/0x10
Aug 21 02:26:59 avalon kernel: [94687.742419]  [<ffffffff810855c7>] ? finish_wait+0x67/0x90
Aug 21 02:26:59 avalon kernel: [94687.742424]  [<ffffffff8110eb2e>] ? congestion_wait+0x7e/0x90
Aug 21 02:26:59 avalon kernel: [94687.742428]  [<ffffffff81085430>] ? autoremove_wake_function+0x0/0x40
Aug 21 02:26:59 avalon kernel: [94687.742432]  [<ffffffff811030f8>] shrink_inactive_list+0x6f8/0x7f0
Aug 21 02:26:59 avalon kernel: [94687.742439]  [<ffffffff8105b034>] ? try_to_wake_up+0x284/0x380
Aug 21 02:26:59 avalon kernel: [94687.742444]  [<ffffffff810fcdea>] ? determine_dirtyable_memory+0x1a/0x30
Aug 21 02:26:59 avalon kernel: [94687.742448]  [<ffffffff810fce97>] ? get_dirty_limits+0x27/0x2f0
Aug 21 02:26:59 avalon kernel: [94687.742452]  [<ffffffff81103281>] shrink_list+0x91/0xf0
Aug 21 02:26:59 avalon kernel: [94687.742456]  [<ffffffff81103477>] shrink_zone+0x197/0x240
Aug 21 02:26:59 avalon kernel: [94687.742460]  [<ffffffff81103592>] shrink_zones+0x72/0x100
Aug 21 02:26:59 avalon kernel: [94687.742464]  [<ffffffff8110369e>] do_try_to_free_pages+0x7e/0x330
Aug 21 02:26:59 avalon kernel: [94687.742468]  [<ffffffff81103a4f>] try_to_free_pages+0x6f/0x80
Aug 21 02:26:59 avalon kernel: [94687.742472]  [<ffffffff81101550>] ? isolate_pages_global+0x0/0x50
Aug 21 02:26:59 avalon kernel: [94687.742476]  [<ffffffff810faa1a>] __alloc_pages_slowpath+0x27a/0x580
Aug 21 02:26:59 avalon kernel: [94687.742481]  [<ffffffff810fae7e>] __alloc_pages_nodemask+0x15e/0x1a0
Aug 21 02:26:59 avalon kernel: [94687.742487]  [<ffffffff8112dba7>] alloc_pages_current+0x87/0xd0
Aug 21 02:26:59 avalon kernel: [94687.742492]  [<ffffffff81133b17>] new_slab+0x2f7/0x310
Aug 21 02:26:59 avalon kernel: [94687.742496]  [<ffffffff81136329>] __slab_alloc+0x169/0x2d0
Aug 21 02:26:59 avalon kernel: [94687.742502]  [<ffffffff81451c24>] ? sock_alloc_send_pskb+0x1c4/0x320
Aug 21 02:26:59 avalon kernel: [94687.742507]  [<ffffffff81137408>] __kmalloc_node_track_caller+0xb8/0x180
Aug 21 02:26:59 avalon kernel: [94687.742511]  [<ffffffff81451c24>] ? sock_alloc_send_pskb+0x1c4/0x320
Aug 21 02:26:59 avalon kernel: [94687.742515]  [<ffffffff81455c6f>] ? __alloc_skb+0x4f/0x190
Aug 21 02:26:59 avalon kernel: [94687.742521]  [<ffffffff81155c10>] ? pollwake+0x0/0x60
Aug 21 02:26:59 avalon kernel: [94687.742524]  [<ffffffff81455ca0>] __alloc_skb+0x80/0x190
Aug 21 02:26:59 avalon kernel: [94687.742529]  [<ffffffff81451c24>] sock_alloc_send_pskb+0x1c4/0x320
Aug 21 02:26:59 avalon kernel: [94687.742534]  [<ffffffff8105353e>] ? __wake_up_sync_key+0x5e/0x80
Aug 21 02:26:59 avalon kernel: [94687.742538]  [<ffffffff81451d95>] sock_alloc_send_skb+0x15/0x20
Aug 21 02:26:59 avalon kernel: [94687.742543]  [<ffffffff814e50f5>] unix_stream_sendmsg+0x275/0x3e0
Aug 21 02:26:59 avalon kernel: [94687.742547]  [<ffffffff810f5ad0>] ? T.804+0x2c0/0x410
Aug 21 02:26:59 avalon kernel: [94687.742551]  [<ffffffff814502fc>] sock_aio_write+0x13c/0x150
Aug 21 02:26:59 avalon kernel: [94687.742556]  [<ffffffff8114392a>] do_sync_write+0xfa/0x140
Aug 21 02:26:59 avalon kernel: [94687.742560]  [<ffffffff81085430>] ? autoremove_wake_function+0x0/0x40
Aug 21 02:26:59 avalon kernel: [94687.742565]  [<ffffffff8108fbc9>] ? ktime_get_ts+0xa9/0xe0
Aug 21 02:26:59 avalon kernel: [94687.742569]  [<ffffffff81154dd8>] ? poll_select_copy_remaining+0xf8/0x150
Aug 21 02:26:59 avalon kernel: [94687.742575]  [<ffffffff81253076>] ? security_file_permission+0x16/0x20
Aug 21 02:26:59 avalon kernel: [94687.742580]  [<ffffffff81143cf4>] vfs_write+0x184/0x1a0
Aug 21 02:26:59 avalon kernel: [94687.742584]  [<ffffffff811444c1>] sys_write+0x51/0x80
Aug 21 02:26:59 avalon kernel: [94687.742590]  [<ffffffff810131b2>] system_call_fastpath+0x16/0x1b
Aug 21 02:28:59 avalon kernel: [94807.447911] INFO: task zvol/0:4930 blocked for more than 120 seconds.
Aug 21 02:28:59 avalon kernel: [94807.447916] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 21 02:28:59 avalon kernel: [94807.447920] zvol/0        D 00000000ffffffff     0  4930      2 0x00000000
Aug 21 02:28:59 avalon kernel: [94807.447927]  ffff880136703c40 0000000000000046 0000000000015bc0 0000000000015bc0
Aug 21 02:28:59 avalon kernel: [94807.447934]  ffff880113a483c0 ffff880136703fd8 0000000000015bc0 ffff880113a48000
Aug 21 02:28:59 avalon kernel: [94807.447940]  0000000000015bc0 ffff880136703fd8 0000000000015bc0 ffff880113a483c0
Aug 21 02:28:59 avalon kernel: [94807.447946] Call Trace:
Aug 21 02:28:59 avalon kernel: [94807.447967]  [<ffffffffa0d31e97>] cv_wait_common+0x97/0xd0 [spl]
Aug 21 02:28:59 avalon kernel: [94807.447976]  [<ffffffff81085430>] ? autoremove_wake_function+0x0/0x40
Aug 21 02:28:59 avalon kernel: [94807.447989]  [<ffffffffa0d31f03>] __cv_wait+0x13/0x20 [spl]
Aug 21 02:28:59 avalon kernel: [94807.448029]  [<ffffffffa0e3955b>] dmu_tx_wait+0x8b/0xf0 [zfs]
Aug 21 02:28:59 avalon kernel: [94807.448065]  [<ffffffffa0e3967a>] dmu_tx_assign+0x6a/0x410 [zfs]
Aug 21 02:28:59 avalon kernel: [94807.448101]  [<ffffffffa0e43c97>] ? dsl_dataset_block_freeable+0x27/0x80 [zfs]
Aug 21 02:28:59 avalon kernel: [94807.448143]  [<ffffffffa0e9c4cd>] zvol_write+0x7d/0x3c0 [zfs]
Aug 21 02:28:59 avalon kernel: [94807.448156]  [<ffffffffa0d2ec76>] taskq_thread+0x1b6/0x320 [spl]
Aug 21 02:28:59 avalon kernel: [94807.448163]  [<ffffffff8105b130>] ? default_wake_function+0x0/0x20
Aug 21 02:28:59 avalon kernel: [94807.448174]  [<ffffffffa0d2eac0>] ? taskq_thread+0x0/0x320 [spl]
Aug 21 02:28:59 avalon kernel: [94807.448179]  [<ffffffff810850b6>] kthread+0x96/0xa0
Aug 21 02:28:59 avalon kernel: [94807.448185]  [<ffffffff810141ea>] child_rip+0xa/0x20
Aug 21 02:28:59 avalon kernel: [94807.448190]  [<ffffffff81085020>] ? kthread+0x0/0xa0
Aug 21 02:28:59 avalon kernel: [94807.448194]  [<ffffffff810141e0>] ? child_rip+0x0/0x20
Aug 21 02:28:59 avalon kernel: [94807.448216] INFO: task txg_quiesce:5220 blocked for more than 120 seconds.
Aug 21 02:28:59 avalon kernel: [94807.448219] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 21 02:28:59 avalon kernel: [94807.448222] txg_quiesce   D 00000000ffffffff     0  5220      2 0x00000000
Aug 21 02:28:59 avalon kernel: [94807.448227]  ffff8800a3c41d90 0000000000000046 0000000000015bc0 0000000000015bc0
Aug 21 02:28:59 avalon kernel: [94807.448233]  ffff880120ec4890 ffff8800a3c41fd8 0000000000015bc0 ffff880120ec44d0
Aug 21 02:28:59 avalon kernel: [94807.448239]  0000000000015bc0 ffff8800a3c41fd8 0000000000015bc0 ffff880120ec4890
Aug 21 02:28:59 avalon kernel: [94807.448244] Call Trace:
Aug 21 02:28:59 avalon kernel: [94807.448257]  [<ffffffffa0d31e97>] cv_wait_common+0x97/0xd0 [spl]
Aug 21 02:28:59 avalon kernel: [94807.448262]  [<ffffffff81085430>] ? autoremove_wake_function+0x0/0x40
Aug 21 02:28:59 avalon kernel: [94807.448275]  [<ffffffffa0d31f03>] __cv_wait+0x13/0x20 [spl]
Aug 21 02:28:59 avalon kernel: [94807.448319]  [<ffffffffa0e71c5b>] txg_quiesce_thread+0x1bb/0x260 [zfs]
Aug 21 02:28:59 avalon kernel: [94807.448364]  [<ffffffffa0e71aa0>] ? txg_quiesce_thread+0x0/0x260 [zfs]
Aug 21 02:28:59 avalon kernel: [94807.448376]  [<ffffffffa0d2e208>] thread_generic_wrapper+0x68/0x80 [spl]
Aug 21 02:28:59 avalon kernel: [94807.448387]  [<ffffffffa0d2e1a0>] ? thread_generic_wrapper+0x0/0x80 [spl]
Aug 21 02:28:59 avalon kernel: [94807.448392]  [<ffffffff810850b6>] kthread+0x96/0xa0
Aug 21 02:28:59 avalon kernel: [94807.448396]  [<ffffffff810141ea>] child_rip+0xa/0x20
Aug 21 02:28:59 avalon kernel: [94807.448401]  [<ffffffff81085020>] ? kthread+0x0/0xa0
Aug 21 02:28:59 avalon kernel: [94807.448406]  [<ffffffff810141e0>] ? child_rip+0x0/0x20
Aug 21 02:28:59 avalon kernel: [94807.448409] INFO: task txg_sync:5221 blocked for more than 120 seconds.
Aug 21 02:28:59 avalon kernel: [94807.448412] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 21 02:28:59 avalon kernel: [94807.448415] txg_sync      D 0000000000000002     0  5221      2 0x00000000
Aug 21 02:28:59 avalon kernel: [94807.448420]  ffff8800a3c43d70 0000000000000046 0000000000015bc0 0000000000015bc0
Aug 21 02:28:59 avalon kernel: [94807.448426]  ffff8801355c4890 ffff8800a3c43fd8 0000000000015bc0 ffff8801355c44d0
Aug 21 02:28:59 avalon kernel: [94807.448431]  0000000000015bc0 ffff8800a3c43fd8 0000000000015bc0 ffff8801355c4890
Aug 21 02:28:59 avalon kernel: [94807.448437] Call Trace:
Aug 21 02:28:59 avalon kernel: [94807.448449]  [<ffffffffa0d31e97>] cv_wait_common+0x97/0xd0 [spl]
Aug 21 02:28:59 avalon kernel: [94807.448454]  [<ffffffff81085430>] ? autoremove_wake_function+0x0/0x40
Aug 21 02:28:59 avalon kernel: [94807.448466]  [<ffffffffa0d31f03>] __cv_wait+0x13/0x20 [spl]
Aug 21 02:28:59 avalon kernel: [94807.448510]  [<ffffffffa0e715cd>] txg_thread_wait+0x1d/0x40 [zfs]
Aug 21 02:28:59 avalon kernel: [94807.448553]  [<ffffffffa0e718b0>] txg_sync_thread+0x1b0/0x3a0 [zfs]
Aug 21 02:28:59 avalon kernel: [94807.448598]  [<ffffffffa0e71700>] ? txg_sync_thread+0x0/0x3a0 [zfs]
Aug 21 02:28:59 avalon kernel: [94807.448609]  [<ffffffffa0d2e208>] thread_generic_wrapper+0x68/0x80 [spl]
Aug 21 02:28:59 avalon kernel: [94807.448620]  [<ffffffffa0d2e1a0>] ? thread_generic_wrapper+0x0/0x80 [spl]
Aug 21 02:28:59 avalon kernel: [94807.448625]  [<ffffffff810850b6>] kthread+0x96/0xa0
Aug 21 02:28:59 avalon kernel: [94807.448630]  [<ffffffff810141ea>] child_rip+0xa/0x20
Aug 21 02:28:59 avalon kernel: [94807.448635]  [<ffffffff81085020>] ? kthread+0x0/0xa0
Aug 21 02:28:59 avalon kernel: [94807.448639]  [<ffffffff810141e0>] ? child_rip+0x0/0x20
Aug 21 02:28:59 avalon kernel: [94807.448661] INFO: task jbd2/md9001-8:6653 blocked for more than 120 seconds.
Aug 21 02:28:59 avalon kernel: [94807.448664] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 21 02:28:59 avalon kernel: [94807.448667] jbd2/md9001-8 D ffff880133a78000     0  6653      2 0x00000000
Aug 21 02:28:59 avalon kernel: [94807.448672]  ffff880136f6bab0 0000000000000046 0000000000015bc0 0000000000015bc0
Aug 21 02:28:59 avalon kernel: [94807.448678]  ffff880133a783c0 ffff880136f6bfd8 0000000000015bc0 ffff880133a78000
Aug 21 02:28:59 avalon kernel: [94807.448683]  0000000000015bc0 ffff880136f6bfd8 0000000000015bc0 ffff880133a783c0
Aug 21 02:28:59 avalon kernel: [94807.448689] Call Trace:
Aug 21 02:28:59 avalon kernel: [94807.448696]  [<ffffffff815420c7>] io_schedule+0x47/0x70
Aug 21 02:28:59 avalon kernel: [94807.448704]  [<ffffffff812a0b3e>] get_request_wait+0xce/0x1a0
Aug 21 02:28:59 avalon kernel: [94807.448709]  [<ffffffff81085430>] ? autoremove_wake_function+0x0/0x40
Aug 21 02:28:59 avalon kernel: [94807.448716]  [<ffffffff81295dab>] ? elv_merge+0x1cb/0x200
Aug 21 02:28:59 avalon kernel: [94807.448721]  [<ffffffff812a1225>] __make_request+0x95/0x4a0
Aug 21 02:28:59 avalon kernel: [94807.448726]  [<ffffffff8129f981>] generic_make_request+0x1b1/0x4f0
Aug 21 02:28:59 avalon kernel: [94807.448734]  [<ffffffff810f6b65>] ? mempool_alloc_slab+0x15/0x20
Aug 21 02:28:59 avalon kernel: [94807.448741]  [<ffffffff8121b06e>] ? jbd2_journal_file_buffer+0x5e/0x90
Aug 21 02:28:59 avalon kernel: [94807.448747]  [<ffffffff8129fd40>] submit_bio+0x80/0x110
Aug 21 02:28:59 avalon kernel: [94807.448754]  [<ffffffff8116c999>] submit_bh+0xf9/0x140
Aug 21 02:28:59 avalon kernel: [94807.448759]  [<ffffffff8121ca80>] jbd2_journal_commit_transaction+0x4d0/0x12b0
Aug 21 02:28:59 avalon kernel: [94807.448765]  [<ffffffff81085430>] ? autoremove_wake_function+0x0/0x40
Aug 21 02:28:59 avalon kernel: [94807.448771]  [<ffffffff81223e1d>] kjournald2+0xbd/0x220
Aug 21 02:28:59 avalon kernel: [94807.448776]  [<ffffffff81085430>] ? autoremove_wake_function+0x0/0x40
Aug 21 02:28:59 avalon kernel: [94807.448781]  [<ffffffff81223d60>] ? kjournald2+0x0/0x220
Aug 21 02:28:59 avalon kernel: [94807.448786]  [<ffffffff810850b6>] kthread+0x96/0xa0
Aug 21 02:28:59 avalon kernel: [94807.448790]  [<ffffffff810141ea>] child_rip+0xa/0x20
Aug 21 02:28:59 avalon kernel: [94807.448795]  [<ffffffff81085020>] ? kthread+0x0/0xa0
Aug 21 02:28:59 avalon kernel: [94807.448799]  [<ffffffff810141e0>] ? child_rip+0x0/0x20
Aug 21 02:28:59 avalon kernel: [94807.448804] INFO: task rsync:6740 blocked for more than 120 seconds.
Aug 21 02:28:59 avalon kernel: [94807.448806] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 21 02:28:59 avalon kernel: [94807.448809] rsync         D 0000000000000002     0  6740   4892 0x00000000
Aug 21 02:28:59 avalon kernel: [94807.448815]  ffff88010e70b308 0000000000000082 0000000000015bc0 0000000000015bc0
Aug 21 02:28:59 avalon kernel: [94807.448820]  ffff880121ebb1a0 ffff88010e70bfd8 0000000000015bc0 ffff880121ebade0
Aug 21 02:28:59 avalon kernel: [94807.448826]  0000000000015bc0 ffff88010e70bfd8 0000000000015bc0 ffff880121ebb1a0
Aug 21 02:28:59 avalon kernel: [94807.448831] Call Trace:
Aug 21 02:28:59 avalon kernel: [94807.448836]  [<ffffffff810f4980>] ? sync_page+0x0/0x50
Aug 21 02:28:59 avalon kernel: [94807.448841]  [<ffffffff815420c7>] io_schedule+0x47/0x70
Aug 21 02:28:59 avalon kernel: [94807.448845]  [<ffffffff810f49bd>] sync_page+0x3d/0x50
Aug 21 02:28:59 avalon kernel: [94807.448850]  [<ffffffff8154292f>] __wait_on_bit+0x5f/0x90
Aug 21 02:28:59 avalon kernel: [94807.448855]  [<ffffffff810f4b73>] wait_on_page_bit+0x73/0x80
Aug 21 02:28:59 avalon kernel: [94807.448860]  [<ffffffff81085470>] ? wake_bit_function+0x0/0x40
Aug 21 02:28:59 avalon kernel: [94807.448866]  [<ffffffff81102576>] shrink_page_list+0x176/0x600
Aug 21 02:28:59 avalon kernel: [94807.448873]  [<ffffffff810397a9>] ? default_spin_lock_flags+0x9/0x10
Aug 21 02:28:59 avalon kernel: [94807.448878]  [<ffffffff810855c7>] ? finish_wait+0x67/0x90
Aug 21 02:28:59 avalon kernel: [94807.448884]  [<ffffffff8110eb2e>] ? congestion_wait+0x7e/0x90
Aug 21 02:28:59 avalon kernel: [94807.448889]  [<ffffffff81085430>] ? autoremove_wake_function+0x0/0x40
Aug 21 02:28:59 avalon kernel: [94807.448894]  [<ffffffff811030f8>] shrink_inactive_list+0x6f8/0x7f0
Aug 21 02:28:59 avalon kernel: [94807.448900]  [<ffffffff8105b034>] ? try_to_wake_up+0x284/0x380
Aug 21 02:28:59 avalon kernel: [94807.448907]  [<ffffffff810fcdea>] ? determine_dirtyable_memory+0x1a/0x30
Aug 21 02:28:59 avalon kernel: [94807.448912]  [<ffffffff810fce97>] ? get_dirty_limits+0x27/0x2f0
Aug 21 02:28:59 avalon kernel: [94807.448917]  [<ffffffff81103281>] shrink_list+0x91/0xf0
Aug 21 02:28:59 avalon kernel: [94807.448922]  [<ffffffff81103477>] shrink_zone+0x197/0x240
Aug 21 02:28:59 avalon kernel: [94807.448926]  [<ffffffff81103592>] shrink_zones+0x72/0x100
Aug 21 02:28:59 avalon kernel: [94807.448931]  [<ffffffff8110369e>] do_try_to_free_pages+0x7e/0x330
Aug 21 02:28:59 avalon kernel: [94807.448936]  [<ffffffff81103a4f>] try_to_free_pages+0x6f/0x80
Aug 21 02:28:59 avalon kernel: [94807.448941]  [<ffffffff81101550>] ? isolate_pages_global+0x0/0x50
Aug 21 02:28:59 avalon kernel: [94807.448946]  [<ffffffff810faa1a>] __alloc_pages_slowpath+0x27a/0x580
Aug 21 02:28:59 avalon kernel: [94807.448952]  [<ffffffff810fae7e>] __alloc_pages_nodemask+0x15e/0x1a0
Aug 21 02:28:59 avalon kernel: [94807.448960]  [<ffffffff8112dba7>] alloc_pages_current+0x87/0xd0
Aug 21 02:28:59 avalon kernel: [94807.448965]  [<ffffffff81133b17>] new_slab+0x2f7/0x310
Aug 21 02:28:59 avalon kernel: [94807.448970]  [<ffffffff81136329>] __slab_alloc+0x169/0x2d0
Aug 21 02:28:59 avalon kernel: [94807.448977]  [<ffffffff81451c24>] ? sock_alloc_send_pskb+0x1c4/0x320
Aug 21 02:28:59 avalon kernel: [94807.448983]  [<ffffffff81137408>] __kmalloc_node_track_caller+0xb8/0x180
Aug 21 02:28:59 avalon kernel: [94807.448989]  [<ffffffff81451c24>] ? sock_alloc_send_pskb+0x1c4/0x320
Aug 21 02:28:59 avalon kernel: [94807.448994]  [<ffffffff81455c6f>] ? __alloc_skb+0x4f/0x190
Aug 21 02:28:59 avalon kernel: [94807.449000]  [<ffffffff81155c10>] ? pollwake+0x0/0x60
Aug 21 02:28:59 avalon kernel: [94807.449004]  [<ffffffff81455ca0>] __alloc_skb+0x80/0x190
Aug 21 02:28:59 avalon kernel: [94807.449010]  [<ffffffff81451c24>] sock_alloc_send_pskb+0x1c4/0x320
Aug 21 02:28:59 avalon kernel: [94807.449015]  [<ffffffff8105353e>] ? __wake_up_sync_key+0x5e/0x80
Aug 21 02:28:59 avalon kernel: [94807.449021]  [<ffffffff81451d95>] sock_alloc_send_skb+0x15/0x20
Aug 21 02:28:59 avalon kernel: [94807.449027]  [<ffffffff814e50f5>] unix_stream_sendmsg+0x275/0x3e0
Aug 21 02:28:59 avalon kernel: [94807.449032]  [<ffffffff810f5ad0>] ? T.804+0x2c0/0x410
Aug 21 02:28:59 avalon kernel: [94807.449037]  [<ffffffff814502fc>] sock_aio_write+0x13c/0x150
Aug 21 02:28:59 avalon kernel: [94807.449044]  [<ffffffff8114392a>] do_sync_write+0xfa/0x140
Aug 21 02:28:59 avalon kernel: [94807.449050]  [<ffffffff81085430>] ? autoremove_wake_function+0x0/0x40
Aug 21 02:28:59 avalon kernel: [94807.449056]  [<ffffffff8108fbc9>] ? ktime_get_ts+0xa9/0xe0
Aug 21 02:28:59 avalon kernel: [94807.449062]  [<ffffffff81154dd8>] ? poll_select_copy_remaining+0xf8/0x150
Aug 21 02:28:59 avalon kernel: [94807.449068]  [<ffffffff81253076>] ? security_file_permission+0x16/0x20
Aug 21 02:28:59 avalon kernel: [94807.449074]  [<ffffffff81143cf4>] vfs_write+0x184/0x1a0
Aug 21 02:28:59 avalon kernel: [94807.449078]  [<ffffffff811444c1>] sys_write+0x51/0x80
Aug 21 02:28:59 avalon kernel: [94807.449085]  [<ffffffff810131b2>] system_call_fastpath+0x16/0x1b
Aug 21 02:30:59 avalon kernel: [94927.153572] INFO: task dbus-daemon:1021 blocked for more than 120 seconds.
Aug 21 02:30:59 avalon kernel: [94927.153577] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 21 02:30:59 avalon kernel: [94927.153581] dbus-daemon   D 00000000ffffffff     0  1021      1 0x00000000
Aug 21 02:30:59 avalon kernel: [94927.153586]  ffff8801129cf258 0000000000000086 0000000000015bc0 0000000000015bc0
Aug 21 02:30:59 avalon kernel: [94927.153592]  ffff8801343d9ab0 ffff8801129cffd8 0000000000015bc0 ffff8801343d96f0
Aug 21 02:30:59 avalon kernel: [94927.153596]  0000000000015bc0 ffff8801129cffd8 0000000000015bc0 ffff8801343d9ab0
Aug 21 02:30:59 avalon kernel: [94927.153601] Call Trace:
Aug 21 02:30:59 avalon kernel: [94927.153613]  [<ffffffff810f4980>] ? sync_page+0x0/0x50
Aug 21 02:30:59 avalon kernel: [94927.153619]  [<ffffffff815420c7>] io_schedule+0x47/0x70
Aug 21 02:30:59 avalon kernel: [94927.153623]  [<ffffffff810f49bd>] sync_page+0x3d/0x50
Aug 21 02:30:59 avalon kernel: [94927.153627]  [<ffffffff8154292f>] __wait_on_bit+0x5f/0x90
Aug 21 02:30:59 avalon kernel: [94927.153631]  [<ffffffff810f4b73>] wait_on_page_bit+0x73/0x80
Aug 21 02:30:59 avalon kernel: [94927.153637]  [<ffffffff81085470>] ? wake_bit_function+0x0/0x40
Aug 21 02:30:59 avalon kernel: [94927.153642]  [<ffffffff815425bd>] ? schedule_timeout+0x19d/0x300
Aug 21 02:30:59 avalon kernel: [94927.153647]  [<ffffffff81102576>] shrink_page_list+0x176/0x600
Aug 21 02:30:59 avalon kernel: [94927.153654]  [<ffffffff810397a9>] ? default_spin_lock_flags+0x9/0x10
Aug 21 02:30:59 avalon kernel: [94927.153658]  [<ffffffff810855c7>] ? finish_wait+0x67/0x90
Aug 21 02:30:59 avalon kernel: [94927.153663]  [<ffffffff8110eb2e>] ? congestion_wait+0x7e/0x90
Aug 21 02:30:59 avalon kernel: [94927.153667]  [<ffffffff81085430>] ? autoremove_wake_function+0x0/0x40
Aug 21 02:30:59 avalon kernel: [94927.153672]  [<ffffffff811030f8>] shrink_inactive_list+0x6f8/0x7f0
Aug 21 02:30:59 avalon kernel: [94927.153679]  [<ffffffff8105b034>] ? try_to_wake_up+0x284/0x380
Aug 21 02:30:59 avalon kernel: [94927.153685]  [<ffffffff8104c874>] ? scale_rt_power+0x24/0x70
Aug 21 02:30:59 avalon kernel: [94927.153690]  [<ffffffff8105b142>] ? default_wake_function+0x12/0x20
Aug 21 02:30:59 avalon kernel: [94927.153694]  [<ffffffff81103281>] shrink_list+0x91/0xf0
Aug 21 02:30:59 avalon kernel: [94927.153697]  [<ffffffff81103477>] shrink_zone+0x197/0x240
Aug 21 02:30:59 avalon kernel: [94927.153702]  [<ffffffff81103592>] shrink_zones+0x72/0x100
Aug 21 02:30:59 avalon kernel: [94927.153706]  [<ffffffff8110369e>] do_try_to_free_pages+0x7e/0x330
Aug 21 02:30:59 avalon kernel: [94927.153710]  [<ffffffff81103a4f>] try_to_free_pages+0x6f/0x80
Aug 21 02:30:59 avalon kernel: [94927.153713]  [<ffffffff81101550>] ? isolate_pages_global+0x0/0x50
Aug 21 02:30:59 avalon kernel: [94927.153718]  [<ffffffff810faa1a>] __alloc_pages_slowpath+0x27a/0x580
Aug 21 02:30:59 avalon kernel: [94927.153723]  [<ffffffff810fae7e>] __alloc_pages_nodemask+0x15e/0x1a0
Aug 21 02:30:59 avalon kernel: [94927.153728]  [<ffffffff8112dba7>] alloc_pages_current+0x87/0xd0
Aug 21 02:30:59 avalon kernel: [94927.153733]  [<ffffffff81133a68>] new_slab+0x248/0x310
Aug 21 02:30:59 avalon kernel: [94927.153737]  [<ffffffff81136329>] __slab_alloc+0x169/0x2d0
Aug 21 02:30:59 avalon kernel: [94927.153742]  [<ffffffff81451c24>] ? sock_alloc_send_pskb+0x1c4/0x320
Aug 21 02:30:59 avalon kernel: [94927.153746]  [<ffffffff81137408>] __kmalloc_node_track_caller+0xb8/0x180
Aug 21 02:30:59 avalon kernel: [94927.153751]  [<ffffffff81451c24>] ? sock_alloc_send_pskb+0x1c4/0x320
Aug 21 02:30:59 avalon kernel: [94927.153755]  [<ffffffff81455c6f>] ? __alloc_skb+0x4f/0x190
Aug 21 02:30:59 avalon kernel: [94927.153758]  [<ffffffff81455ca0>] __alloc_skb+0x80/0x190
Aug 21 02:30:59 avalon kernel: [94927.153763]  [<ffffffff81451c24>] sock_alloc_send_pskb+0x1c4/0x320
Aug 21 02:30:59 avalon kernel: [94927.153767]  [<ffffffff8105353e>] ? __wake_up_sync_key+0x5e/0x80
Aug 21 02:30:59 avalon kernel: [94927.153772]  [<ffffffff81451d95>] sock_alloc_send_skb+0x15/0x20
Aug 21 02:30:59 avalon kernel: [94927.153777]  [<ffffffff814e50f5>] unix_stream_sendmsg+0x275/0x3e0
Aug 21 02:30:59 avalon kernel: [94927.153781]  [<ffffffff814502fc>] sock_aio_write+0x13c/0x150
Aug 21 02:30:59 avalon kernel: [94927.153786]  [<ffffffff814501c0>] ? sock_aio_write+0x0/0x150
Aug 21 02:30:59 avalon kernel: [94927.153792]  [<ffffffff811437eb>] do_sync_readv_writev+0xfb/0x140
Aug 21 02:30:59 avalon kernel: [94927.153796]  [<ffffffff81085430>] ? autoremove_wake_function+0x0/0x40
Aug 21 02:30:59 avalon kernel: [94927.153803]  [<ffffffff81253076>] ? security_file_permission+0x16/0x20
Aug 21 02:30:59 avalon kernel: [94927.153808]  [<ffffffff8114488f>] do_readv_writev+0xcf/0x1f0
Aug 21 02:30:59 avalon kernel: [94927.153813]  [<ffffffff81253076>] ? security_file_permission+0x16/0x20
Aug 21 02:30:59 avalon kernel: [94927.153817]  [<ffffffff811449f8>] vfs_writev+0x48/0x60
Aug 21 02:30:59 avalon kernel: [94927.153821]  [<ffffffff81144b21>] sys_writev+0x51/0xb0
Aug 21 02:30:59 avalon kernel: [94927.153826]  [<ffffffff8115607c>] ? sys_poll+0x7c/0x110
Aug 21 02:30:59 avalon kernel: [94927.153832]  [<ffffffff810131b2>] system_call_fastpath+0x16/0x1b
Aug 21 02:30:59 avalon kernel: [94927.153861] INFO: task zvol/0:4930 blocked for more than 120 seconds.
Aug 21 02:30:59 avalon kernel: [94927.153863] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 21 02:30:59 avalon kernel: [94927.153866] zvol/0        D 00000000ffffffff     0  4930      2 0x00000000
Aug 21 02:30:59 avalon kernel: [94927.153870]  ffff880136703c40 0000000000000046 0000000000015bc0 0000000000015bc0
Aug 21 02:30:59 avalon kernel: [94927.153875]  ffff880113a483c0 ffff880136703fd8 0000000000015bc0 ffff880113a48000
Aug 21 02:30:59 avalon kernel: [94927.153879]  0000000000015bc0 ffff880136703fd8 0000000000015bc0 ffff880113a483c0
Aug 21 02:30:59 avalon kernel: [94927.153884] Call Trace:
Aug 21 02:30:59 avalon kernel: [94927.153899]  [<ffffffffa0d31e97>] cv_wait_common+0x97/0xd0 [spl]
Aug 21 02:30:59 avalon kernel: [94927.153904]  [<ffffffff81085430>] ? autoremove_wake_function+0x0/0x40
Aug 21 02:30:59 avalon kernel: [94927.153914]  [<ffffffffa0d31f03>] __cv_wait+0x13/0x20 [spl]
Aug 21 02:30:59 avalon kernel: [94927.153947]  [<ffffffffa0e3955b>] dmu_tx_wait+0x8b/0xf0 [zfs]
Aug 21 02:30:59 avalon kernel: [94927.153977]  [<ffffffffa0e3967a>] dmu_tx_assign+0x6a/0x410 [zfs]
Aug 21 02:30:59 avalon kernel: [94927.154006]  [<ffffffffa0e43c97>] ? dsl_dataset_block_freeable+0x27/0x80 [zfs]
Aug 21 02:30:59 avalon kernel: [94927.154041]  [<ffffffffa0e9c4cd>] zvol_write+0x7d/0x3c0 [zfs]
Aug 21 02:30:59 avalon kernel: [94927.154051]  [<ffffffffa0d2ec76>] taskq_thread+0x1b6/0x320 [spl]
Aug 21 02:30:59 avalon kernel: [94927.154056]  [<ffffffff8105b130>] ? default_wake_function+0x0/0x20
Aug 21 02:30:59 avalon kernel: [94927.154065]  [<ffffffffa0d2eac0>] ? taskq_thread+0x0/0x320 [spl]
Aug 21 02:30:59 avalon kernel: [94927.154069]  [<ffffffff810850b6>] kthread+0x96/0xa0
Aug 21 02:30:59 avalon kernel: [94927.154074]  [<ffffffff810141ea>] child_rip+0xa/0x20
Aug 21 02:30:59 avalon kernel: [94927.154078]  [<ffffffff81085020>] ? kthread+0x0/0xa0
Aug 21 02:30:59 avalon kernel: [94927.154081]  [<ffffffff810141e0>] ? child_rip+0x0/0x20
Aug 21 02:30:59 avalon kernel: [94927.154101] INFO: task txg_quiesce:5220 blocked for more than 120 seconds.
Aug 21 02:30:59 avalon kernel: [94927.154103] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 21 02:30:59 avalon kernel: [94927.154106] txg_quiesce   D 00000000ffffffff     0  5220      2 0x00000000
Aug 21 02:30:59 avalon kernel: [94927.154110]  ffff8800a3c41d90 0000000000000046 0000000000015bc0 0000000000015bc0
Aug 21 02:30:59 avalon kernel: [94927.154115]  ffff880120ec4890 ffff8800a3c41fd8 0000000000015bc0 ffff880120ec44d0
Aug 21 02:30:59 avalon kernel: [94927.154119]  0000000000015bc0 ffff8800a3c41fd8 0000000000015bc0 ffff880120ec4890
Aug 21 02:30:59 avalon kernel: [94927.154124] Call Trace:
Aug 21 02:30:59 avalon kernel: [94927.154134]  [<ffffffffa0d31e97>] cv_wait_common+0x97/0xd0 [spl]
Aug 21 02:30:59 avalon kernel: [94927.154138]  [<ffffffff81085430>] ? autoremove_wake_function+0x0/0x40
Aug 21 02:30:59 avalon kernel: [94927.154148]  [<ffffffffa0d31f03>] __cv_wait+0x13/0x20 [spl]
Aug 21 02:30:59 avalon kernel: [94927.154184]  [<ffffffffa0e71c5b>] txg_quiesce_thread+0x1bb/0x260 [zfs]
Aug 21 02:30:59 avalon kernel: [94927.154221]  [<ffffffffa0e71aa0>] ? txg_quiesce_thread+0x0/0x260 [zfs]
Aug 21 02:30:59 avalon kernel: [94927.154231]  [<ffffffffa0d2e208>] thread_generic_wrapper+0x68/0x80 [spl]
Aug 21 02:30:59 avalon kernel: [94927.154240]  [<ffffffffa0d2e1a0>] ? thread_generic_wrapper+0x0/0x80 [spl]
Aug 21 02:30:59 avalon kernel: [94927.154244]  [<ffffffff810850b6>] kthread+0x96/0xa0
Aug 21 02:30:59 avalon kernel: [94927.154248]  [<ffffffff810141ea>] child_rip+0xa/0x20
Aug 21 02:30:59 avalon kernel: [94927.154252]  [<ffffffff81085020>] ? kthread+0x0/0xa0
Aug 21 02:30:59 avalon kernel: [94927.154255]  [<ffffffff810141e0>] ? child_rip+0x0/0x20
Aug 21 02:30:59 avalon kernel: [94927.154258] INFO: task txg_sync:5221 blocked for more than 120 seconds.
Aug 21 02:30:59 avalon kernel: [94927.154260] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 21 02:30:59 avalon kernel: [94927.154263] txg_sync      D 0000000000000002     0  5221      2 0x00000000
Aug 21 02:30:59 avalon kernel: [94927.154267]  ffff8800a3c43d70 0000000000000046 0000000000015bc0 0000000000015bc0
Aug 21 02:30:59 avalon kernel: [94927.154271]  ffff8801355c4890 ffff8800a3c43fd8 0000000000015bc0 ffff8801355c44d0
Aug 21 02:30:59 avalon kernel: [94927.154276]  0000000000015bc0 ffff8800a3c43fd8 0000000000015bc0 ffff8801355c4890
Aug 21 02:30:59 avalon kernel: [94927.154280] Call Trace:
Aug 21 02:30:59 avalon kernel: [94927.154290]  [<ffffffffa0d31e97>] cv_wait_common+0x97/0xd0 [spl]
Aug 21 02:30:59 avalon kernel: [94927.154295]  [<ffffffff81085430>] ? autoremove_wake_function+0x0/0x40
Aug 21 02:30:59 avalon kernel: [94927.154305]  [<ffffffffa0d31f03>] __cv_wait+0x13/0x20 [spl]
Aug 21 02:30:59 avalon kernel: [94927.154341]  [<ffffffffa0e715cd>] txg_thread_wait+0x1d/0x40 [zfs]
Aug 21 02:30:59 avalon kernel: [94927.154376]  [<ffffffffa0e718b0>] txg_sync_thread+0x1b0/0x3a0 [zfs]
Aug 21 02:30:59 avalon kernel: [94927.154412]  [<ffffffffa0e71700>] ? txg_sync_thread+0x0/0x3a0 [zfs]
Aug 21 02:30:59 avalon kernel: [94927.154422]  [<ffffffffa0d2e208>] thread_generic_wrapper+0x68/0x80 [spl]
Aug 21 02:30:59 avalon kernel: [94927.154431]  [<ffffffffa0d2e1a0>] ? thread_generic_wrapper+0x0/0x80 [spl]
Aug 21 02:30:59 avalon kernel: [94927.154435]  [<ffffffff810850b6>] kthread+0x96/0xa0
Aug 21 02:30:59 avalon kernel: [94927.154439]  [<ffffffff810141ea>] child_rip+0xa/0x20
Aug 21 02:30:59 avalon kernel: [94927.154443]  [<ffffffff81085020>] ? kthread+0x0/0xa0
Aug 21 02:30:59 avalon kernel: [94927.154447]  [<ffffffff810141e0>] ? child_rip+0x0/0x20