zfs-0.6.4-1_deadlock

Apr 27 11:07:14 backupmaster kernel: INFO: task kswapd0:46 blocked for more than 120 seconds.
Apr 27 11:07:14 backupmaster kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 27 11:07:14 backupmaster kernel: kswapd0         D ffff88021fc13680     0    46      2 0x00000000
Apr 27 11:07:14 backupmaster kernel: ffff880210a73bb0 0000000000000046 ffff880210a73fd8 0000000000013680
Apr 27 11:07:14 backupmaster kernel: ffff880210a73fd8 0000000000013680 ffff880213d64440 ffff8800cf24a498
Apr 27 11:07:14 backupmaster kernel: ffff8800cf24a49c ffff880213d64440 00000000ffffffff ffff8800cf24a4a0
Apr 27 11:07:14 backupmaster kernel: Call Trace:
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8160b029>] schedule_preempt_disabled+0x29/0x70
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81608d55>] __mutex_lock_slowpath+0xc5/0x1c0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff816081bf>] mutex_lock+0x1f/0x2f
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa016ebdd>] zpl_nr_cached_objects+0x2d/0x80 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811ca423>] prune_super+0x63/0x1a0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811690eb>] shrink_slab+0xab/0x300
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811c0751>] ? vmpressure+0x21/0x90
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8116cdf1>] balance_pgdat+0x4b1/0x5e0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8116d093>] kswapd+0x173/0x450
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81098340>] ? wake_up_bit+0x30/0x30
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8116cf20>] ? balance_pgdat+0x5e0/0x5e0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8109738f>] kthread+0xcf/0xe0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff810972c0>] ? kthread_create_on_node+0x140/0x140
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8161497c>] ret_from_fork+0x7c/0xb0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff810972c0>] ? kthread_create_on_node+0x140/0x140
Apr 27 11:07:14 backupmaster kernel: INFO: task txg_sync:1151 blocked for more than 120 seconds.
Apr 27 11:07:14 backupmaster kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 27 11:07:14 backupmaster kernel: txg_sync        D ffff88021fc13680     0  1151      2 0x00000000
Apr 27 11:07:14 backupmaster kernel: ffff88020a5f6df0 0000000000000046 ffff88020a5f7fd8 0000000000013680
Apr 27 11:07:14 backupmaster kernel: ffff88020a5f7fd8 0000000000013680 ffff880035fe38e0 ffff8800cf24a498
Apr 27 11:07:14 backupmaster kernel: ffff8800cf24a49c ffff880035fe38e0 00000000ffffffff ffff8800cf24a4a0
Apr 27 11:07:14 backupmaster kernel: Call Trace:
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8160b029>] schedule_preempt_disabled+0x29/0x70
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81608d55>] __mutex_lock_slowpath+0xc5/0x1c0
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa050a072>] ? xfs_perag_get_tag+0x42/0xe0 [xfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffff816081bf>] mutex_lock+0x1f/0x2f
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa016ebdd>] zpl_nr_cached_objects+0x2d/0x80 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811ca423>] prune_super+0x63/0x1a0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811690eb>] shrink_slab+0xab/0x300
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811c0751>] ? vmpressure+0x21/0x90
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8116c2f2>] do_try_to_free_pages+0x3c2/0x4e0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8116c50c>] try_to_free_pages+0xfc/0x180
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81160bad>] __alloc_pages_nodemask+0x7fd/0xb90
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8119f4e9>] alloc_pages_current+0xa9/0x170
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811a9595>] new_slab+0x275/0x300
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81601a3b>] __slab_alloc+0x315/0x48f
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa000bd2a>] ? spl_kmem_cache_alloc+0xaa/0x180 [spl]
Apr 27 11:07:14 backupmaster kernel: [<ffffffff810a9622>] ? default_wake_function+0x12/0x20
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa000bd2a>] ? spl_kmem_cache_alloc+0xaa/0x180 [spl]
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811abbd3>] kmem_cache_alloc+0x193/0x1d0
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa000bd2a>] ? spl_kmem_cache_alloc+0xaa/0x180 [spl]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa000bd2a>] spl_kmem_cache_alloc+0xaa/0x180 [spl]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa0163233>] zio_buf_alloc+0x23/0x30 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00b8355>] arc_get_data_buf.isra.22+0x345/0x4a0 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00bad2f>] arc_buf_alloc+0x10f/0x140 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00bb504>] arc_read+0x1a4/0xa90 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00d7603>] traverse_prefetch_metadata+0xd3/0xe0 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00d7686>] prefetch_dnode_metadata+0x76/0xe0 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00d80e5>] traverse_visitbp+0x665/0x870 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00d7da2>] traverse_visitbp+0x322/0x870 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00d7da2>] traverse_visitbp+0x322/0x870 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00d7da2>] traverse_visitbp+0x322/0x870 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00d7da2>] traverse_visitbp+0x322/0x870 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00d7da2>] traverse_visitbp+0x322/0x870 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00d7da2>] traverse_visitbp+0x322/0x870 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00d8a51>] traverse_dnode+0x81/0xf0 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00d8207>] traverse_visitbp+0x787/0x870 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa000aad7>] ? spl_kmem_alloc+0xe7/0x1b0 [spl]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00d8471>] traverse_impl+0x181/0x3c0 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00c6b20>] ? dbuf_stats_destroy+0x60/0x60 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00d8af6>] traverse_dataset_destroyed+0x36/0x40 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00c6b20>] ? dbuf_stats_destroy+0x60/0x60 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00c729d>] bptree_iterate+0x1cd/0x350 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00f5680>] ? dsl_scan_prefetch.isra.4+0xf0/0xf0 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00f7ebe>] dsl_scan_sync+0x6ae/0xb80 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa0166dda>] ? zio_wait+0x16a/0x210 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa010adcd>] spa_sync+0x4dd/0xbc0 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8109836b>] ? autoremove_wake_function+0x2b/0x40
Apr 27 11:07:14 backupmaster kernel: [<ffffffff810a2484>] ? __wake_up+0x44/0x50
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa011ccee>] txg_sync_thread+0x37e/0x610 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa011c970>] ? txg_fini+0x2a0/0x2a0 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa000ccc1>] thread_generic_wrapper+0x71/0x80 [spl]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa000cc50>] ? __thread_exit+0x20/0x20 [spl]
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8109738f>] kthread+0xcf/0xe0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff810972c0>] ? kthread_create_on_node+0x140/0x140
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8161497c>] ret_from_fork+0x7c/0xb0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff810972c0>] ? kthread_create_on_node+0x140/0x140
Apr 27 11:07:14 backupmaster kernel: INFO: task irqbalance:1235 blocked for more than 120 seconds.
Apr 27 11:07:14 backupmaster kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 27 11:07:14 backupmaster kernel: irqbalance      D ffff88021fd13680     0  1235      1 0x00000080
Apr 27 11:07:14 backupmaster kernel: ffff8800d1a37790 0000000000000086 ffff8800d1a37fd8 0000000000013680
Apr 27 11:07:14 backupmaster kernel: ffff8800d1a37fd8 0000000000013680 ffff880035fa8000 ffff8800cf24a498
Apr 27 11:07:14 backupmaster kernel: ffff8800cf24a49c ffff880035fa8000 00000000ffffffff ffff8800cf24a4a0
Apr 27 11:07:14 backupmaster kernel: Call Trace:
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8160b029>] schedule_preempt_disabled+0x29/0x70
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81608d55>] __mutex_lock_slowpath+0xc5/0x1c0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff816081bf>] mutex_lock+0x1f/0x2f
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa016ebdd>] zpl_nr_cached_objects+0x2d/0x80 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811ca423>] prune_super+0x63/0x1a0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811690eb>] shrink_slab+0xab/0x300
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811c0751>] ? vmpressure+0x21/0x90
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8116c2f2>] do_try_to_free_pages+0x3c2/0x4e0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8116c50c>] try_to_free_pages+0xfc/0x180
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81160bad>] __alloc_pages_nodemask+0x7fd/0xb90
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8119f4e9>] alloc_pages_current+0xa9/0x170
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8115b53e>] __get_free_pages+0xe/0x50
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81272835>] inode_doinit_with_dentry+0x395/0x680
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8127371c>] selinux_d_instantiate+0x1c/0x20
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81268f7b>] security_d_instantiate+0x1b/0x30
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811de480>] d_instantiate+0x50/0x70
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81233b8f>] proc_lookup_de+0xaf/0xe0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81233bdb>] proc_lookup+0x1b/0x20
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8122f27c>] proc_root_lookup+0x1c/0x40
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811d0b4d>] lookup_real+0x1d/0x50
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811d64c3>] do_last+0xb23/0x1270
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811abdde>] ? kmem_cache_alloc_trace+0x1ce/0x1f0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811d6cd2>] path_openat+0xc2/0x4c0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81012639>] ? __switch_to+0x179/0x4a0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811d810b>] do_filp_open+0x4b/0xb0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811e4b67>] ? __alloc_fd+0xa7/0x130
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811c5f33>] do_sys_open+0xf3/0x1f0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811c604e>] SyS_open+0x1e/0x20
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81614a29>] system_call_fastpath+0x16/0x1b
Apr 27 11:07:14 backupmaster kernel: INFO: task burp:1276 blocked for more than 120 seconds.
Apr 27 11:07:14 backupmaster kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 27 11:07:14 backupmaster kernel: burp            D ffff88021fd93680     0  1276      1 0x00000080
Apr 27 11:07:14 backupmaster kernel: ffff8800d1b07a70 0000000000000086 ffff8800d1b07fd8 0000000000013680
Apr 27 11:07:14 backupmaster kernel: ffff8800d1b07fd8 0000000000013680 ffff88020f338b60 ffff8800cf24a498
Apr 27 11:07:14 backupmaster kernel: ffff8800cf24a49c ffff88020f338b60 00000000ffffffff ffff8800cf24a4a0
Apr 27 11:07:14 backupmaster kernel: Call Trace:
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8160b029>] schedule_preempt_disabled+0x29/0x70
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81608d55>] __mutex_lock_slowpath+0xc5/0x1c0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff816081bf>] mutex_lock+0x1f/0x2f
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa016ebdd>] zpl_nr_cached_objects+0x2d/0x80 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811ca423>] prune_super+0x63/0x1a0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811690eb>] shrink_slab+0xab/0x300
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811c0751>] ? vmpressure+0x21/0x90
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8116c2f2>] do_try_to_free_pages+0x3c2/0x4e0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8116c50c>] try_to_free_pages+0xfc/0x180
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81160bad>] __alloc_pages_nodemask+0x7fd/0xb90
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811c8b00>] ? delayed_fput+0x140/0x140
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8106bf84>] copy_process.part.25+0x134/0x14e0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811c8bd0>] ? get_empty_filp+0x90/0x190
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8106d4ec>] do_fork+0xbc/0x360
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811e4c77>] ? __fd_install+0x47/0x60
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8106d816>] SyS_clone+0x16/0x20
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81614d79>] stub_clone+0x69/0x90
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81614a29>] ? system_call_fastpath+0x16/0x1b
Apr 27 11:07:14 backupmaster kernel: INFO: task ntpd:1310 blocked for more than 120 seconds.
Apr 27 11:07:14 backupmaster kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 27 11:07:14 backupmaster kernel: ntpd            D ffff88021fc13680     0  1310      1 0x00000080
Apr 27 11:07:14 backupmaster kernel: ffff8800cf02bd40 0000000000000086 ffff8800cf02bfd8 0000000000013680
Apr 27 11:07:14 backupmaster kernel: ffff8800cf02bfd8 0000000000013680 ffff8802096f2d80 ffffffff819e2100
Apr 27 11:07:14 backupmaster kernel: ffffffff819e2104 ffff8802096f2d80 00000000ffffffff ffffffff819e2108
Apr 27 11:07:14 backupmaster kernel: Call Trace:
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8160b029>] schedule_preempt_disabled+0x29/0x70
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81608d55>] __mutex_lock_slowpath+0xc5/0x1c0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff816081bf>] mutex_lock+0x1f/0x2f
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8150a545>] rtnl_lock+0x15/0x20
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81511cb5>] dev_ioctl+0x355/0x560
Apr 27 11:07:14 backupmaster kernel: [<ffffffff812847ee>] ? security_netlbl_sid_to_secattr+0x6e/0xb0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff815e9d52>] ? netlbl_domhsh_hash+0x12/0x50
Apr 27 11:07:14 backupmaster kernel: [<ffffffff815e9dbe>] ? netlbl_domhsh_search+0x2e/0xb0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811abdde>] ? kmem_cache_alloc_trace+0x1ce/0x1f0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff814e0eb5>] sock_do_ioctl+0x45/0x50
Apr 27 11:07:14 backupmaster kernel: [<ffffffff814e15b0>] sock_ioctl+0x1f0/0x2c0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811da125>] do_vfs_ioctl+0x2e5/0x4c0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8126f7ae>] ? file_has_perm+0xae/0xc0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811da3a1>] SyS_ioctl+0xa1/0xc0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81614a29>] system_call_fastpath+0x16/0x1b
Apr 27 11:07:14 backupmaster kernel: INFO: task NetworkManager:1366 blocked for more than 120 seconds.
Apr 27 11:07:14 backupmaster kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 27 11:07:14 backupmaster kernel: NetworkManager  D ffff88021fc93680     0  1366      1 0x00000080
Apr 27 11:07:14 backupmaster kernel: ffff880204dc3500 0000000000000086 ffff880204dc3fd8 0000000000013680
Apr 27 11:07:14 backupmaster kernel: ffff880204dc3fd8 0000000000013680 ffff880213c54440 ffff8800cf24a498
Apr 27 11:07:14 backupmaster kernel: ffff8800cf24a49c ffff880213c54440 00000000ffffffff ffff8800cf24a4a0
Apr 27 11:07:14 backupmaster kernel: Call Trace:
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8160b029>] schedule_preempt_disabled+0x29/0x70
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81608d55>] __mutex_lock_slowpath+0xc5/0x1c0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff816081bf>] mutex_lock+0x1f/0x2f
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa016ebdd>] zpl_nr_cached_objects+0x2d/0x80 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811ca423>] prune_super+0x63/0x1a0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811690eb>] shrink_slab+0xab/0x300
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811c0751>] ? vmpressure+0x21/0x90
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8116c2f2>] do_try_to_free_pages+0x3c2/0x4e0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8116c50c>] try_to_free_pages+0xfc/0x180
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81160bad>] __alloc_pages_nodemask+0x7fd/0xb90
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81600d2d>] kmalloc_large_node+0x60/0x8d
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811ae97a>] __kmalloc_node_track_caller+0x20a/0x270
Apr 27 11:07:14 backupmaster kernel: [<ffffffff810b2246>] ? dequeue_entity+0x106/0x510
Apr 27 11:07:14 backupmaster kernel: [<ffffffff814ed3ed>] ? __alloc_skb+0x8d/0x2d0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff814ec6d1>] __kmalloc_reserve.isra.27+0x31/0x90
Apr 27 11:07:14 backupmaster kernel: [<ffffffff814ed3bd>] ? __alloc_skb+0x5d/0x2d0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff814ed3ed>] __alloc_skb+0x8d/0x2d0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8152741f>] ? netlink_lookup+0x3f/0x60
Apr 27 11:07:14 backupmaster kernel: [<ffffffff815286fb>] netlink_alloc_skb+0x6b/0x1e0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81528933>] netlink_dump+0xc3/0x2d0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff815292bb>] __netlink_dump_start+0x1ab/0x230
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8150af50>] ? rtnl_bridge_notify+0x150/0x150
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8150b3c0>] rtnetlink_rcv_msg+0x160/0x270
Apr 27 11:07:14 backupmaster kernel: [<ffffffff812e9836>] ? rhashtable_lookup_compare+0x36/0x70
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8150af50>] ? rtnl_bridge_notify+0x150/0x150
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8150b260>] ? rtnetlink_rcv+0x30/0x30
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8152b1d9>] netlink_rcv_skb+0xa9/0xc0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8150b258>] rtnetlink_rcv+0x28/0x30
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8152a7ed>] netlink_unicast+0xed/0x1b0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8152abe0>] netlink_sendmsg+0x330/0x770
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8126c3f5>] ? sock_has_perm+0x75/0x90
Apr 27 11:07:14 backupmaster kernel: [<ffffffff814e3d50>] sock_sendmsg+0xb0/0xf0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff814e441f>] ? sock_recvmsg+0xbf/0x100
Apr 27 11:07:14 backupmaster kernel: [<ffffffff810d2258>] ? get_futex_key+0x1c8/0x2b0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff814e4189>] ___sys_sendmsg+0x3a9/0x3c0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811dad00>] ? poll_select_copy_remaining+0x150/0x150
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8126f7ae>] ? file_has_perm+0xae/0xc0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff814e5071>] __sys_sendmsg+0x51/0x90
Apr 27 11:07:14 backupmaster kernel: [<ffffffff814e50c2>] SyS_sendmsg+0x12/0x20
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81614a29>] system_call_fastpath+0x16/0x1b
Apr 27 11:07:14 backupmaster kernel: INFO: task nfsd:2044 blocked for more than 120 seconds.
Apr 27 11:07:14 backupmaster kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 27 11:07:14 backupmaster kernel: nfsd            D ffff88021fc13680     0  2044      2 0x00000080
Apr 27 11:07:14 backupmaster kernel: ffff8802036d79d0 0000000000000046 ffff8802036d7fd8 0000000000013680
Apr 27 11:07:14 backupmaster kernel: ffff8802036d7fd8 0000000000013680 ffff8800cf110000 ffff8800cf24a498
Apr 27 11:07:14 backupmaster kernel: ffff8800cf24a49c ffff8800cf110000 00000000ffffffff ffff8800cf24a4a0
Apr 27 11:07:14 backupmaster kernel: Call Trace:
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8160b029>] schedule_preempt_disabled+0x29/0x70
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81608d55>] __mutex_lock_slowpath+0xc5/0x1c0
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa050a072>] ? xfs_perag_get_tag+0x42/0xe0 [xfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffff816081bf>] mutex_lock+0x1f/0x2f
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa016ebdd>] zpl_nr_cached_objects+0x2d/0x80 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811ca423>] prune_super+0x63/0x1a0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811690eb>] shrink_slab+0xab/0x300
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811c0751>] ? vmpressure+0x21/0x90
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8116c2f2>] do_try_to_free_pages+0x3c2/0x4e0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8116c50c>] try_to_free_pages+0xfc/0x180
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81160bad>] __alloc_pages_nodemask+0x7fd/0xb90
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8119f4e9>] alloc_pages_current+0xa9/0x170
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa05dc6cc>] svc_recv+0xdc/0xa80 [sunrpc]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa05dd4e4>] ? svc_send+0xb4/0xe0 [sunrpc]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa0606795>] nfsd+0xd5/0x150 [nfsd]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa06066c0>] ? nfsd_destroy+0x80/0x80 [nfsd]
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8109738f>] kthread+0xcf/0xe0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff810972c0>] ? kthread_create_on_node+0x140/0x140
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8161497c>] ret_from_fork+0x7c/0xb0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff810972c0>] ? kthread_create_on_node+0x140/0x140
Apr 27 11:07:14 backupmaster kernel: INFO: task smbd:2075 blocked for more than 120 seconds.
Apr 27 11:07:14 backupmaster kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 27 11:07:14 backupmaster kernel: smbd            D ffff88021fc13680     0  2075      1 0x00000080
Apr 27 11:07:14 backupmaster kernel: ffff8800cd657a70 0000000000000082 ffff8800cd657fd8 0000000000013680
Apr 27 11:07:14 backupmaster kernel: ffff8800cd657fd8 0000000000013680 ffff88020f33db00 ffff8800cf24a498
Apr 27 11:07:14 backupmaster kernel: ffff8800cf24a49c ffff88020f33db00 00000000ffffffff ffff8800cf24a4a0
Apr 27 11:07:14 backupmaster kernel: Call Trace:
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8160b029>] schedule_preempt_disabled+0x29/0x70
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81608d55>] __mutex_lock_slowpath+0xc5/0x1c0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff816081bf>] mutex_lock+0x1f/0x2f
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa016ebdd>] zpl_nr_cached_objects+0x2d/0x80 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811ca423>] prune_super+0x63/0x1a0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811690eb>] shrink_slab+0xab/0x300
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811c0751>] ? vmpressure+0x21/0x90
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8116c2f2>] do_try_to_free_pages+0x3c2/0x4e0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8116c50c>] try_to_free_pages+0xfc/0x180
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81160bad>] __alloc_pages_nodemask+0x7fd/0xb90
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8106bf84>] copy_process.part.25+0x134/0x14e0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811c8bd0>] ? get_empty_filp+0x90/0x190
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8106d4ec>] do_fork+0xbc/0x360
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811e4c77>] ? __fd_install+0x47/0x60
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8106d816>] SyS_clone+0x16/0x20
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81614d79>] stub_clone+0x69/0x90
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81614a29>] ? system_call_fastpath+0x16/0x1b
Apr 27 11:07:14 backupmaster kernel: INFO: task mysqld:3404 blocked for more than 120 seconds.
Apr 27 11:07:14 backupmaster kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 27 11:07:14 backupmaster kernel: mysqld          D ffff88021fc93680     0  3404   2831 0x00000080
Apr 27 11:07:14 backupmaster kernel: ffff8800cd79b290 0000000000000086 ffff8800cd79bfd8 0000000000013680
Apr 27 11:07:14 backupmaster kernel: ffff8800cd79bfd8 0000000000013680 ffff88020c948b60 ffff8800cf24a498
Apr 27 11:07:14 backupmaster kernel: ffff8800cf24a49c ffff88020c948b60 00000000ffffffff ffff8800cf24a4a0
Apr 27 11:07:14 backupmaster kernel: Call Trace:
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8160b029>] schedule_preempt_disabled+0x29/0x70
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81608d55>] __mutex_lock_slowpath+0xc5/0x1c0
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa050a072>] ? xfs_perag_get_tag+0x42/0xe0 [xfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffff816081bf>] mutex_lock+0x1f/0x2f
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa016ebdd>] zpl_nr_cached_objects+0x2d/0x80 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811ca423>] prune_super+0x63/0x1a0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811690eb>] shrink_slab+0xab/0x300
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811c0751>] ? vmpressure+0x21/0x90
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8116c2f2>] do_try_to_free_pages+0x3c2/0x4e0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8116c50c>] try_to_free_pages+0xfc/0x180
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81160bad>] __alloc_pages_nodemask+0x7fd/0xb90
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8119f4e9>] alloc_pages_current+0xa9/0x170
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811a9595>] new_slab+0x275/0x300
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81601a3b>] __slab_alloc+0x315/0x48f
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa000bd2a>] ? spl_kmem_cache_alloc+0xaa/0x180 [spl]
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811abbd3>] kmem_cache_alloc+0x193/0x1d0
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa000bd2a>] ? spl_kmem_cache_alloc+0xaa/0x180 [spl]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa000bd2a>] spl_kmem_cache_alloc+0xaa/0x180 [spl]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa0163233>] zio_buf_alloc+0x23/0x30 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00b8355>] arc_get_data_buf.isra.22+0x345/0x4a0 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00bad2f>] arc_buf_alloc+0x10f/0x140 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00bb504>] arc_read+0x1a4/0xa90 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00c22e0>] ? dbuf_rele_and_unlock+0x440/0x440 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00c336d>] dbuf_read+0x2cd/0xa10 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00c3f4a>] __dbuf_hold_impl+0x49a/0x520 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00c4043>] dbuf_hold_impl+0x73/0xa0 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00c43ce>] dbuf_hold_level+0x2e/0x60 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00d9b4f>] dmu_tx_check_ioerr+0x3f/0x130 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00d9e86>] dmu_tx_count_write+0x246/0x710 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00c2436>] ? dbuf_rele+0x36/0x40 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811ae07d>] ? __kmalloc_node+0x13d/0x280
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa000ac80>] ? spl_kmem_zalloc+0xe0/0x1a0 [spl]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa000ac80>] ? spl_kmem_zalloc+0xe0/0x1a0 [spl]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa000ac80>] ? spl_kmem_zalloc+0xe0/0x1a0 [spl]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa00da386>] dmu_tx_hold_write+0x36/0x50 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa01564dc>] zfs_write+0x3dc/0xc40 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffff810a691d>] ? ttwu_do_activate.constprop.85+0x5d/0x70
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa016d8bd>] zpl_write+0xbd/0x130 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811c6c5d>] vfs_write+0xbd/0x1e0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811c7852>] SyS_pwrite64+0x92/0xc0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81614a29>] system_call_fastpath+0x16/0x1b
Apr 27 11:07:14 backupmaster kernel: INFO: task qemu-kvm:3385 blocked for more than 120 seconds.
Apr 27 11:07:14 backupmaster kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 27 11:07:14 backupmaster kernel: qemu-kvm        D ffff88021fd93680     0  3385      1 0x00000080
Apr 27 11:07:14 backupmaster kernel: ffff88020236fa70 0000000000000082 ffff88020236ffd8 0000000000013680
Apr 27 11:07:14 backupmaster kernel: ffff88020236ffd8 0000000000013680 ffff880211970b60 ffff8800cf24a498
Apr 27 11:07:14 backupmaster kernel: ffff8800cf24a49c ffff880211970b60 00000000ffffffff ffff8800cf24a4a0
Apr 27 11:07:14 backupmaster kernel: Call Trace:
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8160b029>] schedule_preempt_disabled+0x29/0x70
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81608d55>] __mutex_lock_slowpath+0xc5/0x1c0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff816081bf>] mutex_lock+0x1f/0x2f
Apr 27 11:07:14 backupmaster kernel: [<ffffffffa016ebdd>] zpl_nr_cached_objects+0x2d/0x80 [zfs]
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811ca423>] prune_super+0x63/0x1a0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811690eb>] shrink_slab+0xab/0x300
Apr 27 11:07:14 backupmaster kernel: [<ffffffff811c0751>] ? vmpressure+0x21/0x90
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8116c2f2>] do_try_to_free_pages+0x3c2/0x4e0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8116c50c>] try_to_free_pages+0xfc/0x180
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81160bad>] __alloc_pages_nodemask+0x7fd/0xb90
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8106bf84>] copy_process.part.25+0x134/0x14e0
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8106d4ec>] do_fork+0xbc/0x360
Apr 27 11:07:14 backupmaster kernel: [<ffffffff8106d816>] SyS_clone+0x16/0x20
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81614d79>] stub_clone+0x69/0x90
Apr 27 11:07:14 backupmaster kernel: [<ffffffff81614a29>] ? system_call_fastpath+0x16/0x1b