Skip to content

Instantly share code, notes, and snippets.

@dnozay

dnozay/arcstats.txt

Created February 19, 2014 01:57
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save dnozay/9084691 to your computer and use it in GitHub Desktop.
Save dnozay/9084691 to your computer and use it in GitHub Desktop.
Download ZIP
stuck in zio_wait
Raw
arcstats.txt
** this is 5min BEFORE the hang **
+ cat /proc/spl/kstat/zfs/arcstats
+ tee /zfs-data/jenkins/workspace/syslog.jenkins-slave2/arcstats.txt
4 1 0x01 84 4032 14259512326 11568069705331
name type data
hits 4 85134938
misses 4 697247
demand_data_hits 4 1358021
demand_data_misses 4 83814
demand_metadata_hits 4 30831813
demand_metadata_misses 4 504572
prefetch_data_hits 4 353
prefetch_data_misses 4 10382
prefetch_metadata_hits 4 52944751
prefetch_metadata_misses 4 98479
mru_hits 4 360101
mru_ghost_hits 4 419804
mfu_hits 4 31836570
mfu_ghost_hits 4 159683
deleted 4 399192
recycle_miss 4 333117
mutex_miss 4 230
evict_skip 4 46674058
evict_l2_cached 4 0
evict_l2_eligible 4 14913974272
evict_l2_ineligible 4 1578272768
hash_elements 4 183612
hash_elements_max 4 183643
hash_collisions 4 1102424
hash_chains 4 50369
hash_chain_max 4 15
p 4 734253762
c 4 1383037472
c_min 4 251159808
c_max 4 2009278464
size 4 1382790808
hdr_size 4 53231472
data_size 4 1087153152
other_size 4 242406184
anon_size 4 8786432
anon_evict_data 4 0
anon_evict_metadata 4 0
mru_size 4 444825088
mru_evict_data 4 288960000
mru_evict_metadata 4 50774016
mru_ghost_size 4 391913472
mru_ghost_evict_data 4 131395072
mru_ghost_evict_metadata 4 260518400
mfu_size 4 633541632
mfu_evict_data 4 584250368
mfu_evict_metadata 4 19092480
mfu_ghost_size 4 122526208
mfu_ghost_evict_data 4 45796352
mfu_ghost_evict_metadata 4 76729856
l2_hits 4 0
l2_misses 4 0
l2_feeds 4 0
l2_rw_clash 4 0
l2_read_bytes 4 0
l2_write_bytes 4 0
l2_writes_sent 4 0
l2_writes_done 4 0
l2_writes_error 4 0
l2_writes_hdr_miss 4 0
l2_evict_lock_retry 4 0
l2_evict_reading 4 0
l2_free_on_write 4 0
l2_abort_lowmem 4 0
l2_cksum_bad 4 0
l2_io_error 4 0
l2_size 4 0
l2_asize 4 0
l2_hdr_size 4 0
l2_compress_successes 4 0
l2_compress_zeros 4 0
l2_compress_failures 4 0
memory_throttle_count 4 0
duplicate_buffers 4 0
duplicate_buffers_size 4 0
duplicate_reads 4 0
memory_direct_count 4 6729
memory_indirect_count 4 12298
arc_no_grow 4 0
arc_tempreserve 4 0
arc_loaned_bytes 4 0
arc_prune 4 88
arc_meta_used 4 502282904
arc_meta_limit 4 502319616
arc_meta_max 4 574538120
Raw
messages.txt
Feb 18 14:45:35 jenkins-slave2 kernel: INFO: task kswapd0:59 blocked for more than 120 seconds.
Feb 18 14:45:35 jenkins-slave2 kernel: Tainted: P --------------- 2.6.32-431.3.1.el6.x86_64 #1
Feb 18 14:45:35 jenkins-slave2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 18 14:45:35 jenkins-slave2 kernel: kswapd0 D 0000000000000002 0 59 2 0x00000000
Feb 18 14:45:35 jenkins-slave2 kernel: ffff88013975f6f0 0000000000000046 ffff880100000000 ffffffffa02faa40
Feb 18 14:45:35 jenkins-slave2 kernel: ffff8801052bd4d0 000000021d053378 0000000000000004 00000000a03344e3
Feb 18 14:45:35 jenkins-slave2 kernel: ffff8801396eb058 ffff88013975ffd8 000000000000fbc8 ffff8801396eb058
Feb 18 14:45:35 jenkins-slave2 kernel: Call Trace:
Feb 18 14:45:35 jenkins-slave2 kernel: [<ffffffffa02faa40>] ? vdev_mirror_child_done+0x0/0x30 [zfs]
Feb 18 14:45:35 jenkins-slave2 kernel: [<ffffffff810a70b1>] ? ktime_get_ts+0xb1/0xf0
Feb 18 14:45:35 jenkins-slave2 kernel: [<ffffffff815280b3>] io_schedule+0x73/0xc0
Feb 18 14:45:35 jenkins-slave2 kernel: [<ffffffffa01cd41c>] cv_wait_common+0xac/0x1c0 [spl]
Feb 18 14:45:35 jenkins-slave2 kernel: [<ffffffff8109b2b0>] ? autoremove_wake_function+0x0/0x40
Feb 18 14:45:35 jenkins-slave2 kernel: [<ffffffff81528c5b>] ? mutex_unlock+0x1b/0x20
Feb 18 14:45:35 jenkins-slave2 kernel: [<ffffffffa01cd548>] __cv_wait_io+0x18/0x20 [spl]
Feb 18 14:45:35 jenkins-slave2 kernel: [<ffffffffa0335c3b>] zio_wait+0xfb/0x1b0 [zfs]
Feb 18 14:45:35 jenkins-slave2 kernel: [<ffffffffa029c58d>] dbuf_read+0x3fd/0x750 [zfs]
Feb 18 14:45:35 jenkins-slave2 kernel: [<ffffffffa02a3fc8>] dmu_buf_hold+0x108/0x1d0 [zfs]
Feb 18 14:45:35 jenkins-slave2 kernel: [<ffffffffa030060f>] zap_get_leaf_byblk+0x4f/0x2b0 [zfs]
Feb 18 14:45:35 jenkins-slave2 kernel: [<ffffffffa03008da>] zap_deref_leaf+0x6a/0x80 [zfs]
Feb 18 14:45:35 jenkins-slave2 kernel: [<ffffffffa03010c7>] fzap_remove+0x37/0xb0 [zfs]
Feb 18 14:45:35 jenkins-slave2 kernel: [<ffffffffa030437d>] ? zap_name_alloc+0x8d/0xf0 [zfs]
Feb 18 14:45:35 jenkins-slave2 kernel: [<ffffffffa0306113>] zap_remove_norm+0x153/0x1d0 [zfs]
Feb 18 14:45:35 jenkins-slave2 kernel: [<ffffffffa03061a3>] zap_remove+0x13/0x20 [zfs]
Feb 18 14:45:35 jenkins-slave2 kernel: [<ffffffffa02ff551>] zap_remove_int+0x61/0x90 [zfs]
Feb 18 14:45:35 jenkins-slave2 kernel: [<ffffffffa030efa7>] zfs_rmnode+0x1c7/0x340 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa032c765>] zfs_zinactive+0xa5/0x110 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa032b75f>] zfs_inactive+0x7f/0x220 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa033e11e>] zpl_clear_inode+0xe/0x10 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff811a5bec>] clear_inode+0xac/0x140
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff811a5cc0>] dispose_list+0x40/0x120
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff811a6014>] shrink_icache_memory+0x274/0x2e0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81138aea>] shrink_slab+0x12a/0x1a0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8113bf2a>] balance_pgdat+0x59a/0x820
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8113c2e4>] kswapd+0x134/0x3b0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109b2b0>] ? autoremove_wake_function+0x0/0x40
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8113c1b0>] ? kswapd+0x0/0x3b0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109af06>] kthread+0x96/0xa0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109ae70>] ? kthread+0x0/0xa0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: INFO: task spl_kmem_cache/:706 blocked for more than 120 seconds.
Feb 18 14:45:36 jenkins-slave2 kernel: Tainted: P --------------- 2.6.32-431.3.1.el6.x86_64 #1
Feb 18 14:45:36 jenkins-slave2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 18 14:45:36 jenkins-slave2 kernel: spl_kmem_cach D 0000000000000000 0 706 2 0x00000000
Feb 18 14:45:36 jenkins-slave2 kernel: ffff88013c9992b0 0000000000000046 0000000000000000 ffffffffa02faa40
Feb 18 14:45:36 jenkins-slave2 kernel: ffff8800bad5d6d0 00000002627e8358 0000000000000004 00000000a03344e3
Feb 18 14:45:36 jenkins-slave2 kernel: ffff8801385bdab8 ffff88013c999fd8 000000000000fbc8 ffff8801385bdab8
Feb 18 14:45:36 jenkins-slave2 kernel: Call Trace:
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa02faa40>] ? vdev_mirror_child_done+0x0/0x30 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff815280b3>] io_schedule+0x73/0xc0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01cd41c>] cv_wait_common+0xac/0x1c0 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109b2b0>] ? autoremove_wake_function+0x0/0x40
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01cd548>] __cv_wait_io+0x18/0x20 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa0335c3b>] zio_wait+0xfb/0x1b0 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa029c58d>] dbuf_read+0x3fd/0x750 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa02a3fc8>] dmu_buf_hold+0x108/0x1d0 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa030060f>] zap_get_leaf_byblk+0x4f/0x2b0 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa03008da>] zap_deref_leaf+0x6a/0x80 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa03010c7>] fzap_remove+0x37/0xb0 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa030437d>] ? zap_name_alloc+0x8d/0xf0 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa0306113>] zap_remove_norm+0x153/0x1d0 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa03061a3>] zap_remove+0x13/0x20 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa02ff551>] zap_remove_int+0x61/0x90 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa030efa7>] zfs_rmnode+0x1c7/0x340 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa032c765>] zfs_zinactive+0xa5/0x110 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa032b75f>] zfs_inactive+0x7f/0x220 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa033e11e>] zpl_clear_inode+0xe/0x10 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff811a5bec>] clear_inode+0xac/0x140
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff811a5cc0>] dispose_list+0x40/0x120
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff811a6014>] shrink_icache_memory+0x274/0x2e0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81138aea>] shrink_slab+0x12a/0x1a0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8113aec7>] do_try_to_free_pages+0x3f7/0x610
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8112d2fc>] ? get_page_from_freelist+0x15c/0x870
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8113b2b2>] try_to_free_pages+0x92/0x120
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff811411c0>] ? next_zone+0x30/0x40
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8112f718>] __alloc_pages_nodemask+0x478/0x8d0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81167aaa>] alloc_pages_current+0xaa/0x110
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8112cf4e>] __get_free_pages+0xe/0x50
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8104ec85>] pte_alloc_one_kernel+0x15/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8114634b>] __pte_alloc_kernel+0x1b/0xc0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81157579>] vmap_page_range_noflush+0x309/0x370
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81157612>] map_vm_area+0x32/0x50
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81159080>] __vmalloc_area_node+0x100/0x190
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01c2ee9>] ? kv_alloc+0x59/0x60 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81158f0d>] __vmalloc_node+0xad/0x120
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01c2ee9>] ? kv_alloc+0x59/0x60 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff811592f2>] __vmalloc+0x22/0x30
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01c2ee9>] kv_alloc+0x59/0x60 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01c2f29>] spl_cache_grow_work+0x39/0x410 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81058d53>] ? __wake_up+0x53/0x70
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01c6628>] taskq_thread+0x218/0x4b0 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81527920>] ? thread_return+0x4e/0x76e
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01c6410>] ? taskq_thread+0x0/0x4b0 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109af06>] kthread+0x96/0xa0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109ae70>] ? kthread+0x0/0xa0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: INFO: task arc_adapt:740 blocked for more than 120 seconds.
Feb 18 14:45:36 jenkins-slave2 kernel: Tainted: P --------------- 2.6.32-431.3.1.el6.x86_64 #1
Feb 18 14:45:36 jenkins-slave2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 18 14:45:36 jenkins-slave2 kernel: arc_adapt D 0000000000000001 0 740 2 0x00000000
Feb 18 14:45:36 jenkins-slave2 kernel: ffff88013a8f1880 0000000000000046 0000000000000000 ffffffffa02faa40
Feb 18 14:45:36 jenkins-slave2 kernel: ffff8800b861f250 000000024c721d98 0000000000000004 00000000a03344e3
Feb 18 14:45:36 jenkins-slave2 kernel: ffff88013c138638 ffff88013a8f1fd8 000000000000fbc8 ffff88013c138638
Feb 18 14:45:36 jenkins-slave2 kernel: Call Trace:
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa02faa40>] ? vdev_mirror_child_done+0x0/0x30 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff815280b3>] io_schedule+0x73/0xc0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01cd41c>] cv_wait_common+0xac/0x1c0 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109b2b0>] ? autoremove_wake_function+0x0/0x40
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01cd548>] __cv_wait_io+0x18/0x20 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa0335c3b>] zio_wait+0xfb/0x1b0 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa029c58d>] dbuf_read+0x3fd/0x750 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa02a3fc8>] dmu_buf_hold+0x108/0x1d0 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa030060f>] zap_get_leaf_byblk+0x4f/0x2b0 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa03008da>] zap_deref_leaf+0x6a/0x80 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa03010c7>] fzap_remove+0x37/0xb0 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa030437d>] ? zap_name_alloc+0x8d/0xf0 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa0306113>] zap_remove_norm+0x153/0x1d0 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa03061a3>] zap_remove+0x13/0x20 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa02ff551>] zap_remove_int+0x61/0x90 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa030efa7>] zfs_rmnode+0x1c7/0x340 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa032c765>] zfs_zinactive+0xa5/0x110 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa032b75f>] zfs_inactive+0x7f/0x220 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa033e11e>] zpl_clear_inode+0xe/0x10 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff811a5bec>] clear_inode+0xac/0x140
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff811a5cc0>] dispose_list+0x40/0x120
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff811a6014>] shrink_icache_memory+0x274/0x2e0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa033e2c0>] ? zpl_prune_sbs+0x0/0x60 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa033e313>] zpl_prune_sbs+0x53/0x60 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa0296920>] arc_adjust_meta+0x120/0x1e0 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa02969e0>] ? arc_adapt_thread+0x0/0x160 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa02969e0>] ? arc_adapt_thread+0x0/0x160 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa0296a47>] arc_adapt_thread+0x67/0x160 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01c5478>] thread_generic_wrapper+0x68/0x80 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01c5410>] ? thread_generic_wrapper+0x0/0x80 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109af06>] kthread+0x96/0xa0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109ae70>] ? kthread+0x0/0xa0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: INFO: task z_rd_int/0:899 blocked for more than 120 seconds.
Feb 18 14:45:36 jenkins-slave2 kernel: Tainted: P --------------- 2.6.32-431.3.1.el6.x86_64 #1
Feb 18 14:45:36 jenkins-slave2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 18 14:45:36 jenkins-slave2 kernel: z_rd_int/0 D 0000000000000000 0 899 2 0x00000000
Feb 18 14:45:36 jenkins-slave2 kernel: ffff880138799ce0 0000000000000046 0000000000000000 ffff8800282168a8
Feb 18 14:45:36 jenkins-slave2 kernel: 0000000000001206 ffff88013967b540 ffff8801385bd500 0000000000000000
Feb 18 14:45:36 jenkins-slave2 kernel: ffff88013967baf8 ffff880138799fd8 000000000000fbc8 ffff88013967baf8
Feb 18 14:45:36 jenkins-slave2 kernel: Call Trace:
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8152908e>] __mutex_lock_slowpath+0x13e/0x180
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81528f2b>] mutex_lock+0x2b/0x50
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa02fbdae>] vdev_queue_io_done+0x3e/0x120 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa0334bb8>] zio_vdev_io_done+0x88/0x190 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa0335ab3>] zio_execute+0xb3/0x140 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01c6628>] taskq_thread+0x218/0x4b0 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81527920>] ? thread_return+0x4e/0x76e
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01c6410>] ? taskq_thread+0x0/0x4b0 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109af06>] kthread+0x96/0xa0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109ae70>] ? kthread+0x0/0xa0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: INFO: task z_rd_int/1:900 blocked for more than 120 seconds.
Feb 18 14:45:36 jenkins-slave2 kernel: Tainted: P --------------- 2.6.32-431.3.1.el6.x86_64 #1
Feb 18 14:45:36 jenkins-slave2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 18 14:45:36 jenkins-slave2 kernel: z_rd_int/1 D 0000000000000001 0 900 2 0x00000000
Feb 18 14:45:36 jenkins-slave2 kernel: ffff8801386bdce0 0000000000000046 0000000000000000 ffff8800282968a8
Feb 18 14:45:36 jenkins-slave2 kernel: 000000000000a841 ffff880138648080 ffff880138840ae0 0000000000000000
Feb 18 14:45:36 jenkins-slave2 kernel: ffff880138648638 ffff8801386bdfd8 000000000000fbc8 ffff880138648638
Feb 18 14:45:36 jenkins-slave2 kernel: Call Trace:
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8152908e>] __mutex_lock_slowpath+0x13e/0x180
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81528f2b>] mutex_lock+0x2b/0x50
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa02fbdae>] vdev_queue_io_done+0x3e/0x120 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa0334bb8>] zio_vdev_io_done+0x88/0x190 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa0335ab3>] zio_execute+0xb3/0x140 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01c6628>] taskq_thread+0x218/0x4b0 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81527920>] ? thread_return+0x4e/0x76e
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01c6410>] ? taskq_thread+0x0/0x4b0 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109af06>] kthread+0x96/0xa0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109ae70>] ? kthread+0x0/0xa0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: INFO: task z_rd_int/2:901 blocked for more than 120 seconds.
Feb 18 14:45:36 jenkins-slave2 kernel: Tainted: P --------------- 2.6.32-431.3.1.el6.x86_64 #1
Feb 18 14:45:36 jenkins-slave2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 18 14:45:36 jenkins-slave2 kernel: z_rd_int/2 D 0000000000000002 0 901 2 0x00000000
Feb 18 14:45:36 jenkins-slave2 kernel: ffff88013957bce0 0000000000000046 0000000000000000 0000000000016840
Feb 18 14:45:36 jenkins-slave2 kernel: 0000000000016840 ffff8801394dd540 ffff88013dc58040 ffffffff8160bf40
Feb 18 14:45:36 jenkins-slave2 kernel: ffff8801394ddaf8 ffff88013957bfd8 000000000000fbc8 ffff8801394ddaf8
Feb 18 14:45:36 jenkins-slave2 kernel: Call Trace:
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8152908e>] __mutex_lock_slowpath+0x13e/0x180
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81528f2b>] mutex_lock+0x2b/0x50
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa02fbdae>] vdev_queue_io_done+0x3e/0x120 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa0334bb8>] zio_vdev_io_done+0x88/0x190 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa0335ab3>] zio_execute+0xb3/0x140 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01c6628>] taskq_thread+0x218/0x4b0 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81527920>] ? thread_return+0x4e/0x76e
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01c6410>] ? taskq_thread+0x0/0x4b0 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109af06>] kthread+0x96/0xa0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109ae70>] ? kthread+0x0/0xa0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: INFO: task z_rd_int/3:902 blocked for more than 120 seconds.
Feb 18 14:45:36 jenkins-slave2 kernel: Tainted: P --------------- 2.6.32-431.3.1.el6.x86_64 #1
Feb 18 14:45:36 jenkins-slave2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 18 14:45:36 jenkins-slave2 kernel: z_rd_int/3 D 0000000000000003 0 902 2 0x00000000
Feb 18 14:45:36 jenkins-slave2 kernel: ffff88013c9f9ce0 0000000000000046 0000000000000000 ffffffff81053657
Feb 18 14:45:36 jenkins-slave2 kernel: 0000000000016840 ffff88013c588aa0 ffff88013dc92aa0 ffffffff8160bf40
Feb 18 14:45:36 jenkins-slave2 kernel: ffff88013c589058 ffff88013c9f9fd8 000000000000fbc8 ffff88013c589058
Feb 18 14:45:36 jenkins-slave2 kernel: Call Trace:
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81053657>] ? walk_tg_tree_from+0x67/0xc0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8152908e>] __mutex_lock_slowpath+0x13e/0x180
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81528f2b>] mutex_lock+0x2b/0x50
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa02fbdae>] vdev_queue_io_done+0x3e/0x120 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa0334bb8>] zio_vdev_io_done+0x88/0x190 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa0335ab3>] zio_execute+0xb3/0x140 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01c6628>] taskq_thread+0x218/0x4b0 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81527920>] ? thread_return+0x4e/0x76e
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01c6410>] ? taskq_thread+0x0/0x4b0 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109af06>] kthread+0x96/0xa0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109ae70>] ? kthread+0x0/0xa0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: INFO: task z_wr_iss/0:903 blocked for more than 120 seconds.
Feb 18 14:45:36 jenkins-slave2 kernel: Tainted: P --------------- 2.6.32-431.3.1.el6.x86_64 #1
Feb 18 14:45:36 jenkins-slave2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 18 14:45:36 jenkins-slave2 kernel: z_wr_iss/0 D 0000000000000000 0 903 2 0x00000000
Feb 18 14:45:36 jenkins-slave2 kernel: ffff8801394abbc0 0000000000000046 0000000000000000 0000000000000287
Feb 18 14:45:36 jenkins-slave2 kernel: ffff8801394abb50 ffffffff8109b497 ffff880138ac5800 ffff8801354f0000
Feb 18 14:45:36 jenkins-slave2 kernel: ffff880138b55058 ffff8801394abfd8 000000000000fbc8 ffff880138b55058
Feb 18 14:45:36 jenkins-slave2 kernel: Call Trace:
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109b497>] ? finish_wait+0x67/0x80
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8152908e>] __mutex_lock_slowpath+0x13e/0x180
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81528f2b>] mutex_lock+0x2b/0x50
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa02fbf11>] vdev_queue_io+0x81/0x130 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa0336039>] zio_vdev_io_start+0x1c9/0x2e0 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa0336a13>] zio_nowait+0xb3/0x170 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa02fb1b2>] vdev_mirror_io_start+0x252/0x500 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa02faa40>] ? vdev_mirror_child_done+0x0/0x30 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa033608f>] zio_vdev_io_start+0x21f/0x2e0 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa0335ab3>] zio_execute+0xb3/0x140 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01c6628>] taskq_thread+0x218/0x4b0 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81527920>] ? thread_return+0x4e/0x76e
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01c6410>] ? taskq_thread+0x0/0x4b0 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109af06>] kthread+0x96/0xa0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109ae70>] ? kthread+0x0/0xa0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: INFO: task z_wr_iss/1:904 blocked for more than 120 seconds.
Feb 18 14:45:36 jenkins-slave2 kernel: Tainted: P --------------- 2.6.32-431.3.1.el6.x86_64 #1
Feb 18 14:45:36 jenkins-slave2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 18 14:45:36 jenkins-slave2 kernel: z_wr_iss/1 D 0000000000000001 0 904 2 0x00000000
Feb 18 14:45:36 jenkins-slave2 kernel: ffff88013b38dbc0 0000000000000046 0000000000000000 0000000000000202
Feb 18 14:45:36 jenkins-slave2 kernel: ffff88013b38db50 ffffffff8109b497 ffff880139353000 ffff8801354f0000
Feb 18 14:45:36 jenkins-slave2 kernel: ffff880138841098 ffff88013b38dfd8 000000000000fbc8 ffff880138841098
Feb 18 14:45:36 jenkins-slave2 kernel: Call Trace:
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109b497>] ? finish_wait+0x67/0x80
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8152908e>] __mutex_lock_slowpath+0x13e/0x180
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81528f2b>] mutex_lock+0x2b/0x50
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa02fbf11>] vdev_queue_io+0x81/0x130 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa0336039>] zio_vdev_io_start+0x1c9/0x2e0 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa0336a13>] zio_nowait+0xb3/0x170 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa02fb1b2>] vdev_mirror_io_start+0x252/0x500 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa02faa40>] ? vdev_mirror_child_done+0x0/0x30 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa033608f>] zio_vdev_io_start+0x21f/0x2e0 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa0335ab3>] zio_execute+0xb3/0x140 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01c6628>] taskq_thread+0x218/0x4b0 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81527920>] ? thread_return+0x4e/0x76e
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01c6410>] ? taskq_thread+0x0/0x4b0 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109af06>] kthread+0x96/0xa0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109ae70>] ? kthread+0x0/0xa0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: INFO: task z_wr_iss/2:905 blocked for more than 120 seconds.
Feb 18 14:45:36 jenkins-slave2 kernel: Tainted: P --------------- 2.6.32-431.3.1.el6.x86_64 #1
Feb 18 14:45:36 jenkins-slave2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 18 14:45:36 jenkins-slave2 kernel: z_wr_iss/2 D 0000000000000002 0 905 2 0x00000000
Feb 18 14:45:36 jenkins-slave2 kernel: ffff88013a8cfbc0 0000000000000046 0000000000000000 0000000000000287
Feb 18 14:45:36 jenkins-slave2 kernel: ffff88013a8cfb50 ffffffff8109b497 ffff880139389c00 ffff8801354f0000
Feb 18 14:45:36 jenkins-slave2 kernel: ffff880139bee5f8 ffff88013a8cffd8 000000000000fbc8 ffff880139bee5f8
Feb 18 14:45:36 jenkins-slave2 kernel: Call Trace:
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109b497>] ? finish_wait+0x67/0x80
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8152908e>] __mutex_lock_slowpath+0x13e/0x180
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81528f2b>] mutex_lock+0x2b/0x50
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa02fbf11>] vdev_queue_io+0x81/0x130 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa0336039>] zio_vdev_io_start+0x1c9/0x2e0 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa0336a13>] zio_nowait+0xb3/0x170 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa02fb1b2>] vdev_mirror_io_start+0x252/0x500 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa02faa40>] ? vdev_mirror_child_done+0x0/0x30 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa033608f>] zio_vdev_io_start+0x21f/0x2e0 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa0335ab3>] zio_execute+0xb3/0x140 [zfs]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01c6628>] taskq_thread+0x218/0x4b0 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81527920>] ? thread_return+0x4e/0x76e
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffffa01c6410>] ? taskq_thread+0x0/0x4b0 [spl]
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109af06>] kthread+0x96/0xa0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8109ae70>] ? kthread+0x0/0xa0
Feb 18 14:45:36 jenkins-slave2 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
** didn't notice it was hung until then **
Feb 18 17:25:55 jenkins-slave2 kernel: SysRq : Show State
Feb 18 17:25:55 jenkins-slave2 kernel: task PC stack pid father
Feb 18 17:25:55 jenkins-slave2 kernel: init S 0000000000000001 0 1 0 0x00000000
Feb 18 17:25:55 jenkins-slave2 kernel: ffff88013dc0b908 0000000000000086 0000000000000000 ffff88013dc0b8cc
Feb 18 17:25:55 jenkins-slave2 kernel: 0000000000000003 ffff88013fc23080 ffff880028316840 0000000000000406
Feb 18 17:25:55 jenkins-slave2 kernel: ffff88013dc09ab8 ffff88013dc0bfd8 000000000000fbc8 ffff88013dc09ab8
Feb 18 17:25:55 jenkins-slave2 kernel: Call Trace:
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8152942d>] schedule_hrtimeout_range+0x13d/0x160
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8109b656>] ? add_wait_queue+0x46/0x60
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8119fe65>] ? __pollwait+0x75/0xf0
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8119fe65>] ? __pollwait+0x75/0xf0
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8119fcf9>] poll_schedule_timeout+0x39/0x60
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff811a0dbc>] do_select+0x57c/0x6c0
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff810149b9>] ? read_tsc+0x9/0x20
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8119fdf0>] ? __pollwait+0x0/0xf0
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8119fee0>] ? pollwake+0x0/0x60
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8119fee0>] ? pollwake+0x0/0x60
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8119fee0>] ? pollwake+0x0/0x60
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8119fee0>] ? pollwake+0x0/0x60
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8119fee0>] ? pollwake+0x0/0x60
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8122dad1>] ? avc_has_perm+0x71/0x90
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8114a667>] ? handle_pte_fault+0xf7/0xb00
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff81528f1e>] ? mutex_lock+0x1e/0x50
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff811a108a>] core_sys_select+0x18a/0x2c0
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8122e120>] ? selinux_task_wait+0x40/0x50
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff81226856>] ? security_task_wait+0x16/0x20
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8107586d>] ? wait_consider_task+0x9d/0xb20
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8109b6ac>] ? remove_wait_queue+0x3c/0x50
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8107646f>] ? do_wait+0x17f/0x240
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff811a1417>] sys_select+0x47/0x110
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff81074b70>] ? child_wait_callback+0x0/0x70
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Feb 18 17:25:55 jenkins-slave2 kernel: kthreadd S 0000000000000003 0 2 0 0x00000000
Feb 18 17:25:55 jenkins-slave2 kernel: ffff88013dc11ed0 0000000000000046 0000000000000000 ffffffff8100c1a2
Feb 18 17:25:55 jenkins-slave2 kernel: ffff88013dc08aa0 ffff88013dc08aa0 ffffffff81eb3c88 ffff88012d255c08
Feb 18 17:25:55 jenkins-slave2 kernel: ffff88013dc09058 ffff88013dc11fd8 000000000000fbc8 ffff88013dc09058
Feb 18 17:25:55 jenkins-slave2 kernel: Call Trace:
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8100c1a2>] ? kernel_thread+0x82/0xe0
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8109ae70>] ? kthread+0x0/0xa0
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8109ae65>] kthreadd+0x1b5/0x1c0
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8109acb0>] ? kthreadd+0x0/0x1c0
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Feb 18 17:25:55 jenkins-slave2 kernel: migration/0 S 0000000000000000 0 3 2 0x00000000
Feb 18 17:25:55 jenkins-slave2 kernel: ffff88013dc15e70 0000000000000046 ffff88013dc15dd0 ffff880045016040
Feb 18 17:25:55 jenkins-slave2 kernel: ffff88013dc15e00 ffffffff8106540e ffff880028396840 ffff880045016040
Feb 18 17:25:55 jenkins-slave2 kernel: ffff88013dc085f8 ffff88013dc15fd8 000000000000fbc8 ffff88013dc085f8
Feb 18 17:25:55 jenkins-slave2 kernel: Call Trace:
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8106540e>] ? pull_task+0x4e/0x60
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff81068635>] migration_thread+0x265/0x2e0
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff810683d0>] ? migration_thread+0x0/0x2e0
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8109af06>] kthread+0x96/0xa0
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8109ae70>] ? kthread+0x0/0xa0
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Feb 18 17:25:55 jenkins-slave2 kernel: ksoftirqd/0 S 0000000000000000 0 4 2 0x00000000
Feb 18 17:25:55 jenkins-slave2 kernel: ffff88013dc19ea0 0000000000000046 0000000000000000 ffff88013dc19e64
Feb 18 17:25:55 jenkins-slave2 kernel: 000000000000fbc8 ffff88013fc23080 ffff880028296840 000000000000af5c
Feb 18 17:25:55 jenkins-slave2 kernel: ffff88013dc17af8 ffff88013dc19fd8 000000000000fbc8 ffff88013dc17af8
Feb 18 17:25:55 jenkins-slave2 kernel: Call Trace:
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8107a4f5>] ksoftirqd+0xd5/0x110
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8107a420>] ? ksoftirqd+0x0/0x110
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8109af06>] kthread+0x96/0xa0
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8109ae70>] ? kthread+0x0/0xa0
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Feb 18 17:25:55 jenkins-slave2 kernel: migration/0 S 0000000000000000 0 5 2 0x00000000
Feb 18 17:25:55 jenkins-slave2 kernel: ffff88013dc1bdf0 0000000000000046 0000000000000000 0000000000000000
Feb 18 17:25:55 jenkins-slave2 kernel: 0000000000000000 00000000ffffffff ffffffffa04ab248 0000000000000000
Feb 18 17:25:55 jenkins-slave2 kernel: ffff88013dc17098 ffff88013dc1bfd8 000000000000fbc8 ffff88013dc17098
Feb 18 17:25:55 jenkins-slave2 kernel: Call Trace:
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff810d3c00>] ? stop_machine_cpu_stop+0x0/0xf0
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff810d3995>] cpu_stopper_thread+0x125/0x1b0
Feb 18 17:25:55 jenkins-slave2 kernel: [<ffffffff81527920>] ? thread_return+0x4e/0x76e
** reboot **
Raw
ps.txt
** this is 5min BEFORE the hang **
+ ps max -o pid,f,user,blocked,etime,stat,nwchan:16,wchan:24,comm --sort etime
+ tee /zfs-data/jenkins/workspace/syslog.jenkins-slave2/ps.txt
PID F USER BLOCKED ELAPSED STAT WCHAN WCHAN COMMAND
1 4 root - 03:12:48 - - - init
- 4 root 0000000000000000 03:12:48 Ss ffffff poll_schedule_timeout -
2 1 root - 03:12:48 - - - kthreadd
3 1 root - 03:12:48 - - - migration/0
4 1 root - 03:12:48 - - - ksoftirqd/0
5 1 root - 03:12:48 - - - migration/0
6 5 root - 03:12:48 - - - watchdog/0
7 1 root - 03:12:48 - - - migration/1
8 1 root - 03:12:48 - - - migration/1
9 1 root - 03:12:48 - - - ksoftirqd/1
10 5 root - 03:12:48 - - - watchdog/1
11 1 root - 03:12:48 - - - migration/2
12 1 root - 03:12:48 - - - migration/2
13 1 root - 03:12:48 - - - ksoftirqd/2
14 5 root - 03:12:48 - - - watchdog/2
15 1 root - 03:12:48 - - - migration/3
16 1 root - 03:12:48 - - - migration/3
17 1 root - 03:12:48 - - - ksoftirqd/3
18 5 root - 03:12:48 - - - watchdog/3
19 1 root - 03:12:48 - - - events/0
20 1 root - 03:12:48 - - - events/1
21 1 root - 03:12:48 - - - events/2
22 1 root - 03:12:48 - - - events/3
23 1 root - 03:12:48 - - - cgroup
24 1 root - 03:12:48 - - - khelper
25 1 root - 03:12:48 - - - netns
26 1 root - 03:12:48 - - - async/mgr
27 1 root - 03:12:48 - - - pm
28 1 root - 03:12:48 - - - sync_supers
29 1 root - 03:12:48 - - - bdi-default
30 1 root - 03:12:48 - - - kintegrityd/0
31 1 root - 03:12:48 - - - kintegrityd/1
32 1 root - 03:12:48 - - - kintegrityd/2
33 1 root - 03:12:48 - - - kintegrityd/3
34 1 root - 03:12:48 - - - kblockd/0
35 1 root - 03:12:48 - - - kblockd/1
36 1 root - 03:12:48 - - - kblockd/2
37 1 root - 03:12:48 - - - kblockd/3
38 1 root - 03:12:48 - - - kacpid
39 1 root - 03:12:48 - - - kacpi_notify
40 1 root - 03:12:48 - - - kacpi_hotplug
41 1 root - 03:12:48 - - - ata_aux
42 1 root - 03:12:48 - - - ata_sff/0
43 1 root - 03:12:48 - - - ata_sff/1
44 1 root - 03:12:48 - - - ata_sff/2
45 1 root - 03:12:48 - - - ata_sff/3
46 1 root - 03:12:48 - - - ksuspend_usbd
47 1 root - 03:12:48 - - - khubd
48 5 root - 03:12:48 - - - kseriod
49 1 root - 03:12:48 - - - md/0
50 1 root - 03:12:48 - - - md/1
51 1 root - 03:12:48 - - - md/2
52 1 root - 03:12:48 - - - md/3
53 1 root - 03:12:48 - - - md_misc/0
54 1 root - 03:12:48 - - - md_misc/1
55 1 root - 03:12:48 - - - md_misc/2
56 1 root - 03:12:48 - - - md_misc/3
57 1 root - 03:12:48 - - - linkwatch
58 1 root - 03:12:48 - - - khungtaskd
59 1 root - 03:12:48 - - - kswapd0
60 1 root - 03:12:48 - - - ksmd
61 1 root - 03:12:48 - - - khugepaged
62 1 root - 03:12:48 - - - aio/0
63 1 root - 03:12:48 - - - aio/1
64 1 root - 03:12:48 - - - aio/2
65 1 root - 03:12:48 - - - aio/3
66 1 root - 03:12:48 - - - crypto/0
67 1 root - 03:12:48 - - - crypto/1
68 1 root - 03:12:48 - - - crypto/2
69 1 root - 03:12:48 - - - crypto/3
74 1 root - 03:12:48 - - - kthrotld/0
75 1 root - 03:12:48 - - - kthrotld/1
76 1 root - 03:12:48 - - - kthrotld/2
77 1 root - 03:12:48 - - - kthrotld/3
78 1 root - 03:12:48 - - - pciehpd
80 1 root - 03:12:47 - - - kpsmoused
81 1 root - 03:12:47 - - - usbhid_resumer
111 1 root - 03:12:47 - - - kstriped
198 1 root - 03:12:47 - - - scsi_eh_0
199 1 root - 03:12:47 - - - scsi_eh_1
205 1 root - 03:12:46 - - - mpt_poll_0
206 1 root - 03:12:46 - - - mpt/0
207 1 root - 03:12:46 - - - scsi_eh_2
327 1 root - 03:12:45 - - - jbd2/sda3-8
328 1 root - 03:12:45 - - - ext4-dio-unwrit
337 1 root - 03:12:43 - - - flush-8:0
426 5 root - 03:12:37 - - - udevd
683 1 root - 03:12:35 - - - vmmemctl
706 1 root - 03:12:35 - - - spl_kmem_cache/
707 1 root - 03:12:35 - - - spl_system_task
708 1 root - 03:12:35 - - - spl_system_task
709 1 root - 03:12:35 - - - spl_system_task
710 1 root - 03:12:35 - - - spl_system_task
740 1 root - 03:12:34 - - - arc_adapt
746 1 root - 03:12:34 - - - l2arc_feed
747 1 root - 03:12:34 - - - z_unmount/0
748 1 root - 03:12:34 - - - zvol/0
749 1 root - 03:12:34 - - - zvol/1
750 1 root - 03:12:34 - - - zvol/2
751 1 root - 03:12:34 - - - zvol/3
752 1 root - 03:12:34 - - - zvol/4
753 1 root - 03:12:34 - - - zvol/5
754 1 root - 03:12:34 - - - zvol/6
755 1 root - 03:12:34 - - - zvol/7
756 1 root - 03:12:34 - - - zvol/8
757 1 root - 03:12:34 - - - zvol/9
758 1 root - 03:12:34 - - - zvol/10
759 1 root - 03:12:34 - - - zvol/11
760 1 root - 03:12:34 - - - zvol/12
761 1 root - 03:12:34 - - - zvol/13
762 1 root - 03:12:34 - - - zvol/14
763 1 root - 03:12:34 - - - zvol/15
764 1 root - 03:12:34 - - - zvol/16
765 1 root - 03:12:34 - - - zvol/17
766 1 root - 03:12:34 - - - zvol/18
767 1 root - 03:12:34 - - - zvol/19
768 1 root - 03:12:34 - - - zvol/20
769 1 root - 03:12:34 - - - zvol/21
770 1 root - 03:12:34 - - - zvol/22
771 1 root - 03:12:34 - - - zvol/23
772 1 root - 03:12:34 - - - zvol/24
773 1 root - 03:12:34 - - - zvol/25
774 1 root - 03:12:34 - - - zvol/26
775 1 root - 03:12:34 - - - zvol/27
776 1 root - 03:12:34 - - - zvol/28
777 1 root - 03:12:34 - - - zvol/29
778 1 root - 03:12:34 - - - zvol/30
779 1 root - 03:12:34 - - - zvol/31
889 1 root - 03:12:34 - - - z_null_iss/0
890 1 root - 03:12:34 - - - z_null_int/0
891 1 root - 03:12:34 - - - z_rd_iss/0
892 1 root - 03:12:34 - - - z_rd_iss/1
893 1 root - 03:12:34 - - - z_rd_iss/2
894 1 root - 03:12:34 - - - z_rd_iss/3
895 1 root - 03:12:34 - - - z_rd_iss/4
896 1 root - 03:12:34 - - - z_rd_iss/5
897 1 root - 03:12:34 - - - z_rd_iss/6
898 1 root - 03:12:34 - - - z_rd_iss/7
899 1 root - 03:12:34 - - - z_rd_int/0
900 1 root - 03:12:34 - - - z_rd_int/1
901 1 root - 03:12:34 - - - z_rd_int/2
902 1 root - 03:12:34 - - - z_rd_int/3
903 1 root - 03:12:34 - - - z_wr_iss/0
904 1 root - 03:12:34 - - - z_wr_iss/1
905 1 root - 03:12:34 - - - z_wr_iss/2
906 1 root - 03:12:34 - - - z_wr_iss/3
907 1 root - 03:12:34 - - - z_wr_iss_h/0
908 1 root - 03:12:34 - - - z_wr_iss_h/1
909 1 root - 03:12:34 - - - z_wr_iss_h/2
910 1 root - 03:12:34 - - - z_wr_iss_h/3
911 1 root - 03:12:34 - - - z_wr_iss_h/4
912 1 root - 03:12:34 - - - z_wr_int/0
913 1 root - 03:12:34 - - - z_wr_int/1
914 1 root - 03:12:34 - - - z_wr_int/2
915 1 root - 03:12:34 - - - z_wr_int/3
916 1 root - 03:12:34 - - - z_wr_int/4
917 1 root - 03:12:34 - - - z_wr_int/5
918 1 root - 03:12:34 - - - z_wr_int/6
919 1 root - 03:12:34 - - - z_wr_int/7
920 1 root - 03:12:34 - - - z_wr_int/8
921 1 root - 03:12:34 - - - z_wr_int/9
922 1 root - 03:12:34 - - - z_wr_int/10
923 1 root - 03:12:34 - - - z_wr_int/11
924 1 root - 03:12:34 - - - z_wr_int/12
925 1 root - 03:12:34 - - - z_wr_int/13
926 1 root - 03:12:34 - - - z_wr_int/14
927 1 root - 03:12:34 - - - z_wr_int/15
928 1 root - 03:12:34 - - - z_wr_int_h/0
929 1 root - 03:12:34 - - - z_wr_int_h/1
930 1 root - 03:12:34 - - - z_wr_int_h/2
931 1 root - 03:12:34 - - - z_wr_int_h/3
932 1 root - 03:12:34 - - - z_wr_int_h/4
933 1 root - 03:12:34 - - - z_fr_iss_0/0
934 1 root - 03:12:34 - - - z_fr_iss_0/1
935 1 root - 03:12:34 - - - z_fr_iss_0/2
936 1 root - 03:12:34 - - - z_fr_iss_0/3
937 1 root - 03:12:34 - - - z_fr_iss_1/0
938 1 root - 03:12:34 - - - z_fr_iss_1/1
939 1 root - 03:12:34 - - - z_fr_iss_1/2
940 1 root - 03:12:34 - - - z_fr_iss_1/3
941 1 root - 03:12:34 - - - z_fr_iss_2/0
942 1 root - 03:12:34 - - - z_fr_iss_2/1
943 1 root - 03:12:34 - - - z_fr_iss_2/2
944 1 root - 03:12:34 - - - z_fr_iss_2/3
945 1 root - 03:12:34 - - - z_fr_iss_3/0
946 1 root - 03:12:34 - - - z_fr_iss_3/1
947 1 root - 03:12:34 - - - z_fr_iss_3/2
948 1 root - 03:12:34 - - - z_fr_iss_3/3
949 1 root - 03:12:34 - - - z_fr_iss_4/0
950 1 root - 03:12:34 - - - z_fr_iss_4/1
951 1 root - 03:12:34 - - - z_fr_iss_4/2
952 1 root - 03:12:34 - - - z_fr_iss_4/3
953 1 root - 03:12:34 - - - z_fr_iss_5/0
954 1 root - 03:12:34 - - - z_fr_iss_5/1
955 1 root - 03:12:34 - - - z_fr_iss_5/2
956 1 root - 03:12:34 - - - z_fr_iss_5/3
957 1 root - 03:12:34 - - - z_fr_iss_6/0
958 1 root - 03:12:34 - - - z_fr_iss_6/1
959 1 root - 03:12:34 - - - z_fr_iss_6/2
960 1 root - 03:12:34 - - - z_fr_iss_6/3
961 1 root - 03:12:34 - - - z_fr_iss_7/0
962 1 root - 03:12:34 - - - z_fr_iss_7/1
963 1 root - 03:12:34 - - - z_fr_iss_7/2
964 1 root - 03:12:34 - - - z_fr_iss_7/3
965 1 root - 03:12:34 - - - z_fr_int/0
966 1 root - 03:12:34 - - - z_cl_iss/0
967 1 root - 03:12:34 - - - z_cl_int/0
968 1 root - 03:12:34 - - - z_ioctl_iss/0
969 1 root - 03:12:34 - - - z_ioctl_int/0
971 1 root - 03:12:33 - - - zfs_iput_taskq/
972 1 root - 03:12:31 - - - txg_quiesce
973 1 root - 03:12:31 - - - txg_sync
1022 1 root - 03:12:27 - - - jbd2/sda1-8
1023 1 root - 03:12:27 - - - ext4-dio-unwrit
1060 1 root - 03:12:26 - - - kauditd
1545 5 root - 03:11:40 - - - auditd
- 5 root 0000000000000000 03:11:40 S<sl ffffff ep_poll -
- 1 root 0000000000006a01 03:11:40 S<sl ffffff futex_wait_queue_me -
1561 5 root - 03:11:40 - - - rsyslogd
- 5 root 0000000000000000 03:11:40 Sl ffffff poll_schedule_timeout -
- 1 root fffffffe7febfeff 03:11:40 Sl ffffff futex_wait_queue_me -
- 1 root fffffffe7febfeff 03:11:40 Sl ffffff poll_schedule_timeout -
- 5 root fffffffe7febfeff 03:11:40 Sl ffffff syslog -
1590 5 root - 03:11:39 - - - irqbalance
1604 5 rpc - 03:11:39 - - - rpcbind
1617 1 root - 03:11:38 - - - sssd
1618 0 root - 03:11:38 - - - sssd_be
1619 4 root - 03:11:37 - - - sssd_nss
1620 4 root - 03:11:37 - - - sssd_pam
1637 5 rpcuser - 03:11:37 - - - rpc.statd
1761 5 dbus - 03:11:36 - - - dbus-daemon
- 5 dbus 0000000000000000 03:11:36 Ssl ffffff poll_schedule_timeout -
- 5 dbus 0000000000000000 03:11:36 Ssl ffffff poll_schedule_timeout -
1791 1 root - 03:11:36 - - - acpid
1800 5 68 - 03:11:36 - - - hald
- 5 68 0000000000000000 03:11:36 Ssl ffffff poll_schedule_timeout -
- 5 68 0000000000000000 03:11:36 Ssl ffffff pipe_wait -
1801 0 root - 03:11:36 - - - hald-runner
1830 0 root - 03:11:35 - - - hald-addon-inpu
1851 4 68 - 03:11:35 - - - hald-addon-acpi
1864 5 root - 03:11:35 - - - automount
- 5 root 0000000000030000 03:11:35 Ssl ffffff rt_sigtimedwait -
- 1 root fffffffe7ffbfa07 03:11:35 Ssl ffffff futex_wait_queue_me -
- 1 root fffffffe7ffbfa07 03:11:35 Ssl ffffff futex_wait_queue_me -
- 5 root fffffffe7ffbfa07 03:11:34 Ssl ffffff poll_schedule_timeout -
- 5 root fffffffe7ffbfa07 03:11:34 Ssl ffffff poll_schedule_timeout -
- 5 root fffffffe7ffbfa07 03:11:34 Ssl ffffff poll_schedule_timeout -
1885 5 root - 03:11:34 - - - sshd
1961 4 root - 03:11:33 - - - master
1970 4 postfix - 03:11:33 - - - qmgr
1985 1 root - 03:11:33 - - - abrtd
1993 1 root - 03:11:33 - - - crond
2004 5 root - 03:11:32 - - - atd
2017 4 root - 03:11:32 - - - mingetty
2019 4 root - 03:11:32 - - - mingetty
2021 4 root - 03:11:32 - - - mingetty
2023 4 root - 03:11:32 - - - mingetty
2025 4 root - 03:11:32 - - - mingetty
2027 4 root - 03:11:32 - - - mingetty
2034 5 root - 03:11:32 - - - udevd
2035 5 root - 03:11:32 - - - udevd
3859 1 root - 01:53:36 - - - zil_clean/0
3862 1 root - 01:53:36 - - - zil_clean/0
3871 4 root - 01:53:10 - - - sshd
3874 5 jenkins - 01:53:10 - - - sshd
3892 0 jenkins - 01:53:07 - - - bash
3893 0 jenkins - 01:53:07 - - - java
- 0 jenkins 0000000000000000 01:53:07 Sl ae8e9 futex_wait_queue_me -
- 1 jenkins 0000000000000004 01:53:07 Sl ae8e9 futex_wait_queue_me -
- 1 jenkins 0000000000000004 01:53:07 Sl ae8e9 futex_wait_queue_me -
- 1 jenkins 0000000000000004 01:53:07 Sl ae8e9 futex_wait_queue_me -
- 1 jenkins 0000000000000004 01:53:07 Sl ae8e9 futex_wait_queue_me -
- 1 jenkins 0000000000000004 01:53:07 Sl ae8e9 futex_wait_queue_me -
- 1 jenkins 0000000000000000 01:53:07 Sl ae8e9 futex_wait_queue_me -
- 1 jenkins 0000000000000004 01:53:07 Sl ae8e9 futex_wait_queue_me -
- 1 jenkins 0000000000000004 01:53:07 Sl ae8e9 futex_wait_queue_me -
- 1 jenkins 0000000000000004 01:53:07 Sl ae8e9 futex_wait_queue_me -
- 1 jenkins 0000000000000004 01:53:07 Sl ae8e9 futex_wait_queue_me -
- 1 jenkins 0000000000000004 01:53:07 Sl ae8e9 futex_wait_queue_me -
- 1 jenkins 0000000000000004 01:53:07 Sl ae8e9 futex_wait_queue_me -
- 1 jenkins 0000000000000004 01:53:07 Sl ae8e9 futex_wait_queue_me -
- 1 jenkins 0000000000000004 01:53:06 Sl 1939db pipe_wait -
- 1 jenkins 0000000000000004 01:53:06 Sl ae8e9 futex_wait_queue_me -
- 1 jenkins 0000000000000004 01:53:06 Sl ae8e9 futex_wait_queue_me -
- 1 jenkins 0000000000000004 01:53:05 Sl ae8e9 futex_wait_queue_me -
- 1 jenkins 0000000000000004 01:05:15 Sl ae8e9 futex_wait_queue_me -
- 1 jenkins 0000000000000004 20:41 Sl ae8e9 futex_wait_queue_me -
- 1 jenkins 0000000000000004 17:27 Sl ae8e9 futex_wait_queue_me -
- 1 jenkins 0000000000000004 11:51 Sl 764d4 wait -
- 1 jenkins 0000000000000004 11:51 Sl 1939db pipe_wait -
- 1 jenkins 0000000000000004 09:26 Sl 764d4 wait -
- 1 jenkins 0000000000000004 09:26 Sl 1939db pipe_wait -
- 1 jenkins 0000000000000004 01:17 Sl 764d4 wait -
- 1 jenkins 0000000000000004 01:17 Sl 1939db pipe_wait -
- 1 jenkins 0000000000000004 00:01 Sl 764d4 wait -
- 1 jenkins 0000000000000004 00:01 Sl 1939db pipe_wait -
3977 5 root - 01:52:39 - - - rpciod/0
3978 5 root - 01:52:39 - - - rpciod/1
3979 5 root - 01:52:39 - - - rpciod/2
3980 5 root - 01:52:39 - - - rpciod/3
3982 1 root - 01:52:39 - - - kslowd000
3983 1 root - 01:52:39 - - - kslowd001
3984 1 root - 01:52:39 - - - nfsiod
3985 1 root - 01:52:39 - - - lockd
9687 4 root - 21:49 - - - sshd
9695 4 root - 21:46 - - - bash
17964 0 jenkins - 11:51 - - - sh
19723 0 jenkins - 09:45 - - - make
19935 0 jenkins - 09:26 - - - sh
20257 0 jenkins - 05:48 - - - make
22832 0 jenkins - 01:17 - - - sh
22852 0 jenkins - 01:15 - - - rm
23307 0 jenkins - 00:04 - - - x86_64-linux5.0
23308 0 jenkins - 00:04 - - - cc1
23309 0 jenkins - 00:04 - - - x86_64-linux5.0
23326 0 jenkins - 00:01 - - - sh
23352 0 jenkins - 00:01 - - - ps
23353 0 jenkins - 00:01 - - - tee
23362 0 jenkins - 00:01 - - - cp
25759 4 postfix - 01:31:29 - - - pickup
Raw
slab.txt
** this is 5min BEFORE the hang **
+ cat /proc/spl/kmem/slab
+ tee /zfs-data/jenkins/workspace/syslog.jenkins-slave2/slab.txt
--------------------- cache ------------------------------------------------------- ----- slab ------ ---- object ----- --- emergency ---
name flags size alloc slabsize objsize total alloc max total alloc max dlock alloc max
spl_vn_cache 0x00020 4096 1848 4096 88 1 1 1 21 21 21 0 0 0
spl_vn_file_cache 0x00020 0 0 4096 96 0 0 0 0 0 0 0 0 0
spl_zlib_workspace_cache 0x00140 0 0 8388608 268072 0 0 0 0 0 0 0 0 0
space_seg_cache 0x00020 2867200 1147008 4096 64 700 695 1072 26600 17922 40736 0 0 0
zio_cache 0x00020 9830400 309960 32768 1080 300 16 415 8700 287 12035 0 0 0
zio_link_cache 0x00020 819200 24336 4096 48 200 32 273 9000 507 12285 0 0 0
zio_vdev_cache 0x00040 8388608 6030048 4194304 131088 2 2 2 60 46 46 0 0 0
zio_buf_512 0x00020 176422912 66721280 32768 512 5384 5373 5781 166904 130315 179211 0 0 0
zio_data_buf_512 0x00020 23724032 11475456 32768 512 724 723 1292 22444 22413 40052 0 0 0
zio_buf_1024 0x00040 622592 342016 32768 1024 19 19 26 399 334 546 0 0 0
zio_data_buf_1024 0x00040 1376256 703488 32768 1024 42 42 51 882 687 1071 0 0 0
zio_buf_1536 0x00040 1048576 615936 65536 1536 16 15 18 496 401 558 0 0 0
zio_data_buf_1536 0x00040 2228224 1347072 65536 1536 34 34 51 1054 877 1581 0 0 0
zio_buf_2048 0x00040 1114112 579584 65536 2048 17 17 19 425 283 475 0 0 0
zio_data_buf_2048 0x00040 2949120 1687552 65536 2048 45 45 57 1125 824 1425 0 0 0
zio_buf_2560 0x00040 1179648 637440 65536 2560 18 18 127 378 249 2667 0 0 0
zio_data_buf_2560 0x00040 3211264 2183680 65536 2560 49 49 66 1029 853 1386 0 0 0
zio_buf_3072 0x00040 1376256 964608 65536 3072 21 19 22 378 314 396 0 0 0
zio_data_buf_3072 0x00040 3211264 1523712 65536 3072 49 49 56 882 496 1008 0 0 0
zio_buf_3584 0x00040 1703936 1214976 131072 3584 13 13 14 403 339 434 0 0 0
zio_data_buf_3584 0x00040 3801088 1727488 131072 3584 29 29 30 899 482 930 0 0 0
zio_buf_4096 0x00040 36700160 1003520 262144 4096 140 29 191 4340 245 5921 3 0 0
zio_data_buf_4096 0x00040 4194304 1499136 262144 4096 16 16 18 496 366 558 0 0 0
zio_buf_5120 0x00040 1048576 532480 131072 5120 8 8 11 168 104 231 0 0 0
zio_data_buf_5120 0x00040 3670016 2508800 131072 5120 28 28 34 588 490 714 0 0 0
zio_buf_6144 0x00040 1179648 798720 131072 6144 9 9 9 162 130 162 0 0 0
zio_data_buf_6144 0x00040 3801088 2187264 131072 6144 29 29 34 522 356 612 0 0 0
zio_buf_7168 0x00040 1572864 874496 262144 7168 6 6 6 186 122 186 0 0 0
zio_data_buf_7168 0x00040 3932160 2609152 262144 7168 15 15 20 465 364 620 0 0 0
zio_buf_8192 0x00040 9961472 1384448 262144 8192 38 18 70 798 169 1470 0 0 0
zio_data_buf_8192 0x00040 4718592 2310144 262144 8192 18 18 18 378 282 378 0 0 0
zio_buf_10240 0x00040 2359296 1280000 262144 10240 9 9 9 189 125 189 0 0 0
zio_data_buf_10240 0x00040 7340032 4884480 262144 10240 28 28 41 588 477 861 0 0 0
zio_buf_12288 0x00040 3670016 1880064 524288 12288 7 6 7 217 153 217 0 0 0
zio_data_buf_12288 0x00040 10485760 4841472 524288 12288 20 20 22 620 394 682 0 0 0
zio_buf_14336 0x00040 2621440 2207744 524288 14336 5 5 6 155 154 186 0 0 0
zio_data_buf_14336 0x00040 9961472 6522880 524288 14336 19 19 25 589 455 775 0 0 0
zio_buf_16384 0x00040 279969792 204242944 524288 16384 534 534 638 13350 12466 15950 0 0 0
zio_data_buf_16384 0x00040 9437184 5455872 524288 16384 18 18 21 450 333 525 0 0 0
zio_buf_20480 0x00040 4718592 2560000 524288 20480 9 9 10 189 125 210 0 0 0
zio_data_buf_20480 0x00040 20447232 12513280 524288 20480 39 39 55 819 611 1155 0 0 0
zio_buf_24576 0x00040 5767168 3293184 524288 24576 11 11 11 198 134 198 0 0 0
zio_data_buf_24576 0x00040 13631488 8650752 524288 24576 26 26 32 468 352 576 0 0 0
zio_buf_28672 0x00040 7340032 4386816 1048576 28672 7 7 7 217 153 217 0 0 0
zio_data_buf_28672 0x00040 11534336 6107136 1048576 28672 11 11 11 341 213 341 0 0 0
zio_buf_32768 0x00040 6291456 4456448 1048576 32768 6 6 6 168 136 168 0 0 0
zio_data_buf_32768 0x00040 9437184 7208960 1048576 32768 9 9 9 252 220 252 0 0 0
zio_buf_36864 0x00040 6291456 5529600 1048576 36864 6 6 6 150 150 150 0 0 0
zio_data_buf_36864 0x00040 9437184 7114752 1048576 36864 9 9 9 225 193 225 0 0 0
zio_buf_40960 0x00040 8388608 6225920 1048576 40960 8 8 8 184 152 184 0 0 0
zio_data_buf_40960 0x00040 11534336 9052160 1048576 40960 11 11 11 253 221 253 0 0 0
zio_buf_45056 0x00040 8388608 3244032 1048576 45056 8 8 8 168 72 168 0 0 0
zio_data_buf_45056 0x00040 9437184 7073792 1048576 45056 9 9 9 189 157 189 0 0 0
zio_buf_49152 0x00040 9437184 6832128 1048576 49152 9 9 9 171 139 171 0 0 0
zio_data_buf_49152 0x00040 8388608 7471104 1048576 49152 8 8 8 152 152 152 0 0 0
zio_buf_53248 0x00040 9437184 6922240 1048576 53248 9 9 9 162 130 162 0 0 0
zio_data_buf_53248 0x00040 9437184 6922240 1048576 53248 9 9 9 162 130 162 0 0 0
zio_buf_57344 0x00040 10485760 7913472 1048576 57344 10 10 11 170 138 187 0 0 0
zio_data_buf_57344 0x00040 10485760 9748480 1048576 57344 10 10 10 170 170 170 0 0 0
zio_buf_61440 0x00040 12582912 9461760 2097152 61440 6 6 6 186 154 186 0 0 0
zio_data_buf_61440 0x00040 12582912 11427840 2097152 61440 6 6 6 186 186 186 0 0 0
zio_buf_65536 0x00040 12582912 9699328 2097152 65536 6 6 6 180 148 180 0 0 0
zio_data_buf_65536 0x00040 10485760 9830400 2097152 65536 5 5 5 150 150 150 0 0 0
zio_buf_69632 0x00040 12582912 9469952 2097152 69632 6 6 6 168 136 168 0 0 0
zio_data_buf_69632 0x00040 10485760 9748480 2097152 69632 5 5 5 140 140 140 0 0 0
zio_buf_73728 0x00040 12582912 11501568 2097152 73728 6 6 6 156 156 156 0 0 0
zio_data_buf_73728 0x00040 10485760 9584640 2097152 73728 5 5 5 130 130 130 0 0 0
zio_buf_77824 0x00040 12582912 11673600 2097152 77824 6 6 6 150 150 150 0 0 0
zio_data_buf_77824 0x00040 14680064 11673600 2097152 77824 7 6 6 175 150 150 0 0 0
zio_buf_81920 0x00040 10485760 9830400 2097152 81920 5 5 5 120 120 120 0 0 0
zio_data_buf_81920 0x00040 16777216 13107200 2097152 81920 8 8 8 192 160 192 0 0 0
zio_buf_86016 0x00040 8388608 7913472 2097152 86016 4 4 4 92 92 92 0 0 0
zio_data_buf_86016 0x00040 12582912 11870208 2097152 86016 6 6 6 138 138 138 0 0 0
zio_buf_90112 0x00040 8388608 7929856 2097152 90112 4 4 4 88 88 88 0 0 0
zio_data_buf_90112 0x00040 12582912 11894784 2097152 90112 6 6 6 132 132 132 0 0 0
zio_buf_94208 0x00040 10485760 9891840 2097152 94208 5 5 5 105 105 105 0 0 0
zio_data_buf_94208 0x00040 8388608 7913472 2097152 94208 4 4 4 84 84 84 0 0 0
zio_buf_98304 0x00040 10485760 9830400 2097152 98304 5 5 5 100 100 100 1 0 0
zio_data_buf_98304 0x00040 10485760 9830400 2097152 98304 5 5 5 100 100 100 0 0 0
zio_buf_102400 0x00040 6291456 5836800 2097152 102400 3 3 3 57 57 57 0 0 0
zio_data_buf_102400 0x00040 8388608 7782400 2097152 102400 4 4 4 76 76 76 0 0 0
zio_buf_106496 0x00040 6291456 5750784 2097152 106496 3 3 3 54 54 54 0 0 0
zio_data_buf_106496 0x00040 8388608 7667712 2097152 106496 4 4 4 72 72 72 0 0 0
zio_buf_110592 0x00040 8388608 7962624 2097152 110592 4 4 4 72 72 72 0 0 0
zio_data_buf_110592 0x00040 10485760 9953280 2097152 110592 5 5 5 90 90 90 0 0 0
zio_buf_114688 0x00040 8388608 7798784 2097152 114688 4 4 4 68 68 68 0 0 0
zio_data_buf_114688 0x00040 8388608 7798784 2097152 114688 4 4 4 68 68 68 0 0 0
zio_buf_118784 0x00040 4194304 4038656 2097152 118784 2 2 2 34 34 34 0 0 0
zio_data_buf_118784 0x00040 8388608 8077312 2097152 118784 4 4 4 68 68 68 0 0 0
zio_buf_122880 0x00040 8388608 7864320 2097152 122880 4 4 4 64 64 64 0 0 0
zio_data_buf_122880 0x00040 8388608 7864320 2097152 122880 4 4 4 64 64 64 0 0 0
zio_buf_126976 0x00040 16777216 15745024 4194304 126976 4 4 4 124 124 124 0 0 0
zio_data_buf_126976 0x00040 16777216 15745024 4194304 126976 4 4 4 124 124 124 0 0 0
zio_buf_131072 0x00040 37748736 6291456 4194304 131072 9 5 11 279 48 326 0 0 0
zio_data_buf_131072 0x00040 809500672 781189120 4194304 131072 193 193 349 5983 5960 10812 0 0 0
lz4_cache 0x00040 2097152 2031616 524288 16384 4 4 4 124 124 124 0 0 0
sa_cache 0x00020 9199616 5926800 4096 80 2246 2245 3848 74118 74085 126984 0 0 0
spill_cache 0x00040 0 0 4194304 131072 0 0 0 0 0 0 0 0 0
dnode_t 0x00020 194494464 154805248 16384 928 11871 11758 15555 189936 166816 248880 0 0 0
dmu_buf_impl_t 0x00020 70598656 49356720 8192 280 8618 8558 9442 215450 176274 236050 0 0 0
arc_buf_hdr_t 0x00020 58032128 48617712 8192 264 7084 7083 7083 184184 184158 184158 0 0 0
arc_buf_t 0x00020 7290880 4150368 4096 72 1780 1779 2796 64080 57644 100656 0 0 0
zil_lwb_cache 0x00020 102400 76800 4096 200 25 24 29 400 384 464 0 0 0
zfs_znode_cache 0x00020 80969728 76471200 32768 1032 2471 2470 4233 74130 74100 126990 0 0 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment