Skip to content

Instantly share code, notes, and snippets.

@kohsuke
Created September 20, 2013 22:18
Show Gist options
  • Save kohsuke/6644657 to your computer and use it in GitHub Desktop.
Save kohsuke/6644657 to your computer and use it in GitHub Desktop.
[91195.388948] br0: port 1(tap0) entering forwarding state
[91280.260024] BUG: soft lockup - CPU#0 stuck for 22s! [zvol/16:309]
[91280.261103] Modules linked in: xt_conntrack tun usb_storage netconsole configfs nfsd iptable_filter ipt_MASQUERADE xt_comment xt_tcpudp iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack ip_tables bridge stp llc ip6table_filter ip6_tables x_tables nls_cp437 vfat fat zfs(P) zunicode(P) zavl(P) zcommon(P) znvpair(P) spl zlib_deflate arc4 b43 mac80211 radeon cfg80211 i2c_algo_bit ttm snd_hda_codec_hdmi drm_kms_helper drm snd_hda_codec_realtek ssb btusb bluetooth rfkill mmc_core pcmcia pcspkr i2c_i801 snd_hda_intel snd_hda_codec snd_hwdep snd_pcm i2c_core snd_page_alloc snd_timer applesmc snd ioatdma pcmcia_core input_polldev soundcore shpchp pci_hotplug evdev e1000e iTCO_wdt iTCO_vendor_support i7core_edac edac_core processor button dca nfs nfs_acl auth_rpcgss fscache lockd sunrpc kvm_intel kvm ext4 crc16 jbd2 mbcache sr_mod sd_mod cdrom uhci_hcd ahci libahci libata firewire_ohci ehci_hcd scsi_mod firewire_core crc_itu_t usbcore
[91280.271391] CPU 0
[91280.271405] Modules linked in: xt_conntrack tun usb_storage netconsole configfs nfsd iptable_filter ipt_MASQUERADE xt_comment xt_tcpudp iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack ip_tables bridge stp llc ip6table_filter ip6_tables x_tables nls_cp437 vfat fat zfs(P) zunicode(P) zavl(P) zcommon(P) znvpair(P) spl zlib_deflate arc4 b43 mac80211 radeon cfg80211 i2c_algo_bit ttm snd_hda_codec_hdmi drm_kms_helper drm snd_hda_codec_realtek ssb btusb bluetooth rfkill mmc_core pcmcia pcspkr i2c_i801 snd_hda_intel snd_hda_codec snd_hwdep snd_pcm i2c_core snd_page_alloc snd_timer applesmc snd ioatdma pcmcia_core input_polldev soundcore shpchp pci_hotplug evdev e1000e iTCO_wdt iTCO_vendor_support i7core_edac edac_core processor button dca nfs nfs_acl auth_rpcgss fscache lockd sunrpc kvm_intel kvm ext4 crc16 jbd2 mbcache sr_mod sd_mod cdrom uhci_hcd ahci libahci libata firewire_ohci ehci_hcd scsi_mod firewire_core crc_itu_t usbcore
[91280.284678]
[91280.286193] Pid: 309, comm: zvol/16 Tainted: P 3.0.84-1-lts #1 Apple Inc. MacPro5,1/Mac-F221BEC8
[91280.287782] RIP: 0010:[<ffffffff8141858b>] [<ffffffff8141858b>] __mutex_lock_slowpath+0x20b/0x2d0
[91280.289422] RSP: 0018:ffff880465df1aa0 EFLAGS: 00000246
[91280.291064] RAX: ffff880465bdca88 RBX: 0000000000015ab9 RCX: ffff880465bdca88
[91280.292782] RDX: ffff880465bdca88 RSI: ffff880465df1ac0 RDI: 0000000000000246
[91280.294393] RBP: ffff880465df1b10 R08: ffff880487212940 R09: 0000000000000246
[91280.295997] R10: 0000000000015ab9 R11: 0000000000012940 R12: ffffffff81421dce
[91280.297626] R13: ffff880465df1a90 R14: ffffffff8141990e R15: 0000000000000000
[91280.299265] FS: 0000000000000000(0000) GS:ffff880487200000(0000) knlGS:0000000000000000
[91280.300954] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[91280.302762] CR2: 000000010e6c5000 CR3: 0000000002893000 CR4: 00000000000026e0
[91280.304551] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[91280.306252] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[91280.307964] Process zvol/16 (pid: 309, threadinfo ffff880465df0000, task ffff880465dcf710)
[91280.309701] Stack:
[91280.311466] 0000000000000000 00000016d8f00000 ffff880465bdca88 0000000000000246
[91280.313277] ffff880465df1ac0 ffff880465df1ac0 0000000000000000 ffff880465df1ac0
[91280.315351] 0000000000000000 ffff880465bdca80 ffff880465bdca80 0000000000020000
[91280.317171] Call Trace:
[91280.318957] [<ffffffff81418666>] mutex_lock+0x16/0x30
[91280.320792] [<ffffffffa07ec138>] metaslab_alloc+0x328/0xb80 [zfs]
[91280.322595] [<ffffffff81421196>] ? ftrace_call+0x5/0x2b
[91280.324460] [<ffffffffa084e677>] zio_alloc_zil+0x57/0x170 [zfs]
[91280.326486] [<ffffffffa0847ae9>] zil_lwb_write_start+0x2f9/0x3a0 [zfs]
[91280.328307] [<ffffffffa0847f93>] zil_commit.part.12+0x403/0x720 [zfs]
[91280.330121] [<ffffffff81421196>] ? ftrace_call+0x5/0x2b
[91280.331949] [<ffffffffa0848887>] zil_commit+0x17/0x20 [zfs]
[91280.333768] [<ffffffffa0856132>] zvol_write+0x192/0x480 [zfs]
[91280.335578] [<ffffffffa00a8bcf>] taskq_thread+0x23f/0x4c0 [spl]
[91280.337379] [<ffffffff8104fc70>] ? wake_up_process+0x50/0x50
[91280.339154] [<ffffffffa00a8990>] ? taskq_cancel_id+0x1f0/0x1f0 [spl]
[91280.340910] [<ffffffff8107a0ec>] kthread+0x8c/0xa0
[91280.342632] [<ffffffff81422524>] kernel_thread_helper+0x4/0x10
[91280.344329] [<ffffffff8107a060>] ? kthread_flush_work_fn+0x20/0x20
[91280.345997] [<ffffffff81422520>] ? gs_change+0x13/0x13
[91280.347698] Code: 65 48 8b 04 25 80 cd 00 00 49 89 47 18 48 8b 45 a0 49 3b 47 08 4c 8b 4d a8 75 07 41 c7 07 00 00 00 00 41 fe 47 04 4c 89 cf 57 9d
[91280.349631] 66 90 66 90 48 8d 7d b0 e8 87 4f c7 ff 48 83 c4 48 5b 41 5c
[91280.353192] Call Trace:
[91280.354960] [<ffffffff81418666>] mutex_lock+0x16/0x30
[91280.356701] [<ffffffffa07ec138>] metaslab_alloc+0x328/0xb80 [zfs]
[91280.358587] [<ffffffff81421196>] ? ftrace_call+0x5/0x2b
[91280.360338] [<ffffffffa084e677>] zio_alloc_zil+0x57/0x170 [zfs]
[91280.361999] [<ffffffffa0847ae9>] zil_lwb_write_start+0x2f9/0x3a0 [zfs]
[91280.363638] [<ffffffffa0847f93>] zil_commit.part.12+0x403/0x720 [zfs]
[91280.365243] [<ffffffff81421196>] ? ftrace_call+0x5/0x2b
[91280.366845] [<ffffffffa0848887>] zil_commit+0x17/0x20 [zfs]
[91280.368439] [<ffffffffa0856132>] zvol_write+0x192/0x480 [zfs]
[91280.370029] [<ffffffffa00a8bcf>] taskq_thread+0x23f/0x4c0 [spl]
[91280.371985] [<ffffffff8104fc70>] ? wake_up_process+0x50/0x50
[91280.373738] [<ffffffffa00a8990>] ? taskq_cancel_id+0x1f0/0x1f0 [spl]
[91280.375287] [<ffffffff8107a0ec>] kthread+0x8c/0xa0
[91280.376810] [<ffffffff81422524>] kernel_thread_helper+0x4/0x10
[91280.378527] [<ffffffff8107a060>] ? kthread_flush_work_fn+0x20/0x20
[91280.380058] [<ffffffff81422520>] ? gs_change+0x13/0x13
[91402.924818] INFO: task txg_sync:560 blocked for more than 120 seconds.
[91402.926922] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[91402.929040] txg_sync D ffff88046f2e27b0 0 560 2 0x00000000
[91402.931159] ffff880460521790 0000000000000046 ffff880400000000 ffff880460521fd8
[91402.933289] 0000000000012940 ffff880460521fd8 0000000000012940 ffff88046ce34770
[91402.935469] 0000000000000000 0000000000000001 0000000000000000 ffffffff00000000
[91402.937591] Call Trace:
[91402.939678] [<ffffffff81421196>] ? ftrace_call+0x5/0x2b
[91402.941762] [<ffffffff81421196>] ? ftrace_call+0x5/0x2b
[91402.943793] [<ffffffff814170ef>] schedule+0x3f/0x60
[91402.945847] [<ffffffff814172cf>] io_schedule+0x8f/0xe0
[91402.947852] [<ffffffffa00ae846>] cv_wait_common+0xa6/0x1c0 [spl]
[91402.949855] [<ffffffff8107aae0>] ? wake_up_bit+0x30/0x30
[91402.951840] [<ffffffffa00ae9b8>] __cv_wait_io+0x18/0x20 [spl]
[91402.953831] [<ffffffffa084d30b>] zio_wait+0x11b/0x1f0 [zfs]
[91402.955872] [<ffffffffa07b88fd>] dmu_buf_hold_array_by_dnode+0x1fd/0x610 [zfs]
[91402.957892] [<ffffffffa07b8ded>] dmu_buf_hold_array+0x5d/0x80 [zfs]
[91402.959897] [<ffffffffa07b9284>] dmu_write.part.7+0x44/0x150 [zfs]
[91402.961890] [<ffffffff81421196>] ? ftrace_call+0x5/0x2b
[91402.963882] [<ffffffffa07b93a5>] dmu_write+0x15/0x20 [zfs]
[91402.965933] [<ffffffffa0804ca2>] space_map_sync+0x382/0x540 [zfs]
[91402.967950] [<ffffffffa07ea7f0>] metaslab_condense+0x1b0/0x320 [zfs]
[91402.969944] [<ffffffff8113b7ed>] ? __kmalloc+0x12d/0x160
[91402.971936] [<ffffffffa00a2776>] ? kmem_alloc_debug+0x256/0x500 [spl]
[91402.973957] [<ffffffffa0803760>] ? spa_debug_enabled+0x20/0x20 [zfs]
[91402.976034] [<ffffffffa07eb66c>] metaslab_sync+0x29c/0x380 [zfs]
[91402.978065] [<ffffffffa080a3e3>] vdev_sync+0x73/0x140 [zfs]
[91402.980090] [<ffffffffa07f68bb>] spa_sync+0x49b/0xb10 [zfs]
[91402.982092] [<ffffffff81421196>] ? ftrace_call+0x5/0x2b
[91402.984029] [<ffffffffa080623d>] txg_sync_thread+0x33d/0x5c0 [zfs]
[91402.985917] [<ffffffffa0805f00>] ? txg_delay+0x160/0x160 [zfs]
[91402.987709] [<ffffffffa00a6c0a>] thread_generic_wrapper+0x7a/0x90 [spl]
[91402.989472] [<ffffffffa00a6b90>] ? __thread_exit+0xa0/0xa0 [spl]
[91402.991222] [<ffffffff8107a0ec>] kthread+0x8c/0xa0
[91402.992962] [<ffffffff81422524>] kernel_thread_helper+0x4/0x10
[91402.994715] [<ffffffff8107a060>] ? kthread_flush_work_fn+0x20/0x20
[91402.996462] [<ffffffff81422520>] ? gs_change+0x13/0x13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment