Skip to content

Instantly share code, notes, and snippets.

@crowtrobot

crowtrobot/dmesg

Created April 12, 2014 23:31
Show Gist options
  • Save crowtrobot/10562026 to your computer and use it in GitHub Desktop.
Save crowtrobot/10562026 to your computer and use it in GitHub Desktop.
Download ZIP
second run dmesg
Raw
dmesg
[ 14.779694] Loading iSCSI transport class v2.0-870.
[ 14.782311] iscsi: registered transport (tcp)
[ 14.809085] Bluetooth: Core ver 2.16
[ 14.809106] NET: Registered protocol family 31
[ 14.809107] Bluetooth: HCI device and connection manager initialized
[ 14.809114] Bluetooth: HCI socket layer initialized
[ 14.809117] Bluetooth: L2CAP socket layer initialized
[ 14.809123] Bluetooth: SCO socket layer initialized
[ 14.815544] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[ 14.815547] Bluetooth: BNEP filters: protocol multicast
[ 14.815553] Bluetooth: BNEP socket layer initialized
[ 14.816541] Bluetooth: RFCOMM TTY layer initialized
[ 14.816550] Bluetooth: RFCOMM socket layer initialized
[ 14.816552] Bluetooth: RFCOMM ver 1.11
[ 14.819079] iscsi: registered transport (iser)
[ 14.832933] ppdev: user-space parallel port driver
[ 14.833348] init: avahi-cups-reload main process (1576) terminated with status 1
[ 14.957063] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[ 14.957322] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[ 15.047090] r8169 0000:0a:00.0 eth0: link down
[ 15.047134] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 15.047724] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 15.216559] init: udev-fallback-graphics main process (1800) terminated with status 1
[ 15.574340] init: mdm main process (2368) killed by TERM signal
[ 15.699364] Rounding down aligned max_sectors from 4294967295 to 4294967288
[ 16.750321] Rounding down aligned max_sectors from 4294967295 to 4294967288
[ 16.901575] qla2xxx [0000:00:00.0]-0005: : QLogic Fibre Channel HBA Driver: 8.04.00.08-k.
[ 17.108857] vboxdrv: Found 6 processor cores.
[ 17.109063] vboxdrv: fAsync=0 offMin=0x4ab offMax=0x3512
[ 17.109121] vboxdrv: TSC mode is 'synchronous', kernel timer mode is 'normal'.
[ 17.109123] vboxdrv: Successfully loaded version 4.3.10 (interface 0x001a0007).
[ 17.316861] vboxpci: IOMMU found
[ 17.693774] init: plymouth-stop pre-start process (2960) terminated with status 1
[ 18.642305] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[ 18.642484] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[ 64.609978] init: tty1 main process ended, respawning
[ 201.270536] init: smbd main process (1491) killed by TERM signal
[ 313.868626] ------------[ cut here ]------------
[ 313.868637] WARNING: at /build/buildd/linux-3.8.0/kernel/watchdog.c:246 watchdog_overflow_callback+0x9c/0xd0()
[ 313.868642] Hardware name: To be filled by O.E.M.
[ 313.868645] Watchdog detected hard LOCKUP on cpu 1
[ 313.868647] Modules linked in: pci_stub vboxpci(OF) vboxnetadp(OF) vboxnetflt(OF) vboxdrv(OF) ib_srpt tcm_qla2xxx qla2xxx tcm_loop tcm_fc libfc scsi_transport_fc(F) scsi_tgt(F) iscsi_target_mod target_core_pscsi target_core_file target_core_iblock target_core_mod configfs(F) parport_pc(F) ppdev(F) ib_iser rdma_cm bnep ib_addr rfcomm iw_cm bluetooth ib_cm ib_sa ib_mad ib_core iscsi_tcp(F) libiscsi_tcp(F) libiscsi(F) scsi_transport_iscsi(F) nfsv4(F) binfmt_misc(F) nfsd(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lockd(F) sunrpc(F) fscache(F) snd_hda_codec_hdmi eeepc_wmi asus_wmi sparse_keymap video(F) mxm_wmi joydev(F) dm_multipath(F) scsi_dh(F) kvm_amd(F) kvm(F) ghash_clmulni_intel(F) aesni_intel(F) aes_x86_64(F) xts(F) lrw(F) gf128mul(F) ablk_helper(F) cryptd(F) microcode(F) psmouse(F) serio_raw(F) snd_hda_codec_realtek edac_core edac_mce_amd fam15h_power k10temp snd_hda_intel snd_hda_codec sp5100_tco snd_hwdep(F) snd_pcm(F) i2c_piix4 snd_seq_midi(F) snd_seq_midi_event(F) snd_page_alloc(F) snd_rawmidi(F) snd_seq(F) snd_seq_device(F) snd_timer(F) snd(F) nvidia(POF) soundcore(F) mac_hid wmi it87 hwmon_vid lp(F) parport(F) zfs(POF) zunicode(POF) zavl(POF) zcommon(POF) znvpair(POF) spl(OF) xfs(F) btrfs(F) zlib_deflate(F) libcrc32c(F) dm_raid45 xor(F) dm_mirror(F) dm_region_hash(F) dm_log(F) hid_generic hid_microsoft usbhid hid usb_storage(F) ahci(F) libahci(F) r8169 e1000e(F)
[ 313.868774] Pid: 6459, comm: zfs Tainted: PF O 3.8.0-35-generic #50-Ubuntu
[ 313.868778] Call Trace:
[ 313.868781] <NMI> [<ffffffff81058a3f>] warn_slowpath_common+0x7f/0xc0
[ 313.868795] [<ffffffff81058b3c>] warn_slowpath_fmt+0x4c/0x50
[ 313.868803] [<ffffffff8101bb89>] ? sched_clock+0x9/0x10
[ 313.868809] [<ffffffff810e9a30>] ? touch_nmi_watchdog+0x80/0x80
[ 313.868815] [<ffffffff810e9acc>] watchdog_overflow_callback+0x9c/0xd0
[ 313.868822] [<ffffffff81126c5d>] __perf_event_overflow+0x9d/0x230
[ 313.868828] [<ffffffff81123dc4>] ? perf_event_update_userpage+0x24/0x110
[ 313.868834] [<ffffffff81127874>] perf_event_overflow+0x14/0x20
[ 313.868840] [<ffffffff8102582f>] x86_pmu_handle_irq+0xff/0x150
[ 313.868849] [<ffffffff816cfc6d>] perf_event_nmi_handler+0x1d/0x20
[ 313.868855] [<ffffffff816cf419>] nmi_handle.isra.0+0x59/0x90
[ 313.868862] [<ffffffff816cf5d9>] do_nmi+0x189/0x360
[ 313.868868] [<ffffffff816cea41>] end_repeat_nmi+0x1e/0x2e
[ 313.868892] [<ffffffffa02d183b>] ? kmem_free_debug+0x4b/0x150 [spl]
[ 313.868901] [<ffffffff81044ac2>] ? __ticket_spin_lock+0x22/0x30
[ 313.868907] [<ffffffff81044ac2>] ? __ticket_spin_lock+0x22/0x30
[ 313.868913] [<ffffffff81044ac2>] ? __ticket_spin_lock+0x22/0x30
[ 313.868916] <<EOE>> [<ffffffff816cde65>] _raw_spin_lock_irq+0x15/0x20
[ 313.868927] [<ffffffff816cdca6>] rwsem_down_failed_common+0x46/0x170
[ 313.868933] [<ffffffff816cb7ad>] ? mutex_lock+0x1d/0x50
[ 313.868939] [<ffffffff816cde05>] rwsem_down_read_failed+0x15/0x17
[ 313.868946] [<ffffffff813555d4>] call_rwsem_down_read_failed+0x14/0x30
[ 313.868967] [<ffffffffa02d183b>] ? kmem_free_debug+0x4b/0x150 [spl]
[ 313.868977] [<ffffffff816cc0e0>] ? down_read+0x20/0x23
[ 313.869055] [<ffffffffa03b7210>] zap_get_leaf_byblk+0xd0/0x2c0 [zfs]
[ 313.869122] [<ffffffffa03b6c0b>] ? zap_idx_to_blk+0x13b/0x150 [zfs]
[ 313.869190] [<ffffffffa03b746d>] zap_deref_leaf+0x6d/0x80 [zfs]
[ 313.869258] [<ffffffffa03b9021>] fzap_cursor_retrieve+0xe1/0x280 [zfs]
[ 313.869325] [<ffffffffa03bc848>] zap_cursor_retrieve+0x68/0x2e0 [zfs]
[ 313.869387] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 313.869454] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 313.869526] [<ffffffffa03b86d1>] zap_value_search+0x91/0xd0 [zfs]
[ 313.869591] [<ffffffffa03e779d>] zfs_obj_to_path_impl+0x1bd/0x2b0 [zfs]
[ 313.869655] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 313.869718] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 313.869782] [<ffffffffa03e3261>] ? zfs_obj_to_stats_impl+0xa1/0xb0 [zfs]
[ 313.869848] [<ffffffffa03e79eb>] zfs_obj_to_stats+0xab/0xd0 [zfs]
[ 313.869914] [<ffffffffa03ca6d4>] zfs_ioc_obj_to_stats+0x84/0xa0 [zfs]
[ 313.869979] [<ffffffffa03d0879>] zfsdev_ioctl+0x4b9/0x530 [zfs]
[ 313.869986] [<ffffffff811a6659>] do_vfs_ioctl+0x99/0x570
[ 313.869993] [<ffffffff811a6bc1>] sys_ioctl+0x91/0xb0
[ 313.869999] [<ffffffff816d671d>] system_call_fastpath+0x1a/0x1f
[ 313.870004] ---[ end trace 91c7ca51c92e39b0 ]---
[ 336.940107] INFO: rcu_sched detected stalls on CPUs/tasks: { 1} (detected by 5, t=15002 jiffies, g=32254, c=32253, q=17734)
[ 336.940118] sending NMI to all CPUs:
[ 336.940140] NMI backtrace for cpu 5
[ 336.940144] CPU 5
[ 336.940153] Pid: 0, comm: swapper/5 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 336.940159] RIP: 0010:[<ffffffff81354161>] [<ffffffff81354161>] __const_udelay+0x1/0x30
[ 336.940172] RSP: 0018:ffff88063ed43dd8 EFLAGS: 00000046
[ 336.940178] RAX: 0000000000000000 RBX: 0000000000002710 RCX: 0000000000000004
[ 336.940182] RDX: 0000000000000c00 RSI: 0000000000000100 RDI: 0000000000418958
[ 336.940187] RBP: ffff88063ed43df0 R08: 000000000000000a R09: 0000000000000000
[ 336.940191] R10: 0000000000000479 R11: 0000000000000002 R12: ffffffff81ce70c0
[ 336.940196] R13: ffff88063ed4ee60 R14: ffffffff81c3f3c0 R15: ffffffff81c3f480
[ 336.940202] FS: 00007fa534ff9700(0000) GS:ffff88063ed40000(0000) knlGS:0000000000000000
[ 336.940207] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 336.940211] CR2: 00007f8d1701c1ac CR3: 00000005e1941000 CR4: 00000000000407e0
[ 336.940216] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 336.940221] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 336.940226] Process swapper/5 (pid: 0, threadinfo ffff88061cd8a000, task ffff88061cd90000)
[ 336.940230] Stack:
[ 336.940234] ffff88063ed43df0 ffffffff81039d6a ffffffff81c3f3c0 ffff88063ed43e50
[ 336.940245] ffffffff810f2935 0000000000004546 0000000000000082 0000000100002488
[ 336.940255] 0000000000000005 ffff88061cd8a000 ffff88061cd90000 0000000000000005
[ 336.940265] Call Trace:
[ 336.940270] <IRQ>
[ 336.940274] [<ffffffff81039d6a>] ? arch_trigger_all_cpu_backtrace+0x6a/0x90
[ 336.940292] [<ffffffff810f2935>] rcu_pending+0x565/0x5b0
[ 336.940301] [<ffffffff810f4a0b>] rcu_check_callbacks+0x9b/0x120
[ 336.940311] [<ffffffff8106a928>] update_process_times+0x48/0x90
[ 336.940319] [<ffffffff810b3a8e>] tick_sched_handle.isra.8+0x2e/0x70
[ 336.940327] [<ffffffff810b3c1c>] tick_sched_timer+0x4c/0x80
[ 336.940336] [<ffffffff810819d9>] __run_hrtimer+0x79/0x1c0
[ 336.940344] [<ffffffff810b3bd0>] ? tick_sched_do_timer+0x40/0x40
[ 336.940353] [<ffffffff810822c7>] hrtimer_interrupt+0xe7/0x230
[ 336.940363] [<ffffffff81574a50>] ? centrino_target+0x370/0x370
[ 336.940373] [<ffffffff816d8519>] smp_apic_timer_interrupt+0x69/0x99
[ 336.940382] [<ffffffff816d745d>] apic_timer_interrupt+0x6d/0x80
[ 336.940386] <EOI>
[ 336.940390] [<ffffffff81091a9d>] ? sched_clock_cpu+0xbd/0x110
[ 336.940404] [<ffffffff815754a8>] ? cpuidle_wrap_enter+0x58/0xa0
[ 336.940414] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 336.940422] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 336.940431] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 336.940440] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 336.940445] Code: ff c8 5d c3 66 0f 1f 44 00 00 66 66 66 66 90 55 48 89 e5 ff 15 49 25 92 00 5d c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 <48> 8d 04 bd 00 00 00 00 65 48 8b 14 25 18 3e 01 00 48 8d 0c 12
[ 336.940551] NMI backtrace for cpu 4
[ 336.940555] CPU 4
[ 336.940562] Pid: 0, comm: swapper/4 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 336.940567] RIP: 0010:[<ffffffff813e3eb7>] [<ffffffff813e3eb7>] acpi_idle_do_entry+0x21/0x2b
[ 336.940578] RSP: 0018:ffff88061cd89e10 EFLAGS: 00000093
[ 336.940583] RAX: 0000004e72f64d00 RBX: ffff88061ce310a0 RCX: 000000000000c648
[ 336.940587] RDX: 0000000000000815 RSI: ffff88063ed00000 RDI: ffff88061ce310a0
[ 336.940591] RBP: ffff88061cd89e10 R08: 000000000000333c R09: 0000000000000018
[ 336.940595] R10: 0000000000000937 R11: 0000000000000000 R12: 0000000000000002
[ 336.940600] R13: ffff88061cd88000 R14: ffff88061ce31000 R15: ffffffff81574a50
[ 336.940605] FS: 00007fa5167fc700(0000) GS:ffff88063ed00000(0000) knlGS:0000000000000000
[ 336.940610] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 336.940614] CR2: 00007f8c9c002068 CR3: 00000005e1941000 CR4: 00000000000407e0
[ 336.940619] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 336.940623] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 336.940628] Process swapper/4 (pid: 0, threadinfo ffff88061cd88000, task ffff88061cd7dd00)
[ 336.940631] Stack:
[ 336.940634] ffff88061cd89e40 ffffffff813e3faa ffff88061ca45600 ffffffff81c7e060
[ 336.940646] 0000004e8e6093ea 0000000000000002 ffff88061cd89e50 ffffffff81574a69
[ 336.940657] ffff88061cd89eb0 ffffffff81575499 7735940000000001 ffff88061ca45600
[ 336.940668] Call Trace:
[ 336.940677] [<ffffffff813e3faa>] acpi_idle_enter_simple+0x87/0xc5
[ 336.940686] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 336.940694] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 336.940703] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 336.940711] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 336.940720] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 336.940729] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 336.940734] Code: 00 83 88 3c e0 ff ff 04 5d c3 8a 47 08 55 48 89 e5 3c 01 75 07 e8 3a 35 c5 ff eb 17 3c 02 75 07 e8 a2 ff ff ff eb 0c 8b 57 04 ec <48> 8b 15 86 e1 b2 00 ed 5d c3 66 66 66 66 90 55 48 63 c2 48 c1
[ 336.940854] NMI backtrace for cpu 1
[ 336.940861] CPU 1
[ 336.940871] Pid: 6459, comm: zfs Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 336.940877] RIP: 0010:[<ffffffff81044ac2>] [<ffffffff81044ac2>] __ticket_spin_lock+0x22/0x30
[ 336.940892] RSP: 0018:ffff8805d42e3958 EFLAGS: 00000006
[ 336.940898] RAX: 0000000000002d4b RBX: ffff8805dce22e80 RCX: 0000000000000000
[ 336.940903] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88051fac2488
[ 336.940907] RBP: ffff8805d42e3958 R08: 0000000000016d40 R09: ffff88063ec56d40
[ 336.940912] R10: ffffea0017696200 R11: ffffffffa02d183b R12: ffff88051fac2480
[ 336.940917] R13: ffff88051fac2488 R14: ffffffffffffffff R15: 0000000000000001
[ 336.940923] FS: 00007f1ac9a97700(0000) GS:ffff88063ec40000(0000) knlGS:0000000000000000
[ 336.940927] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 336.940932] CR2: 00007f2a6f2e0000 CR3: 00000005ddf2a000 CR4: 00000000000407e0
[ 336.940937] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 336.940941] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 336.940947] Process zfs (pid: 6459, threadinfo ffff8805d42e2000, task ffff8805dce22e80)
[ 336.940950] Stack:
[ 336.940954] ffff8805d42e3968 ffffffff816cde65 ffff8805d42e39d0 ffffffff816cdca6
[ 336.940966] 0000000000000000 0000000000000000 00000000d42e39d0 ffffffff00000000
[ 336.940976] ffffffff816cb7ad ffff8805cc05b970 ffff88051fac2480 0000000000000002
[ 336.940986] Call Trace:
[ 336.940998] [<ffffffff816cde65>] _raw_spin_lock_irq+0x15/0x20
[ 336.941007] [<ffffffff816cdca6>] rwsem_down_failed_common+0x46/0x170
[ 336.941016] [<ffffffff816cb7ad>] ? mutex_lock+0x1d/0x50
[ 336.941025] [<ffffffff816cde05>] rwsem_down_read_failed+0x15/0x17
[ 336.941035] [<ffffffff813555d4>] call_rwsem_down_read_failed+0x14/0x30
[ 336.941077] [<ffffffffa02d183b>] ? kmem_free_debug+0x4b/0x150 [spl]
[ 336.941091] [<ffffffff816cc0e0>] ? down_read+0x20/0x23
[ 336.941175] [<ffffffffa03b7210>] zap_get_leaf_byblk+0xd0/0x2c0 [zfs]
[ 336.941248] [<ffffffffa03b6c0b>] ? zap_idx_to_blk+0x13b/0x150 [zfs]
[ 336.941321] [<ffffffffa03b746d>] zap_deref_leaf+0x6d/0x80 [zfs]
[ 336.941393] [<ffffffffa03b9021>] fzap_cursor_retrieve+0xe1/0x280 [zfs]
[ 336.941466] [<ffffffffa03bc848>] zap_cursor_retrieve+0x68/0x2e0 [zfs]
[ 336.941534] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 336.941602] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 336.941678] [<ffffffffa03b86d1>] zap_value_search+0x91/0xd0 [zfs]
[ 336.941750] [<ffffffffa03e779d>] zfs_obj_to_path_impl+0x1bd/0x2b0 [zfs]
[ 336.941820] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 336.941888] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 336.941958] [<ffffffffa03e3261>] ? zfs_obj_to_stats_impl+0xa1/0xb0 [zfs]
[ 336.942030] [<ffffffffa03e79eb>] zfs_obj_to_stats+0xab/0xd0 [zfs]
[ 336.942101] [<ffffffffa03ca6d4>] zfs_ioc_obj_to_stats+0x84/0xa0 [zfs]
[ 336.942173] [<ffffffffa03d0879>] zfsdev_ioctl+0x4b9/0x530 [zfs]
[ 336.942182] [<ffffffff811a6659>] do_vfs_ioctl+0x99/0x570
[ 336.942191] [<ffffffff811a6bc1>] sys_ioctl+0x91/0xb0
[ 336.942201] [<ffffffff816d671d>] system_call_fastpath+0x1a/0x1f
[ 336.942205] Code: 84 00 00 00 00 00 0f 1f 00 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c1 c1 e9 10 66 39 c1 89 ca 74 11 0f 1f 80 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89
[ 336.942313] NMI backtrace for cpu 0
[ 336.942318] CPU 0
[ 336.942325] Pid: 0, comm: swapper/0 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 336.942330] RIP: 0010:[<ffffffff813e3eb7>] [<ffffffff813e3eb7>] acpi_idle_do_entry+0x21/0x2b
[ 336.942342] RSP: 0018:ffffffff81c01e10 EFLAGS: 00000093
[ 336.942347] RAX: 0000004e72b8df00 RBX: ffff88061ce308a0 RCX: 000000000000c648
[ 336.942351] RDX: 0000000000000815 RSI: ffff88063ec00000 RDI: ffff88061ce308a0
[ 336.942356] RBP: ffffffff81c01e10 R08: 0000000000002e8b R09: 0000000000000018
[ 336.942360] R10: 0000000000005d76 R11: 0000000000000000 R12: 0000000000000002
[ 336.942365] R13: ffffffff81c00000 R14: ffff88061ce30800 R15: ffffffff81574a50
[ 336.942370] FS: 00007f149d07d840(0000) GS:ffff88063ec00000(0000) knlGS:0000000000000000
[ 336.942375] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 336.942380] CR2: 00007f149d098000 CR3: 00000005dccb7000 CR4: 00000000000407f0
[ 336.942385] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 336.942389] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 336.942394] Process swapper/0 (pid: 0, threadinfo ffffffff81c00000, task ffffffff81c15440)
[ 336.942398] Stack:
[ 336.942401] ffffffff81c01e40 ffffffff813e3faa ffff880616740600 ffffffff81c7e060
[ 336.942411] 0000004e8e22fce5 0000000000000002 ffffffff81c01e50 ffffffff81574a69
[ 336.942422] ffffffff81c01eb0 ffffffff81575499 7735940000000001 ffff880616740600
[ 336.942432] Call Trace:
[ 336.942441] [<ffffffff813e3faa>] acpi_idle_enter_simple+0x87/0xc5
[ 336.942452] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 336.942460] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 336.942469] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 336.942477] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 336.942487] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 336.942496] [<ffffffff816abc32>] rest_init+0x72/0x80
[ 336.942506] [<ffffffff81d05be9>] start_kernel+0x3d1/0x3dc
[ 336.942515] [<ffffffff81d0566e>] ? do_early_param+0x8e/0x8e
[ 336.942524] [<ffffffff81d05355>] x86_64_start_reservations+0x130/0x133
[ 336.942533] [<ffffffff81d05458>] x86_64_start_kernel+0x100/0x10f
[ 336.942537] Code: 00 83 88 3c e0 ff ff 04 5d c3 8a 47 08 55 48 89 e5 3c 01 75 07 e8 3a 35 c5 ff eb 17 3c 02 75 07 e8 a2 ff ff ff eb 0c 8b 57 04 ec <48> 8b 15 86 e1 b2 00 ed 5d c3 66 66 66 66 90 55 48 63 c2 48 c1
[ 336.942644] NMI backtrace for cpu 2
[ 336.942652] CPU 2
[ 336.942662] Pid: 0, comm: swapper/2 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 336.942668] RIP: 0010:[<ffffffff813e3eb7>] [<ffffffff813e3eb7>] acpi_idle_do_entry+0x21/0x2b
[ 336.942683] RSP: 0018:ffff88061cd85e10 EFLAGS: 00000093
[ 336.942688] RAX: 0000004e723ff600 RBX: ffff88061cdb88a0 RCX: 000000000000c648
[ 336.942693] RDX: 0000000000000815 RSI: ffff88063ec80000 RDI: ffff88061cdb88a0
[ 336.942697] RBP: ffff88061cd85e10 R08: 0000000000002e4f R09: 0000000000000018
[ 336.942702] R10: 0000000000002ebc R11: 0000000000000000 R12: 0000000000000002
[ 336.942706] R13: ffff88061cd84000 R14: ffff88061cdb8800 R15: ffffffff81574a50
[ 336.942712] FS: 00007f3fa5ffb700(0000) GS:ffff88063ec80000(0000) knlGS:0000000000000000
[ 336.942717] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 336.942721] CR2: 00007f8d183f9000 CR3: 00000005d4e3c000 CR4: 00000000000407e0
[ 336.942726] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 336.942731] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 336.942736] Process swapper/2 (pid: 0, threadinfo ffff88061cd84000, task ffff88061cd7ae80)
[ 336.942740] Stack:
[ 336.942743] ffff88061cd85e40 ffffffff813e3faa ffff88061c8c7200 ffffffff81c7e060
[ 336.942755] 0000004e8da9c0be 0000000000000002 ffff88061cd85e50 ffffffff81574a69
[ 336.942766] ffff88061cd85eb0 ffffffff81575499 7735940000000001 ffff88061c8c7200
[ 336.942776] Call Trace:
[ 336.942787] [<ffffffff813e3faa>] acpi_idle_enter_simple+0x87/0xc5
[ 336.942797] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 336.942805] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 336.942814] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 336.942822] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 336.942832] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 336.942841] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 336.942846] Code: 00 83 88 3c e0 ff ff 04 5d c3 8a 47 08 55 48 89 e5 3c 01 75 07 e8 3a 35 c5 ff eb 17 3c 02 75 07 e8 a2 ff ff ff eb 0c 8b 57 04 ec <48> 8b 15 86 e1 b2 00 ed 5d c3 66 66 66 66 90 55 48 63 c2 48 c1
[ 336.942952] NMI backtrace for cpu 3
[ 336.942955] CPU 3
[ 336.942961] Pid: 0, comm: swapper/3 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 336.942964] RIP: 0010:[<ffffffff813e3eb7>] [<ffffffff813e3eb7>] acpi_idle_do_entry+0x21/0x2b
[ 336.942972] RSP: 0018:ffff88061cd87e10 EFLAGS: 00000093
[ 336.942976] RAX: 0000004e72f68900 RBX: ffff8806167d54a0 RCX: 000000000000c648
[ 336.942979] RDX: 0000000000000815 RSI: ffff88063ecc0000 RDI: ffff8806167d54a0
[ 336.942982] RBP: ffff88061cd87e10 R08: 0000000000002e52 R09: 0000000000000018
[ 336.942986] R10: 0000000000002e76 R11: 0000000000000000 R12: 0000000000000002
[ 336.942989] R13: ffff88061cd86000 R14: ffff8806167d5400 R15: ffffffff81574a50
[ 336.942994] FS: 00007f44cebfd7c0(0000) GS:ffff88063ecc0000(0000) knlGS:0000000000000000
[ 336.942997] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 336.943001] CR2: 0000000002142846 CR3: 00000005ef311000 CR4: 00000000000407e0
[ 336.943020] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 336.943022] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 336.943025] Process swapper/3 (pid: 0, threadinfo ffff88061cd86000, task ffff88061cd7c5c0)
[ 336.943025] Stack:
[ 336.943031] ffff88061cd87e40 ffffffff813e3faa ffff880616006400 ffffffff81c7e060
[ 336.943038] 0000004e8e60d014 0000000000000002 ffff88061cd87e50 ffffffff81574a69
[ 336.943051] ffff88061cd87eb0 ffffffff81575499 7735940000000001 ffff880616006400
[ 336.943054] Call Trace:
[ 336.943061] [<ffffffff813e3faa>] acpi_idle_enter_simple+0x87/0xc5
[ 336.943069] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 336.943076] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 336.943085] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 336.943093] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 336.943101] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 336.943106] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 336.943111] Code: 00 83 88 3c e0 ff ff 04 5d c3 8a 47 08 55 48 89 e5 3c 01 75 07 e8 3a 35 c5 ff eb 17 3c 02 75 07 e8 a2 ff ff ff eb 0c 8b 57 04 ec <48> 8b 15 86 e1 b2 00 ed 5d c3 66 66 66 66 90 55 48 63 c2 48 c1
[ 480.335714] INFO: task txg_sync:718 blocked for more than 120 seconds.
[ 480.335719] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 480.335724] txg_sync D ffff88063ec93ec0 0 718 2 0x00000000
[ 480.335731] ffff8805f56c1c50 0000000000000046 ffff880615bb5d00 ffff8805f56c1fd8
[ 480.335739] ffff8805f56c1fd8 ffff8805f56c1fd8 ffff88061cd7ae80 ffff880615bb5d00
[ 480.335745] ffff88061be1e550 ffff88061be1e530 ffff88061be1e558 0000000000000000
[ 480.335752] Call Trace:
[ 480.335765] [<ffffffff816ccf39>] schedule+0x29/0x70
[ 480.335811] [<ffffffffa02dd63d>] cv_wait_common+0xfd/0x1b0 [spl]
[ 480.335819] [<ffffffff8107def0>] ? finish_wait+0x80/0x80
[ 480.335843] [<ffffffffa02dd745>] __cv_wait+0x15/0x20 [spl]
[ 480.335917] [<ffffffffa038eaa5>] rrw_enter_write+0x35/0x70 [zfs]
[ 480.335979] [<ffffffffa038eaf3>] rrw_enter+0x13/0x30 [zfs]
[ 480.336043] [<ffffffffa039811e>] spa_sync+0x81e/0xb20 [zfs]
[ 480.336051] [<ffffffff810ac1c8>] ? ktime_get_ts+0x48/0xe0
[ 480.336119] [<ffffffffa03a885a>] txg_sync_thread+0x3aa/0x600 [zfs]
[ 480.336191] [<ffffffffa03a84b0>] ? txg_init+0x270/0x270 [zfs]
[ 480.336213] [<ffffffffa02d5b78>] thread_generic_wrapper+0x78/0x90 [spl]
[ 480.336234] [<ffffffffa02d5b00>] ? __thread_create+0x340/0x340 [spl]
[ 480.336239] [<ffffffff8107d5e0>] kthread+0xc0/0xd0
[ 480.336245] [<ffffffff8107d520>] ? kthread_create_on_node+0x120/0x120
[ 480.336252] [<ffffffff816d666c>] ret_from_fork+0x7c/0xb0
[ 480.336257] [<ffffffff8107d520>] ? kthread_create_on_node+0x120/0x120
[ 480.336291] INFO: task zfs:6459 blocked for more than 120 seconds.
[ 480.336294] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 480.336297] zfs D 0000000000000000 0 6459 4544 0x00000008
[ 480.336302] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 480.336309] ffff8805d42e3888 ffffffff81044b59 ffff8805d42e38a8 ffff8805d42e38a8
[ 480.336315] 0000000000000282 ffff8805d8cc9ec8 ffff8805d42e3958 ffff8805d42e38f8
[ 480.336321] Call Trace:
[ 480.336330] [<ffffffff81044b59>] ? default_spin_lock_flags+0x9/0x10
[ 480.336337] [<ffffffff8117e07d>] ? __kmalloc+0x13d/0x170
[ 480.336358] [<ffffffffa02d29c6>] ? kmem_alloc_debug+0x96/0x3b0 [spl]
[ 480.336379] [<ffffffffa02d29c6>] ? kmem_alloc_debug+0x96/0x3b0 [spl]
[ 480.336385] [<ffffffff816cb7ad>] ? mutex_lock+0x1d/0x50
[ 480.336432] [<ffffffffa0354ff1>] ? dbuf_find+0xe1/0xf0 [zfs]
[ 480.336438] [<ffffffff81044b59>] ? default_spin_lock_flags+0x9/0x10
[ 480.336485] [<ffffffffa03571e9>] ? dbuf_read+0x579/0x7f0 [zfs]
[ 480.336505] [<ffffffffa02d183b>] ? kmem_free_debug+0x4b/0x150 [spl]
[ 480.336512] [<ffffffff816cde65>] ? _raw_spin_lock_irq+0x15/0x20
[ 480.336517] [<ffffffff816cdca6>] ? rwsem_down_failed_common+0x46/0x170
[ 480.336523] [<ffffffff816cb7ad>] ? mutex_lock+0x1d/0x50
[ 480.336529] [<ffffffff816cde05>] ? rwsem_down_read_failed+0x15/0x17
[ 480.336536] [<ffffffff813555d4>] ? call_rwsem_down_read_failed+0x14/0x30
[ 480.336556] [<ffffffffa02d183b>] ? kmem_free_debug+0x4b/0x150 [spl]
[ 480.336566] [<ffffffff816cc0e0>] ? down_read+0x20/0x23
[ 480.336631] [<ffffffffa03b7210>] ? zap_get_leaf_byblk+0xd0/0x2c0 [zfs]
[ 480.336697] [<ffffffffa03b6c0b>] ? zap_idx_to_blk+0x13b/0x150 [zfs]
[ 480.336766] [<ffffffffa03b746d>] ? zap_deref_leaf+0x6d/0x80 [zfs]
[ 480.336834] [<ffffffffa03b9021>] ? fzap_cursor_retrieve+0xe1/0x280 [zfs]
[ 480.336901] [<ffffffffa03bc848>] ? zap_cursor_retrieve+0x68/0x2e0 [zfs]
[ 480.336964] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 480.337027] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 480.337098] [<ffffffffa03b86d1>] ? zap_value_search+0x91/0xd0 [zfs]
[ 480.337164] [<ffffffffa03e779d>] ? zfs_obj_to_path_impl+0x1bd/0x2b0 [zfs]
[ 480.337228] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 480.337290] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 480.337354] [<ffffffffa03e3261>] ? zfs_obj_to_stats_impl+0xa1/0xb0 [zfs]
[ 480.337420] [<ffffffffa03e79eb>] ? zfs_obj_to_stats+0xab/0xd0 [zfs]
[ 480.337485] [<ffffffffa03ca6d4>] ? zfs_ioc_obj_to_stats+0x84/0xa0 [zfs]
[ 480.337550] [<ffffffffa03d0879>] ? zfsdev_ioctl+0x4b9/0x530 [zfs]
[ 480.337557] [<ffffffff811a6659>] ? do_vfs_ioctl+0x99/0x570
[ 480.337563] [<ffffffff811a6bc1>] ? sys_ioctl+0x91/0xb0
[ 480.337570] [<ffffffff816d671d>] ? system_call_fastpath+0x1a/0x1f
[ 516.504679] INFO: rcu_sched detected stalls on CPUs/tasks: { 1} (detected by 3, t=60007 jiffies, g=32254, c=32253, q=51221)
[ 516.504691] sending NMI to all CPUs:
[ 516.504703] NMI backtrace for cpu 1
[ 516.504711] CPU 1
[ 516.504720] Pid: 6459, comm: zfs Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 516.504726] RIP: 0010:[<ffffffff81044ac2>] [<ffffffff81044ac2>] __ticket_spin_lock+0x22/0x30
[ 516.504741] RSP: 0018:ffff8805d42e3958 EFLAGS: 00000006
[ 516.504746] RAX: 0000000000002d4b RBX: ffff8805dce22e80 RCX: 0000000000000000
[ 516.504751] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88051fac2488
[ 516.504756] RBP: ffff8805d42e3958 R08: 0000000000016d40 R09: ffff88063ec56d40
[ 516.504760] R10: ffffea0017696200 R11: ffffffffa02d183b R12: ffff88051fac2480
[ 516.504765] R13: ffff88051fac2488 R14: ffffffffffffffff R15: 0000000000000001
[ 516.504771] FS: 00007f1ac9a97700(0000) GS:ffff88063ec40000(0000) knlGS:0000000000000000
[ 516.504776] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 516.504780] CR2: 00007f2a6f2e0000 CR3: 00000005ddf2a000 CR4: 00000000000407e0
[ 516.504785] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 516.504790] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 516.504795] Process zfs (pid: 6459, threadinfo ffff8805d42e2000, task ffff8805dce22e80)
[ 516.504799] Stack:
[ 516.504803] ffff8805d42e3968 ffffffff816cde65 ffff8805d42e39d0 ffffffff816cdca6
[ 516.504814] 0000000000000000 0000000000000000 00000000d42e39d0 ffffffff00000000
[ 516.504825] ffffffff816cb7ad ffff8805cc05b970 ffff88051fac2480 0000000000000002
[ 516.504835] Call Trace:
[ 516.504847] [<ffffffff816cde65>] _raw_spin_lock_irq+0x15/0x20
[ 516.504856] [<ffffffff816cdca6>] rwsem_down_failed_common+0x46/0x170
[ 516.504865] [<ffffffff816cb7ad>] ? mutex_lock+0x1d/0x50
[ 516.504874] [<ffffffff816cde05>] rwsem_down_read_failed+0x15/0x17
[ 516.504884] [<ffffffff813555d4>] call_rwsem_down_read_failed+0x14/0x30
[ 516.504923] [<ffffffffa02d183b>] ? kmem_free_debug+0x4b/0x150 [spl]
[ 516.504936] [<ffffffff816cc0e0>] ? down_read+0x20/0x23
[ 516.505020] [<ffffffffa03b7210>] zap_get_leaf_byblk+0xd0/0x2c0 [zfs]
[ 516.505091] [<ffffffffa03b6c0b>] ? zap_idx_to_blk+0x13b/0x150 [zfs]
[ 516.505165] [<ffffffffa03b746d>] zap_deref_leaf+0x6d/0x80 [zfs]
[ 516.505237] [<ffffffffa03b9021>] fzap_cursor_retrieve+0xe1/0x280 [zfs]
[ 516.505309] [<ffffffffa03bc848>] zap_cursor_retrieve+0x68/0x2e0 [zfs]
[ 516.505376] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 516.505444] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 516.505521] [<ffffffffa03b86d1>] zap_value_search+0x91/0xd0 [zfs]
[ 516.505592] [<ffffffffa03e779d>] zfs_obj_to_path_impl+0x1bd/0x2b0 [zfs]
[ 516.505662] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 516.505730] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 516.505800] [<ffffffffa03e3261>] ? zfs_obj_to_stats_impl+0xa1/0xb0 [zfs]
[ 516.505872] [<ffffffffa03e79eb>] zfs_obj_to_stats+0xab/0xd0 [zfs]
[ 516.505943] [<ffffffffa03ca6d4>] zfs_ioc_obj_to_stats+0x84/0xa0 [zfs]
[ 516.506014] [<ffffffffa03d0879>] zfsdev_ioctl+0x4b9/0x530 [zfs]
[ 516.506024] [<ffffffff811a6659>] do_vfs_ioctl+0x99/0x570
[ 516.506033] [<ffffffff811a6bc1>] sys_ioctl+0x91/0xb0
[ 516.506043] [<ffffffff816d671d>] system_call_fastpath+0x1a/0x1f
[ 516.506047] Code: 84 00 00 00 00 00 0f 1f 00 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c1 c1 e9 10 66 39 c1 89 ca 74 11 0f 1f 80 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89
[ 516.506153] NMI backtrace for cpu 0
[ 516.506157] CPU 0
[ 516.506165] Pid: 50, comm: kworker/0:1 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 516.506170] RIP: 0010:[<ffffffff816cde55>] [<ffffffff816cde55>] _raw_spin_lock_irq+0x5/0x20
[ 516.506180] RSP: 0018:ffff88061673fde8 EFLAGS: 00000202
[ 516.506185] RAX: ffff88063ec13ec0 RBX: ffff88061cfcba00 RCX: 0000000000000002
[ 516.506189] RDX: ffff88063ec00000 RSI: ffffffff816ccf39 RDI: ffff88063ec13ec0
[ 516.506194] RBP: ffff88061673fe48 R08: 0000000000000001 R09: 0000000000000004
[ 516.506198] R10: ffffffff81ce70a0 R11: 0000000000000001 R12: ffff88063ec13ec0
[ 516.506203] R13: 0000000000000000 R14: 0000000000013e80 R15: ffff88061c9ac5c0
[ 516.506208] FS: 00007fa5177fe700(0000) GS:ffff88063ec00000(0000) knlGS:0000000000000000
[ 516.506213] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 516.506217] CR2: 00007f8d10000020 CR3: 00000005e1941000 CR4: 00000000000407f0
[ 516.506222] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 516.506226] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 516.506231] Process kworker/0:1 (pid: 50, threadinfo ffff88061673e000, task ffff88061c9ac5c0)
[ 516.506234] Stack:
[ 516.506238] ffffffff816cc4b0 ffff88061c9ac5c0 ffff88061673ffd8 ffff88061673ffd8
[ 516.506248] ffff88061673ffd8 ffffffff81c15440 ffff88061c9ac5c0 ffff88061cfcba00
[ 516.506259] ffff88063ec0e590 0000000000000000 0000000000013e80 ffff88061c9ac5c0
[ 516.506268] Call Trace:
[ 516.506277] [<ffffffff816cc4b0>] ? __schedule+0xb0/0x800
[ 516.506286] [<ffffffff816ccf39>] schedule+0x29/0x70
[ 516.506296] [<ffffffff81078307>] worker_thread+0x237/0x490
[ 516.506305] [<ffffffff810780d0>] ? manage_workers+0x2b0/0x2b0
[ 516.506313] [<ffffffff8107d5e0>] kthread+0xc0/0xd0
[ 516.506322] [<ffffffff8107d520>] ? kthread_create_on_node+0x120/0x120
[ 516.506330] [<ffffffff816d666c>] ret_from_fork+0x7c/0xb0
[ 516.506338] [<ffffffff8107d520>] ? kthread_create_on_node+0x120/0x120
[ 516.506343] Code: d2 89 d0 5d c3 90 66 66 66 66 90 55 48 89 e5 e8 62 6c 97 ff 66 90 5d c3 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 <55> 48 89 e5 fa 66 66 90 66 66 90 e8 3b 6c 97 ff 66 90 5d c3 0f
[ 516.506447] NMI backtrace for cpu 3
[ 516.506452] CPU 3
[ 516.506461] Pid: 0, comm: swapper/3 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 516.506466] RIP: 0010:[<ffffffff81354172>] [<ffffffff81354172>] __const_udelay+0x12/0x30
[ 516.506480] RSP: 0018:ffff88063ecc3dd8 EFLAGS: 00000046
[ 516.506485] RAX: 0000000001062560 RBX: 0000000000002710 RCX: 0000000000000004
[ 516.506489] RDX: 0000000000d5a27c RSI: 0000000000000100 RDI: 0000000000418958
[ 516.506494] RBP: ffff88063ecc3df0 R08: 000000000000000a R09: 0000000000000000
[ 516.506499] R10: 000000000000058e R11: 0000000000000002 R12: ffffffff81ce70c0
[ 516.506503] R13: ffff88063eccee60 R14: ffffffff81c3f3c0 R15: ffffffff81c3f480
[ 516.506509] FS: 00007f887ab47700(0000) GS:ffff88063ecc0000(0000) knlGS:0000000000000000
[ 516.506514] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 516.506519] CR2: 000000000213d728 CR3: 00000005c62fe000 CR4: 00000000000407e0
[ 516.506523] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 516.506528] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 516.506533] Process swapper/3 (pid: 0, threadinfo ffff88061cd86000, task ffff88061cd7c5c0)
[ 516.506537] Stack:
[ 516.506540] ffff88063ecc3df0 ffffffff81039d6a ffffffff81c3f3c0 ffff88063ecc3e50
[ 516.506552] ffffffff810f2935 000000000000c815 0000000000000082 0000000100000086
[ 516.506562] 0000000000000003 ffff88061cd86000 ffff88061cd7c5c0 0000000000000003
[ 516.506572] Call Trace:
[ 516.506577] <IRQ>
[ 516.506581] [<ffffffff81039d6a>] ? arch_trigger_all_cpu_backtrace+0x6a/0x90
[ 516.506599] [<ffffffff810f2935>] rcu_pending+0x565/0x5b0
[ 516.506608] [<ffffffff810f4a0b>] rcu_check_callbacks+0x9b/0x120
[ 516.506617] [<ffffffff8106a928>] update_process_times+0x48/0x90
[ 516.506626] [<ffffffff810b3a8e>] tick_sched_handle.isra.8+0x2e/0x70
[ 516.506634] [<ffffffff810b3c1c>] tick_sched_timer+0x4c/0x80
[ 516.506643] [<ffffffff810819d9>] __run_hrtimer+0x79/0x1c0
[ 516.506651] [<ffffffff810b3bd0>] ? tick_sched_do_timer+0x40/0x40
[ 516.506659] [<ffffffff810822c7>] hrtimer_interrupt+0xe7/0x230
[ 516.506669] [<ffffffff81574a50>] ? centrino_target+0x370/0x370
[ 516.506679] [<ffffffff816d8519>] smp_apic_timer_interrupt+0x69/0x99
[ 516.506688] [<ffffffff816d745d>] apic_timer_interrupt+0x6d/0x80
[ 516.506692] <EOI>
[ 516.506697] [<ffffffff81043fa6>] ? native_safe_halt+0x6/0x10
[ 516.506713] [<ffffffff813e3769>] arch_safe_halt+0x9/0xd
[ 516.506722] [<ffffffff813e3e7d>] acpi_safe_halt+0x2a/0x43
[ 516.506730] [<ffffffff813e3eb1>] acpi_idle_do_entry+0x1b/0x2b
[ 516.506738] [<ffffffff813e3f04>] acpi_idle_enter_c1+0x43/0x62
[ 516.506746] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 516.506754] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 516.506763] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 516.506771] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 516.506780] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 516.506789] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 516.506794] Code: 89 e5 ff 15 49 25 92 00 5d c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 8d 04 bd 00 00 00 00 65 48 8b 14 25 18 3e 01 00 <48> 8d 0c 12 48 c1 e2 06 48 89 e5 48 29 ca f7 e2 48 8d 7a 01 ff
[ 516.506901] NMI backtrace for cpu 2
[ 516.506904] CPU 2
[ 516.506911] Pid: 0, comm: swapper/2 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 516.506916] RIP: 0010:[<ffffffff813e3eb7>] [<ffffffff813e3eb7>] acpi_idle_do_entry+0x21/0x2b
[ 516.506926] RSP: 0018:ffff88061cd85e10 EFLAGS: 00000093
[ 516.506931] RAX: 0000007841d87f00 RBX: ffff88061cdb88a0 RCX: 000000000000c648
[ 516.506935] RDX: 0000000000000815 RSI: ffff88063ec80000 RDI: ffff88061cdb88a0
[ 516.506939] RBP: ffff88061cd85e10 R08: 0000000000002e88 R09: 0000000000000018
[ 516.506943] R10: 0000000000002975 R11: 0000000000000000 R12: 0000000000000002
[ 516.506948] R13: ffff88061cd84000 R14: ffff88061cdb8800 R15: ffffffff81574a50
[ 516.506953] FS: 00007fa5167fc700(0000) GS:ffff88063ec80000(0000) knlGS:0000000000000000
[ 516.506958] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 516.506962] CR2: 00007f149d098000 CR3: 00000005e1941000 CR4: 00000000000407e0
[ 516.506966] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 516.506971] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 516.506975] Process swapper/2 (pid: 0, threadinfo ffff88061cd84000, task ffff88061cd7ae80)
[ 516.506979] Stack:
[ 516.506982] ffff88061cd85e40 ffffffff813e3faa ffff88061c8c7200 ffffffff81c7e060
[ 516.506994] 000000787867328d 0000000000000002 ffff88061cd85e50 ffffffff81574a69
[ 516.507005] ffff88061cd85eb0 ffffffff81575499 7735940000000001 ffff88061c8c7200
[ 516.507016] Call Trace:
[ 516.507025] [<ffffffff813e3faa>] acpi_idle_enter_simple+0x87/0xc5
[ 516.507034] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 516.507042] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 516.507052] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 516.507060] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 516.507069] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 516.507077] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 516.507082] Code: 00 83 88 3c e0 ff ff 04 5d c3 8a 47 08 55 48 89 e5 3c 01 75 07 e8 3a 35 c5 ff eb 17 3c 02 75 07 e8 a2 ff ff ff eb 0c 8b 57 04 ec <48> 8b 15 86 e1 b2 00 ed 5d c3 66 66 66 66 90 55 48 63 c2 48 c1
[ 516.507203] NMI backtrace for cpu 5
[ 516.507211] CPU 5
[ 516.507220] Pid: 0, comm: swapper/5 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 516.507226] RIP: 0010:[<ffffffff81043fa6>] [<ffffffff81043fa6>] native_safe_halt+0x6/0x10
[ 516.507240] RSP: 0018:ffff88061cd8bde8 EFLAGS: 00000246
[ 516.507245] RAX: 0000000000000000 RBX: ffff88062600e06c RCX: ffffffff81c7e060
[ 516.507250] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88062600e06c
[ 516.507254] RBP: ffff88061cd8bde8 R08: 0000000000002e4e R09: 000000000000001c
[ 516.507259] R10: 0000000000002e76 R11: 0000000000000000 R12: 0000000000000001
[ 516.507263] R13: ffff88062600e000 R14: 0000000000000001 R15: ffffffff81574a50
[ 516.507269] FS: 00007fa896657700(0000) GS:ffff88063ed40000(0000) knlGS:0000000000000000
[ 516.507274] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 516.507279] CR2: 00000000018f6000 CR3: 00000005ad530000 CR4: 00000000000407e0
[ 516.507283] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 516.507288] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 516.507293] Process swapper/5 (pid: 0, threadinfo ffff88061cd8a000, task ffff88061cd90000)
[ 516.507297] Stack:
[ 516.507300] ffff88061cd8bdf8 ffffffff813e3769 ffff88061cd8be08 ffffffff813e3e7d
[ 516.507312] ffff88061cd8be18 ffffffff813e3eb1 ffff88061cd8be40 ffffffff813e3f04
[ 516.507322] ffff880616086000 ffffffff81c7e060 000000787868156d ffff88061cd8be50
[ 516.507332] Call Trace:
[ 516.507344] [<ffffffff813e3769>] arch_safe_halt+0x9/0xd
[ 516.507353] [<ffffffff813e3e7d>] acpi_safe_halt+0x2a/0x43
[ 516.507362] [<ffffffff813e3eb1>] acpi_idle_do_entry+0x1b/0x2b
[ 516.507370] [<ffffffff813e3f04>] acpi_idle_enter_c1+0x43/0x62
[ 516.507380] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 516.507388] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 516.507397] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 516.507405] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 516.507414] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 516.507423] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 516.507429] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 516.507536] NMI backtrace for cpu 4
[ 516.507539] CPU 4
[ 516.507545] Pid: 0, comm: swapper/4 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 516.507548] RIP: 0010:[<ffffffff813e3eb7>] [<ffffffff813e3eb7>] acpi_idle_do_entry+0x21/0x2b
[ 516.507555] RSP: 0018:ffff88061cd89e10 EFLAGS: 00000093
[ 516.507558] RAX: 00000078419c6000 RBX: ffff88061ce310a0 RCX: 000000000000c648
[ 516.507561] RDX: 0000000000000815 RSI: ffff88063ed00000 RDI: ffff88061ce310a0
[ 516.507564] RBP: ffff88061cd89e10 R08: 0000000000002e55 R09: 0000000000000018
[ 516.507567] R10: 0000000000002e6d R11: 0000000000000000 R12: 0000000000000002
[ 516.507569] R13: ffff88061cd88000 R14: ffff88061ce31000 R15: ffffffff81574a50
[ 516.507573] FS: 00007f2055fb6740(0000) GS:ffff88063ed00000(0000) knlGS:0000000000000000
[ 516.507577] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 516.507579] CR2: 00007fd699e45000 CR3: 00000005ddef3000 CR4: 00000000000407e0
[ 516.507582] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 516.507585] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 516.507588] Process swapper/4 (pid: 0, threadinfo ffff88061cd88000, task ffff88061cd7dd00)
[ 516.507591] Stack:
[ 516.507593] ffff88061cd89e40 ffffffff813e3faa ffff88061ca45600 ffffffff81c7e060
[ 516.507599] 00000078782af3e5 0000000000000002 ffff88061cd89e50 ffffffff81574a69
[ 516.507605] ffff88061cd89eb0 ffffffff81575499 7735940000000001 ffff88061ca45600
[ 516.507611] Call Trace:
[ 516.507618] [<ffffffff813e3faa>] acpi_idle_enter_simple+0x87/0xc5
[ 516.507624] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 516.507630] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 516.507636] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 516.507642] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 516.507648] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 516.507653] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 516.507657] Code: 00 83 88 3c e0 ff ff 04 5d c3 8a 47 08 55 48 89 e5 3c 01 75 07 e8 3a 35 c5 ff eb 17 3c 02 75 07 e8 a2 ff ff ff eb 0c 8b 57 04 ec <48> 8b 15 86 e1 b2 00 ed 5d c3 66 66 66 66 90 55 48 63 c2 48 c1
[ 600.085872] INFO: task txg_sync:718 blocked for more than 120 seconds.
[ 600.085879] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 600.085883] txg_sync D ffff88063ec93ec0 0 718 2 0x00000000
[ 600.085892] ffff8805f56c1c50 0000000000000046 ffff880615bb5d00 ffff8805f56c1fd8
[ 600.085901] ffff8805f56c1fd8 ffff8805f56c1fd8 ffff88061cd7ae80 ffff880615bb5d00
[ 600.085909] ffff88061be1e550 ffff88061be1e530 ffff88061be1e558 0000000000000000
[ 600.085917] Call Trace:
[ 600.085930] [<ffffffff816ccf39>] schedule+0x29/0x70
[ 600.085969] [<ffffffffa02dd63d>] cv_wait_common+0xfd/0x1b0 [spl]
[ 600.085976] [<ffffffff8107def0>] ? finish_wait+0x80/0x80
[ 600.086000] [<ffffffffa02dd745>] __cv_wait+0x15/0x20 [spl]
[ 600.086072] [<ffffffffa038eaa5>] rrw_enter_write+0x35/0x70 [zfs]
[ 600.086134] [<ffffffffa038eaf3>] rrw_enter+0x13/0x30 [zfs]
[ 600.086199] [<ffffffffa039811e>] spa_sync+0x81e/0xb20 [zfs]
[ 600.086207] [<ffffffff810ac1c8>] ? ktime_get_ts+0x48/0xe0
[ 600.086274] [<ffffffffa03a885a>] txg_sync_thread+0x3aa/0x600 [zfs]
[ 600.086346] [<ffffffffa03a84b0>] ? txg_init+0x270/0x270 [zfs]
[ 600.086368] [<ffffffffa02d5b78>] thread_generic_wrapper+0x78/0x90 [spl]
[ 600.086390] [<ffffffffa02d5b00>] ? __thread_create+0x340/0x340 [spl]
[ 600.086395] [<ffffffff8107d5e0>] kthread+0xc0/0xd0
[ 600.086401] [<ffffffff8107d520>] ? kthread_create_on_node+0x120/0x120
[ 600.086407] [<ffffffff816d666c>] ret_from_fork+0x7c/0xb0
[ 600.086413] [<ffffffff8107d520>] ? kthread_create_on_node+0x120/0x120
[ 600.086447] INFO: task zfs:6459 blocked for more than 120 seconds.
[ 600.086450] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 600.086453] zfs D 0000000000000000 0 6459 4544 0x00000008
[ 600.086458] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 600.086465] ffff8805d42e3888 ffffffff81044b59 ffff8805d42e38a8 ffff8805d42e38a8
[ 600.086471] 0000000000000282 ffff8805d8cc9ec8 ffff8805d42e3958 ffff8805d42e38f8
[ 600.086478] Call Trace:
[ 600.086486] [<ffffffff81044b59>] ? default_spin_lock_flags+0x9/0x10
[ 600.086494] [<ffffffff8117e07d>] ? __kmalloc+0x13d/0x170
[ 600.086515] [<ffffffffa02d29c6>] ? kmem_alloc_debug+0x96/0x3b0 [spl]
[ 600.086536] [<ffffffffa02d29c6>] ? kmem_alloc_debug+0x96/0x3b0 [spl]
[ 600.086542] [<ffffffff816cb7ad>] ? mutex_lock+0x1d/0x50
[ 600.086588] [<ffffffffa0354ff1>] ? dbuf_find+0xe1/0xf0 [zfs]
[ 600.086595] [<ffffffff81044b59>] ? default_spin_lock_flags+0x9/0x10
[ 600.086643] [<ffffffffa03571e9>] ? dbuf_read+0x579/0x7f0 [zfs]
[ 600.086663] [<ffffffffa02d183b>] ? kmem_free_debug+0x4b/0x150 [spl]
[ 600.086669] [<ffffffff816cde65>] ? _raw_spin_lock_irq+0x15/0x20
[ 600.086675] [<ffffffff816cdca6>] ? rwsem_down_failed_common+0x46/0x170
[ 600.086681] [<ffffffff816cb7ad>] ? mutex_lock+0x1d/0x50
[ 600.086687] [<ffffffff816cde05>] ? rwsem_down_read_failed+0x15/0x17
[ 600.086693] [<ffffffff813555d4>] ? call_rwsem_down_read_failed+0x14/0x30
[ 600.086713] [<ffffffffa02d183b>] ? kmem_free_debug+0x4b/0x150 [spl]
[ 600.086723] [<ffffffff816cc0e0>] ? down_read+0x20/0x23
[ 600.086790] [<ffffffffa03b7210>] ? zap_get_leaf_byblk+0xd0/0x2c0 [zfs]
[ 600.086857] [<ffffffffa03b6c0b>] ? zap_idx_to_blk+0x13b/0x150 [zfs]
[ 600.086925] [<ffffffffa03b746d>] ? zap_deref_leaf+0x6d/0x80 [zfs]
[ 600.086992] [<ffffffffa03b9021>] ? fzap_cursor_retrieve+0xe1/0x280 [zfs]
[ 600.087060] [<ffffffffa03bc848>] ? zap_cursor_retrieve+0x68/0x2e0 [zfs]
[ 600.087123] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 600.087185] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 600.087255] [<ffffffffa03b86d1>] ? zap_value_search+0x91/0xd0 [zfs]
[ 600.087320] [<ffffffffa03e779d>] ? zfs_obj_to_path_impl+0x1bd/0x2b0 [zfs]
[ 600.087386] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 600.087449] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 600.087512] [<ffffffffa03e3261>] ? zfs_obj_to_stats_impl+0xa1/0xb0 [zfs]
[ 600.087577] [<ffffffffa03e79eb>] ? zfs_obj_to_stats+0xab/0xd0 [zfs]
[ 600.087641] [<ffffffffa03ca6d4>] ? zfs_ioc_obj_to_stats+0x84/0xa0 [zfs]
[ 600.087707] [<ffffffffa03d0879>] ? zfsdev_ioctl+0x4b9/0x530 [zfs]
[ 600.087714] [<ffffffff811a6659>] ? do_vfs_ioctl+0x99/0x570
[ 600.087720] [<ffffffff811a6bc1>] ? sys_ioctl+0x91/0xb0
[ 600.087726] [<ffffffff816d671d>] ? system_call_fastpath+0x1a/0x1f
[ 696.174489] INFO: rcu_sched detected stalls on CPUs/tasks: { 1} (detected by 3, t=105012 jiffies, g=32254, c=32253, q=63035)
[ 696.174500] sending NMI to all CPUs:
[ 696.174521] NMI backtrace for cpu 1
[ 696.174528] CPU 1
[ 696.174538] Pid: 6459, comm: zfs Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 696.174544] RIP: 0010:[<ffffffff81044ac2>] [<ffffffff81044ac2>] __ticket_spin_lock+0x22/0x30
[ 696.174559] RSP: 0018:ffff8805d42e3958 EFLAGS: 00000006
[ 696.174564] RAX: 0000000000002d4b RBX: ffff8805dce22e80 RCX: 0000000000000000
[ 696.174569] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88051fac2488
[ 696.174574] RBP: ffff8805d42e3958 R08: 0000000000016d40 R09: ffff88063ec56d40
[ 696.174579] R10: ffffea0017696200 R11: ffffffffa02d183b R12: ffff88051fac2480
[ 696.174583] R13: ffff88051fac2488 R14: ffffffffffffffff R15: 0000000000000001
[ 696.174589] FS: 00007f1ac9a97700(0000) GS:ffff88063ec40000(0000) knlGS:0000000000000000
[ 696.174594] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 696.174599] CR2: 00007f2a6f2e0000 CR3: 00000005ddf2a000 CR4: 00000000000407e0
[ 696.174604] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 696.174609] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 696.174614] Process zfs (pid: 6459, threadinfo ffff8805d42e2000, task ffff8805dce22e80)
[ 696.174618] Stack:
[ 696.174621] ffff8805d42e3968 ffffffff816cde65 ffff8805d42e39d0 ffffffff816cdca6
[ 696.174633] 0000000000000000 0000000000000000 00000000d42e39d0 ffffffff00000000
[ 696.174643] ffffffff816cb7ad ffff8805cc05b970 ffff88051fac2480 0000000000000002
[ 696.174654] Call Trace:
[ 696.174665] [<ffffffff816cde65>] _raw_spin_lock_irq+0x15/0x20
[ 696.174674] [<ffffffff816cdca6>] rwsem_down_failed_common+0x46/0x170
[ 696.174683] [<ffffffff816cb7ad>] ? mutex_lock+0x1d/0x50
[ 696.174692] [<ffffffff816cde05>] rwsem_down_read_failed+0x15/0x17
[ 696.174702] [<ffffffff813555d4>] call_rwsem_down_read_failed+0x14/0x30
[ 696.174742] [<ffffffffa02d183b>] ? kmem_free_debug+0x4b/0x150 [spl]
[ 696.174755] [<ffffffff816cc0e0>] ? down_read+0x20/0x23
[ 696.174839] [<ffffffffa03b7210>] zap_get_leaf_byblk+0xd0/0x2c0 [zfs]
[ 696.174912] [<ffffffffa03b6c0b>] ? zap_idx_to_blk+0x13b/0x150 [zfs]
[ 696.174984] [<ffffffffa03b746d>] zap_deref_leaf+0x6d/0x80 [zfs]
[ 696.175057] [<ffffffffa03b9021>] fzap_cursor_retrieve+0xe1/0x280 [zfs]
[ 696.175129] [<ffffffffa03bc848>] zap_cursor_retrieve+0x68/0x2e0 [zfs]
[ 696.175197] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 696.175264] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 696.175344] [<ffffffffa03b86d1>] zap_value_search+0x91/0xd0 [zfs]
[ 696.175416] [<ffffffffa03e779d>] zfs_obj_to_path_impl+0x1bd/0x2b0 [zfs]
[ 696.175486] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 696.175554] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 696.175623] [<ffffffffa03e3261>] ? zfs_obj_to_stats_impl+0xa1/0xb0 [zfs]
[ 696.175695] [<ffffffffa03e79eb>] zfs_obj_to_stats+0xab/0xd0 [zfs]
[ 696.175765] [<ffffffffa03ca6d4>] zfs_ioc_obj_to_stats+0x84/0xa0 [zfs]
[ 696.175837] [<ffffffffa03d0879>] zfsdev_ioctl+0x4b9/0x530 [zfs]
[ 696.175846] [<ffffffff811a6659>] do_vfs_ioctl+0x99/0x570
[ 696.175855] [<ffffffff811a6bc1>] sys_ioctl+0x91/0xb0
[ 696.175865] [<ffffffff816d671d>] system_call_fastpath+0x1a/0x1f
[ 696.175870] Code: 84 00 00 00 00 00 0f 1f 00 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c1 c1 e9 10 66 39 c1 89 ca 74 11 0f 1f 80 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89
[ 696.175976] NMI backtrace for cpu 0
[ 696.175981] CPU 0
[ 696.175988] Pid: 0, comm: swapper/0 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 696.175993] RIP: 0010:[<ffffffff813e3eb7>] [<ffffffff813e3eb7>] acpi_idle_do_entry+0x21/0x2b
[ 696.176005] RSP: 0018:ffffffff81c01e10 EFLAGS: 00000093
[ 696.176009] RAX: 000000a216ff2d00 RBX: ffff88061ce308a0 RCX: 000000000000c648
[ 696.176014] RDX: 0000000000000815 RSI: ffff88063ec00000 RDI: ffff88061ce308a0
[ 696.176018] RBP: ffffffff81c01e10 R08: 0000000000002ea1 R09: 0000000000000018
[ 696.176023] R10: 0000000000002ec6 R11: 0000000000000000 R12: 0000000000000002
[ 696.176027] R13: ffffffff81c00000 R14: ffff88061ce30800 R15: ffffffff81574a50
[ 696.176033] FS: 00007fd1622d6740(0000) GS:ffff88063ec00000(0000) knlGS:0000000000000000
[ 696.176038] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 696.176042] CR2: 00007f8d10000020 CR3: 00000005ddefd000 CR4: 00000000000407f0
[ 696.176047] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 696.176051] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 696.176056] Process swapper/0 (pid: 0, threadinfo ffffffff81c00000, task ffffffff81c15440)
[ 696.176060] Stack:
[ 696.176063] ffffffff81c01e40 ffffffff813e3faa ffff880616740600 ffffffff81c7e060
[ 696.176074] 000000a2626e183b 0000000000000002 ffffffff81c01e50 ffffffff81574a69
[ 696.176084] ffffffff81c01eb0 ffffffff81575499 7735940000000001 ffff880616740600
[ 696.176094] Call Trace:
[ 696.176103] [<ffffffff813e3faa>] acpi_idle_enter_simple+0x87/0xc5
[ 696.176113] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 696.176122] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 696.176130] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 696.176139] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 696.176148] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 696.176158] [<ffffffff816abc32>] rest_init+0x72/0x80
[ 696.176168] [<ffffffff81d05be9>] start_kernel+0x3d1/0x3dc
[ 696.176176] [<ffffffff81d0566e>] ? do_early_param+0x8e/0x8e
[ 696.176186] [<ffffffff81d05355>] x86_64_start_reservations+0x130/0x133
[ 696.176195] [<ffffffff81d05458>] x86_64_start_kernel+0x100/0x10f
[ 696.176199] Code: 00 83 88 3c e0 ff ff 04 5d c3 8a 47 08 55 48 89 e5 3c 01 75 07 e8 3a 35 c5 ff eb 17 3c 02 75 07 e8 a2 ff ff ff eb 0c 8b 57 04 ec <48> 8b 15 86 e1 b2 00 ed 5d c3 66 66 66 66 90 55 48 63 c2 48 c1
[ 696.176303] NMI backtrace for cpu 2
[ 696.176308] CPU 2
[ 696.176317] Pid: 0, comm: swapper/2 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 696.176323] RIP: 0010:[<ffffffff813e3eb7>] [<ffffffff813e3eb7>] acpi_idle_do_entry+0x21/0x2b
[ 696.176337] RSP: 0018:ffff88061cd85e10 EFLAGS: 00000093
[ 696.176343] RAX: 000000a216df4c00 RBX: ffff88061cdb88a0 RCX: 000000000000c648
[ 696.176348] RDX: 0000000000000815 RSI: ffff88063ec80000 RDI: ffff88061cdb88a0
[ 696.176352] RBP: ffff88061cd85e10 R08: 000000000000272a R09: 0000000000000018
[ 696.176357] R10: 0000000000002729 R11: 0000000000000000 R12: 0000000000000002
[ 696.176361] R13: ffff88061cd84000 R14: ffff88061cdb8800 R15: ffffffff81574a50
[ 696.176367] FS: 00007f8879344700(0000) GS:ffff88063ec80000(0000) knlGS:0000000000000000
[ 696.176372] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 696.176377] CR2: 00007f88577fdfb8 CR3: 00000005c62fe000 CR4: 00000000000407e0
[ 696.176382] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 696.176387] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 696.176392] Process swapper/2 (pid: 0, threadinfo ffff88061cd84000, task ffff88061cd7ae80)
[ 696.176395] Stack:
[ 696.176399] ffff88061cd85e40 ffffffff813e3faa ffff88061c8c7200 ffffffff81c7e060
[ 696.176411] 000000a2624e2a7c 0000000000000002 ffff88061cd85e50 ffffffff81574a69
[ 696.176421] ffff88061cd85eb0 ffffffff81575499 7735940000000001 ffff88061c8c7200
[ 696.176431] Call Trace:
[ 696.176442] [<ffffffff813e3faa>] acpi_idle_enter_simple+0x87/0xc5
[ 696.176452] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 696.176461] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 696.176470] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 696.176478] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 696.176487] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 696.176496] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 696.176502] Code: 00 83 88 3c e0 ff ff 04 5d c3 8a 47 08 55 48 89 e5 3c 01 75 07 e8 3a 35 c5 ff eb 17 3c 02 75 07 e8 a2 ff ff ff eb 0c 8b 57 04 ec <48> 8b 15 86 e1 b2 00 ed 5d c3 66 66 66 66 90 55 48 63 c2 48 c1
[ 696.176608] NMI backtrace for cpu 3
[ 696.176611] CPU 3
[ 696.176617] Pid: 0, comm: swapper/3 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 696.176620] RIP: 0010:[<ffffffff81354161>] [<ffffffff81354161>] __const_udelay+0x1/0x30
[ 696.176628] RSP: 0018:ffff88063ecc3dd8 EFLAGS: 00000046
[ 696.176630] RAX: 0000000000000000 RBX: 0000000000002710 RCX: 0000000000000004
[ 696.176633] RDX: 0000000000000c00 RSI: 0000000000000100 RDI: 0000000000418958
[ 696.176636] RBP: ffff88063ecc3df0 R08: 000000000000000a R09: 0000000000000000
[ 696.176639] R10: 00000000000006a8 R11: 0000000000000002 R12: ffffffff81ce70c0
[ 696.176642] R13: ffff88063eccee60 R14: ffffffff81c3f3c0 R15: ffffffff81c3f480
[ 696.176645] FS: 00007f3fc198a700(0000) GS:ffff88063ecc0000(0000) knlGS:0000000000000000
[ 696.176648] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 696.176651] CR2: 00007f8844007a88 CR3: 00000005d4e3c000 CR4: 00000000000407e0
[ 696.176654] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 696.176657] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 696.176660] Process swapper/3 (pid: 0, threadinfo ffff88061cd86000, task ffff88061cd7c5c0)
[ 696.176662] Stack:
[ 696.176665] ffff88063ecc3df0 ffffffff81039d6a ffffffff81c3f3c0 ffff88063ecc3e50
[ 696.176671] ffffffff810f2935 000000000000f63b 0000000000000082 000000013ecc3e28
[ 696.176677] 0000000000000003 ffff88061cd86000 ffff88061cd7c5c0 0000000000000003
[ 696.176683] Call Trace:
[ 696.176686] <IRQ>
[ 696.176688] [<ffffffff81039d6a>] ? arch_trigger_all_cpu_backtrace+0x6a/0x90
[ 696.176700] [<ffffffff810f2935>] rcu_pending+0x565/0x5b0
[ 696.176707] [<ffffffff810f4a0b>] rcu_check_callbacks+0x9b/0x120
[ 696.176714] [<ffffffff8106a928>] update_process_times+0x48/0x90
[ 696.176720] [<ffffffff810b3a8e>] tick_sched_handle.isra.8+0x2e/0x70
[ 696.176725] [<ffffffff810b3c1c>] tick_sched_timer+0x4c/0x80
[ 696.176732] [<ffffffff810819d9>] __run_hrtimer+0x79/0x1c0
[ 696.176737] [<ffffffff810b3bd0>] ? tick_sched_do_timer+0x40/0x40
[ 696.176743] [<ffffffff810822c7>] hrtimer_interrupt+0xe7/0x230
[ 696.176749] [<ffffffff81574a50>] ? centrino_target+0x370/0x370
[ 696.176756] [<ffffffff816d8519>] smp_apic_timer_interrupt+0x69/0x99
[ 696.176763] [<ffffffff816d745d>] apic_timer_interrupt+0x6d/0x80
[ 696.176765] <EOI>
[ 696.176768] [<ffffffff81091a9d>] ? sched_clock_cpu+0xbd/0x110
[ 696.176778] [<ffffffff815754a8>] ? cpuidle_wrap_enter+0x58/0xa0
[ 696.176785] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 696.176790] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 696.176796] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 696.176802] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 696.176806] Code: ff c8 5d c3 66 0f 1f 44 00 00 66 66 66 66 90 55 48 89 e5 ff 15 49 25 92 00 5d c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 <48> 8d 04 bd 00 00 00 00 65 48 8b 14 25 18 3e 01 00 48 8d 0c 12
[ 696.176864] NMI backtrace for cpu 5
[ 696.176871] CPU 5
[ 696.176881] Pid: 0, comm: swapper/5 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 696.176887] RIP: 0010:[<ffffffff813e3eb7>] [<ffffffff813e3eb7>] acpi_idle_do_entry+0x21/0x2b
[ 696.176902] RSP: 0018:ffff88061cd8be10 EFLAGS: 00000093
[ 696.176907] RAX: 000000a216c1ff00 RBX: ffff88062600e0a0 RCX: 000000000000c648
[ 696.176912] RDX: 0000000000000815 RSI: ffff88063ed40000 RDI: ffff88062600e0a0
[ 696.176916] RBP: ffff88061cd8be10 R08: 0000000000002eab R09: 0000000000000018
[ 696.176921] R10: 00000000000007e7 R11: 0000000000000000 R12: 0000000000000002
[ 696.176925] R13: ffff88061cd8a000 R14: ffff88062600e000 R15: ffffffff81574a50
[ 696.176931] FS: 00007f887ab47700(0000) GS:ffff88063ed40000(0000) knlGS:0000000000000000
[ 696.176936] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 696.176940] CR2: 00007f88440028c8 CR3: 00000005c62fe000 CR4: 00000000000407e0
[ 696.176945] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 696.176950] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 696.176955] Process swapper/5 (pid: 0, threadinfo ffff88061cd8a000, task ffff88061cd90000)
[ 696.176959] Stack:
[ 696.176962] ffff88061cd8be40 ffffffff813e3faa ffff880616086000 ffffffff81c7e060
[ 696.176974] 000000a26230d1eb 0000000000000002 ffff88061cd8be50 ffffffff81574a69
[ 696.176985] ffff88061cd8beb0 ffffffff81575499 7735940000000001 ffff880616086000
[ 696.176995] Call Trace:
[ 696.177006] [<ffffffff813e3faa>] acpi_idle_enter_simple+0x87/0xc5
[ 696.177015] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 696.177024] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 696.177033] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 696.177041] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 696.177050] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 696.177059] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 696.177064] Code: 00 83 88 3c e0 ff ff 04 5d c3 8a 47 08 55 48 89 e5 3c 01 75 07 e8 3a 35 c5 ff eb 17 3c 02 75 07 e8 a2 ff ff ff eb 0c 8b 57 04 ec <48> 8b 15 86 e1 b2 00 ed 5d c3 66 66 66 66 90 55 48 63 c2 48 c1
[ 696.177170] NMI backtrace for cpu 4
[ 696.177173] CPU 4
[ 696.177179] Pid: 0, comm: swapper/4 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 696.177183] RIP: 0010:[<ffffffff813e3eb7>] [<ffffffff813e3eb7>] acpi_idle_do_entry+0x21/0x2b
[ 696.177189] RSP: 0018:ffff88061cd89e10 EFLAGS: 00000093
[ 696.177193] RAX: 000000a216db1d00 RBX: ffff88061ce310a0 RCX: 000000000000c648
[ 696.177195] RDX: 0000000000000815 RSI: ffff88063ed00000 RDI: ffff88061ce310a0
[ 696.177198] RBP: ffff88061cd89e10 R08: 000000000000283c R09: 0000000000000018
[ 696.177201] R10: 0000000000000b78 R11: 0000000000000000 R12: 0000000000000002
[ 696.177204] R13: ffff88061cd88000 R14: ffff88061ce31000 R15: ffffffff81574a50
[ 696.177207] FS: 00007fa534ff9700(0000) GS:ffff88063ed00000(0000) knlGS:0000000000000000
[ 696.177210] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 696.177213] CR2: 00007fd699e45000 CR3: 00000005e1941000 CR4: 00000000000407e0
[ 696.177216] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 696.177219] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 696.177222] Process swapper/4 (pid: 0, threadinfo ffff88061cd88000, task ffff88061cd7dd00)
[ 696.177224] Stack:
[ 696.177226] ffff88061cd89e40 ffffffff813e3faa ffff88061ca45600 ffffffff81c7e060
[ 696.177233] 000000a26249f948 0000000000000002 ffff88061cd89e50 ffffffff81574a69
[ 696.177239] ffff88061cd89eb0 ffffffff81575499 7735940000000001 ffff88061ca45600
[ 696.177245] Call Trace:
[ 696.177251] [<ffffffff813e3faa>] acpi_idle_enter_simple+0x87/0xc5
[ 696.177258] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 696.177263] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 696.177270] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 696.177275] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 696.177281] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 696.177287] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 696.177291] Code: 00 83 88 3c e0 ff ff 04 5d c3 8a 47 08 55 48 89 e5 3c 01 75 07 e8 3a 35 c5 ff eb 17 3c 02 75 07 e8 a2 ff ff ff eb 0c 8b 57 04 ec <48> 8b 15 86 e1 b2 00 ed 5d c3 66 66 66 66 90 55 48 63 c2 48 c1
[ 719.865045] INFO: task txg_sync:718 blocked for more than 120 seconds.
[ 719.865051] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 719.865055] txg_sync D ffff88063ec93ec0 0 718 2 0x00000000
[ 719.865061] ffff8805f56c1c50 0000000000000046 ffff880615bb5d00 ffff8805f56c1fd8
[ 719.865069] ffff8805f56c1fd8 ffff8805f56c1fd8 ffff88061cd7ae80 ffff880615bb5d00
[ 719.865076] ffff88061be1e550 ffff88061be1e530 ffff88061be1e558 0000000000000000
[ 719.865082] Call Trace:
[ 719.865094] [<ffffffff816ccf39>] schedule+0x29/0x70
[ 719.865134] [<ffffffffa02dd63d>] cv_wait_common+0xfd/0x1b0 [spl]
[ 719.865141] [<ffffffff8107def0>] ? finish_wait+0x80/0x80
[ 719.865165] [<ffffffffa02dd745>] __cv_wait+0x15/0x20 [spl]
[ 719.865238] [<ffffffffa038eaa5>] rrw_enter_write+0x35/0x70 [zfs]
[ 719.865300] [<ffffffffa038eaf3>] rrw_enter+0x13/0x30 [zfs]
[ 719.865364] [<ffffffffa039811e>] spa_sync+0x81e/0xb20 [zfs]
[ 719.865372] [<ffffffff810ac1c8>] ? ktime_get_ts+0x48/0xe0
[ 719.865440] [<ffffffffa03a885a>] txg_sync_thread+0x3aa/0x600 [zfs]
[ 719.865512] [<ffffffffa03a84b0>] ? txg_init+0x270/0x270 [zfs]
[ 719.865533] [<ffffffffa02d5b78>] thread_generic_wrapper+0x78/0x90 [spl]
[ 719.865555] [<ffffffffa02d5b00>] ? __thread_create+0x340/0x340 [spl]
[ 719.865560] [<ffffffff8107d5e0>] kthread+0xc0/0xd0
[ 719.865566] [<ffffffff8107d520>] ? kthread_create_on_node+0x120/0x120
[ 719.865572] [<ffffffff816d666c>] ret_from_fork+0x7c/0xb0
[ 719.865578] [<ffffffff8107d520>] ? kthread_create_on_node+0x120/0x120
[ 719.865616] INFO: task zfs:6459 blocked for more than 120 seconds.
[ 719.865618] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 719.865621] zfs D 0000000000000000 0 6459 4544 0x00000008
[ 719.865627] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 719.865633] ffff8805d42e3888 ffffffff81044b59 ffff8805d42e38a8 ffff8805d42e38a8
[ 719.865639] 0000000000000282 ffff8805d8cc9ec8 ffff8805d42e3958 ffff8805d42e38f8
[ 719.865645] Call Trace:
[ 719.865653] [<ffffffff81044b59>] ? default_spin_lock_flags+0x9/0x10
[ 719.865662] [<ffffffff8117e07d>] ? __kmalloc+0x13d/0x170
[ 719.865682] [<ffffffffa02d29c6>] ? kmem_alloc_debug+0x96/0x3b0 [spl]
[ 719.865704] [<ffffffffa02d29c6>] ? kmem_alloc_debug+0x96/0x3b0 [spl]
[ 719.865710] [<ffffffff816cb7ad>] ? mutex_lock+0x1d/0x50
[ 719.865756] [<ffffffffa0354ff1>] ? dbuf_find+0xe1/0xf0 [zfs]
[ 719.865763] [<ffffffff81044b59>] ? default_spin_lock_flags+0x9/0x10
[ 719.865810] [<ffffffffa03571e9>] ? dbuf_read+0x579/0x7f0 [zfs]
[ 719.865830] [<ffffffffa02d183b>] ? kmem_free_debug+0x4b/0x150 [spl]
[ 719.865836] [<ffffffff816cde65>] ? _raw_spin_lock_irq+0x15/0x20
[ 719.865842] [<ffffffff816cdca6>] ? rwsem_down_failed_common+0x46/0x170
[ 719.865847] [<ffffffff816cb7ad>] ? mutex_lock+0x1d/0x50
[ 719.865853] [<ffffffff816cde05>] ? rwsem_down_read_failed+0x15/0x17
[ 719.865859] [<ffffffff813555d4>] ? call_rwsem_down_read_failed+0x14/0x30
[ 719.865879] [<ffffffffa02d183b>] ? kmem_free_debug+0x4b/0x150 [spl]
[ 719.865889] [<ffffffff816cc0e0>] ? down_read+0x20/0x23
[ 719.865955] [<ffffffffa03b7210>] ? zap_get_leaf_byblk+0xd0/0x2c0 [zfs]
[ 719.866022] [<ffffffffa03b6c0b>] ? zap_idx_to_blk+0x13b/0x150 [zfs]
[ 719.866090] [<ffffffffa03b746d>] ? zap_deref_leaf+0x6d/0x80 [zfs]
[ 719.866158] [<ffffffffa03b9021>] ? fzap_cursor_retrieve+0xe1/0x280 [zfs]
[ 719.866227] [<ffffffffa03bc848>] ? zap_cursor_retrieve+0x68/0x2e0 [zfs]
[ 719.866293] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 719.866356] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 719.866427] [<ffffffffa03b86d1>] ? zap_value_search+0x91/0xd0 [zfs]
[ 719.866493] [<ffffffffa03e779d>] ? zfs_obj_to_path_impl+0x1bd/0x2b0 [zfs]
[ 719.866558] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 719.866621] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 719.866686] [<ffffffffa03e3261>] ? zfs_obj_to_stats_impl+0xa1/0xb0 [zfs]
[ 719.866752] [<ffffffffa03e79eb>] ? zfs_obj_to_stats+0xab/0xd0 [zfs]
[ 719.866817] [<ffffffffa03ca6d4>] ? zfs_ioc_obj_to_stats+0x84/0xa0 [zfs]
[ 719.866882] [<ffffffffa03d0879>] ? zfsdev_ioctl+0x4b9/0x530 [zfs]
[ 719.866890] [<ffffffff811a6659>] ? do_vfs_ioctl+0x99/0x570
[ 719.866896] [<ffffffff811a6bc1>] ? sys_ioctl+0x91/0xb0
[ 719.866903] [<ffffffff816d671d>] ? system_call_fastpath+0x1a/0x1f
[ 839.667169] INFO: task txg_sync:718 blocked for more than 120 seconds.
[ 839.667175] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 839.667179] txg_sync D ffff88063ec93ec0 0 718 2 0x00000000
[ 839.667186] ffff8805f56c1c50 0000000000000046 ffff880615bb5d00 ffff8805f56c1fd8
[ 839.667194] ffff8805f56c1fd8 ffff8805f56c1fd8 ffff88061cd7ae80 ffff880615bb5d00
[ 839.667201] ffff88061be1e550 ffff88061be1e530 ffff88061be1e558 0000000000000000
[ 839.667207] Call Trace:
[ 839.667219] [<ffffffff816ccf39>] schedule+0x29/0x70
[ 839.667258] [<ffffffffa02dd63d>] cv_wait_common+0xfd/0x1b0 [spl]
[ 839.667266] [<ffffffff8107def0>] ? finish_wait+0x80/0x80
[ 839.667290] [<ffffffffa02dd745>] __cv_wait+0x15/0x20 [spl]
[ 839.667361] [<ffffffffa038eaa5>] rrw_enter_write+0x35/0x70 [zfs]
[ 839.667423] [<ffffffffa038eaf3>] rrw_enter+0x13/0x30 [zfs]
[ 839.667487] [<ffffffffa039811e>] spa_sync+0x81e/0xb20 [zfs]
[ 839.667495] [<ffffffff810ac1c8>] ? ktime_get_ts+0x48/0xe0
[ 839.667562] [<ffffffffa03a885a>] txg_sync_thread+0x3aa/0x600 [zfs]
[ 839.667634] [<ffffffffa03a84b0>] ? txg_init+0x270/0x270 [zfs]
[ 839.667656] [<ffffffffa02d5b78>] thread_generic_wrapper+0x78/0x90 [spl]
[ 839.667677] [<ffffffffa02d5b00>] ? __thread_create+0x340/0x340 [spl]
[ 839.667682] [<ffffffff8107d5e0>] kthread+0xc0/0xd0
[ 839.667688] [<ffffffff8107d520>] ? kthread_create_on_node+0x120/0x120
[ 839.667695] [<ffffffff816d666c>] ret_from_fork+0x7c/0xb0
[ 839.667700] [<ffffffff8107d520>] ? kthread_create_on_node+0x120/0x120
[ 839.667735] INFO: task zfs:6459 blocked for more than 120 seconds.
[ 839.667737] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 839.667740] zfs D 0000000000000000 0 6459 4544 0x00000008
[ 839.667745] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 839.667752] ffff8805d42e3888 ffffffff81044b59 ffff8805d42e38a8 ffff8805d42e38a8
[ 839.667758] 0000000000000282 ffff8805d8cc9ec8 ffff8805d42e3958 ffff8805d42e38f8
[ 839.667764] Call Trace:
[ 839.667773] [<ffffffff81044b59>] ? default_spin_lock_flags+0x9/0x10
[ 839.667781] [<ffffffff8117e07d>] ? __kmalloc+0x13d/0x170
[ 839.667802] [<ffffffffa02d29c6>] ? kmem_alloc_debug+0x96/0x3b0 [spl]
[ 839.667823] [<ffffffffa02d29c6>] ? kmem_alloc_debug+0x96/0x3b0 [spl]
[ 839.667828] [<ffffffff816cb7ad>] ? mutex_lock+0x1d/0x50
[ 839.667874] [<ffffffffa0354ff1>] ? dbuf_find+0xe1/0xf0 [zfs]
[ 839.667881] [<ffffffff81044b59>] ? default_spin_lock_flags+0x9/0x10
[ 839.667928] [<ffffffffa03571e9>] ? dbuf_read+0x579/0x7f0 [zfs]
[ 839.667948] [<ffffffffa02d183b>] ? kmem_free_debug+0x4b/0x150 [spl]
[ 839.667954] [<ffffffff816cde65>] ? _raw_spin_lock_irq+0x15/0x20
[ 839.667960] [<ffffffff816cdca6>] ? rwsem_down_failed_common+0x46/0x170
[ 839.667966] [<ffffffff816cb7ad>] ? mutex_lock+0x1d/0x50
[ 839.667972] [<ffffffff816cde05>] ? rwsem_down_read_failed+0x15/0x17
[ 839.667978] [<ffffffff813555d4>] ? call_rwsem_down_read_failed+0x14/0x30
[ 839.667999] [<ffffffffa02d183b>] ? kmem_free_debug+0x4b/0x150 [spl]
[ 839.668008] [<ffffffff816cc0e0>] ? down_read+0x20/0x23
[ 839.668074] [<ffffffffa03b7210>] ? zap_get_leaf_byblk+0xd0/0x2c0 [zfs]
[ 839.668141] [<ffffffffa03b6c0b>] ? zap_idx_to_blk+0x13b/0x150 [zfs]
[ 839.668209] [<ffffffffa03b746d>] ? zap_deref_leaf+0x6d/0x80 [zfs]
[ 839.668276] [<ffffffffa03b9021>] ? fzap_cursor_retrieve+0xe1/0x280 [zfs]
[ 839.668343] [<ffffffffa03bc848>] ? zap_cursor_retrieve+0x68/0x2e0 [zfs]
[ 839.668409] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 839.668472] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 839.668544] [<ffffffffa03b86d1>] ? zap_value_search+0x91/0xd0 [zfs]
[ 839.668609] [<ffffffffa03e779d>] ? zfs_obj_to_path_impl+0x1bd/0x2b0 [zfs]
[ 839.668673] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 839.668736] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 839.668800] [<ffffffffa03e3261>] ? zfs_obj_to_stats_impl+0xa1/0xb0 [zfs]
[ 839.668866] [<ffffffffa03e79eb>] ? zfs_obj_to_stats+0xab/0xd0 [zfs]
[ 839.668931] [<ffffffffa03ca6d4>] ? zfs_ioc_obj_to_stats+0x84/0xa0 [zfs]
[ 839.668997] [<ffffffffa03d0879>] ? zfsdev_ioctl+0x4b9/0x530 [zfs]
[ 839.669004] [<ffffffff811a6659>] ? do_vfs_ioctl+0x99/0x570
[ 839.669010] [<ffffffff811a6bc1>] ? sys_ioctl+0x91/0xb0
[ 839.669016] [<ffffffff816d671d>] ? system_call_fastpath+0x1a/0x1f
[ 875.898541] INFO: rcu_sched detected stalls on CPUs/tasks: { 1} (detected by 0, t=150017 jiffies, g=32254, c=32253, q=76681)
[ 875.898556] sending NMI to all CPUs:
[ 875.898576] NMI backtrace for cpu 1
[ 875.898580] CPU 1
[ 875.898589] Pid: 6459, comm: zfs Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 875.898595] RIP: 0010:[<ffffffff81044ac2>] [<ffffffff81044ac2>] __ticket_spin_lock+0x22/0x30
[ 875.898608] RSP: 0018:ffff8805d42e3958 EFLAGS: 00000006
[ 875.898614] RAX: 0000000000002d4b RBX: ffff8805dce22e80 RCX: 0000000000000000
[ 875.898619] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88051fac2488
[ 875.898624] RBP: ffff8805d42e3958 R08: 0000000000016d40 R09: ffff88063ec56d40
[ 875.898628] R10: ffffea0017696200 R11: ffffffffa02d183b R12: ffff88051fac2480
[ 875.898633] R13: ffff88051fac2488 R14: ffffffffffffffff R15: 0000000000000001
[ 875.898639] FS: 00007f1ac9a97700(0000) GS:ffff88063ec40000(0000) knlGS:0000000000000000
[ 875.898644] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 875.898649] CR2: 00007f2a6f2e0000 CR3: 00000005ddf2a000 CR4: 00000000000407e0
[ 875.898653] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 875.898658] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 875.898664] Process zfs (pid: 6459, threadinfo ffff8805d42e2000, task ffff8805dce22e80)
[ 875.898667] Stack:
[ 875.898671] ffff8805d42e3968 ffffffff816cde65 ffff8805d42e39d0 ffffffff816cdca6
[ 875.898682] 0000000000000000 0000000000000000 00000000d42e39d0 ffffffff00000000
[ 875.898693] ffffffff816cb7ad ffff8805cc05b970 ffff88051fac2480 0000000000000002
[ 875.898703] Call Trace:
[ 875.898714] [<ffffffff816cde65>] _raw_spin_lock_irq+0x15/0x20
[ 875.898723] [<ffffffff816cdca6>] rwsem_down_failed_common+0x46/0x170
[ 875.898732] [<ffffffff816cb7ad>] ? mutex_lock+0x1d/0x50
[ 875.898741] [<ffffffff816cde05>] rwsem_down_read_failed+0x15/0x17
[ 875.898751] [<ffffffff813555d4>] call_rwsem_down_read_failed+0x14/0x30
[ 875.898784] [<ffffffffa02d183b>] ? kmem_free_debug+0x4b/0x150 [spl]
[ 875.898797] [<ffffffff816cc0e0>] ? down_read+0x20/0x23
[ 875.898877] [<ffffffffa03b7210>] zap_get_leaf_byblk+0xd0/0x2c0 [zfs]
[ 875.898949] [<ffffffffa03b6c0b>] ? zap_idx_to_blk+0x13b/0x150 [zfs]
[ 875.899023] [<ffffffffa03b746d>] zap_deref_leaf+0x6d/0x80 [zfs]
[ 875.899097] [<ffffffffa03b9021>] fzap_cursor_retrieve+0xe1/0x280 [zfs]
[ 875.899169] [<ffffffffa03bc848>] zap_cursor_retrieve+0x68/0x2e0 [zfs]
[ 875.899237] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 875.899305] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 875.899383] [<ffffffffa03b86d1>] zap_value_search+0x91/0xd0 [zfs]
[ 875.899454] [<ffffffffa03e779d>] zfs_obj_to_path_impl+0x1bd/0x2b0 [zfs]
[ 875.899524] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 875.899592] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 875.899662] [<ffffffffa03e3261>] ? zfs_obj_to_stats_impl+0xa1/0xb0 [zfs]
[ 875.899733] [<ffffffffa03e79eb>] zfs_obj_to_stats+0xab/0xd0 [zfs]
[ 875.899804] [<ffffffffa03ca6d4>] zfs_ioc_obj_to_stats+0x84/0xa0 [zfs]
[ 875.899876] [<ffffffffa03d0879>] zfsdev_ioctl+0x4b9/0x530 [zfs]
[ 875.899885] [<ffffffff811a6659>] do_vfs_ioctl+0x99/0x570
[ 875.899894] [<ffffffff811a6bc1>] sys_ioctl+0x91/0xb0
[ 875.899904] [<ffffffff816d671d>] system_call_fastpath+0x1a/0x1f
[ 875.899908] Code: 84 00 00 00 00 00 0f 1f 00 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c1 c1 e9 10 66 39 c1 89 ca 74 11 0f 1f 80 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89
[ 875.900016] NMI backtrace for cpu 0
[ 875.900020] CPU 0
[ 875.900027] Pid: 0, comm: swapper/0 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 875.900032] RIP: 0010:[<ffffffff81356bf3>] [<ffffffff81356bf3>] __bitmap_empty+0x13/0x80
[ 875.900042] RSP: 0018:ffff88063ec03dd8 EFLAGS: 00000002
[ 875.900046] RAX: 0000000000000000 RBX: 0000000000002710 RCX: 0000000000000004
[ 875.900051] RDX: 0000000000000c00 RSI: 0000000000000100 RDI: ffffffff81ce6b80
[ 875.900055] RBP: ffff88063ec03dd8 R08: 000000000000000a R09: 0000000000000000
[ 875.900060] R10: 00000000000007fd R11: 0000000000000002 R12: ffffffff81ce70c0
[ 875.900064] R13: ffff88063ec0ee60 R14: ffffffff81c3f3c0 R15: ffffffff81c3f480
[ 875.900070] FS: 00007fa5167fc700(0000) GS:ffff88063ec00000(0000) knlGS:0000000000000000
[ 875.900075] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 875.900079] CR2: 00007f8d10000020 CR3: 00000005e1941000 CR4: 00000000000407f0
[ 875.900084] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 875.900089] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 875.900094] Process swapper/0 (pid: 0, threadinfo ffffffff81c00000, task ffffffff81c15440)
[ 875.900097] Stack:
[ 875.900100] ffff88063ec03df0 ffffffff81039d80 ffffffff81c3f3c0 ffff88063ec03e50
[ 875.900111] ffffffff810f2935 0000000000012b89 0000000000000082 0000000100000086
[ 875.900121] 0000000000000000 ffffffff81c00000 ffffffff81c15440 0000000000000000
[ 875.900131] Call Trace:
[ 875.900135] <IRQ>
[ 875.900139] [<ffffffff81039d80>] arch_trigger_all_cpu_backtrace+0x80/0x90
[ 875.900155] [<ffffffff810f2935>] rcu_pending+0x565/0x5b0
[ 875.900164] [<ffffffff810f4a0b>] rcu_check_callbacks+0x9b/0x120
[ 875.900174] [<ffffffff8106a928>] update_process_times+0x48/0x90
[ 875.900183] [<ffffffff810b3a8e>] tick_sched_handle.isra.8+0x2e/0x70
[ 875.900190] [<ffffffff810b3c1c>] tick_sched_timer+0x4c/0x80
[ 875.900199] [<ffffffff810819d9>] __run_hrtimer+0x79/0x1c0
[ 875.900207] [<ffffffff810b3bd0>] ? tick_sched_do_timer+0x40/0x40
[ 875.900216] [<ffffffff810822c7>] hrtimer_interrupt+0xe7/0x230
[ 875.900226] [<ffffffff81574a50>] ? centrino_target+0x370/0x370
[ 875.900235] [<ffffffff816d8519>] smp_apic_timer_interrupt+0x69/0x99
[ 875.900244] [<ffffffff816d745d>] apic_timer_interrupt+0x6d/0x80
[ 875.900248] <EOI>
[ 875.900251] [<ffffffff81091a9d>] ? sched_clock_cpu+0xbd/0x110
[ 875.900266] [<ffffffff815754a8>] ? cpuidle_wrap_enter+0x58/0xa0
[ 875.900276] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 875.900284] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 875.900293] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 875.900302] [<ffffffff816abc32>] rest_init+0x72/0x80
[ 875.900312] [<ffffffff81d05be9>] start_kernel+0x3d1/0x3dc
[ 875.900320] [<ffffffff81d0566e>] ? do_early_param+0x8e/0x8e
[ 875.900330] [<ffffffff81d05355>] x86_64_start_reservations+0x130/0x133
[ 875.900338] [<ffffffff81d05458>] x86_64_start_kernel+0x100/0x10f
[ 875.900342] Code: 4c 89 4c 24 40 c7 04 24 10 00 00 00 48 89 44 24 10 e8 32 ff ff ff c9 c3 8d 4e 3f 85 f6 55 0f 49 ce 48 89 e5 c1 f9 06 85 c9 7e 61 <48> 83 3f 00 75 57 48 8d 57 08 31 c0 eb 12 0f 1f 80 00 00 00 00
[ 875.900449] NMI backtrace for cpu 2
[ 875.900457] CPU 2
[ 875.900466] Pid: 0, comm: swapper/2 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 875.900472] RIP: 0010:[<ffffffff813e3eb7>] [<ffffffff813e3eb7>] acpi_idle_do_entry+0x21/0x2b
[ 875.900487] RSP: 0018:ffff88061cd85e10 EFLAGS: 00000093
[ 875.900492] RAX: 000000cbef263500 RBX: ffff88061cdb88a0 RCX: 000000000000c648
[ 875.900497] RDX: 0000000000000815 RSI: ffff88063ec80000 RDI: ffff88061cdb88a0
[ 875.900502] RBP: ffff88061cd85e10 R08: 0000000000002ea8 R09: 0000000000000018
[ 875.900506] R10: 0000000000000a4a R11: 0000000000000000 R12: 0000000000000002
[ 875.900511] R13: ffff88061cd84000 R14: ffff88061cdb8800 R15: ffffffff81574a50
[ 875.900517] FS: 00007f3fa7fff700(0000) GS:ffff88063ec80000(0000) knlGS:0000000000000000
[ 875.900522] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 875.900526] CR2: 00000000018f7000 CR3: 00000005d4e3c000 CR4: 00000000000407e0
[ 875.900531] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 875.900536] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 875.900541] Process swapper/2 (pid: 0, threadinfo ffff88061cd84000, task ffff88061cd7ae80)
[ 875.900545] Stack:
[ 875.900548] ffff88061cd85e40 ffffffff813e3faa ffff88061c8c7200 ffffffff81c7e060
[ 875.900560] 000000cc4c38195d 0000000000000002 ffff88061cd85e50 ffffffff81574a69
[ 875.900570] ffff88061cd85eb0 ffffffff81575499 7735940000000001 ffff88061c8c7200
[ 875.900581] Call Trace:
[ 875.900592] [<ffffffff813e3faa>] acpi_idle_enter_simple+0x87/0xc5
[ 875.900602] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 875.900611] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 875.900620] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 875.900628] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 875.900637] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 875.900646] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 875.900652] Code: 00 83 88 3c e0 ff ff 04 5d c3 8a 47 08 55 48 89 e5 3c 01 75 07 e8 3a 35 c5 ff eb 17 3c 02 75 07 e8 a2 ff ff ff eb 0c 8b 57 04 ec <48> 8b 15 86 e1 b2 00 ed 5d c3 66 66 66 66 90 55 48 63 c2 48 c1
[ 875.900758] NMI backtrace for cpu 3
[ 875.900760] CPU 3
[ 875.900767] Pid: 0, comm: swapper/3 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 875.900770] RIP: 0010:[<ffffffff81043fa6>] [<ffffffff81043fa6>] native_safe_halt+0x6/0x10
[ 875.900777] RSP: 0018:ffff88061cd87de8 EFLAGS: 00000246
[ 875.900780] RAX: 0000000000000000 RBX: ffff8806167d546c RCX: ffffffff81c7e060
[ 875.900783] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8806167d546c
[ 875.900786] RBP: ffff88061cd87de8 R08: 0000000000002e8d R09: 000000000000001c
[ 875.900788] R10: 0000000000001b8e R11: ffff88061cd87e48 R12: 0000000000000001
[ 875.900791] R13: ffff8806167d5400 R14: 0000000000000001 R15: ffffffff81574a50
[ 875.900795] FS: 00007f88577fe700(0000) GS:ffff88063ecc0000(0000) knlGS:0000000000000000
[ 875.900798] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 875.900800] CR2: 00007f8844007a88 CR3: 00000005c62fe000 CR4: 00000000000407e0
[ 875.900803] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 875.900806] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 875.900809] Process swapper/3 (pid: 0, threadinfo ffff88061cd86000, task ffff88061cd7c5c0)
[ 875.900811] Stack:
[ 875.900813] ffff88061cd87df8 ffffffff813e3769 ffff88061cd87e08 ffffffff813e3e7d
[ 875.900820] ffff88061cd87e18 ffffffff813e3eb1 ffff88061cd87e40 ffffffff813e3f04
[ 875.900826] ffff880616006400 ffffffff81c7e060 000000cc4bfb7537 ffff88061cd87e50
[ 875.900832] Call Trace:
[ 875.900838] [<ffffffff813e3769>] arch_safe_halt+0x9/0xd
[ 875.900844] [<ffffffff813e3e7d>] acpi_safe_halt+0x2a/0x43
[ 875.900850] [<ffffffff813e3eb1>] acpi_idle_do_entry+0x1b/0x2b
[ 875.900855] [<ffffffff813e3f04>] acpi_idle_enter_c1+0x43/0x62
[ 875.900861] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 875.900867] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 875.900873] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 875.900878] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 875.900885] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 875.900890] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 875.900894] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 875.900951] NMI backtrace for cpu 4
[ 875.900959] CPU 4
[ 875.900968] Pid: 0, comm: swapper/4 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 875.900974] RIP: 0010:[<ffffffff813e3eb7>] [<ffffffff813e3eb7>] acpi_idle_do_entry+0x21/0x2b
[ 875.900988] RSP: 0018:ffff88061cd89e10 EFLAGS: 00000093
[ 875.900994] RAX: 000000cbef644900 RBX: ffff88061ce310a0 RCX: 000000000000c648
[ 875.900998] RDX: 0000000000000815 RSI: ffff88063ed00000 RDI: ffff88061ce310a0
[ 875.901003] RBP: ffff88061cd89e10 R08: 0000000000002e5c R09: 0000000000000018
[ 875.901007] R10: 0000000000000878 R11: 0000000000000000 R12: 0000000000000002
[ 875.901012] R13: ffff88061cd88000 R14: ffff88061ce31000 R15: ffffffff81574a50
[ 875.901018] FS: 00007f8879344700(0000) GS:ffff88063ed00000(0000) knlGS:0000000000000000
[ 875.901023] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 875.901027] CR2: 00007f88577fdfb8 CR3: 00000005c62fe000 CR4: 00000000000407e0
[ 875.901032] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 875.901037] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 875.901042] Process swapper/4 (pid: 0, threadinfo ffff88061cd88000, task ffff88061cd7dd00)
[ 875.901045] Stack:
[ 875.901049] ffff88061cd89e40 ffffffff813e3faa ffff88061ca45600 ffffffff81c7e060
[ 875.901061] 000000cc4c7647db 0000000000000002 ffff88061cd89e50 ffffffff81574a69
[ 875.901071] ffff88061cd89eb0 ffffffff81575499 7735940000000001 ffff88061ca45600
[ 875.901082] Call Trace:
[ 875.901092] [<ffffffff813e3faa>] acpi_idle_enter_simple+0x87/0xc5
[ 875.901102] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 875.901111] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 875.901120] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 875.901128] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 875.901137] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 875.901146] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 875.901151] Code: 00 83 88 3c e0 ff ff 04 5d c3 8a 47 08 55 48 89 e5 3c 01 75 07 e8 3a 35 c5 ff eb 17 3c 02 75 07 e8 a2 ff ff ff eb 0c 8b 57 04 ec <48> 8b 15 86 e1 b2 00 ed 5d c3 66 66 66 66 90 55 48 63 c2 48 c1
[ 875.901257] NMI backtrace for cpu 5
[ 875.901260] CPU 5
[ 875.901266] Pid: 0, comm: swapper/5 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 875.901269] RIP: 0010:[<ffffffff813e3eb7>] [<ffffffff813e3eb7>] acpi_idle_do_entry+0x21/0x2b
[ 875.901276] RSP: 0018:ffff88061cd8be10 EFLAGS: 00000093
[ 875.901279] RAX: 000000cbef275c00 RBX: ffff88062600e0a0 RCX: 000000000000c648
[ 875.901282] RDX: 0000000000000815 RSI: ffff88063ed40000 RDI: ffff88062600e0a0
[ 875.901285] RBP: ffff88061cd8be10 R08: 0000000000002e5c R09: 0000000000000018
[ 875.901288] R10: 0000000000002ea5 R11: 0000000000000000 R12: 0000000000000002
[ 875.901290] R13: ffff88061cd8a000 R14: ffff88062600e000 R15: ffffffff81574a50
[ 875.901294] FS: 00007fa5177fe700(0000) GS:ffff88063ed40000(0000) knlGS:0000000000000000
[ 875.901297] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 875.901300] CR2: 00007f88440028c8 CR3: 00000005e1941000 CR4: 00000000000407e0
[ 875.901303] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 875.901305] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 875.901309] Process swapper/5 (pid: 0, threadinfo ffff88061cd8a000, task ffff88061cd90000)
[ 875.901311] Stack:
[ 875.901313] ffff88061cd8be40 ffffffff813e3faa ffff880616086000 ffffffff81c7e060
[ 875.901319] 000000cc4c393ff4 0000000000000002 ffff88061cd8be50 ffffffff81574a69
[ 875.901325] ffff88061cd8beb0 ffffffff81575499 7735940000000001 ffff880616086000
[ 875.901331] Call Trace:
[ 875.901337] [<ffffffff813e3faa>] acpi_idle_enter_simple+0x87/0xc5
[ 875.901343] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 875.901349] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 875.901355] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 875.901361] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 875.901367] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 875.901372] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 875.901376] Code: 00 83 88 3c e0 ff ff 04 5d c3 8a 47 08 55 48 89 e5 3c 01 75 07 e8 3a 35 c5 ff eb 17 3c 02 75 07 e8 a2 ff ff ff eb 0c 8b 57 04 ec <48> 8b 15 86 e1 b2 00 ed 5d c3 66 66 66 66 90 55 48 63 c2 48 c1
[ 959.462938] INFO: task txg_sync:718 blocked for more than 120 seconds.
[ 959.462943] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 959.462947] txg_sync D ffff88063ec93ec0 0 718 2 0x00000000
[ 959.462954] ffff8805f56c1c50 0000000000000046 ffff880615bb5d00 ffff8805f56c1fd8
[ 959.462962] ffff8805f56c1fd8 ffff8805f56c1fd8 ffff88061cd7ae80 ffff880615bb5d00
[ 959.462969] ffff88061be1e550 ffff88061be1e530 ffff88061be1e558 0000000000000000
[ 959.462975] Call Trace:
[ 959.462987] [<ffffffff816ccf39>] schedule+0x29/0x70
[ 959.463025] [<ffffffffa02dd63d>] cv_wait_common+0xfd/0x1b0 [spl]
[ 959.463033] [<ffffffff8107def0>] ? finish_wait+0x80/0x80
[ 959.463057] [<ffffffffa02dd745>] __cv_wait+0x15/0x20 [spl]
[ 959.463127] [<ffffffffa038eaa5>] rrw_enter_write+0x35/0x70 [zfs]
[ 959.463190] [<ffffffffa038eaf3>] rrw_enter+0x13/0x30 [zfs]
[ 959.463254] [<ffffffffa039811e>] spa_sync+0x81e/0xb20 [zfs]
[ 959.463262] [<ffffffff810ac1c8>] ? ktime_get_ts+0x48/0xe0
[ 959.463329] [<ffffffffa03a885a>] txg_sync_thread+0x3aa/0x600 [zfs]
[ 959.463400] [<ffffffffa03a84b0>] ? txg_init+0x270/0x270 [zfs]
[ 959.463421] [<ffffffffa02d5b78>] thread_generic_wrapper+0x78/0x90 [spl]
[ 959.463442] [<ffffffffa02d5b00>] ? __thread_create+0x340/0x340 [spl]
[ 959.463447] [<ffffffff8107d5e0>] kthread+0xc0/0xd0
[ 959.463453] [<ffffffff8107d520>] ? kthread_create_on_node+0x120/0x120
[ 959.463460] [<ffffffff816d666c>] ret_from_fork+0x7c/0xb0
[ 959.463465] [<ffffffff8107d520>] ? kthread_create_on_node+0x120/0x120
[ 959.463501] INFO: task zfs:6459 blocked for more than 120 seconds.
[ 959.463503] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 959.463506] zfs D 0000000000000000 0 6459 4544 0x00000008
[ 959.463512] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 959.463518] ffff8805d42e3888 ffffffff81044b59 ffff8805d42e38a8 ffff8805d42e38a8
[ 959.463524] 0000000000000282 ffff8805d8cc9ec8 ffff8805d42e3958 ffff8805d42e38f8
[ 959.463531] Call Trace:
[ 959.463539] [<ffffffff81044b59>] ? default_spin_lock_flags+0x9/0x10
[ 959.463547] [<ffffffff8117e07d>] ? __kmalloc+0x13d/0x170
[ 959.463568] [<ffffffffa02d29c6>] ? kmem_alloc_debug+0x96/0x3b0 [spl]
[ 959.463588] [<ffffffffa02d29c6>] ? kmem_alloc_debug+0x96/0x3b0 [spl]
[ 959.463594] [<ffffffff816cb7ad>] ? mutex_lock+0x1d/0x50
[ 959.463640] [<ffffffffa0354ff1>] ? dbuf_find+0xe1/0xf0 [zfs]
[ 959.463647] [<ffffffff81044b59>] ? default_spin_lock_flags+0x9/0x10
[ 959.463694] [<ffffffffa03571e9>] ? dbuf_read+0x579/0x7f0 [zfs]
[ 959.463715] [<ffffffffa02d183b>] ? kmem_free_debug+0x4b/0x150 [spl]
[ 959.463721] [<ffffffff816cde65>] ? _raw_spin_lock_irq+0x15/0x20
[ 959.463727] [<ffffffff816cdca6>] ? rwsem_down_failed_common+0x46/0x170
[ 959.463732] [<ffffffff816cb7ad>] ? mutex_lock+0x1d/0x50
[ 959.463738] [<ffffffff816cde05>] ? rwsem_down_read_failed+0x15/0x17
[ 959.463745] [<ffffffff813555d4>] ? call_rwsem_down_read_failed+0x14/0x30
[ 959.463765] [<ffffffffa02d183b>] ? kmem_free_debug+0x4b/0x150 [spl]
[ 959.463775] [<ffffffff816cc0e0>] ? down_read+0x20/0x23
[ 959.463840] [<ffffffffa03b7210>] ? zap_get_leaf_byblk+0xd0/0x2c0 [zfs]
[ 959.463907] [<ffffffffa03b6c0b>] ? zap_idx_to_blk+0x13b/0x150 [zfs]
[ 959.463976] [<ffffffffa03b746d>] ? zap_deref_leaf+0x6d/0x80 [zfs]
[ 959.464044] [<ffffffffa03b9021>] ? fzap_cursor_retrieve+0xe1/0x280 [zfs]
[ 959.464111] [<ffffffffa03bc848>] ? zap_cursor_retrieve+0x68/0x2e0 [zfs]
[ 959.464177] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 959.464240] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 959.464311] [<ffffffffa03b86d1>] ? zap_value_search+0x91/0xd0 [zfs]
[ 959.464377] [<ffffffffa03e779d>] ? zfs_obj_to_path_impl+0x1bd/0x2b0 [zfs]
[ 959.464442] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 959.464505] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 959.464569] [<ffffffffa03e3261>] ? zfs_obj_to_stats_impl+0xa1/0xb0 [zfs]
[ 959.464634] [<ffffffffa03e79eb>] ? zfs_obj_to_stats+0xab/0xd0 [zfs]
[ 959.464699] [<ffffffffa03ca6d4>] ? zfs_ioc_obj_to_stats+0x84/0xa0 [zfs]
[ 959.464765] [<ffffffffa03d0879>] ? zfsdev_ioctl+0x4b9/0x530 [zfs]
[ 959.464772] [<ffffffff811a6659>] ? do_vfs_ioctl+0x99/0x570
[ 959.464779] [<ffffffff811a6bc1>] ? sys_ioctl+0x91/0xb0
[ 959.464785] [<ffffffff816d671d>] ? system_call_fastpath+0x1a/0x1f
[ 1055.608158] INFO: rcu_sched detected stalls on CPUs/tasks: { 1} (detected by 4, t=195023 jiffies, g=32254, c=32253, q=86429)
[ 1055.608169] sending NMI to all CPUs:
[ 1055.608189] NMI backtrace for cpu 1
[ 1055.608197] CPU 1
[ 1055.608206] Pid: 6459, comm: zfs Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 1055.608212] RIP: 0010:[<ffffffff81044ac2>] [<ffffffff81044ac2>] __ticket_spin_lock+0x22/0x30
[ 1055.608226] RSP: 0018:ffff8805d42e3958 EFLAGS: 00000006
[ 1055.608232] RAX: 0000000000002d4b RBX: ffff8805dce22e80 RCX: 0000000000000000
[ 1055.608237] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88051fac2488
[ 1055.608242] RBP: ffff8805d42e3958 R08: 0000000000016d40 R09: ffff88063ec56d40
[ 1055.608246] R10: ffffea0017696200 R11: ffffffffa02d183b R12: ffff88051fac2480
[ 1055.608251] R13: ffff88051fac2488 R14: ffffffffffffffff R15: 0000000000000001
[ 1055.608257] FS: 00007f1ac9a97700(0000) GS:ffff88063ec40000(0000) knlGS:0000000000000000
[ 1055.608262] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1055.608267] CR2: 00007f2a6f2e0000 CR3: 00000005ddf2a000 CR4: 00000000000407e0
[ 1055.608272] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1055.608277] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1055.608282] Process zfs (pid: 6459, threadinfo ffff8805d42e2000, task ffff8805dce22e80)
[ 1055.608286] Stack:
[ 1055.608289] ffff8805d42e3968 ffffffff816cde65 ffff8805d42e39d0 ffffffff816cdca6
[ 1055.608301] 0000000000000000 0000000000000000 00000000d42e39d0 ffffffff00000000
[ 1055.608311] ffffffff816cb7ad ffff8805cc05b970 ffff88051fac2480 0000000000000002
[ 1055.608322] Call Trace:
[ 1055.608333] [<ffffffff816cde65>] _raw_spin_lock_irq+0x15/0x20
[ 1055.608342] [<ffffffff816cdca6>] rwsem_down_failed_common+0x46/0x170
[ 1055.608351] [<ffffffff816cb7ad>] ? mutex_lock+0x1d/0x50
[ 1055.608360] [<ffffffff816cde05>] rwsem_down_read_failed+0x15/0x17
[ 1055.608369] [<ffffffff813555d4>] call_rwsem_down_read_failed+0x14/0x30
[ 1055.608405] [<ffffffffa02d183b>] ? kmem_free_debug+0x4b/0x150 [spl]
[ 1055.608418] [<ffffffff816cc0e0>] ? down_read+0x20/0x23
[ 1055.608499] [<ffffffffa03b7210>] zap_get_leaf_byblk+0xd0/0x2c0 [zfs]
[ 1055.608571] [<ffffffffa03b6c0b>] ? zap_idx_to_blk+0x13b/0x150 [zfs]
[ 1055.608645] [<ffffffffa03b746d>] zap_deref_leaf+0x6d/0x80 [zfs]
[ 1055.608717] [<ffffffffa03b9021>] fzap_cursor_retrieve+0xe1/0x280 [zfs]
[ 1055.608790] [<ffffffffa03bc848>] zap_cursor_retrieve+0x68/0x2e0 [zfs]
[ 1055.608857] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 1055.608926] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 1055.609003] [<ffffffffa03b86d1>] zap_value_search+0x91/0xd0 [zfs]
[ 1055.609073] [<ffffffffa03e779d>] zfs_obj_to_path_impl+0x1bd/0x2b0 [zfs]
[ 1055.609143] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 1055.609211] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 1055.609281] [<ffffffffa03e3261>] ? zfs_obj_to_stats_impl+0xa1/0xb0 [zfs]
[ 1055.609352] [<ffffffffa03e79eb>] zfs_obj_to_stats+0xab/0xd0 [zfs]
[ 1055.609422] [<ffffffffa03ca6d4>] zfs_ioc_obj_to_stats+0x84/0xa0 [zfs]
[ 1055.609494] [<ffffffffa03d0879>] zfsdev_ioctl+0x4b9/0x530 [zfs]
[ 1055.609504] [<ffffffff811a6659>] do_vfs_ioctl+0x99/0x570
[ 1055.609513] [<ffffffff811a6bc1>] sys_ioctl+0x91/0xb0
[ 1055.609522] [<ffffffff816d671d>] system_call_fastpath+0x1a/0x1f
[ 1055.609527] Code: 84 00 00 00 00 00 0f 1f 00 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c1 c1 e9 10 66 39 c1 89 ca 74 11 0f 1f 80 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89
[ 1055.609634] NMI backtrace for cpu 0
[ 1055.609638] CPU 0
[ 1055.609645] Pid: 0, comm: swapper/0 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 1055.609650] RIP: 0010:[<ffffffff813e3eb7>] [<ffffffff813e3eb7>] acpi_idle_do_entry+0x21/0x2b
[ 1055.609662] RSP: 0018:ffffffff81c01e10 EFLAGS: 00000093
[ 1055.609667] RAX: 000000f5c6e2d500 RBX: ffff88061ce308a0 RCX: 000000000000c648
[ 1055.609672] RDX: 0000000000000815 RSI: ffff88063ec00000 RDI: ffff88061ce308a0
[ 1055.609676] RBP: ffffffff81c01e10 R08: 00000000000020c4 R09: 0000000000000018
[ 1055.609681] R10: 0000000000000b71 R11: 0000000000000000 R12: 0000000000000002
[ 1055.609685] R13: ffffffff81c00000 R14: ffff88061ce30800 R15: ffffffff81574a50
[ 1055.609691] FS: 00007fa5167fc700(0000) GS:ffff88063ec00000(0000) knlGS:0000000000000000
[ 1055.609696] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1055.609700] CR2: 00007f8d10000020 CR3: 00000005e1941000 CR4: 00000000000407f0
[ 1055.609705] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1055.609709] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1055.609714] Process swapper/0 (pid: 0, threadinfo ffffffff81c00000, task ffffffff81c15440)
[ 1055.609718] Stack:
[ 1055.609721] ffffffff81c01e40 ffffffff813e3faa ffff880616740600 ffffffff81c7e060
[ 1055.609732] 000000f636b2a839 0000000000000002 ffffffff81c01e50 ffffffff81574a69
[ 1055.609742] ffffffff81c01eb0 ffffffff81575499 7735940000000001 ffff880616740600
[ 1055.609752] Call Trace:
[ 1055.609761] [<ffffffff813e3faa>] acpi_idle_enter_simple+0x87/0xc5
[ 1055.609771] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 1055.609779] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 1055.609788] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 1055.609797] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 1055.609806] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 1055.609816] [<ffffffff816abc32>] rest_init+0x72/0x80
[ 1055.609825] [<ffffffff81d05be9>] start_kernel+0x3d1/0x3dc
[ 1055.609834] [<ffffffff81d0566e>] ? do_early_param+0x8e/0x8e
[ 1055.609844] [<ffffffff81d05355>] x86_64_start_reservations+0x130/0x133
[ 1055.609852] [<ffffffff81d05458>] x86_64_start_kernel+0x100/0x10f
[ 1055.609856] Code: 00 83 88 3c e0 ff ff 04 5d c3 8a 47 08 55 48 89 e5 3c 01 75 07 e8 3a 35 c5 ff eb 17 3c 02 75 07 e8 a2 ff ff ff eb 0c 8b 57 04 ec <48> 8b 15 86 e1 b2 00 ed 5d c3 66 66 66 66 90 55 48 63 c2 48 c1
[ 1055.609961] NMI backtrace for cpu 5
[ 1055.609966] CPU 5
[ 1055.609975] Pid: 0, comm: swapper/5 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 1055.609980] RIP: 0010:[<ffffffff813e3eb7>] [<ffffffff813e3eb7>] acpi_idle_do_entry+0x21/0x2b
[ 1055.609995] RSP: 0018:ffff88061cd8be10 EFLAGS: 00000093
[ 1055.610000] RAX: 000000f5c670fb00 RBX: ffff88062600e0a0 RCX: 000000000000c648
[ 1055.610005] RDX: 0000000000000815 RSI: ffff88063ed40000 RDI: ffff88062600e0a0
[ 1055.610009] RBP: ffff88061cd8be10 R08: 0000000000003df5 R09: 0000000000000018
[ 1055.610014] R10: 0000000000003e5f R11: 0000000000000000 R12: 0000000000000002
[ 1055.610018] R13: ffff88061cd8a000 R14: ffff88062600e000 R15: ffffffff81574a50
[ 1055.610024] FS: 00007fa5177fe700(0000) GS:ffff88063ed40000(0000) knlGS:0000000000000000
[ 1055.610029] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1055.610034] CR2: 00007f88440028c8 CR3: 00000005e1941000 CR4: 00000000000407e0
[ 1055.610039] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1055.610043] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1055.610049] Process swapper/5 (pid: 0, threadinfo ffff88061cd8a000, task ffff88061cd90000)
[ 1055.610052] Stack:
[ 1055.610056] ffff88061cd8be40 ffffffff813e3faa ffff880616086000 ffffffff81c7e060
[ 1055.610067] 000000f636409938 0000000000000002 ffff88061cd8be50 ffffffff81574a69
[ 1055.610078] ffff88061cd8beb0 ffffffff81575499 7735940000000001 ffff880616086000
[ 1055.610088] Call Trace:
[ 1055.610099] [<ffffffff813e3faa>] acpi_idle_enter_simple+0x87/0xc5
[ 1055.610109] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 1055.610117] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 1055.610126] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 1055.610134] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 1055.610144] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 1055.610152] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 1055.610158] Code: 00 83 88 3c e0 ff ff 04 5d c3 8a 47 08 55 48 89 e5 3c 01 75 07 e8 3a 35 c5 ff eb 17 3c 02 75 07 e8 a2 ff ff ff eb 0c 8b 57 04 ec <48> 8b 15 86 e1 b2 00 ed 5d c3 66 66 66 66 90 55 48 63 c2 48 c1
[ 1055.610263] NMI backtrace for cpu 4
[ 1055.610266] CPU 4
[ 1055.610272] Pid: 0, comm: swapper/4 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 1055.610276] RIP: 0010:[<ffffffff81354161>] [<ffffffff81354161>] __const_udelay+0x1/0x30
[ 1055.610284] RSP: 0018:ffff88063ed03dd8 EFLAGS: 00000046
[ 1055.610286] RAX: 0000000000000000 RBX: 0000000000002710 RCX: 0000000000000004
[ 1055.610289] RDX: 0000000000000c00 RSI: 0000000000000100 RDI: 0000000000418958
[ 1055.610292] RBP: ffff88063ed03df0 R08: 000000000000000a R09: 0000000000000000
[ 1055.610295] R10: 0000000000000915 R11: 0000000000000002 R12: ffffffff81ce70c0
[ 1055.610297] R13: ffff88063ed0ee60 R14: ffffffff81c3f3c0 R15: ffffffff81c3f480
[ 1055.610301] FS: 00007f8879344700(0000) GS:ffff88063ed00000(0000) knlGS:0000000000000000
[ 1055.610304] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1055.610307] CR2: 00007fd699e45000 CR3: 00000005c62fe000 CR4: 00000000000407e0
[ 1055.610310] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1055.610313] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1055.610316] Process swapper/4 (pid: 0, threadinfo ffff88061cd88000, task ffff88061cd7dd00)
[ 1055.610318] Stack:
[ 1055.610320] ffff88063ed03df0 ffffffff81039d6a ffffffff81c3f3c0 ffff88063ed03e50
[ 1055.610326] ffffffff810f2935 000000000001519d 0000000000000082 000000010002e3bc
[ 1055.610332] 0000000000000004 ffff88061cd88000 ffff88061cd7dd00 0000000000000004
[ 1055.610338] Call Trace:
[ 1055.610341] <IRQ>
[ 1055.610343] [<ffffffff81039d6a>] ? arch_trigger_all_cpu_backtrace+0x6a/0x90
[ 1055.610356] [<ffffffff810f2935>] rcu_pending+0x565/0x5b0
[ 1055.610363] [<ffffffff810f4a0b>] rcu_check_callbacks+0x9b/0x120
[ 1055.610369] [<ffffffff8106a928>] update_process_times+0x48/0x90
[ 1055.610376] [<ffffffff810b3a8e>] tick_sched_handle.isra.8+0x2e/0x70
[ 1055.610381] [<ffffffff810b3c1c>] tick_sched_timer+0x4c/0x80
[ 1055.610387] [<ffffffff810819d9>] __run_hrtimer+0x79/0x1c0
[ 1055.610392] [<ffffffff810b3bd0>] ? tick_sched_do_timer+0x40/0x40
[ 1055.610398] [<ffffffff810822c7>] hrtimer_interrupt+0xe7/0x230
[ 1055.610405] [<ffffffff81574a50>] ? centrino_target+0x370/0x370
[ 1055.610411] [<ffffffff816d8519>] smp_apic_timer_interrupt+0x69/0x99
[ 1055.610418] [<ffffffff816d745d>] apic_timer_interrupt+0x6d/0x80
[ 1055.610420] <EOI>
[ 1055.610422] [<ffffffff81091a9d>] ? sched_clock_cpu+0xbd/0x110
[ 1055.610433] [<ffffffff815754a8>] ? cpuidle_wrap_enter+0x58/0xa0
[ 1055.610440] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 1055.610446] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 1055.610452] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 1055.610457] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 1055.610461] Code: ff c8 5d c3 66 0f 1f 44 00 00 66 66 66 66 90 55 48 89 e5 ff 15 49 25 92 00 5d c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 <48> 8d 04 bd 00 00 00 00 65 48 8b 14 25 18 3e 01 00 48 8d 0c 12
[ 1055.610519] NMI backtrace for cpu 3
[ 1055.610527] CPU 3
[ 1055.610536] Pid: 0, comm: swapper/3 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 1055.610542] RIP: 0010:[<ffffffff813e3eb7>] [<ffffffff813e3eb7>] acpi_idle_do_entry+0x21/0x2b
[ 1055.610556] RSP: 0018:ffff88061cd87e10 EFLAGS: 00000093
[ 1055.610561] RAX: 000000f5c670f700 RBX: ffff8806167d54a0 RCX: 000000000000c648
[ 1055.610566] RDX: 0000000000000815 RSI: ffff88063ecc0000 RDI: ffff8806167d54a0
[ 1055.610571] RBP: ffff88061cd87e10 R08: 0000000000002e54 R09: 0000000000000018
[ 1055.610575] R10: 00000000000009e7 R11: 0000000000000000 R12: 0000000000000002
[ 1055.610580] R13: ffff88061cd86000 R14: ffff8806167d5400 R15: ffffffff81574a50
[ 1055.610586] FS: 00007f887ab47700(0000) GS:ffff88063ecc0000(0000) knlGS:0000000000000000
[ 1055.610591] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1055.610595] CR2: 0000000000449e70 CR3: 00000005c62fe000 CR4: 00000000000407e0
[ 1055.610600] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1055.610604] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1055.610610] Process swapper/3 (pid: 0, threadinfo ffff88061cd86000, task ffff88061cd7c5c0)
[ 1055.610613] Stack:
[ 1055.610617] ffff88061cd87e40 ffffffff813e3faa ffff880616006400 ffffffff81c7e060
[ 1055.610629] 000000f6364095e9 0000000000000002 ffff88061cd87e50 ffffffff81574a69
[ 1055.610639] ffff88061cd87eb0 ffffffff81575499 7735940000000001 ffff880616006400
[ 1055.610649] Call Trace:
[ 1055.610660] [<ffffffff813e3faa>] acpi_idle_enter_simple+0x87/0xc5
[ 1055.610670] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 1055.610678] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 1055.610688] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 1055.610696] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 1055.610705] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 1055.610714] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 1055.610719] Code: 00 83 88 3c e0 ff ff 04 5d c3 8a 47 08 55 48 89 e5 3c 01 75 07 e8 3a 35 c5 ff eb 17 3c 02 75 07 e8 a2 ff ff ff eb 0c 8b 57 04 ec <48> 8b 15 86 e1 b2 00 ed 5d c3 66 66 66 66 90 55 48 63 c2 48 c1
[ 1055.610827] NMI backtrace for cpu 2
[ 1055.610830] CPU 2
[ 1055.610836] Pid: 0, comm: swapper/2 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 1055.610839] RIP: 0010:[<ffffffff813e3eb7>] [<ffffffff813e3eb7>] acpi_idle_do_entry+0x21/0x2b
[ 1055.610847] RSP: 0018:ffff88061cd85e10 EFLAGS: 00000093
[ 1055.610850] RAX: 000000f5c670c400 RBX: ffff88061cdb88a0 RCX: 000000000000c648
[ 1055.610853] RDX: 0000000000000815 RSI: ffff88063ec80000 RDI: ffff88061cdb88a0
[ 1055.610855] RBP: ffff88061cd85e10 R08: 0000000000003e09 R09: 0000000000000018
[ 1055.610858] R10: 0000000000002e78 R11: 0000000000000000 R12: 0000000000000002
[ 1055.610861] R13: ffff88061cd84000 R14: ffff88061cdb8800 R15: ffffffff81574a50
[ 1055.610865] FS: 00007f3fa7fff700(0000) GS:ffff88063ec80000(0000) knlGS:0000000000000000
[ 1055.610868] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1055.610871] CR2: 00000000018f7000 CR3: 00000005d4e3c000 CR4: 00000000000407e0
[ 1055.610873] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1055.610876] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1055.610880] Process swapper/2 (pid: 0, threadinfo ffff88061cd84000, task ffff88061cd7ae80)
[ 1055.610882] Stack:
[ 1055.610884] ffff88061cd85e40 ffffffff813e3faa ffff88061c8c7200 ffffffff81c7e060
[ 1055.610890] 000000f636406194 0000000000000002 ffff88061cd85e50 ffffffff81574a69
[ 1055.610896] ffff88061cd85eb0 ffffffff81575499 7735940000000001 ffff88061c8c7200
[ 1055.610902] Call Trace:
[ 1055.610909] [<ffffffff813e3faa>] acpi_idle_enter_simple+0x87/0xc5
[ 1055.610915] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 1055.610920] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 1055.610927] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 1055.610932] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 1055.610939] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 1055.610944] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 1055.610948] Code: 00 83 88 3c e0 ff ff 04 5d c3 8a 47 08 55 48 89 e5 3c 01 75 07 e8 3a 35 c5 ff eb 17 3c 02 75 07 e8 a2 ff ff ff eb 0c 8b 57 04 ec <48> 8b 15 86 e1 b2 00 ed 5d c3 66 66 66 66 90 55 48 63 c2 48 c1
[ 1235.299277] INFO: rcu_sched detected stalls on CPUs/tasks: { 1} (detected by 0, t=240028 jiffies, g=32254, c=32253, q=96261)
[ 1235.299292] sending NMI to all CPUs:
[ 1235.299301] NMI backtrace for cpu 1
[ 1235.299306] CPU 1
[ 1235.299315] Pid: 6459, comm: zfs Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 1235.299321] RIP: 0010:[<ffffffff81044ac2>] [<ffffffff81044ac2>] __ticket_spin_lock+0x22/0x30
[ 1235.299336] RSP: 0018:ffff8805d42e3958 EFLAGS: 00000006
[ 1235.299341] RAX: 0000000000002d4b RBX: ffff8805dce22e80 RCX: 0000000000000000
[ 1235.299346] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88051fac2488
[ 1235.299351] RBP: ffff8805d42e3958 R08: 0000000000016d40 R09: ffff88063ec56d40
[ 1235.299356] R10: ffffea0017696200 R11: ffffffffa02d183b R12: ffff88051fac2480
[ 1235.299360] R13: ffff88051fac2488 R14: ffffffffffffffff R15: 0000000000000001
[ 1235.299366] FS: 00007f1ac9a97700(0000) GS:ffff88063ec40000(0000) knlGS:0000000000000000
[ 1235.299372] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1235.299376] CR2: 00007f2a6f2e0000 CR3: 00000005ddf2a000 CR4: 00000000000407e0
[ 1235.299381] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1235.299386] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1235.299392] Process zfs (pid: 6459, threadinfo ffff8805d42e2000, task ffff8805dce22e80)
[ 1235.299395] Stack:
[ 1235.299399] ffff8805d42e3968 ffffffff816cde65 ffff8805d42e39d0 ffffffff816cdca6
[ 1235.299411] 0000000000000000 0000000000000000 00000000d42e39d0 ffffffff00000000
[ 1235.299421] ffffffff816cb7ad ffff8805cc05b970 ffff88051fac2480 0000000000000002
[ 1235.299432] Call Trace:
[ 1235.299443] [<ffffffff816cde65>] _raw_spin_lock_irq+0x15/0x20
[ 1235.299453] [<ffffffff816cdca6>] rwsem_down_failed_common+0x46/0x170
[ 1235.299462] [<ffffffff816cb7ad>] ? mutex_lock+0x1d/0x50
[ 1235.299471] [<ffffffff816cde05>] rwsem_down_read_failed+0x15/0x17
[ 1235.299481] [<ffffffff813555d4>] call_rwsem_down_read_failed+0x14/0x30
[ 1235.299521] [<ffffffffa02d183b>] ? kmem_free_debug+0x4b/0x150 [spl]
[ 1235.299534] [<ffffffff816cc0e0>] ? down_read+0x20/0x23
[ 1235.299616] [<ffffffffa03b7210>] zap_get_leaf_byblk+0xd0/0x2c0 [zfs]
[ 1235.299689] [<ffffffffa03b6c0b>] ? zap_idx_to_blk+0x13b/0x150 [zfs]
[ 1235.299763] [<ffffffffa03b746d>] zap_deref_leaf+0x6d/0x80 [zfs]
[ 1235.299836] [<ffffffffa03b9021>] fzap_cursor_retrieve+0xe1/0x280 [zfs]
[ 1235.299909] [<ffffffffa03bc848>] zap_cursor_retrieve+0x68/0x2e0 [zfs]
[ 1235.299977] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 1235.300044] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 1235.300121] [<ffffffffa03b86d1>] zap_value_search+0x91/0xd0 [zfs]
[ 1235.300192] [<ffffffffa03e779d>] zfs_obj_to_path_impl+0x1bd/0x2b0 [zfs]
[ 1235.300261] [<ffffffffa0392723>] ? sa_lookup_impl+0x13/0x20 [zfs]
[ 1235.300329] [<ffffffffa0392780>] ? sa_bulk_lookup+0x40/0x60 [zfs]
[ 1235.300399] [<ffffffffa03e3261>] ? zfs_obj_to_stats_impl+0xa1/0xb0 [zfs]
[ 1235.300471] [<ffffffffa03e79eb>] zfs_obj_to_stats+0xab/0xd0 [zfs]
[ 1235.300542] [<ffffffffa03ca6d4>] zfs_ioc_obj_to_stats+0x84/0xa0 [zfs]
[ 1235.300612] [<ffffffffa03d0879>] zfsdev_ioctl+0x4b9/0x530 [zfs]
[ 1235.300622] [<ffffffff811a6659>] do_vfs_ioctl+0x99/0x570
[ 1235.300631] [<ffffffff811a6bc1>] sys_ioctl+0x91/0xb0
[ 1235.300641] [<ffffffff816d671d>] system_call_fastpath+0x1a/0x1f
[ 1235.300645] Code: 84 00 00 00 00 00 0f 1f 00 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c1 c1 e9 10 66 39 c1 89 ca 74 11 0f 1f 80 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89
[ 1235.300752] NMI backtrace for cpu 0
[ 1235.300756] CPU 0
[ 1235.300763] Pid: 0, comm: swapper/0 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 1235.300768] RIP: 0010:[<ffffffff81354186>] [<ffffffff81354186>] __const_udelay+0x26/0x30
[ 1235.300778] RSP: 0018:ffff88063ec03dd8 EFLAGS: 00000813
[ 1235.300782] RAX: 00000000e934eb00 RBX: 0000000000002710 RCX: 0000000001ab44f8
[ 1235.300787] RDX: 000000000034fb57 RSI: 0000000000000100 RDI: 000000000034fb58
[ 1235.300791] RBP: ffff88063ec03dd8 R08: 000000000000000a R09: 0000000000000000
[ 1235.300796] R10: 00000000000009ea R11: 0000000000000002 R12: ffffffff81ce70c0
[ 1235.300800] R13: ffff88063ec0ee60 R14: ffffffff81c3f3c0 R15: ffffffff81c3f480
[ 1235.300806] FS: 00007fd1622d6740(0000) GS:ffff88063ec00000(0000) knlGS:0000000000000000
[ 1235.300811] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1235.300815] CR2: 00007f8d10000020 CR3: 00000005ddefd000 CR4: 00000000000407f0
[ 1235.300820] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1235.300824] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1235.300829] Process swapper/0 (pid: 0, threadinfo ffffffff81c00000, task ffffffff81c15440)
[ 1235.300833] Stack:
[ 1235.300836] ffff88063ec03df0 ffffffff81039d6a ffffffff81c3f3c0 ffff88063ec03e50
[ 1235.300847] ffffffff810f2935 0000000000017805 0000000000000082 0000000100039386
[ 1235.300857] 0000000000000000 ffffffff81c00000 ffffffff81c15440 0000000000000000
[ 1235.300867] Call Trace:
[ 1235.300871] <IRQ>
[ 1235.300875] [<ffffffff81039d6a>] arch_trigger_all_cpu_backtrace+0x6a/0x90
[ 1235.300891] [<ffffffff810f2935>] rcu_pending+0x565/0x5b0
[ 1235.300900] [<ffffffff810f4a0b>] rcu_check_callbacks+0x9b/0x120
[ 1235.300909] [<ffffffff8106a928>] update_process_times+0x48/0x90
[ 1235.300918] [<ffffffff810b3a8e>] tick_sched_handle.isra.8+0x2e/0x70
[ 1235.300926] [<ffffffff810b3c1c>] tick_sched_timer+0x4c/0x80
[ 1235.300935] [<ffffffff810819d9>] __run_hrtimer+0x79/0x1c0
[ 1235.300943] [<ffffffff810b3bd0>] ? tick_sched_do_timer+0x40/0x40
[ 1235.300952] [<ffffffff810822c7>] hrtimer_interrupt+0xe7/0x230
[ 1235.300961] [<ffffffff81574a50>] ? centrino_target+0x370/0x370
[ 1235.300971] [<ffffffff816d8519>] smp_apic_timer_interrupt+0x69/0x99
[ 1235.300979] [<ffffffff816d745d>] apic_timer_interrupt+0x6d/0x80
[ 1235.300983] <EOI>
[ 1235.300988] [<ffffffff81043fa6>] ? native_safe_halt+0x6/0x10
[ 1235.301004] [<ffffffff813e3769>] arch_safe_halt+0x9/0xd
[ 1235.301012] [<ffffffff813e3e7d>] acpi_safe_halt+0x2a/0x43
[ 1235.301020] [<ffffffff813e3eb1>] acpi_idle_do_entry+0x1b/0x2b
[ 1235.301029] [<ffffffff813e3f04>] acpi_idle_enter_c1+0x43/0x62
[ 1235.301037] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 1235.301045] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 1235.301054] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 1235.301062] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 1235.301072] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 1235.301081] [<ffffffff816abc32>] rest_init+0x72/0x80
[ 1235.301090] [<ffffffff81d05be9>] start_kernel+0x3d1/0x3dc
[ 1235.301099] [<ffffffff81d0566e>] ? do_early_param+0x8e/0x8e
[ 1235.301108] [<ffffffff81d05355>] x86_64_start_reservations+0x130/0x133
[ 1235.301116] [<ffffffff81d05458>] x86_64_start_kernel+0x100/0x10f
[ 1235.301121] Code: 00 00 00 00 00 55 48 8d 04 bd 00 00 00 00 65 48 8b 14 25 18 3e 01 00 48 8d 0c 12 48 c1 e2 06 48 89 e5 48 29 ca f7 e2 48 8d 7a 01 <ff> 15 0c 25 92 00 5d c3 66 90 66 66 66 66 90 55 65 48 8b 14 25
[ 1235.301228] NMI backtrace for cpu 2
[ 1235.301237] CPU 2
[ 1235.301246] Pid: 0, comm: swapper/2 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 1235.301252] RIP: 0010:[<ffffffff813e3eb7>] [<ffffffff813e3eb7>] acpi_idle_do_entry+0x21/0x2b
[ 1235.301267] RSP: 0018:ffff88061cd85e10 EFLAGS: 00000093
[ 1235.301272] RAX: 0000011f9c649300 RBX: ffff88061cdb88a0 RCX: 000000000000c648
[ 1235.301277] RDX: 0000000000000815 RSI: ffff88063ec80000 RDI: ffff88061cdb88a0
[ 1235.301282] RBP: ffff88061cd85e10 R08: 0000000000002e56 R09: 0000000000000018
[ 1235.301286] R10: 0000000000002e71 R11: 0000000000000000 R12: 0000000000000002
[ 1235.301291] R13: ffff88061cd84000 R14: ffff88061cdb8800 R15: ffffffff81574a50
[ 1235.301297] FS: 00007fa534ff9700(0000) GS:ffff88063ec80000(0000) knlGS:0000000000000000
[ 1235.301301] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1235.301306] CR2: 00007f3fcaa47000 CR3: 00000005e1941000 CR4: 00000000000407e0
[ 1235.301311] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1235.301316] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1235.301321] Process swapper/2 (pid: 0, threadinfo ffff88061cd84000, task ffff88061cd7ae80)
[ 1235.301324] Stack:
[ 1235.301328] ffff88061cd85e40 ffffffff813e3faa ffff88061c8c7200 ffffffff81c7e060
[ 1235.301340] 000001201fcdac04 0000000000000002 ffff88061cd85e50 ffffffff81574a69
[ 1235.301350] ffff88061cd85eb0 ffffffff81575499 7735940000000001 ffff88061c8c7200
[ 1235.301361] Call Trace:
[ 1235.301372] [<ffffffff813e3faa>] acpi_idle_enter_simple+0x87/0xc5
[ 1235.301382] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 1235.301390] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 1235.301399] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 1235.301407] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 1235.301417] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 1235.301426] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 1235.301432] Code: 00 83 88 3c e0 ff ff 04 5d c3 8a 47 08 55 48 89 e5 3c 01 75 07 e8 3a 35 c5 ff eb 17 3c 02 75 07 e8 a2 ff ff ff eb 0c 8b 57 04 ec <48> 8b 15 86 e1 b2 00 ed 5d c3 66 66 66 66 90 55 48 63 c2 48 c1
[ 1235.301538] NMI backtrace for cpu 3
[ 1235.301540] CPU 3
[ 1235.301546] Pid: 0, comm: swapper/3 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 1235.301550] RIP: 0010:[<ffffffff813e3eb7>] [<ffffffff813e3eb7>] acpi_idle_do_entry+0x21/0x2b
[ 1235.301558] RSP: 0018:ffff88061cd87e10 EFLAGS: 00000093
[ 1235.301562] RAX: 0000011f9d583100 RBX: ffff8806167d54a0 RCX: 000000000000c648
[ 1235.301566] RDX: 0000000000000815 RSI: ffff88063ecc0000 RDI: ffff8806167d54a0
[ 1235.301569] RBP: ffff88061cd87e10 R08: 0000000000002e5b R09: 0000000000000018
[ 1235.301573] R10: 0000000000002e76 R11: 0000000000000000 R12: 0000000000000002
[ 1235.301577] R13: ffff88061cd86000 R14: ffff8806167d5400 R15: ffffffff81574a50
[ 1235.301586] FS: 00007f887ab47700(0000) GS:ffff88063ecc0000(0000) knlGS:0000000000000000
[ 1235.301600] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1235.301614] CR2: 00007f79d9e1f000 CR3: 00000005c62fe000 CR4: 00000000000407e0
[ 1235.301627] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1235.301641] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1235.301653] Process swapper/3 (pid: 0, threadinfo ffff88061cd86000, task ffff88061cd7c5c0)
[ 1235.301663] Stack:
[ 1235.301673] ffff88061cd87e40 ffffffff813e3faa ffff880616006400 ffffffff81c7e060
[ 1235.301717] 0000012020c1be13 0000000000000002 ffff88061cd87e50 ffffffff81574a69
[ 1235.301724] ffff88061cd87eb0 ffffffff81575499 7735940000000001 ffff880616006400
[ 1235.301732] Call Trace:
[ 1235.301740] [<ffffffff813e3faa>] acpi_idle_enter_simple+0x87/0xc5
[ 1235.301748] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 1235.301755] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 1235.301764] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 1235.301772] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 1235.301780] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 1235.301787] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 1235.301792] Code: 00 83 88 3c e0 ff ff 04 5d c3 8a 47 08 55 48 89 e5 3c 01 75 07 e8 3a 35 c5 ff eb 17 3c 02 75 07 e8 a2 ff ff ff eb 0c 8b 57 04 ec <48> 8b 15 86 e1 b2 00 ed 5d c3 66 66 66 66 90 55 48 63 c2 48 c1
[ 1235.301867] NMI backtrace for cpu 5
[ 1235.301875] CPU 5
[ 1235.301885] Pid: 0, comm: swapper/5 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 1235.301891] RIP: 0010:[<ffffffff813e3eb7>] [<ffffffff813e3eb7>] acpi_idle_do_entry+0x21/0x2b
[ 1235.301906] RSP: 0018:ffff88061cd8be10 EFLAGS: 00000093
[ 1235.301911] RAX: 0000011f9cde4700 RBX: ffff88062600e0a0 RCX: 000000000000c648
[ 1235.301916] RDX: 0000000000000815 RSI: ffff88063ed40000 RDI: ffff88062600e0a0
[ 1235.301920] RBP: ffff88061cd8be10 R08: 0000000000002e5f R09: 0000000000000018
[ 1235.301925] R10: 0000000000002e7b R11: 0000000000000000 R12: 0000000000000002
[ 1235.301929] R13: ffff88061cd8a000 R14: ffff88062600e000 R15: ffffffff81574a50
[ 1235.301935] FS: 00007f3fc0947700(0000) GS:ffff88063ed40000(0000) knlGS:0000000000000000
[ 1235.301940] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1235.301945] CR2: 00007f79b80690e8 CR3: 00000005d4e3c000 CR4: 00000000000407e0
[ 1235.301949] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1235.301954] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1235.301969] Process swapper/5 (pid: 0, threadinfo ffff88061cd8a000, task ffff88061cd90000)
[ 1235.301981] Stack:
[ 1235.301993] ffff88061cd8be40 ffffffff813e3faa ffff880616086000 ffffffff81c7e060
[ 1235.302032] 0000012020479a0d 0000000000000002 ffff88061cd8be50 ffffffff81574a69
[ 1235.302072] ffff88061cd8beb0 ffffffff81575499 7735940000000001 ffff880616086000
[ 1235.302088] Call Trace:
[ 1235.302098] [<ffffffff813e3faa>] acpi_idle_enter_simple+0x87/0xc5
[ 1235.302108] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 1235.302117] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 1235.302126] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 1235.302134] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 1235.302144] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 1235.302153] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 1235.302158] Code: 00 83 88 3c e0 ff ff 04 5d c3 8a 47 08 55 48 89 e5 3c 01 75 07 e8 3a 35 c5 ff eb 17 3c 02 75 07 e8 a2 ff ff ff eb 0c 8b 57 04 ec <48> 8b 15 86 e1 b2 00 ed 5d c3 66 66 66 66 90 55 48 63 c2 48 c1
[ 1235.302310] NMI backtrace for cpu 4
[ 1235.302322] CPU 4
[ 1235.302334] Pid: 0, comm: swapper/4 Tainted: PF W O 3.8.0-35-generic #50-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0
[ 1235.302346] RIP: 0010:[<ffffffff813e3eb7>] [<ffffffff813e3eb7>] acpi_idle_do_entry+0x21/0x2b
[ 1235.302368] RSP: 0018:ffff88061cd89e10 EFLAGS: 00000093
[ 1235.302376] RAX: 0000011f9d1b3d00 RBX: ffff88061ce310a0 RCX: 000000000000c648
[ 1235.302380] RDX: 0000000000000815 RSI: ffff88063ed00000 RDI: ffff88061ce310a0
[ 1235.302384] RBP: ffff88061cd89e10 R08: 0000000000002e5e R09: 0000000000000018
[ 1235.302388] R10: 0000000000002e6f R11: 0000000000000000 R12: 0000000000000002
[ 1235.302391] R13: ffff88061cd88000 R14: ffff88061ce31000 R15: ffffffff81574a50
[ 1235.302396] FS: 00007fa5167fc700(0000) GS:ffff88063ed00000(0000) knlGS:0000000000000000
[ 1235.302400] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1235.302404] CR2: 00007fd699e45000 CR3: 00000005e1941000 CR4: 00000000000407e0
[ 1235.302408] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1235.302411] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1235.302416] Process swapper/4 (pid: 0, threadinfo ffff88061cd88000, task ffff88061cd7dd00)
[ 1235.302419] Stack:
[ 1235.302421] ffff88061cd89e40 ffffffff813e3faa ffff88061ca45600 ffffffff81c7e060
[ 1235.302430] 000001202084ac75 0000000000000002 ffff88061cd89e50 ffffffff81574a69
[ 1235.302438] ffff88061cd89eb0 ffffffff81575499 7735940000000001 ffff88061ca45600
[ 1235.302445] Call Trace:
[ 1235.302452] [<ffffffff813e3faa>] acpi_idle_enter_simple+0x87/0xc5
[ 1235.302458] [<ffffffff81574a69>] cpuidle_enter+0x19/0x20
[ 1235.302464] [<ffffffff81575499>] cpuidle_wrap_enter+0x49/0xa0
[ 1235.302470] [<ffffffff81575500>] cpuidle_enter_tk+0x10/0x20
[ 1235.302476] [<ffffffff815750e5>] cpuidle_idle_call+0xa5/0x270
[ 1235.302482] [<ffffffff8101d62f>] cpu_idle+0xaf/0x120
[ 1235.302488] [<ffffffff816b7620>] start_secondary+0x1e0/0x1e5
[ 1235.302492] Code: 00 83 88 3c e0 ff ff 04 5d c3 8a 47 08 55 48 89 e5 3c 01 75 07 e8 3a 35 c5 ff eb 17 3c 02 75 07 e8 a2 ff ff ff eb 0c 8b 57 04 ec <48> 8b 15 86 e1 b2 00 ed 5d c3 66 66 66 66 90 55 48 63 c2 48 c1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment