Skip to content

Instantly share code, notes, and snippets.

@saurabhnanda

saurabhnanda/kernel stacktraces

Last active April 16, 2019 14:41
Show Gist options
  • Save saurabhnanda/f35e99229950ffbb905934a6a6dc1f80 to your computer and use it in GitHub Desktop.
Save saurabhnanda/f35e99229950ffbb905934a6a6dc1f80 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
kernel stacktraces
Apr 14 07:31:24 14333 kernel: [843556.475850] INFO: rcu_sched detected stalls on CPUs/tasks:
Apr 14 07:31:24 14333 kernel: [843556.475854] 0-...!: (1 GPs behind) idle=ffa/1/0 softirq=7326149/7326150 fqs=0
Apr 14 07:31:24 14333 kernel: [843556.475855] (detected by 1, t=34565 jiffies, g=5573741, c=5573740, q=262)
Apr 14 07:31:24 14333 kernel: [843556.475859] Sending NMI from CPU 1 to CPUs 0:
Apr 14 07:31:24 14333 kernel: [843556.476867] NMI backtrace for cpu 0
Apr 14 07:31:24 14333 kernel: [843556.476868] CPU: 0 PID: 0 Comm: swapper/0 Tainted: P O L 4.15.0-47-generic #50-Ubuntu
Apr 14 07:31:24 14333 kernel: [843556.476869] Hardware name: BHYVE, BIOS 1.00 03/14/2014
Apr 14 07:31:24 14333 kernel: [843556.476869] RIP: 0010:io_serial_in+0x18/0x20
Apr 14 07:31:24 14333 kernel: [843556.476869] RSP: 0018:ffff9b1a3fc03c10 EFLAGS: 00000002
Apr 14 07:31:24 14333 kernel: [843556.476870] RAX: 0000000084a02960 RBX: ffffffff85efee20 RCX: 0000000000000000
Apr 14 07:31:24 14333 kernel: [843556.476871] RDX: 00000000000003fd RSI: 0000000000000005 RDI: ffffffff85efee20
Apr 14 07:31:24 14333 kernel: [843556.476872] RBP: ffff9b1a3fc03c10 R08: 0000000000000000 R09: 0000000000001d7d
Apr 14 07:31:24 14333 kernel: [843556.476872] R10: 000000000000003e R11: ffffffff85d5380e R12: 0000000000002710
Apr 14 07:31:24 14333 kernel: [843556.476873] R13: 0000000000000020 R14: ffffffff85d53818 R15: ffffffff85efee20
Apr 14 07:31:24 14333 kernel: [843556.476873] FS: 0000000000000000(0000) GS:ffff9b1a3fc00000(0000) knlGS:0000000000000000
Apr 14 07:31:24 14333 kernel: [843556.476874] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 14 07:31:24 14333 kernel: [843556.476874] CR2: 000000c4202c5000 CR3: 000000002140a003 CR4: 00000000000606f0
Apr 14 07:31:24 14333 kernel: [843556.476874] Call Trace:
Apr 14 07:31:24 14333 kernel: [843556.476875] <IRQ>
Apr 14 07:31:24 14333 kernel: [843556.476875] wait_for_xmitr+0x47/0xb0
Apr 14 07:31:24 14333 kernel: [843556.476875] serial8250_console_putchar+0x1c/0x40
Apr 14 07:31:24 14333 kernel: [843556.476876] ? wait_for_xmitr+0xb0/0xb0
Apr 14 07:31:24 14333 kernel: [843556.476876] uart_console_write+0x4c/0x60
Apr 14 07:31:24 14333 kernel: [843556.476876] serial8250_console_write+0x10a/0x2a0
Apr 14 07:31:25 14333 kernel: [843556.476877] ? msg_print_text+0x80/0x100
Apr 14 07:31:25 14333 kernel: [843556.476877] univ8250_console_write+0x25/0x30
Apr 14 07:31:25 14333 kernel: [843556.476877] console_unlock+0x34b/0x4e0
Apr 14 07:31:25 14333 kernel: [843556.476878] vprintk_emit+0x333/0x3a0
Apr 14 07:31:25 14333 kernel: [843556.476878] vprintk_default+0x29/0x50
Apr 14 07:31:25 14333 kernel: [843556.476878] vprintk_func+0x27/0x60
Apr 14 07:31:25 14333 kernel: [843556.476879] printk+0x52/0x6e
Apr 14 07:31:25 14333 kernel: [843556.476879] rcu_check_callbacks+0x6b6/0x900
Apr 14 07:31:25 14333 kernel: [843556.476879] ? update_wall_time+0x474/0x6f0
Apr 14 07:31:25 14333 kernel: [843556.476880] ? tick_sched_do_timer+0x50/0x50
Apr 14 07:31:25 14333 kernel: [843556.476880] update_process_times+0x2f/0x60
Apr 14 07:31:25 14333 kernel: [843556.476880] tick_sched_handle+0x26/0x70
Apr 14 07:31:25 14333 kernel: [843556.476881] ? tick_sched_do_timer+0x3f/0x50
Apr 14 07:31:25 14333 kernel: [843556.476881] tick_sched_timer+0x39/0x80
Apr 14 07:31:25 14333 kernel: [843556.476881] __hrtimer_run_queues+0xdf/0x230
Apr 14 07:31:25 14333 kernel: [843556.476882] hrtimer_interrupt+0xa0/0x1d0
Apr 14 07:31:25 14333 kernel: [843556.476882] smp_apic_timer_interrupt+0x6f/0x130
Apr 14 07:31:25 14333 kernel: [843556.476882] apic_timer_interrupt+0x84/0x90
Apr 14 07:31:25 14333 kernel: [843556.476883] </IRQ>
Apr 14 07:31:25 14333 kernel: [843556.476883] RIP: 0010:native_safe_halt+0x6/0x10
Apr 14 07:31:25 14333 kernel: [843556.476883] RSP: 0018:ffffffff85803e28 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff11
Apr 14 07:31:25 14333 kernel: [843556.476884] RAX: ffffffff84da0060 RBX: 0000000000000000 RCX: 0000000000000000
Apr 14 07:31:25 14333 kernel: [843556.476885] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Apr 14 07:31:25 14333 kernel: [843556.476885] RBP: ffffffff85803e28 R08: 0000000000000002 R09: 0000000000000001
Apr 14 07:31:25 14333 kernel: [843556.476885] R10: ffffffff85803da0 R11: 0000000000000000 R12: 0000000000000000
Apr 14 07:31:25 14333 kernel: [843556.476886] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Apr 14 07:31:25 14333 kernel: [843556.476886] ? __cpuidle_text_start+0x8/0x8
Apr 14 07:31:25 14333 kernel: [843556.476886] default_idle+0x20/0x100
Apr 14 07:31:25 14333 kernel: [843556.476887] arch_cpu_idle+0x15/0x20
Apr 14 07:31:25 14333 kernel: [843556.476887] default_idle_call+0x23/0x30
Apr 14 07:31:25 14333 kernel: [843556.476887] do_idle+0x172/0x1f0
Apr 14 07:31:25 14333 kernel: [843556.476888] cpu_startup_entry+0x73/0x80
Apr 14 07:31:25 14333 kernel: [843556.476888] rest_init+0xae/0xb0
Apr 14 07:31:25 14333 kernel: [843556.476888] start_kernel+0x4dc/0x4fd
Apr 14 07:31:25 14333 kernel: [843556.476889] x86_64_start_reservations+0x24/0x26
Apr 14 07:31:25 14333 kernel: [843556.476889] x86_64_start_kernel+0x74/0x77
Apr 14 07:31:25 14333 kernel: [843556.476889] secondary_startup_64+0xa5/0xb0
Apr 14 07:31:25 14333 kernel: [843556.476890] Code: 48 89 e5 d3 e6 48 63 f6 48 03 77 10 8b 06 5d c3 0f 1f 00 0f 1f 44 00 00 0f b6 8f a1 00 00 00 8b 57 08 55 48 89 e5 d3 e6 01 f2 ec <0f> b6 c0 5d c3 0f 1f 00 0f 1f 44 00 00 0f b6 8f a1 00 00 00 89
Apr 14 07:31:25 14333 kernel: [843556.476904] rcu_sched kthread starved for 34565 jiffies! g5573741 c5573740 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=1
Apr 14 07:31:25 14333 kernel: [843556.476905] rcu_sched I 0 8 2 0x80000000
Apr 14 07:31:25 14333 kernel: [843556.476908] Call Trace:
Apr 14 07:31:25 14333 kernel: [843556.476917] __schedule+0x291/0x8a0
Apr 14 07:31:25 14333 kernel: [843556.476921] ? __switch_to_asm+0x34/0x70
Apr 14 07:31:25 14333 kernel: [843556.476923] ? __switch_to_asm+0x40/0x70
Apr 14 07:31:25 14333 kernel: [843556.476924] schedule+0x2c/0x80
Apr 14 07:31:25 14333 kernel: [843556.476926] schedule_timeout+0x15d/0x350
Apr 14 07:31:25 14333 kernel: [843556.476932] ? __next_timer_interrupt+0xe0/0xe0
Apr 14 07:31:25 14333 kernel: [843556.476934] rcu_gp_kthread+0x53a/0x960
Apr 14 07:31:25 14333 kernel: [843556.476938] kthread+0x121/0x140
Apr 14 07:31:25 14333 kernel: [843556.476940] ? rcu_note_context_switch+0x150/0x150
Apr 14 07:31:25 14333 kernel: [843556.476942] ? kthread_create_worker_on_cpu+0x70/0x70
Apr 14 07:31:25 14333 kernel: [843556.476944] ret_from_fork+0x35/0x40
Apr 14 07:31:25 14333 kernel: [843556.477839] NMI backtrace for cpu 0
Apr 14 07:31:25 14333 kernel: [843556.477839] CPU: 0 PID: 0 Comm: swapper/0 Tainted: P O L 4.15.0-47-generic #50-Ubuntu
Apr 14 07:31:25 14333 kernel: [843556.477839] Hardware name: BHYVE, BIOS 1.00 03/14/2014
Apr 14 07:31:25 14333 kernel: [843556.477839] Call Trace:
Apr 14 07:31:25 14333 kernel: [843556.477839] <IRQ>
Apr 14 07:31:25 14333 kernel: [843556.477839] dump_stack+0x63/0x8b
Apr 14 07:31:25 14333 kernel: [843556.477839] nmi_cpu_backtrace+0xca/0xd0
Apr 14 07:31:25 14333 kernel: [843556.477839] ? lapic_can_unplug_cpu+0xb0/0xb0
Apr 14 07:31:25 14333 kernel: [843556.477839] nmi_trigger_cpumask_backtrace+0xe6/0x130
Apr 14 07:31:25 14333 kernel: [843556.477839] arch_trigger_cpumask_backtrace+0x19/0x20
Apr 14 07:31:25 14333 kernel: [843556.477839] rcu_dump_cpu_stacks+0xa3/0xdb
Apr 14 07:31:25 14333 kernel: [843556.477839] rcu_check_callbacks+0x6c6/0x900
Apr 14 07:31:25 14333 kernel: [843556.477839] ? update_wall_time+0x474/0x6f0
Apr 14 07:31:25 14333 kernel: [843556.477839] ? tick_sched_do_timer+0x50/0x50
Apr 14 07:31:25 14333 kernel: [843556.477839] update_process_times+0x2f/0x60
Apr 14 07:31:25 14333 kernel: [843556.477839] tick_sched_handle+0x26/0x70
Apr 14 07:31:25 14333 kernel: [843556.477839] ? tick_sched_do_timer+0x3f/0x50
Apr 14 07:31:25 14333 kernel: [843556.477839] tick_sched_timer+0x39/0x80
Apr 14 07:31:25 14333 kernel: [843556.477839] __hrtimer_run_queues+0xdf/0x230
Apr 14 07:31:25 14333 kernel: [843556.477839] hrtimer_interrupt+0xa0/0x1d0
Apr 14 07:31:25 14333 kernel: [843556.477839] smp_apic_timer_interrupt+0x6f/0x130
Apr 14 07:31:25 14333 kernel: [843556.477839] apic_timer_interrupt+0x84/0x90
Apr 14 07:31:25 14333 kernel: [843556.477839] </IRQ>
Apr 14 07:31:25 14333 kernel: [843556.477839] RIP: 0010:native_safe_halt+0x6/0x10
Apr 14 07:31:25 14333 kernel: [843556.477839] RSP: 0018:ffffffff85803e28 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff11
Apr 14 07:31:25 14333 kernel: [843556.477839] RAX: ffffffff84da0060 RBX: 0000000000000000 RCX: 0000000000000000
Apr 14 07:31:25 14333 kernel: [843556.477839] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Apr 14 07:31:25 14333 kernel: [843556.477839] RBP: ffffffff85803e28 R08: 0000000000000002 R09: 0000000000000001
Apr 14 07:31:25 14333 kernel: [843556.477839] R10: ffffffff85803da0 R11: 0000000000000000 R12: 0000000000000000
Apr 14 07:31:25 14333 kernel: [843556.477839] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Apr 14 07:31:25 14333 kernel: [843556.477839] ? __cpuidle_text_start+0x8/0x8
Apr 14 07:31:25 14333 kernel: [843556.477839] default_idle+0x20/0x100
Apr 14 07:31:25 14333 kernel: [843556.477839] arch_cpu_idle+0x15/0x20
Apr 14 07:31:25 14333 kernel: [843556.477839] default_idle_call+0x23/0x30
Apr 14 07:31:25 14333 kernel: [843556.477839] do_idle+0x172/0x1f0
Apr 14 07:31:25 14333 kernel: [843556.477839] cpu_startup_entry+0x73/0x80
Apr 14 07:31:25 14333 kernel: [843556.477839] rest_init+0xae/0xb0
Apr 14 07:31:25 14333 kernel: [843556.477839] start_kernel+0x4dc/0x4fd
Apr 14 07:31:25 14333 kernel: [843556.477839] x86_64_start_reservations+0x24/0x26
Apr 14 07:31:25 14333 kernel: [843556.477839] x86_64_start_kernel+0x74/0x77
Apr 14 07:31:25 14333 kernel: [843556.477839] secondary_startup_64+0xa5/0xb0
Raw
strace output
execve("/sbin/zfs", ["zfs", "list"], 0x7ffc3138b268 /* 25 vars */) = 0
brk(NULL) = 0x5589b4639000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=23289, ...}) = 0
mmap(NULL, 23289, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f79ad400000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/libnvpair.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360>\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=75760, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f79ad3fe000
mmap(NULL, 2170960, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f79acfcc000
mprotect(0x7f79acfde000, 2093056, PROT_NONE) = 0
mmap(0x7f79ad1dd000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x11000) = 0x7f79ad1dd000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/libuutil.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240P\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=69576, ...}) = 0
mmap(NULL, 2169344, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f79acdba000
mprotect(0x7f79acdca000, 2093056, PROT_NONE) = 0
mmap(0x7f79acfc9000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xf000) = 0x7f79acfc9000
mmap(0x7f79acfcb000, 2560, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f79acfcb000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/libzpool.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\261\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=2087776, ...}) = 0
mmap(NULL, 5748760, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f79ac83e000
mprotect(0x7f79aca26000, 2093056, PROT_NONE) = 0
mmap(0x7f79acc25000, 94208, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e7000) = 0x7f79acc25000
mmap(0x7f79acc3c000, 1562648, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f79acc3c000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/libzfs.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\242\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=293008, ...}) = 0
mmap(NULL, 2388576, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f79ac5f6000
mprotect(0x7f79ac63c000, 2097152, PROT_NONE) = 0
mmap(0x7f79ac83c000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x46000) = 0x7f79ac83c000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/libzfs_core.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\26\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=18272, ...}) = 0
mmap(NULL, 2113784, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f79ac3f1000
mprotect(0x7f79ac3f5000, 2093056, PROT_NONE) = 0
mmap(0x7f79ac5f4000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f79ac5f4000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000b\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=144976, ...}) = 0
mmap(NULL, 2221184, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f79ac1d2000
mprotect(0x7f79ac1ec000, 2093056, PROT_NONE) = 0
mmap(0x7f79ac3eb000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19000) = 0x7f79ac3eb000
mmap(0x7f79ac3ed000, 13440, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f79ac3ed000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\34\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2030544, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f79ad3fc000
mmap(NULL, 4131552, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f79abde1000
mprotect(0x7f79abfc8000, 2097152, PROT_NONE) = 0
mmap(0x7f79ac1c8000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e7000) = 0x7f79ac1c8000
mmap(0x7f79ac1ce000, 15072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f79ac1ce000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\"\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=31680, ...}) = 0
mmap(NULL, 2128864, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f79abbd9000
mprotect(0x7f79abbe0000, 2093056, PROT_NONE) = 0
mmap(0x7f79abddf000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f79abddf000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libuuid.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@\26\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=27112, ...}) = 0
mmap(NULL, 2122112, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f79ab9d2000
mprotect(0x7f79ab9d8000, 2093056, PROT_NONE) = 0
mmap(0x7f79abbd7000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f79abbd7000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\37\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=116960, ...}) = 0
mmap(NULL, 2212016, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f79ab7b5000
mprotect(0x7f79ab7d1000, 2093056, PROT_NONE) = 0
mmap(0x7f79ab9d0000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b000) = 0x7f79ab9d0000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\16\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=14560, ...}) = 0
mmap(NULL, 2109712, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f79ab5b1000
mprotect(0x7f79ab5b4000, 2093056, PROT_NONE) = 0
mmap(0x7f79ab7b3000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f79ab7b3000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200\272\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=1700792, ...}) = 0
mmap(NULL, 3789144, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f79ab213000
mprotect(0x7f79ab3b0000, 2093056, PROT_NONE) = 0
mmap(0x7f79ab5af000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19c000) = 0x7f79ab5af000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libblkid.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000\230\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=311720, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f79ad3fa000
mmap(NULL, 2411776, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f79aafc6000
mprotect(0x7f79ab00d000, 2097152, PROT_NONE) = 0
mmap(0x7f79ab20d000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x47000) = 0x7f79ab20d000
mmap(0x7f79ab212000, 3328, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f79ab212000
close(3) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f79ad3f8000
mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f79ad3f4000
arch_prctl(ARCH_SET_FS, 0x7f79ad3f5780) = 0
mprotect(0x7f79ac1c8000, 16384, PROT_READ) = 0
mprotect(0x7f79abbd7000, 4096, PROT_READ) = 0
mprotect(0x7f79ab20d000, 16384, PROT_READ) = 0
mprotect(0x7f79ab5af000, 4096, PROT_READ) = 0
mprotect(0x7f79ab7b3000, 4096, PROT_READ) = 0
mprotect(0x7f79ab9d0000, 4096, PROT_READ) = 0
mprotect(0x7f79ac3eb000, 4096, PROT_READ) = 0
mprotect(0x7f79abddf000, 4096, PROT_READ) = 0
mprotect(0x7f79ad1dd000, 4096, PROT_READ) = 0
mprotect(0x7f79acfc9000, 4096, PROT_READ) = 0
mprotect(0x7f79ac5f4000, 4096, PROT_READ) = 0
mprotect(0x7f79acc25000, 20480, PROT_READ) = 0
mprotect(0x7f79ac83c000, 4096, PROT_READ) = 0
mprotect(0x5589b26fd000, 4096, PROT_READ) = 0
mprotect(0x7f79ad406000, 4096, PROT_READ) = 0
munmap(0x7f79ad400000, 23289) = 0
set_tid_address(0x7f79ad3f5a50) = 6505
set_robust_list(0x7f79ad3f5a60, 24) = 0
rt_sigaction(SIGRTMIN, {sa_handler=0x7f79ac1d7cb0, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7f79ac1e4890}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {sa_handler=0x7f79ac1d7d50, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f79ac1e4890}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
brk(NULL) = 0x5589b4639000
brk(0x5589b465a000) = 0x5589b465a000
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=1683056, ...}) = 0
mmap(NULL, 1683056, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f79ad259000
close(3) = 0
access("/run/systemd/container", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/sys/module/zfs/version", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
read(3, "0.7.5-1ubuntu16.4\n", 4096) = 18
close(3) = 0
access("/sys/module/zfs/version", F_OK) = 0
openat(AT_FDCWD, "/sys/module/zfs/version", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
read(3, "0.7.5-1ubuntu16.4\n", 4096) = 18
close(3) = 0
access("/sys/module/zfs/version", F_OK) = 0
clock_gettime(CLOCK_MONOTONIC, {tv_sec=1048407, tv_nsec=943215771}) = 0
openat(AT_FDCWD, "/dev/zfs", O_RDWR) = 3
close(3) = 0
openat(AT_FDCWD, "/dev/zfs", O_RDWR) = 3
openat(AT_FDCWD, "/proc/self/mounts", O_RDONLY) = 4
openat(AT_FDCWD, "/etc/dfs/sharetab", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/dev/zfs", O_RDWR) = 5
mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f79ad218000
ioctl(3, _IOC(0, 0x5a, 0x4, 0)
Raw
zfs-stuck-processes
# ps axu|egrep -i 'zfs|zpool'
root 2206 0.0 0.2 37488 2064 ? D Apr14 0:00 zpool status
root 9165 0.0 0.2 37488 2128 ? D 03:14 0:00 zpool status
root 9299 0.0 0.1 13136 1096 pts/5 S+ 03:14 0:00 grep -E --color=auto -i zfs|zpool
root 11458 0.0 0.2 37540 3000 ? Ds Apr14 0:02 zfs recv -F rsynctank/barman
root 13120 0.0 0.1 37436 1280 ? D Apr14 0:00 zfs list
root 13247 0.0 0.1 37436 1280 ? Ds Apr14 0:00 zfs list -H -o name -t filesystem volume rsynctank/barman
root 13330 0.0 0.1 37436 1356 ? Ds Apr14 0:00 zfs list -H -o name -t filesystem volume rsynctank/barman
root 13672 0.0 0.1 37488 1424 ? D Apr14 0:00 zpool status
root 13832 0.0 0.2 11592 2636 ? Ss Apr14 0:00 bash -c /usr/bin/mbuffer -q -s 256k -W 600 -m 1G|zfs recv -F rsynctank/fileserver
root 13834 0.0 0.1 37436 1336 ? D Apr14 0:00 zfs recv -F rsynctank/fileserver
root 13890 0.0 0.1 37436 1568 ? Ds Apr14 0:00 zfs list -H -o name -t snapshot -s creation -d 1 rsynctank/barman
root 14000 0.0 0.2 37436 2188 ? Ds Apr14 0:00 zfs recv -F rsynctank/logs
root 14087 0.0 0.2 37436 2216 ? Ds Apr14 0:00 zfs list -H -o name -t snapshot -s creation -d 1 rsynctank/barman
root 14185 0.0 0.2 37436 2204 ? D Apr14 0:00 zfs list
root 16246 0.0 0.2 37436 2076 ? Ds Apr14 0:00 zfs list -H -o name -t snapshot -s creation -d 1 rsynctank/fileserver
root 18695 0.0 0.0 4628 900 ? Ss Apr14 0:00 /bin/sh -c [ $(date +%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/scrub ] && /usr/lib/zfs-linux/scrub
root 18697 0.0 0.0 4628 784 ? S Apr14 0:00 /bin/sh -eu /usr/lib/zfs-linux/scrub
root 18700 0.0 0.0 4628 88 ? S Apr14 0:00 /bin/sh -eu /usr/lib/zfs-linux/scrub
root 18701 0.0 0.3 37228 3456 ? D Apr14 0:00 zpool scrub rsynctank
root 20066 0.0 0.1 37436 1244 ? Ds Apr14 0:00 zfs list -H -o name -t snapshot -s creation -d 1 rsynctank/fileserver
root 22326 0.0 0.1 37436 1380 ? Ds Apr14 0:00 zfs list -H -o name -t snapshot -s creation -d 1 rsynctank/logs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment