Skip to content

Instantly share code, notes, and snippets.

@therealkenc
Created July 8, 2020 11:31
Show Gist options
  • Save therealkenc/d82d5020222d794dc2d41a86c0cbc4ef to your computer and use it in GitHub Desktop.
Save therealkenc/d82d5020222d794dc2d41a86c0cbc4ef to your computer and use it in GitHub Desktop.
[ 2838.721141] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[ 2838.721141] rcu: 1-...!: (3287 GPs behind) idle=74a/1/0x4000000000000002 softirq=320/320 fqs=732
[ 2838.721141] rcu: 2-...!: (1 GPs behind) idle=cfa/1/0x4000000000000000 softirq=869/4545 fqs=732
[ 2838.721141] rcu: 5-...!: (1 GPs behind) idle=ffa/1/0x4000000000000000 softirq=288/288 fqs=732
[ 2838.721141] rcu: 6-...!: (1 GPs behind) idle=48e/1/0x4000000000000000 softirq=475/475 fqs=732
[ 2838.721141] rcu: 8-...!: (1 GPs behind) idle=26e/1/0x4000000000000000 softirq=562/562 fqs=732
[ 2838.721141] rcu: 10-...!: (1 GPs behind) idle=f2a/1/0x4000000000000000 softirq=2795/2796 fqs=732
[ 2838.721141] rcu: 24-...!: (1 GPs behind) idle=31e/0/0x1 softirq=2217/2218 fqs=732
[ 2838.721141] rcu: 26-...!: (1 GPs behind) idle=966/0/0x1 softirq=2421/2421 fqs=732
[ 2838.721141] rcu: (detected by 26, t=1464 jiffies, g=25105, q=1072)
[ 2838.721141] Sending NMI from CPU 26 to CPUs 1:
[ 3018.488316] NMI backtrace for cpu 1
[ 3018.488317] CPU: 1 PID: 16 Comm: ksoftirqd/1 Not tainted 4.19.121-microsoft-standard #1
[ 3018.488317] RIP: 0010:account_system_index_time+0x20/0x90
[ 3018.488318] Code: 01 30 48 01 70 40 c3 0f 1f 00 0f 1f 44 00 00 41 54 41 89 d4 55 48 89 f5 53 48 8b 87 a8 06 00 00 48 89 fb 48 01 b7 a8 05 00 00 <0f> b6 90 10 01 00 00 84 d2 74 1e 48 83 bf b0 06 00 00 00 74 14 48
[ 3018.488318] RSP: 0018:ffff889d35e43ed0 EFLAGS: 00000002
[ 3018.488319] RAX: ffff888107454400 RBX: ffff888107498e00 RCX: 0000000000989680
[ 3018.488319] RDX: 0000000000000002 RSI: 0000000000989680 RDI: ffff888107498e00
[ 3018.488319] RBP: 0000000000989680 R08: 0000000000000002 R09: 009c09a8304b73a3
[ 3018.488319] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
[ 3018.488319] R13: ffff889d35e5b3c0 R14: ffffffff81145b40 R15: ffff889d35e5b4f8
[ 3018.488320] FS: 0000000000000000(0000) GS:ffff889d35e40000(0000) knlGS:0000000000000000
[ 3018.488320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3018.488320] CR2: 0000000000200e0a CR3: 0000001c75648000 CR4: 0000000000340ea0
[ 3018.488320] Call Trace:
[ 3018.488320] <IRQ>
[ 3018.488320] update_process_times+0x1c/0x60
[ 3018.488321] tick_sched_timer+0x37/0x70
[ 3018.488321] __hrtimer_run_queues+0x100/0x280
[ 3018.488321] hrtimer_interrupt+0xf4/0x210
[ 3018.488321] hv_stimer0_isr+0x20/0x30
[ 3018.488321] hv_stimer0_vector_handler+0x3b/0x70
[ 3018.488321] hv_stimer0_callback_vector+0xf/0x20
[ 3018.488322] </IRQ>
[ 3018.488322] RIP: 0010:finish_task_switch+0x75/0x270
[ 3018.488322] Code: 8b 7e 10 65 48 8b 1c 25 c0 4c 01 00 0f 1f 44 00 00 0f 1f 44 00 00 41 c7 46 38 00 00 00 00 41 c6 04 24 00 fb 66 0f 1f 44 00 00 <65> 48 8b 04 25 c0 4c 01 00 0f 1f 44 00 00 4d 85 ed 74 21 65 48 8b
[ 3018.488322] RSP: 0018:ffffc9000019be48 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff12
[ 3018.488323] RAX: 0000000080000000 RBX: ffff888107498e00 RCX: 0000000000000000
[ 3018.488323] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff888107dd2a00
[ 3018.488323] RBP: ffffc9000019be70 R08: 00000000000000f3 R09: 0000000000000000
[ 3018.488323] R10: 0000000000000000 R11: 000002a5821747b1 R12: ffff889d35e5f600
[ 3018.488324] R13: ffff889d2ad99100 R14: ffff888107dd2a00 R15: 0000000000000000
[ 3018.488324] __schedule+0x298/0x840
[ 3018.488324] ? sort_range+0x20/0x20
[ 3018.488324] schedule+0x2c/0x80
[ 3018.488324] smpboot_thread_fn+0x10b/0x160
[ 3018.488324] kthread+0x113/0x130
[ 3018.488325] ? kthread_create_worker_on_cpu+0x70/0x70
[ 3018.488325] ret_from_fork+0x35/0x40
[ 3018.488325] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 0.000 msecs
[ 2838.721141] Sending NMI from CPU 26 to CPUs 2:
[ 2838.721141] NMI backtrace for cpu 2
[ 2838.721141] CPU: 2 PID: 21 Comm: ksoftirqd/2 Not tainted 4.19.121-microsoft-standard #1
[ 2838.721141] RIP: 0010:queued_spin_lock_slowpath+0x20/0x190
[ 2838.721141] Code: 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f 1f 44 00 00 ba 01 00 00 00 8b 07 85 c0 75 09 f0 0f b1 17 85 c0 75 f2 c3 f3 90 <eb> ed 81 fe 00 01 00 00 74 37 81 e6 00 ff ff ff 75 5f f0 0f ba 2f
[ 2838.721141] RSP: 0018:ffffc900001c3e00 EFLAGS: 00000002
[ 2838.721141] RAX: 0000000000000001 RBX: 0000000000000246 RCX: 0000000000000006
[ 2838.721141] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffffffff82444280
[ 2838.721141] RBP: ffffffff82444080 R08: 0000000000000003 R09: ffff889d35e9f600
[ 2838.721141] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff82444280
[ 2838.721141] R13: ffffffff82444280 R14: 7fffffffffffffff R15: 0000000000000009
[ 2838.721141] FS: 0000000000000000(0000) GS:ffff889d35e80000(0000) knlGS:0000000000000000
[ 2838.721141] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2838.721141] CR2: 0000000000000000 CR3: 0000001c801aa000 CR4: 0000000000340ea0
[ 2838.721141] Call Trace:
[ 2838.721141] _raw_spin_lock_irqsave+0x32/0x40
[ 2838.721141] rcu_process_callbacks+0xbe/0x4b0
[ 2838.721141] ? rebalance_domains+0x26b/0x2c0
[ 2838.721141] __do_softirq+0xde/0x2de
[ 2838.721141] ? sort_range+0x20/0x20
[ 2838.721141] run_ksoftirqd+0x26/0x40
[ 2838.721141] smpboot_thread_fn+0xc5/0x160
[ 2838.721141] kthread+0x113/0x130
[ 2838.721141] ? kthread_create_worker_on_cpu+0x70/0x70
[ 2838.721141] ret_from_fork+0x35/0x40
[ 2838.721141] Sending NMI from CPU 26 to CPUs 5:
[ 3018.490089] NMI backtrace for cpu 5 skipped: idling at native_safe_halt+0xe/0x10
[ 2838.721141] Sending NMI from CPU 26 to CPUs 6:
[ 3018.491110] NMI backtrace for cpu 6 skipped: idling at native_safe_halt+0xe/0x10
[ 2838.721141] Sending NMI from CPU 26 to CPUs 8:
[ 3018.492138] NMI backtrace for cpu 8 skipped: idling at native_safe_halt+0xe/0x10
[ 2838.721141] Sending NMI from CPU 26 to CPUs 10:
[ 3018.493166] NMI backtrace for cpu 10 skipped: idling at native_safe_halt+0xe/0x10
[ 2838.721141] NMI backtrace for cpu 26
[ 2838.721141] CPU: 26 PID: 0 Comm: swapper/26 Not tainted 4.19.121-microsoft-standard #1
[ 2838.721141] Call Trace:
[ 2838.721141] <IRQ>
[ 2838.721141] dump_stack+0x66/0x90
[ 2838.721141] nmi_cpu_backtrace.cold.3+0x13/0x50
[ 2838.721141] ? lapic_can_unplug_cpu.cold.31+0x40/0x40
[ 2838.721141] nmi_trigger_cpumask_backtrace+0xc8/0xca
[ 2838.721141] rcu_dump_cpu_stacks+0x9b/0xcb
[ 2838.721141] rcu_check_callbacks.cold.82+0x296/0x359
[ 2838.721141] ? tick_sched_do_timer+0x60/0x60
[ 2838.721141] update_process_times+0x28/0x60
[ 2838.721141] tick_sched_timer+0x37/0x70
[ 2838.721141] __hrtimer_run_queues+0x100/0x280
[ 2838.721141] hrtimer_interrupt+0xf4/0x210
[ 2838.721141] hv_stimer0_isr+0x20/0x30
[ 2838.721141] hv_stimer0_vector_handler+0x3b/0x70
[ 2838.721141] hv_stimer0_callback_vector+0xf/0x20
[ 2838.721141] </IRQ>
[ 2838.721141] RIP: 0010:native_safe_halt+0xe/0x10
[ 2838.721141] Code: 1f 44 00 00 e9 c5 fe ff ff 0f 0b eb 94 0f 0b e9 04 ff ff ff 90 90 90 90 90 90 90 90 e9 07 00 00 00 0f 00 2d e4 23 53 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d d4 23 53 00 f4 c3 90 90 0f 1f 44 00
[ 2838.721141] RSP: 0018:ffffc90000153ec8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff12
[ 2838.721141] RAX: ffffffff81ae6440 RBX: 000000000000001a RCX: ffffffff82441c10
[ 2838.721141] RDX: 0000000000007962 RSI: ffffffff8243f878 RDI: 000000000000001a
[ 2838.721141] RBP: 000000000000001a R08: 0000000000000002 R09: 009c09a8304b73a3
[ 2838.721141] R10: 00000acde75815b0 R11: 0000000000000000 R12: ffff888107468e00
[ 2838.721141] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 2838.721141] ? __cpuidle_text_start+0x8/0x8
[ 2838.721141] default_idle+0x1c/0x140
[ 2838.721141] do_idle+0xe5/0x110
[ 2838.721141] cpu_startup_entry+0x6f/0x80
[ 2838.721141] start_secondary+0x195/0x1d0
[ 2838.721141] secondary_startup_64+0xa4/0xb0
[ 2838.721141] rcu: rcu_sched kthread starved for 652 jiffies! g25105 f0x0 RCU_GP_DOING_FQS(6) ->state=0x0 ->cpu=22
[ 2838.721141] rcu: RCU grace-period kthread stack dump:
[ 2838.721141] rcu_sched I 0 10 2 0x80000000
[ 2838.721141] Call Trace:
[ 2838.721141] ? __schedule+0x290/0x840
[ 2838.721141] schedule+0x2c/0x80
[ 2838.721141] schedule_timeout+0x166/0x380
[ 2838.721141] ? __next_timer_interrupt+0xc0/0xc0
[ 2838.721141] rcu_gp_kthread+0x3e7/0x800
[ 2838.721141] ? rcu_process_callbacks+0x4b0/0x4b0
[ 2838.721141] kthread+0x113/0x130
[ 2838.721141] ? kthread_create_worker_on_cpu+0x70/0x70
[ 2838.721141] ret_from_fork+0x35/0x40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment