Skip to content

Instantly share code, notes, and snippets.

@discordianfish

discordianfish/3.11.0-15-generic Secret

Last active August 29, 2015 13:55
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save discordianfish/7886354d9a19b2084775 to your computer and use it in GitHub Desktop.
Save discordianfish/7886354d9a19b2084775 to your computer and use it in GitHub Desktop.
Download ZIP
Rapid ghost container restarts cause kernel lockup
Raw
3.11.0-15-generic
[ 797.035435] BUG: soft lockup - CPU#0 stuck for 23s! [supervisord:8323]
[ 797.042613] Kernel panic - not syncing: softlockup: hung tasks
[ 797.048831] CPU: 0 PID: 8323 Comm: supervisord Tainted: GF I 3.11.0-15-generic #25~precise1-Ubuntu
[ 797.051439] BUG: soft lockup - CPU#2 stuck for 23s! [supervisord:8414]
[ 797.051457] Modules linked in: veth xt_nat xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc vesafb(F) ext2 psmouse lp joydev parport gpio_ich serio_raw hid_generic dcdbas wmi usbhid acpi_power_meter hid i7core_edac lpc_ich edac_core mac_hid ses enclosure bnx2
[ 797.051459] CPU: 2 PID: 8414 Comm: supervisord Tainted: GF I 3.11.0-15-generic #25~precise1-Ubuntu
[ 797.051460] Hardware name: Dell Inc. PowerEdge R710/0YDJK3, BIOS 6.3.0 07/24/2012
[ 797.051462] task: ffff88060b935dc0 ti: ffff88061713a000 task.ti: ffff88061713a000
[ 797.051466] RIP: 0010:[<ffffffff81050ff5>] [<ffffffff81050ff5>] __ticket_spin_lock+0x25/0x30
[ 797.051468] RSP: 0018:ffff88061713bc18 EFLAGS: 00000283
[ 797.051469] RAX: 00000000000097ae RBX: ffff880326960300 RCX: 0000000000000000
[ 797.051470] RDX: 00000000000097b6 RSI: ffffffff81822f11 RDI: ffffffff81c06384
[ 797.051471] RBP: ffff88061713bc18 R08: ffff88061713bc60 R09: ffffffff81822f10
[ 797.051472] R10: ffffffff811c3296 R11: ffff9b8b8c96888b R12: ffffffffa0168645
[ 797.051473] R13: ffff88061713bbe8 R14: ffff880623cba000 R15: 0000000000000001
[ 797.051474] FS: 00007f4fec739700(0000) GS:ffff88063fc20000(0000) knlGS:0000000000000000
[ 797.051476] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 797.051477] CR2: 000000c2105570c8 CR3: 000000062c119000 CR4: 00000000000007e0
[ 797.051477] Stack:
[ 797.051481] ffff88061713bc28 ffffffff8174759e ffff88061713bc78 ffffffff811cc1b0
[ 797.051484] ffff88061713bc98 00001fff1713bdb8 ffff88061713bd08 0000000000000000
[ 797.051487] 0000000000000000 ffff880633049fff ffff88061713bc98 ffff88061713be10
[ 797.051487] Call Trace:
[ 797.051491] [<ffffffff8174759e>] _raw_spin_lock+0xe/0x20
[ 797.051494] [<ffffffff811cc1b0>] d_absolute_path+0x60/0xd0
[ 797.051497] [<ffffffff81326f2c>] d_namespace_path+0x4c/0x2b0
[ 797.051500] [<ffffffff811d3434>] ? mntput+0x24/0x40
[ 797.051503] [<ffffffff811be052>] ? path_put+0x22/0x30
[ 797.051505] [<ffffffff813271da>] aa_path_name+0x4a/0x110
[ 797.051508] [<ffffffff8132f7d9>] aa_path_perm+0x89/0x240
[ 797.051511] [<ffffffff8132da4f>] common_perm+0x5f/0x110
[ 797.051513] [<ffffffff8132db54>] apparmor_inode_getattr+0x54/0x60
[ 797.051517] [<ffffffff812f0ba0>] security_inode_getattr+0x20/0x30
[ 797.051519] [<ffffffff811b8fbf>] vfs_getattr+0x2f/0x80
[ 797.051521] [<ffffffff811b9074>] vfs_fstatat+0x64/0xb0
[ 797.051523] [<ffffffff811b918b>] vfs_stat+0x1b/0x20
[ 797.051525] [<ffffffff811b91a5>] SYSC_newstat+0x15/0x30
[ 797.051529] [<ffffffff81748995>] ? do_device_not_available+0x25/0x50
[ 797.051532] [<ffffffff817521ce>] ? device_not_available+0x1e/0x30
[ 797.051534] [<ffffffff811b92fe>] SyS_newstat+0xe/0x10
[ 797.051537] [<ffffffff8175099d>] system_call_fastpath+0x1a/0x1f
[ 797.051557] Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1
[ 797.087447] BUG: soft lockup - CPU#4 stuck for 23s! [docker:8498]
[ 797.087464] Modules linked in: veth xt_nat xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc vesafb(F) ext2 psmouse lp joydev parport gpio_ich serio_raw hid_generic dcdbas wmi usbhid acpi_power_meter hid i7core_edac lpc_ich edac_core mac_hid ses enclosure bnx2
[ 797.087467] CPU: 4 PID: 8498 Comm: docker Tainted: GF I 3.11.0-15-generic #25~precise1-Ubuntu
[ 797.087468] Hardware name: Dell Inc. PowerEdge R710/0YDJK3, BIOS 6.3.0 07/24/2012
[ 797.087469] task: ffff88061727c650 ti: ffff880608f12000 task.ti: ffff880608f12000
[ 797.087473] RIP: 0010:[<ffffffff81050ff2>] [<ffffffff81050ff2>] __ticket_spin_lock+0x22/0x30
[ 797.087475] RSP: 0018:ffff880608f13c88 EFLAGS: 00000287
[ 797.087476] RAX: 00000000000097ae RBX: ffffffff81b2b078 RCX: 0000000000000000
[ 797.087477] RDX: 00000000000097b5 RSI: ffffffff81822f11 RDI: ffffffff81c06384
[ 797.087478] RBP: ffff880608f13c88 R08: ffff880608f13cd0 R09: ffffffff81822f10
[ 797.087479] R10: 0000000000003735 R11: 0000000000000000 R12: 0000000000000014
[ 797.087480] R13: ffff88062c465058 R14: ffff88062c4650b0 R15: 0000000000000000
[ 797.087481] FS: 00007fce14ff9700(0000) GS:ffff88063fc40000(0000) knlGS:0000000000000000
[ 797.087483] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 797.087484] CR2: 00000000006040e0 CR3: 0000000612349000 CR4: 00000000000007e0
[ 797.087484] Stack:
[ 797.087488] ffff880608f13c98 ffffffff8174759e ffff880608f13ce8 ffffffff811cc1b0
[ 797.087491] ffff880608f13cb8 00001fff8174759e ffff880608f13ce8 0000000000000000
[ 797.087494] 0000000000000000 ffff880633051fff ffff88062c468240 ffff88032e741c10
[ 797.087494] Call Trace:
[ 797.087498] [<ffffffff8174759e>] _raw_spin_lock+0xe/0x20
[ 797.087500] [<ffffffff811cc1b0>] d_absolute_path+0x60/0xd0
[ 797.087503] [<ffffffff81326f2c>] d_namespace_path+0x4c/0x2b0
[ 797.087506] [<ffffffff813271da>] aa_path_name+0x4a/0x110
[ 797.087509] [<ffffffff813278de>] apparmor_bprm_set_creds+0x11e/0x910
[ 797.087513] [<ffffffff811777c5>] ? __vma_link_rb+0x105/0x120
[ 797.087515] [<ffffffff811778af>] ? vma_link+0xcf/0xe0
[ 797.087519] [<ffffffff812f0803>] security_bprm_set_creds+0x13/0x20
[ 797.087521] [<ffffffff811b9c88>] prepare_binprm+0xb8/0x1e0
[ 797.087524] [<ffffffff811bb878>] do_execve_common.isra.23+0x268/0x370
[ 797.087526] [<ffffffff811bb998>] do_execve+0x18/0x20
[ 797.087528] [<ffffffff811bbc1d>] SyS_execve+0x3d/0x60
[ 797.087531] [<ffffffff81750f49>] stub_execve+0x69/0xa0
[ 797.087551] Code: 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89
[ 797.119454] BUG: soft lockup - CPU#6 stuck for 22s! [gunicorn:7155]
[ 797.119472] Modules linked in: veth xt_nat xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc vesafb(F) ext2 psmouse lp joydev parport gpio_ich serio_raw hid_generic dcdbas wmi usbhid acpi_power_meter hid i7core_edac lpc_ich edac_core mac_hid ses enclosure bnx2
[ 797.119474] CPU: 6 PID: 7155 Comm: gunicorn Tainted: GF I 3.11.0-15-generic #25~precise1-Ubuntu
[ 797.119475] Hardware name: Dell Inc. PowerEdge R710/0YDJK3, BIOS 6.3.0 07/24/2012
[ 797.119476] task: ffff88032c269770 ti: ffff88032d914000 task.ti: ffff88032d914000
[ 797.119481] RIP: 0010:[<ffffffff81050ff2>] [<ffffffff81050ff2>] __ticket_spin_lock+0x22/0x30
[ 797.119482] RSP: 0018:ffff88032d915c28 EFLAGS: 00000283
[ 797.119483] RAX: 00000000000097ae RBX: 00001f903ffe7b08 RCX: 0000000000000000
[ 797.119484] RDX: 00000000000097b3 RSI: ffffffff81822f11 RDI: ffffffff81c06384
[ 797.119485] RBP: ffff88032d915c28 R08: ffff88032d915c70 R09: ffffffff81822f10
[ 797.119486] R10: 00007fff1efc4a40 R11: 0000000000000246 R12: ffffffff811cc1ee
[ 797.119487] R13: ffff88032d915bd8 R14: ffffffff8109139a R15: ffff88032d915b88
[ 797.119489] FS: 00007eff609cb700(0000) GS:ffff88063fc60000(0000) knlGS:0000000000000000
[ 797.119490] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 797.119491] CR2: 000000c20ffc9f98 CR3: 000000032b66e000 CR4: 00000000000007e0
[ 797.119492] Stack:
[ 797.119496] ffff88032d915c38 ffffffff8174759e ffff88032d915c88 ffffffff811cc1b0
[ 797.119498] 0000000000000000 00001fff00000000 ffff88032d914007 0000000000000000
[ 797.119501] 0000000000000000 ffff880633059fff ffff88032d915d80 ffff88032d915e20
[ 797.119502] Call Trace:
[ 797.119505] [<ffffffff8174759e>] _raw_spin_lock+0xe/0x20
[ 797.119508] [<ffffffff811cc1b0>] d_absolute_path+0x60/0xd0
[ 797.119511] [<ffffffff81326f2c>] d_namespace_path+0x4c/0x2b0
[ 797.119514] [<ffffffff813271da>] aa_path_name+0x4a/0x110
[ 797.119516] [<ffffffff8132f7d9>] aa_path_perm+0x89/0x240
[ 797.119519] [<ffffffff811c6bb7>] ? set_fd_set+0x27/0x30
[ 797.119521] [<ffffffff8132da4f>] common_perm+0x5f/0x110
[ 797.119524] [<ffffffff8132db54>] apparmor_inode_getattr+0x54/0x60
[ 797.119527] [<ffffffff812f0ba0>] security_inode_getattr+0x20/0x30
[ 797.119529] [<ffffffff811b8fbf>] vfs_getattr+0x2f/0x80
[ 797.119531] [<ffffffff811b922d>] vfs_fstat+0x3d/0x70
[ 797.119533] [<ffffffff811b9275>] SYSC_newfstat+0x15/0x30
[ 797.119536] [<ffffffff811c800e>] ? SyS_select+0xce/0x100
[ 797.119538] [<ffffffff811b932e>] SyS_newfstat+0xe/0x10
[ 797.119541] [<ffffffff8175099d>] system_call_fastpath+0x1a/0x1f
[ 797.119560] Code: 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89
[ 797.155463] BUG: soft lockup - CPU#8 stuck for 22s! [docker:8496]
[ 797.155478] Modules linked in: veth xt_nat xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc vesafb(F) ext2 psmouse lp joydev parport gpio_ich serio_raw hid_generic dcdbas wmi usbhid acpi_power_meter hid i7core_edac lpc_ich edac_core mac_hid ses enclosure bnx2
[ 797.155480] CPU: 8 PID: 8496 Comm: docker Tainted: GF I 3.11.0-15-generic #25~precise1-Ubuntu
[ 797.155481] Hardware name: Dell Inc. PowerEdge R710/0YDJK3, BIOS 6.3.0 07/24/2012
[ 797.155482] task: ffff88062d535dc0 ti: ffff88061bf30000 task.ti: ffff88061bf30000
[ 797.155485] RIP: 0010:[<ffffffff81050ff5>] [<ffffffff81050ff5>] __ticket_spin_lock+0x25/0x30
[ 797.155486] RSP: 0018:ffff88061bf31c88 EFLAGS: 00000287
[ 797.155487] RAX: 00000000000097ae RBX: ffffffff81b2b078 RCX: 0000000000000000
[ 797.155488] RDX: 00000000000097b4 RSI: ffffffff81822f11 RDI: ffffffff81c06384
[ 797.155488] RBP: ffff88061bf31c88 R08: ffff88061bf31cd0 R09: ffffffff81822f10
[ 797.155489] R10: 0000000000003735 R11: 0000000000000001 R12: 0000000000000014
[ 797.155490] R13: ffff88062c465058 R14: ffff88062c4650b0 R15: 0000000000000000
[ 797.155491] FS: 00007fce157fa700(0000) GS:ffff88063fc80000(0000) knlGS:0000000000000000
[ 797.155492] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 797.155493] CR2: 000000c21000f020 CR3: 000000062d0e3000 CR4: 00000000000007e0
[ 797.155494] Stack:
[ 797.155497] ffff88061bf31c98 ffffffff8174759e ffff88061bf31ce8 ffffffff811cc1b0
[ 797.155500] ffff88061bf31cb8 00001fff8174759e ffff88061bf31ce8 0000000000000000
[ 797.155503] 0000000000000000 ffff880633061fff ffff88062c468240 ffff8806225a8010
[ 797.155503] Call Trace:
[ 797.155506] [<ffffffff8174759e>] _raw_spin_lock+0xe/0x20
[ 797.155508] [<ffffffff811cc1b0>] d_absolute_path+0x60/0xd0
[ 797.155511] [<ffffffff81326f2c>] d_namespace_path+0x4c/0x2b0
[ 797.155513] [<ffffffff813271da>] aa_path_name+0x4a/0x110
[ 797.155515] [<ffffffff813278de>] apparmor_bprm_set_creds+0x11e/0x910
[ 797.155518] [<ffffffff811777c5>] ? __vma_link_rb+0x105/0x120
[ 797.155520] [<ffffffff811778af>] ? vma_link+0xcf/0xe0
[ 797.155524] [<ffffffff812f0803>] security_bprm_set_creds+0x13/0x20
[ 797.155526] [<ffffffff811b9c88>] prepare_binprm+0xb8/0x1e0
[ 797.155528] [<ffffffff811bb878>] do_execve_common.isra.23+0x268/0x370
[ 797.155531] [<ffffffff811bb998>] do_execve+0x18/0x20
[ 797.155532] [<ffffffff811bbc1d>] SyS_execve+0x3d/0x60
[ 797.155535] [<ffffffff81750f49>] stub_execve+0x69/0xa0
[ 797.155552] Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1
[ 797.191470] BUG: soft lockup - CPU#10 stuck for 22s! [docker:8426]
[ 797.191487] Modules linked in: veth xt_nat xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc vesafb(F) ext2 psmouse lp joydev parport gpio_ich serio_raw hid_generic dcdbas wmi usbhid acpi_power_meter hid i7core_edac lpc_ich edac_core mac_hid ses enclosure bnx2
[ 797.191489] CPU: 10 PID: 8426 Comm: docker Tainted: GF I 3.11.0-15-generic #25~precise1-Ubuntu
[ 797.191490] Hardware name: Dell Inc. PowerEdge R710/0YDJK3, BIOS 6.3.0 07/24/2012
[ 797.191491] task: ffff88061eb89770 ti: ffff88061e9c0000 task.ti: ffff88061e9c0000
[ 797.191495] RIP: 0010:[<ffffffff81050ff2>] [<ffffffff81050ff2>] __ticket_spin_lock+0x22/0x30
[ 797.191496] RSP: 0018:ffff88061e9c1d68 EFLAGS: 00000283
[ 797.191497] RAX: 00000000000097ae RBX: ffff88061e9c1db0 RCX: 0000000000001000
[ 797.191498] RDX: 00000000000097b1 RSI: ffff88061ead600e RDI: ffffffff81c06384
[ 797.191499] RBP: ffff88061e9c1d68 R08: 000000000000000a R09: 000000000000fffe
[ 797.191500] R10: 0000000000000000 R11: 0000000000000246 R12: ffff88061ead600c
[ 797.191501] R13: ffff88032cb05408 R14: 000000008204f7cd R15: 000000000000026c
[ 797.191502] FS: 00007fce15ffb700(0000) GS:ffff88063fca0000(0000) knlGS:0000000000000000
[ 797.191503] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 797.191504] CR2: 000000c2102781b1 CR3: 000000032ccb0000 CR4: 00000000000007e0
[ 797.191505] Stack:
[ 797.191509] ffff88061e9c1d78 ffffffff8174759e ffff88061e9c1dc8 ffffffff811cc3b6
[ 797.191512] ffff88062bd42d80 00000ff222535220 ffff88061e9c1df8 0000000000000000
[ 797.191515] ffff88033fecd400 ffff88032a422300 ffff88061ead600e ffffffff81b11dec
[ 797.191515] Call Trace:
[ 797.191518] [<ffffffff8174759e>] _raw_spin_lock+0xe/0x20
[ 797.191520] [<ffffffff811cc3b6>] dentry_path+0x36/0xe0
[ 797.191524] [<ffffffff811d77e1>] seq_dentry+0x51/0xd0
[ 797.191529] [<ffffffff811f322a>] show_mountinfo+0xca/0x2b0
[ 797.191532] [<ffffffff811d1db6>] m_show+0x16/0x20
[ 797.191534] [<ffffffff811d6d73>] seq_read+0x163/0x3e0
[ 797.191539] [<ffffffff811b42e4>] vfs_read+0xb4/0x180
[ 797.191542] [<ffffffff811b4562>] SyS_read+0x52/0xa0
[ 797.191544] [<ffffffff8175099d>] system_call_fastpath+0x1a/0x1f
[ 797.191565] Code: 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89
[ 797.211477] BUG: soft lockup - CPU#11 stuck for 22s! [twistd:8488]
[ 797.211499] Modules linked in: veth xt_nat xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc vesafb(F) ext2 psmouse lp joydev parport gpio_ich serio_raw hid_generic dcdbas wmi usbhid acpi_power_meter hid i7core_edac lpc_ich edac_core mac_hid ses enclosure bnx2
[ 797.211502] CPU: 11 PID: 8488 Comm: twistd Tainted: GF I 3.11.0-15-generic #25~precise1-Ubuntu
[ 797.211503] Hardware name: Dell Inc. PowerEdge R710/0YDJK3, BIOS 6.3.0 07/24/2012
[ 797.211504] task: ffff88062e571770 ti: ffff88060b8a0000 task.ti: ffff88060b8a0000
[ 797.211512] RIP: 0010:[<ffffffff81050ff2>] [<ffffffff81050ff2>] __ticket_spin_lock+0x22/0x30
[ 797.211513] RSP: 0018:ffff88060b8a1a18 EFLAGS: 00000283
[ 797.211514] RAX: 00000000000097ae RBX: 0000000000000000 RCX: 0000000000000000
[ 797.211515] RDX: 00000000000097b0 RSI: ffffffff81822f11 RDI: ffffffff81c06384
[ 797.211516] RBP: ffff88060b8a1a18 R08: ffff88060b8a1a60 R09: ffffffff81822f10
[ 797.211517] R10: 0100000100000301 R11: 9a93ff868fd18c9a R12: ffffffff81173ff0
[ 797.211518] R13: ffff88060b8a19e8 R14: ffff880321baf0c8 R15: ffff8802ff41d400
[ 797.211519] FS: 00007f94832df700(0000) GS:ffff880333ca0000(0000) knlGS:0000000000000000
[ 797.211521] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 797.211522] CR2: 0000000002920b48 CR3: 000000032e32d000 CR4: 00000000000007e0
[ 797.211522] Stack:
[ 797.211526] ffff88060b8a1a28 ffffffff8174759e ffff88060b8a1a78 ffffffff811cc1b0
[ 797.211529] 0000000000000000 00001fff000280da ffff88060b8a1b08 0000000000000000
[ 797.211532] 0000000000000000 ffff88063306dfff 0000000000000000 ffff8802ffd1b510
[ 797.211532] Call Trace:
[ 797.211538] [<ffffffff8174759e>] _raw_spin_lock+0xe/0x20
[ 797.211542] [<ffffffff811cc1b0>] d_absolute_path+0x60/0xd0
[ 797.211547] [<ffffffff81326f2c>] d_namespace_path+0x4c/0x2b0
[ 797.211550] [<ffffffff813271da>] aa_path_name+0x4a/0x110
[ 797.211553] [<ffffffff8132f7d9>] aa_path_perm+0x89/0x240
[ 797.211556] [<ffffffff817340c5>] ? check_acl+0x5c/0x133
[ 797.211559] [<ffffffff8132e214>] apparmor_file_open+0x184/0x1f0
[ 797.211571] [<ffffffffa0180c83>] ? aufs_permission+0x193/0x300 [aufs]
[ 797.211573] [<ffffffff812f0f69>] security_file_open+0x29/0x90
[ 797.211577] [<ffffffff811b1483>] do_dentry_open+0xf3/0x2a0
[ 797.211579] [<ffffffff811b1939>] vfs_open+0x49/0x50
[ 797.211582] [<ffffffff811c137c>] do_last+0x14c/0x780
[ 797.211585] [<ffffffff811c2e44>] path_openat+0xc4/0x4f0
[ 797.211587] [<ffffffff811c3296>] ? final_putname+0x26/0x50
[ 797.211590] [<ffffffff811c34cb>] ? putname+0x2b/0x40
[ 797.211592] [<ffffffff811c4033>] do_filp_open+0x43/0xa0
[ 797.211595] [<ffffffff811d1005>] ? __alloc_fd+0xd5/0x160
[ 797.211598] [<ffffffff811b2e56>] do_sys_open+0x136/0x2a0
[ 797.211600] [<ffffffff811b2fde>] SyS_open+0x1e/0x20
[ 797.211604] [<ffffffff8175099d>] system_call_fastpath+0x1a/0x1f
[ 797.211624] Code: 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89
[ 797.267487] BUG: soft lockup - CPU#14 stuck for 22s! [twistd:8484]
[ 797.267505] Modules linked in: veth xt_nat xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc vesafb(F) ext2 psmouse lp joydev parport gpio_ich serio_raw hid_generic dcdbas wmi usbhid acpi_power_meter hid i7core_edac lpc_ich edac_core mac_hid ses enclosure bnx2
[ 797.267507] CPU: 14 PID: 8484 Comm: twistd Tainted: GF I 3.11.0-15-generic #25~precise1-Ubuntu
[ 797.267508] Hardware name: Dell Inc. PowerEdge R710/0YDJK3, BIOS 6.3.0 07/24/2012
[ 797.267509] task: ffff88060b930000 ti: ffff88060b8b2000 task.ti: ffff88060b8b2000
[ 797.267513] RIP: 0010:[<ffffffff81050ff5>] [<ffffffff81050ff5>] __ticket_spin_lock+0x25/0x30
[ 797.267514] RSP: 0018:ffff88060b8b3a18 EFLAGS: 00000297
[ 797.267515] RAX: 00000000000097ae RBX: 0000000000000000 RCX: 0000000000000000
[ 797.267516] RDX: 00000000000097af RSI: ffffffff81822f11 RDI: ffffffff81c06384
[ 797.267517] RBP: ffff88060b8b3a18 R08: ffff88060b8b3a60 R09: ffffffff81822f10
[ 797.267518] R10: 0001000001003f03 R11: ff868fd1868f909c R12: ffffffff81173ff0
[ 797.267519] R13: ffff88060b8b39e8 R14: ffff88062d12a8c8 R15: ffff880608cf7000
[ 797.267521] FS: 00007fec63284700(0000) GS:ffff88063fce0000(0000) knlGS:0000000000000000
[ 797.267522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 797.267523] CR2: 0000000001230128 CR3: 0000000608ce9000 CR4: 00000000000007e0
[ 797.267524] Stack:
[ 797.267527] ffff88060b8b3a28 ffffffff8174759e ffff88060b8b3a78 ffffffff811cc1b0
[ 797.267530] 0000000000000000 00001fff000280da ffff88060b8b3b08 0000000000000000
[ 797.267533] 0000000000000000 ffff880633079fff 0000000000000000 ffff88032dcfab10
[ 797.267533] Call Trace:
[ 797.267536] [<ffffffff8174759e>] _raw_spin_lock+0xe/0x20
[ 797.267539] [<ffffffff811cc1b0>] d_absolute_path+0x60/0xd0
[ 797.267542] [<ffffffff81326f2c>] d_namespace_path+0x4c/0x2b0
[ 797.267545] [<ffffffff813271da>] aa_path_name+0x4a/0x110
[ 797.267547] [<ffffffff8132f7d9>] aa_path_perm+0x89/0x240
[ 797.267550] [<ffffffff817340c5>] ? check_acl+0x5c/0x133
[ 797.267553] [<ffffffff8132e214>] apparmor_file_open+0x184/0x1f0
[ 797.267563] [<ffffffffa0180c83>] ? aufs_permission+0x193/0x300 [aufs]
[ 797.267566] [<ffffffff812f0f69>] security_file_open+0x29/0x90
[ 797.267569] [<ffffffff811b1483>] do_dentry_open+0xf3/0x2a0
[ 797.267571] [<ffffffff811b1939>] vfs_open+0x49/0x50
[ 797.267574] [<ffffffff811c137c>] do_last+0x14c/0x780
[ 797.267577] [<ffffffff811c2e44>] path_openat+0xc4/0x4f0
[ 797.267579] [<ffffffff811c3296>] ? final_putname+0x26/0x50
[ 797.267582] [<ffffffff811c34cb>] ? putname+0x2b/0x40
[ 797.267585] [<ffffffff811c4033>] do_filp_open+0x43/0xa0
[ 797.267587] [<ffffffff811d1005>] ? __alloc_fd+0xd5/0x160
[ 797.267590] [<ffffffff811b2e56>] do_sys_open+0x136/0x2a0
[ 797.267593] [<ffffffff811b2fde>] SyS_open+0x1e/0x20
[ 797.267595] [<ffffffff8175099d>] system_call_fastpath+0x1a/0x1f
[ 797.267615] Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1
[ 797.287493] BUG: soft lockup - CPU#15 stuck for 22s! [docker:8495]
[ 797.287510] Modules linked in: veth xt_nat xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc vesafb(F) ext2 psmouse lp joydev parport gpio_ich serio_raw hid_generic dcdbas wmi usbhid acpi_power_meter hid i7core_edac lpc_ich edac_core mac_hid ses enclosure bnx2
[ 797.287512] CPU: 15 PID: 8495 Comm: docker Tainted: GF I 3.11.0-15-generic #25~precise1-Ubuntu
[ 797.287513] Hardware name: Dell Inc. PowerEdge R710/0YDJK3, BIOS 6.3.0 07/24/2012
[ 797.287515] task: ffff88061bfa0000 ti: ffff88062cc7e000 task.ti: ffff88062cc7e000
[ 797.287519] RIP: 0010:[<ffffffff81050ff5>] [<ffffffff81050ff5>] __ticket_spin_lock+0x25/0x30
[ 797.287520] RSP: 0018:ffff88062cc7fc88 EFLAGS: 00000287
[ 797.287521] RAX: 00000000000097ae RBX: ffffffff81b2b078 RCX: 0000000000000000
[ 797.287522] RDX: 00000000000097b2 RSI: ffffffff81822f11 RDI: ffffffff81c06384
[ 797.287523] RBP: ffff88062cc7fc88 R08: ffff88062cc7fcd0 R09: ffffffff81822f10
[ 797.287524] R10: 000000000000292f R11: 0000000000000001 R12: 0000000000000014
[ 797.287525] R13: ffff88062c465058 R14: ffff88062c4650b0 R15: 0000000000000000
[ 797.287527] FS: 00007fce16ffd700(0000) GS:ffff880333ce0000(0000) knlGS:0000000000000000
[ 797.287528] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 797.287529] CR2: 00007f5c7ee23120 CR3: 000000062ce11000 CR4: 00000000000007e0
[ 797.287529] Stack:
[ 797.287533] ffff88062cc7fc98 ffffffff8174759e ffff88062cc7fce8 ffffffff811cc1b0
[ 797.287536] ffff88062cc7fcb8 00001fff8174759e ffff88062cc7fce8 0000000000000000
[ 797.287538] 0000000000000000 ffff88063307dfff ffff88062c468240 ffff8806170ca110
[ 797.287539] Call Trace:
[ 797.287542] [<ffffffff8174759e>] _raw_spin_lock+0xe/0x20
[ 797.287545] [<ffffffff811cc1b0>] d_absolute_path+0x60/0xd0
[ 797.287548] [<ffffffff81326f2c>] d_namespace_path+0x4c/0x2b0
[ 797.287550] [<ffffffff813271da>] aa_path_name+0x4a/0x110
[ 797.287553] [<ffffffff813278de>] apparmor_bprm_set_creds+0x11e/0x910
[ 797.287556] [<ffffffff811777c5>] ? __vma_link_rb+0x105/0x120
[ 797.287558] [<ffffffff811778af>] ? vma_link+0xcf/0xe0
[ 797.287563] [<ffffffff812f0803>] security_bprm_set_creds+0x13/0x20
[ 797.287565] [<ffffffff811b9c88>] prepare_binprm+0xb8/0x1e0
[ 797.287567] [<ffffffff811bb878>] do_execve_common.isra.23+0x268/0x370
[ 797.287570] [<ffffffff811bb998>] do_execve+0x18/0x20
[ 797.287572] [<ffffffff811bbc1d>] SyS_execve+0x3d/0x60
[ 797.287575] [<ffffffff81750f49>] stub_execve+0x69/0xa0
[ 797.287595] Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1
[ 801.068350] BUG: soft lockup - CPU#3 stuck for 23s! [redis-server:7277]
[ 801.068368] Modules linked in: veth xt_nat xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc vesafb(F) ext2 psmouse lp joydev parport gpio_ich serio_raw hid_generic dcdbas wmi usbhid acpi_power_meter hid i7core_edac lpc_ich edac_core mac_hid ses enclosure bnx2
[ 801.068370] CPU: 3 PID: 7277 Comm: redis-server Tainted: GF I 3.11.0-15-generic #25~precise1-Ubuntu
[ 801.068371] Hardware name: Dell Inc. PowerEdge R710/0YDJK3, BIOS 6.3.0 07/24/2012
[ 801.068372] task: ffff880321a5ddc0 ti: ffff88032a6ee000 task.ti: ffff88032a6ee000
[ 801.068376] RIP: 0010:[<ffffffff81050ff2>] [<ffffffff81050ff2>] __ticket_spin_lock+0x22/0x30
[ 801.068377] RSP: 0018:ffff88032a6efc18 EFLAGS: 00000287
[ 801.068378] RAX: 00000000000097ae RBX: ffff88062c7d9b00 RCX: 0000000000000000
[ 801.068379] RDX: 00000000000097b8 RSI: ffffffff81822f11 RDI: ffffffff81c06384
[ 801.068380] RBP: ffff88032a6efc18 R08: ffff88032a6efc60 R09: ffffffff81822f10
[ 801.068381] R10: ffffffff811c3296 R11: 909d9091ffaaff9a R12: ffffffffa0168645
[ 801.068382] R13: ffff88032a6efbe8 R14: ffff88062d170800 R15: 0000000000000001
[ 801.068383] FS: 00007fdccc4df700(0000) GS:ffff880333c20000(0000) knlGS:0000000000000000
[ 801.068385] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 801.068386] CR2: 00000000019c9100 CR3: 000000062dff3000 CR4: 00000000000007e0
[ 801.068386] Stack:
[ 801.068390] ffff88032a6efc28 ffffffff8174759e ffff88032a6efc78 ffffffff811cc1b0
[ 801.068393] 0000000000014580 00001fff2a6efdb8 ffff88032a6efd08 0000000000000000
[ 801.068396] 0000000000000000 ffff88063304dfff ffff88032a6efc98 ffff88032a6efe10
[ 801.068396] Call Trace:
[ 801.068399] [<ffffffff8174759e>] _raw_spin_lock+0xe/0x20
[ 801.068402] [<ffffffff811cc1b0>] d_absolute_path+0x60/0xd0
[ 801.068405] [<ffffffff81326f2c>] d_namespace_path+0x4c/0x2b0
[ 801.068408] [<ffffffff811d3434>] ? mntput+0x24/0x40
[ 801.068410] [<ffffffff811be052>] ? path_put+0x22/0x30
[ 801.068413] [<ffffffff813271da>] aa_path_name+0x4a/0x110
[ 801.068416] [<ffffffff8132f7d9>] aa_path_perm+0x89/0x240
[ 801.068418] [<ffffffff8132da4f>] common_perm+0x5f/0x110
[ 801.068421] [<ffffffff8132db54>] apparmor_inode_getattr+0x54/0x60
[ 801.068424] [<ffffffff812f0ba0>] security_inode_getattr+0x20/0x30
[ 801.068426] [<ffffffff811b8fbf>] vfs_getattr+0x2f/0x80
[ 801.068428] [<ffffffff811b9074>] vfs_fstatat+0x64/0xb0
[ 801.068430] [<ffffffff811b918b>] vfs_stat+0x1b/0x20
[ 801.068432] [<ffffffff811b91a5>] SYSC_newstat+0x15/0x30
[ 801.068437] [<ffffffff811fc316>] ? SyS_epoll_wait+0xd6/0xf0
[ 801.068439] [<ffffffff811b92fe>] SyS_newstat+0xe/0x10
[ 801.068441] [<ffffffff8175099d>] system_call_fastpath+0x1a/0x1f
[ 801.068462] Code: 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89
[ 804.443151] Hardware name: Dell Inc. PowerEdge R710/0YDJK3, BIOS 6.3.0 07/24/2012
[ 804.468446] 0000000000000017 ffff88063fc03dd8 ffffffff8173bc5e 000000000000524d
[ 804.493945] ffffffff81b0a60f ffff88063fc03e58 ffffffff8172e8d8 ffff88063fc03e58
[ 804.519041] ffffffff00000008 ffff88063fc03e68 ffff88063fc03e08 ffff88063fc03e27
[ 804.544618] Call Trace:
[ 804.556053] <IRQ> [<ffffffff8173bc5e>] dump_stack+0x46/0x58
[ 804.570286] [<ffffffff8172e8d8>] panic+0xc1/0x1d7
[ 804.583212] [<ffffffff810fbe17>] watchdog_timer_fn+0x167/0x170
[ 804.599149] [<ffffffff8108c8b8>] __run_hrtimer+0x78/0x230
[ 804.615008] [<ffffffff810fbcb0>] ? watchdog+0x30/0x30
[ 804.631636] [<ffffffff8108d0bf>] hrtimer_interrupt+0xff/0x240
[ 804.646650] [<ffffffff810437bb>] local_apic_timer_interrupt+0x3b/0x60
[ 804.662384] [<ffffffff81752d65>] smp_apic_timer_interrupt+0x45/0x60
[ 804.679850] [<ffffffff817516dd>] apic_timer_interrupt+0x6d/0x80
[ 804.696966] <EOI> [<ffffffff811c3296>] ? final_putname+0x26/0x50
[ 804.713545] [<ffffffff81050ff5>] ? __ticket_spin_lock+0x25/0x30
[ 804.729232] [<ffffffff8174759e>] _raw_spin_lock+0xe/0x20
[ 804.744017] [<ffffffff811cc1b0>] d_absolute_path+0x60/0xd0
[ 804.759860] [<ffffffff81326f2c>] d_namespace_path+0x4c/0x2b0
[ 804.776874] [<ffffffff811d3434>] ? mntput+0x24/0x40
[ 804.793245] [<ffffffff811be052>] ? path_put+0x22/0x30
[ 804.807905] [<ffffffff813271da>] aa_path_name+0x4a/0x110
[ 804.823793] [<ffffffff8132f7d9>] aa_path_perm+0x89/0x240
[ 804.839583] [<ffffffff8132da4f>] common_perm+0x5f/0x110
[ 804.855775] [<ffffffff8132db54>] apparmor_inode_getattr+0x54/0x60
[ 804.872384] [<ffffffff812f0ba0>] security_inode_getattr+0x20/0x30
[ 804.887173] [<ffffffff811b8fbf>] vfs_getattr+0x2f/0x80
[ 804.900936] [<ffffffff811b9074>] vfs_fstatat+0x64/0xb0
[ 804.915506] [<ffffffff811b918b>] vfs_stat+0x1b/0x20
[ 804.931460] [<ffffffff811b91a5>] SYSC_newstat+0x15/0x30
[ 804.945891] [<ffffffff8174bde7>] ? do_page_fault+0x37/0x70
[ 804.960574] [<ffffffff81747e98>] ? page_fault+0x28/0x30
[ 804.972770] [<ffffffff811b92fe>] SyS_newstat+0xe/0x10
[ 804.989091] [<ffffffff8175099d>] system_call_fastpath+0x1a/0x1f
[ 806.064978] Shutting down cpus with NMI
[ 806.077541] ------------[ cut here ]------------
[ 806.089265] WARNING: CPU: 0 PID: 75 at /build/buildd/linux-lts-saucy-3.11.0/kernel/time/tick-broadcast.c:603 tick_handle_oneshot_broadcast+0x1c6/0x1f0()
[ 806.118377] Modules linked in: veth xt_nat xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc vesafb(F) ext2 psmouse lp joydev parport gpio_ich serio_raw hid_generic dcdbas wmi usbhid acpi_power_meter hid i7core_edac lpc_ich edac_core mac_hid ses enclosure bnx2
[ 806.191272] CPU: 0 PID: 75 Comm: watchdog/0 Tainted: GF I 3.11.0-15-generic #25~precise1-Ubuntu
[ 806.216266] Hardware name: Dell Inc. PowerEdge R710/0YDJK3, BIOS 6.3.0 07/24/2012
[ 806.239564] 000000000000025b ffff88062e6999e0 ffffffff8173bc5e 0000000000005ba4
[ 806.263834] 0000000000000000 ffff88062e699a20 ffffffff810653ac ffff88062e699a00
[ 806.288959] 7fffffffffffffff 000000000000e940 0000000000000000 ffffffff81ef9f40
[ 806.315146] Call Trace:
[ 806.327058] [<ffffffff8173bc5e>] dump_stack+0x46/0x58
[ 806.341762] [<ffffffff810653ac>] warn_slowpath_common+0x8c/0xc0
[ 806.357331] [<ffffffff810653fa>] warn_slowpath_null+0x1a/0x20
[ 806.372848] [<ffffffff810c46f6>] tick_handle_oneshot_broadcast+0x1c6/0x1f0
[ 806.389538] [<ffffffff81016525>] timer_interrupt+0x15/0x20
[ 806.404926] [<ffffffff810fcc6d>] handle_irq_event_percpu+0x5d/0x210
[ 806.421215] [<ffffffff810fce6e>] handle_irq_event+0x4e/0x80
[ 806.436719] [<ffffffff810ff814>] handle_edge_irq+0x84/0x130
[ 806.452143] [<ffffffff81015db2>] handle_irq+0x22/0x40
[ 806.467227] [<ffffffff81752c9a>] do_IRQ+0x5a/0xe0
[ 806.481613] [<ffffffff81747bad>] common_interrupt+0x6d/0x6d
[ 806.496936] [<ffffffff8106a4d8>] ? __do_softirq+0x98/0x280
[ 806.512077] [<ffffffff8106a49b>] ? __do_softirq+0x5b/0x280
[ 806.526758] [<ffffffff8174759e>] ? _raw_spin_lock+0xe/0x20
[ 806.540997] [<ffffffff8175239c>] call_softirq+0x1c/0x30
[ 806.554922] [<ffffffff81015e35>] do_softirq+0x65/0xa0
[ 806.568345] [<ffffffff8106a83e>] irq_exit+0x9e/0xc0
[ 806.581354] [<ffffffff81752ca3>] do_IRQ+0x63/0xe0
[ 806.594228] [<ffffffff81747bad>] common_interrupt+0x6d/0x6d
[ 806.607561] [<ffffffff81095bb8>] ? finish_task_switch+0x58/0xf0
[ 806.621005] [<ffffffff81095bb2>] ? finish_task_switch+0x52/0xf0
[ 806.634164] [<ffffffff8174534a>] __schedule+0x3da/0x730
[ 806.646959] [<ffffffff8101bbf3>] ? native_sched_clock+0x13/0x80
[ 806.660294] [<ffffffff81746179>] schedule+0x29/0x70
[ 806.671923] [<ffffffff81091585>] smpboot_thread_fn+0xc5/0x180
[ 806.684176] [<ffffffff810914c0>] ? lg_global_lock+0x70/0x70
[ 806.696195] [<ffffffff81088fe0>] kthread+0xc0/0xd0
[ 806.707145] [<ffffffff81088f20>] ? flush_kthread_worker+0xb0/0xb0
[ 806.719026] [<ffffffff817508ec>] ret_from_fork+0x7c/0xb0
[ 806.730278] [<ffffffff81088f20>] ? flush_kthread_worker+0xb0/0xb0
[ 806.742569] ---[ end trace aae4bf6a95ee82b0 ]---
Raw
3.5.0-45-generic
[ 1920.727350] BUG: soft lockup - CPU#0 stuck for 22s! [node:18932]
[ 1920.733970] Kernel panic - not syncing: softlockup: hung tasks
[ 1920.740103] Pid: 18932, comm: node Not tainted 3.5.0-45-generic #68~precise1-Ubuntu
[ 1920.748154] Call Trace:
[ 1920.750718] <IRQ> [<ffffffff81686452>] panic+0xc1/0x1d7
[ 1920.756438] [<ffffffff810e1bb7>] watchdog_timer_fn+0x177/0x180
[ 1920.762716] [<ffffffff8107c3d8>] __run_hrtimer+0x78/0x1f0
[ 1920.768487] [<ffffffff810e1a40>] ? __touch_watchdog+0x30/0x30
[ 1920.774629] [<ffffffff8105bccd>] ? __do_softirq+0xfd/0x210
[ 1920.780488] [<ffffffff8107cc67>] hrtimer_interrupt+0xf7/0x240
[ 1920.786622] [<ffffffff816a8cdc>] ? call_softirq+0x1c/0x30
[ 1920.792402] [<ffffffff816a9639>] smp_apic_timer_interrupt+0x69/0x99
[ 1920.799085] [<ffffffff816a838a>] apic_timer_interrupt+0x6a/0x70
[ 1920.805400] <EOI> [<ffffffff81040a50>] ? __ticket_spin_lock+0x20/0x30
[ 1920.812396] [<ffffffff81584799>] ? dev_queue_xmit+0x199/0x470
[ 1920.818529] [<ffffffff8169f09e>] _raw_spin_lock+0xe/0x20
[ 1920.824240] [<ffffffff811a11a0>] d_absolute_path+0x60/0xd0
[ 1920.827157] BUG: soft lockup - CPU#1 stuck for 22s! [docker:20564]
[ 1920.827181] CPU 1 Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1920.827202]
[ 1920.827204] Pid: 20564, comm: docker Not tainted 3.5.0-45-generic #68~precise1-Ubuntu Dell Inc. PowerEdge R710/0YDJK3
[ 1920.827207] RIP: 0010:[<ffffffff81040a52>] [<ffffffff81040a52>] __ticket_spin_lock+0x22/0x30
[ 1920.827214] RSP: 0018:ffff880325b99c08 EFLAGS: 00000283
[ 1920.827215] RAX: 000000000000ab71 RBX: 00000000000284d0 RCX: 0000000000000000
[ 1920.827216] RDX: 000000000000ab80 RSI: ffffffff8181b7d1 RDI: ffffffff81c043c4
[ 1920.827217] RBP: ffff880325b99c08 R08: ffff880325b99c50 R09: ffffffff8181b7d0
[ 1920.827218] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88033fffcd00
[ 1920.827218] R13: ffffea000c508080 R14: ffff880333c16e70 R15: ffff88033fffcd00
[ 1920.827220] FS: 00007fd2a27be700(0000) GS:ffff880333c00000(0000) knlGS:0000000000000000
[ 1920.827221] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1920.827222] CR2: 000000c21000f020 CR3: 0000000328394000 CR4: 00000000000007e0
[ 1920.827223] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1920.827224] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1920.827226] Process docker (pid: 20564, threadinfo ffff880325b98000, task ffff88032d5d2e00)
[ 1920.827226] Stack:
[ 1920.827227] ffff880325b99c18 ffffffff8169f09e ffff880325b99c68 ffffffff811a11a0
[ 1920.827231] ffff880325b99cb4 000000ff00000014 ffff880325b99d66 0000000000000000
[ 1920.827234] 0000000000000000 ffff8803270850ff ffff880325b99c98 ffff8803281b1b10
[ 1920.827238] Call Trace:
[ 1920.827239] [<ffffffff8169f09e>] _raw_spin_lock+0xe/0x20
[ 1920.827245] [<ffffffff811a11a0>] d_absolute_path+0x60/0xd0
[ 1920.827249] [<ffffffff812f077c>] d_namespace_path+0x4c/0x2b0
[ 1920.827253] [<ffffffff8169f09e>] ? _raw_spin_lock+0xe/0x20
[ 1920.827255] [<ffffffff8119fc7e>] ? dput+0x1ee/0x290
[ 1920.827258] [<ffffffff812f0a24>] get_name_to_buffer+0x44/0xe0
[ 1920.827260] [<ffffffff812f0b1f>] aa_path_name+0x5f/0xe0
[ 1920.827263] [<ffffffff812f1227>] apparmor_bprm_set_creds+0x107/0x780
[ 1920.827265] [<ffffffff8115515f>] ? vma_link+0xcf/0xe0
[ 1920.827269] [<ffffffff812bbda3>] security_bprm_set_creds+0x13/0x20
[ 1920.827273] [<ffffffff8118ec48>] prepare_binprm+0xb8/0x1e0
[ 1920.827276] [<ffffffff81190c63>] do_execve_common.isra.30+0x243/0x350
[ 1920.827279] [<ffffffff81190d8b>] do_execve+0x1b/0x20
[ 1920.827281] [<ffffffff8101d6c7>] sys_execve+0x47/0x70
[ 1920.827285] [<ffffffff816a7d2c>] stub_execve+0x6c/0xc0
[ 1920.827289] Code: 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89
[ 1920.843126] BUG: soft lockup - CPU#2 stuck for 22s! [gunicorn:18657]
[ 1920.843127] Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1920.843150] CPU 2 Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1920.843171]
[ 1920.843173] Pid: 18657, comm: gunicorn Not tainted 3.5.0-45-generic #68~precise1-Ubuntu Dell Inc. PowerEdge R710/0YDJK3
[ 1920.843176] RIP: 0010:[<ffffffff81040a55>] [<ffffffff81040a55>] __ticket_spin_lock+0x25/0x30
[ 1920.843180] RSP: 0018:ffff88062e211bf8 EFLAGS: 00000293
[ 1920.843181] RAX: 000000000000ab71 RBX: ffffffff811a11de RCX: 0000000000000000
[ 1920.843182] RDX: 000000000000ab7d RSI: ffffffff8181b7d1 RDI: ffffffff81c043c4
[ 1920.843183] RBP: ffff88062e211bf8 R08: ffff88062e211c40 R09: ffffffff8181b7d0
[ 1920.843183] R10: ffff88062a5abc00 R11: ffff88062a71af58 R12: ffffffff8108086a
[ 1920.843184] R13: ffff88062e211b68 R14: 0000000000010004 R15: ffff88062e211ca0
[ 1920.843186] FS: 00007fcaf2a80700(0000) GS:ffff88063fc20000(0000) knlGS:0000000000000000
[ 1920.843187] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1920.843188] CR2: 000000c2102ab000 CR3: 0000000612149000 CR4: 00000000000007e0
[ 1920.843189] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1920.843190] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1920.843192] Process gunicorn (pid: 18657, threadinfo ffff88062e210000, task ffff88062a14c500)
[ 1920.843192] Stack:
[ 1920.843193] ffff88062e211c08 ffffffff8169f09e ffff88062e211c58 ffffffff811a11a0
[ 1920.843197] ffff880606343400 000000ff00000100 ffff88062e211c68 0000000000000000
[ 1920.843200] 0000000000000000 ffff8806063420ff ffffffff812f6748 ffff88062e211e40
[ 1920.843203] Call Trace:
[ 1920.843204] [<ffffffff8169f09e>] _raw_spin_lock+0xe/0x20
[ 1920.843207] [<ffffffff811a11a0>] d_absolute_path+0x60/0xd0
[ 1920.843210] [<ffffffff812f6748>] ? compute_perms.isra.2+0x98/0x1e0
[ 1920.843213] [<ffffffff812f077c>] d_namespace_path+0x4c/0x2b0
[ 1920.843216] [<ffffffff812f0a24>] get_name_to_buffer+0x44/0xe0
[ 1920.843218] [<ffffffff812f0b1f>] aa_path_name+0x5f/0xe0
[ 1920.843221] [<ffffffff812f6b19>] aa_path_perm+0x89/0x180
[ 1920.843223] [<ffffffff812f582b>] common_perm+0x5b/0x70
[ 1920.843226] [<ffffffff812f5894>] apparmor_inode_getattr+0x54/0x60
[ 1920.843229] [<ffffffff812bc160>] security_inode_getattr+0x20/0x30
[ 1920.843232] [<ffffffff8118d82f>] vfs_getattr+0x2f/0x80
[ 1920.843235] [<ffffffff8118d964>] vfs_fstat+0x34/0x60
[ 1920.843237] [<ffffffff8118db2a>] sys_newfstat+0x1a/0x40
[ 1920.843239] [<ffffffff8119c9e2>] ? sys_select+0xd2/0x100
[ 1920.843242] [<ffffffff816a78e9>] system_call_fastpath+0x16/0x1b
[ 1920.843245] Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1
[ 1920.859096] BUG: soft lockup - CPU#3 stuck for 22s! [node:18940]
[ 1920.859096] Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1920.859119] CPU 3 Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1920.859140]
[ 1920.859142] Pid: 18940, comm: node Not tainted 3.5.0-45-generic #68~precise1-Ubuntu Dell Inc. PowerEdge R710/0YDJK3
[ 1920.859145] RIP: 0010:[<ffffffff81040a55>] [<ffffffff81040a55>] __ticket_spin_lock+0x25/0x30
[ 1920.859149] RSP: 0018:ffff8806063059f8 EFLAGS: 00000297
[ 1920.859150] RAX: 000000000000ab71 RBX: 0000000000000000 RCX: 0000000000000000
[ 1920.859151] RDX: 000000000000ab7b RSI: ffffffff8181b7d1 RDI: ffffffff81c043c4
[ 1920.859151] RBP: ffff8806063059f8 R08: ffff880606305a40 R09: ffffffff8181b7d0
[ 1920.859152] R10: 0000000000000000 R11: 9e8a8bff8b9e8b8c R12: 0000000000000008
[ 1920.859153] R13: 0000000000000000 R14: ffffffff8149e015 R15: ffff880606305978
[ 1920.859155] FS: 00007f975f275740(0000) GS:ffff880333c20000(0000) knlGS:0000000000000000
[ 1920.859156] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1920.859157] CR2: 00007fd2a40c2718 CR3: 00000003089d3000 CR4: 00000000000007e0
[ 1920.859158] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1920.859159] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1920.859160] Process node (pid: 18940, threadinfo ffff880606304000, task ffff88062d889700)
[ 1920.859161] Stack:
[ 1920.859161] ffff880606305a08 ffffffff8169f09e ffff880606305a58 ffffffff811a11a0
[ 1920.859165] ffffffffffffffff 000000ff2f23c100 ffff880606305a68 0000000000000000
[ 1920.859168] 0000000000000000 ffff880320e6d9ff ffff880606305a68 ffff880320e6c010
[ 1920.859171] Call Trace:
[ 1920.859172] [<ffffffff8169f09e>] _raw_spin_lock+0xe/0x20
[ 1920.859176] [<ffffffff811a11a0>] d_absolute_path+0x60/0xd0
[ 1920.859178] [<ffffffff812f077c>] d_namespace_path+0x4c/0x2b0
[ 1920.859181] [<ffffffff810911c2>] ? update_sd_lb_stats+0x3e2/0x5f0
[ 1920.859185] [<ffffffff812f0a24>] get_name_to_buffer+0x44/0xe0
[ 1920.859188] [<ffffffff812f0b1f>] aa_path_name+0x5f/0xe0
[ 1920.859190] [<ffffffff81574a0b>] ? skb_release_data.part.47+0xcb/0x110
[ 1920.859194] [<ffffffff812f6b19>] aa_path_perm+0x89/0x180
[ 1920.859196] [<ffffffff812bb6ec>] ? security_inode_permission+0x1c/0x30
[ 1920.859200] [<ffffffff812f61a3>] apparmor_file_open+0x103/0x170
[ 1920.859203] [<ffffffff812bc555>] security_file_open+0x25/0x80
[ 1920.859206] [<ffffffff811868b7>] do_dentry_open+0x107/0x2b0
[ 1920.859210] [<ffffffff812bb6ec>] ? security_inode_permission+0x1c/0x30
[ 1920.859212] [<ffffffff81188168>] nameidata_to_filp+0xa8/0xf0
[ 1920.859215] [<ffffffff81194435>] ? may_open+0x75/0x120
[ 1920.859218] [<ffffffff81195bcc>] do_last+0x4bc/0x850
[ 1920.859221] [<ffffffff81198296>] path_openat+0xd6/0x440
[ 1920.859224] [<ffffffff81198722>] do_filp_open+0x42/0xa0
[ 1920.859226] [<ffffffff8169f09e>] ? _raw_spin_lock+0xe/0x20
[ 1920.859229] [<ffffffff811a5e61>] ? alloc_fd+0xd1/0x120
[ 1920.859232] [<ffffffff811882ac>] do_sys_open+0xfc/0x240
[ 1920.859235] [<ffffffff81188411>] sys_open+0x21/0x30
[ 1920.859237] [<ffffffff816a78e9>] system_call_fastpath+0x16/0x1b
[ 1920.859241] Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1
[ 1920.879057] BUG: soft lockup - CPU#4 stuck for 22s! [docker:20109]
[ 1920.879058] Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1920.879081] CPU 4 Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1920.879103]
[ 1920.879105] Pid: 20109, comm: docker Not tainted 3.5.0-45-generic #68~precise1-Ubuntu Dell Inc. PowerEdge R710/0YDJK3
[ 1920.879107] RIP: 0010:[<ffffffff81040a52>] [<ffffffff81040a52>] __ticket_spin_lock+0x22/0x30
[ 1920.879111] RSP: 0018:ffff8806061ddd78 EFLAGS: 00000297
[ 1920.879112] RAX: 000000000000ab71 RBX: ffff88062ae3d00c RCX: 0000000000001000
[ 1920.879113] RDX: 000000000000ab72 RSI: ffff88062ae3d00e RDI: ffffffff81c043c4
[ 1920.879114] RBP: ffff8806061ddd78 R08: 000000000000000a R09: 000000000000fffe
[ 1920.879115] R10: 0000000000000000 R11: 0000000000000246 R12: 00000000a9a8a5ff
[ 1920.879116] R13: ffff88062e5e0000 R14: ffffffff8114e9b2 R15: ffff8806061ddd58
[ 1920.879117] FS: 00007fa0324d2700(0000) GS:ffff88063fc40000(0000) knlGS:0000000000000000
[ 1920.879118] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1920.879119] CR2: 000000c210347000 CR3: 000000062dab1000 CR4: 00000000000007e0
[ 1920.879120] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1920.879121] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1920.879123] Process docker (pid: 20109, threadinfo ffff8806061dc000, task ffff8806122e1700)
[ 1920.879123] Stack:
[ 1920.879124] ffff8806061ddd88 ffffffff8169f09e ffff8806061dddd8 ffffffff811a1416
[ 1920.879127] ffff8806061dde08 00000ff2811aace8 0000000000000008 0000000000000000
[ 1920.879131] ffff8806061dde18 ffff880615e45d40 ffff88062ae3d00e ffffffff81a2ccaa
[ 1920.879134] Call Trace:
[ 1920.879135] [<ffffffff8169f09e>] _raw_spin_lock+0xe/0x20
[ 1920.879138] [<ffffffff811a1416>] dentry_path+0x36/0xe0
[ 1920.879141] [<ffffffff811ab8c1>] seq_dentry+0x51/0xd0
[ 1920.879145] [<ffffffff811c6ce2>] show_mountinfo+0xd2/0x2c0
[ 1920.879150] [<ffffffff811a64c6>] m_show+0x16/0x20
[ 1920.879153] [<ffffffff811ab43b>] seq_read+0x16b/0x400
[ 1920.879155] [<ffffffff81189120>] vfs_read+0xb0/0x180
[ 1920.879159] [<ffffffff8118923a>] sys_read+0x4a/0x90
[ 1920.879161] [<ffffffff816a78e9>] system_call_fastpath+0x16/0x1b
[ 1920.879164] Code: 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89
[ 1920.895026] BUG: soft lockup - CPU#5 stuck for 22s! [node:18935]
[ 1920.895027] Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1920.895050] CPU 5 Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1920.895071]
[ 1920.895073] Pid: 18935, comm: node Not tainted 3.5.0-45-generic #68~precise1-Ubuntu Dell Inc. PowerEdge R710/0YDJK3
[ 1920.895075] RIP: 0010:[<ffffffff81040a52>] [<ffffffff81040a52>] __ticket_spin_lock+0x22/0x30
[ 1920.895079] RSP: 0018:ffff88062bb359f8 EFLAGS: 00000297
[ 1920.895080] RAX: 000000000000ab71 RBX: 0000000000000000 RCX: 0000000000000000
[ 1920.895081] RDX: 000000000000ab7c RSI: ffffffff8181b7d1 RDI: ffffffff81c043c4
[ 1920.895082] RBP: ffff88062bb359f8 R08: ffff88062bb35a40 R09: ffffffff8181b7d0
[ 1920.895083] R10: 0000000000000000 R11: ffff90ff8b9e8b8c R12: 0000000000000008
[ 1920.895084] R13: 0000000000000000 R14: ffffffff8149e015 R15: ffff88062bb35978
[ 1920.895085] FS: 00007fa035cd2740(0000) GS:ffff880333c40000(0000) knlGS:0000000000000000
[ 1920.895086] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1920.895087] CR2: 000000c21000f020 CR3: 0000000312480000 CR4: 00000000000007e0
[ 1920.895088] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1920.895089] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1920.895091] Process node (pid: 18935, threadinfo ffff88062bb34000, task ffff88062d88ae00)
[ 1920.895091] Stack:
[ 1920.895092] ffff88062bb35a08 ffffffff8169f09e ffff88062bb35a58 ffffffff811a11a0
[ 1920.895095] ffffffffffffffff 000000ff2f23cf40 ffff88062bb35a68 0000000000000000
[ 1920.895098] 0000000000000000 ffff8803126f22ff ffff88062bb35a68 ffff8803126f2610
[ 1920.895102] Call Trace:
[ 1920.895103] [<ffffffff8169f09e>] _raw_spin_lock+0xe/0x20
[ 1920.895106] [<ffffffff811a11a0>] d_absolute_path+0x60/0xd0
[ 1920.895109] [<ffffffff812f077c>] d_namespace_path+0x4c/0x2b0
[ 1920.895111] [<ffffffff810911c2>] ? update_sd_lb_stats+0x3e2/0x5f0
[ 1920.895115] [<ffffffff812f0a24>] get_name_to_buffer+0x44/0xe0
[ 1920.895117] [<ffffffff812f0b1f>] aa_path_name+0x5f/0xe0
[ 1920.895119] [<ffffffff81574a0b>] ? skb_release_data.part.47+0xcb/0x110
[ 1920.895122] [<ffffffff812f6b19>] aa_path_perm+0x89/0x180
[ 1920.895124] [<ffffffff812bb6ec>] ? security_inode_permission+0x1c/0x30
[ 1920.895127] [<ffffffff812f61a3>] apparmor_file_open+0x103/0x170
[ 1920.895130] [<ffffffff812bc555>] security_file_open+0x25/0x80
[ 1920.895133] [<ffffffff811868b7>] do_dentry_open+0x107/0x2b0
[ 1920.895135] [<ffffffff812bb6ec>] ? security_inode_permission+0x1c/0x30
[ 1920.895138] [<ffffffff81188168>] nameidata_to_filp+0xa8/0xf0
[ 1920.895140] [<ffffffff81194435>] ? may_open+0x75/0x120
[ 1920.895143] [<ffffffff81195bcc>] do_last+0x4bc/0x850
[ 1920.895145] [<ffffffff81198296>] path_openat+0xd6/0x440
[ 1920.895148] [<ffffffff81198722>] do_filp_open+0x42/0xa0
[ 1920.895151] [<ffffffff8169f09e>] ? _raw_spin_lock+0xe/0x20
[ 1920.895153] [<ffffffff811a5e61>] ? alloc_fd+0xd1/0x120
[ 1920.895156] [<ffffffff811882ac>] do_sys_open+0xfc/0x240
[ 1920.895159] [<ffffffff81188411>] sys_open+0x21/0x30
[ 1920.895161] [<ffffffff816a78e9>] system_call_fastpath+0x16/0x1b
[ 1920.895164] Code: 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89
[ 1920.914988] BUG: soft lockup - CPU#6 stuck for 22s! [redis-server:18908]
[ 1920.914989] Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1920.915012] CPU 6 Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1920.915033]
[ 1920.915035] Pid: 18908, comm: redis-server Not tainted 3.5.0-45-generic #68~precise1-Ubuntu Dell Inc. PowerEdge R710/0YDJK3
[ 1920.915037] RIP: 0010:[<ffffffff81040a55>] [<ffffffff81040a55>] __ticket_spin_lock+0x25/0x30
[ 1920.915041] RSP: 0018:ffff8806063c3be8 EFLAGS: 00000293
[ 1920.915042] RAX: 000000000000ab71 RBX: ffff8803262e5800 RCX: 0000000000000000
[ 1920.915043] RDX: 000000000000ab73 RSI: ffffffff8181b7d1 RDI: ffffffff81c043c4
[ 1920.915044] RBP: ffff8806063c3be8 R08: ffff8806063c3c30 R09: ffffffff8181b7d0
[ 1920.915045] R10: ffff88062a5abc00 R11: 8b9a8c8a8faaff9a R12: ffffffffa01ff2b9
[ 1920.915046] R13: ffff8806063c3b68 R14: ffffffff812bb6ec R15: ffff8806063c3b48
[ 1920.915048] FS: 00007faeb5f9b700(0000) GS:ffff88063fc60000(0000) knlGS:0000000000000000
[ 1920.915049] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1920.915050] CR2: 00007fda8376b000 CR3: 000000060630d000 CR4: 00000000000007e0
[ 1920.915051] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1920.915052] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1920.915053] Process redis-server (pid: 18908, threadinfo ffff8806063c2000, task ffff88062a03c500)
[ 1920.915054] Stack:
[ 1920.915054] ffff8806063c3bf8 ffffffff8169f09e ffff8806063c3c48 ffffffff811a11a0
[ 1920.915058] ffff8806063c3c28 000000ffa01e78f8 ffff8806063c3dc8 0000000000000000
[ 1920.915061] 0000000000000000 ffff88031dc51bff ffff880625eaf005 ffff8806063c3e30
[ 1920.915064] Call Trace:
[ 1920.915065] [<ffffffff8169f09e>] _raw_spin_lock+0xe/0x20
[ 1920.915069] [<ffffffff811a11a0>] d_absolute_path+0x60/0xd0
[ 1920.915071] [<ffffffff812f077c>] d_namespace_path+0x4c/0x2b0
[ 1920.915074] [<ffffffff8108086a>] ? lg_local_unlock+0x1a/0x20
[ 1920.915078] [<ffffffff811a7b76>] ? mntput_no_expire+0x46/0x160
[ 1920.915081] [<ffffffff812f0a24>] get_name_to_buffer+0x44/0xe0
[ 1920.915084] [<ffffffff812f0b1f>] aa_path_name+0x5f/0xe0
[ 1920.915086] [<ffffffff81040ae9>] ? default_spin_lock_flags+0x9/0x10
[ 1920.915089] [<ffffffff812f6b19>] aa_path_perm+0x89/0x180
[ 1920.915091] [<ffffffff812f582b>] common_perm+0x5b/0x70
[ 1920.915094] [<ffffffff812f5894>] apparmor_inode_getattr+0x54/0x60
[ 1920.915097] [<ffffffff812bc160>] security_inode_getattr+0x20/0x30
[ 1920.915099] [<ffffffff8118d82f>] vfs_getattr+0x2f/0x80
[ 1920.915102] [<ffffffff8118d8ce>] vfs_fstatat+0x4e/0x70
[ 1920.915104] [<ffffffff8118d92b>] vfs_stat+0x1b/0x20
[ 1920.915106] [<ffffffff8118da6a>] sys_newstat+0x1a/0x40
[ 1920.915108] [<ffffffff816a022e>] ? do_device_not_available+0xe/0x10
[ 1920.915112] [<ffffffff816a8a7b>] ? device_not_available+0x1b/0x20
[ 1920.915115] [<ffffffff816a78e9>] system_call_fastpath+0x16/0x1b
[ 1920.915118] Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1
[ 1920.930957] BUG: soft lockup - CPU#7 stuck for 23s! [node:18934]
[ 1920.930958] Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1920.930981] CPU 7 Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1920.931002]
[ 1920.931004] Pid: 18934, comm: node Not tainted 3.5.0-45-generic #68~precise1-Ubuntu Dell Inc. PowerEdge R710/0YDJK3
[ 1920.931007] RIP: 0010:[<ffffffff81040a52>] [<ffffffff81040a52>] __ticket_spin_lock+0x22/0x30
[ 1920.931011] RSP: 0018:ffff88061211b9f8 EFLAGS: 00000297
[ 1920.931012] RAX: 000000000000ab71 RBX: 0000000000000000 RCX: 0000000000000000
[ 1920.931013] RDX: 000000000000ab77 RSI: ffffffff8181b7d1 RDI: ffffffff81c043c4
[ 1920.931014] RBP: ffff88061211b9f8 R08: ffff88061211ba40 R09: ffffffff8181b7d0
[ 1920.931014] R10: 0000000000000000 R11: 8d9a8aff8b9e8b8c R12: 0000000000000008
[ 1920.931015] R13: 0000000000000000 R14: ffffffff8149e015 R15: ffff88061211b978
[ 1920.931017] FS: 00007f17aa2ad740(0000) GS:ffff880333c60000(0000) knlGS:0000000000000000
[ 1920.931018] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1920.931019] CR2: 00007f98b214f3ec CR3: 00000003143f0000 CR4: 00000000000007e0
[ 1920.931020] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1920.931021] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1920.931022] Process node (pid: 18934, threadinfo ffff88061211a000, task ffff88062d888000)
[ 1920.931023] Stack:
[ 1920.931023] ffff88061211ba08 ffffffff8169f09e ffff88061211ba58 ffffffff811a11a0
[ 1920.931027] ffffffffffffffff 000000ff2f23cec0 ffff88061211ba68 0000000000000000
[ 1920.931030] 0000000000000000 ffff88032e31b3ff ffff88061211ba68 ffff88032e31a110
[ 1920.931033] Call Trace:
[ 1920.931034] [<ffffffff8169f09e>] _raw_spin_lock+0xe/0x20
[ 1920.931038] [<ffffffff811a11a0>] d_absolute_path+0x60/0xd0
[ 1920.931040] [<ffffffff812f077c>] d_namespace_path+0x4c/0x2b0
[ 1920.931043] [<ffffffff810911c2>] ? update_sd_lb_stats+0x3e2/0x5f0
[ 1920.931046] [<ffffffff812f0a24>] get_name_to_buffer+0x44/0xe0
[ 1920.931049] [<ffffffff812f0b1f>] aa_path_name+0x5f/0xe0
[ 1920.931051] [<ffffffff81574a0b>] ? skb_release_data.part.47+0xcb/0x110
[ 1920.931054] [<ffffffff812f6b19>] aa_path_perm+0x89/0x180
[ 1920.931056] [<ffffffff812bb6ec>] ? security_inode_permission+0x1c/0x30
[ 1920.931059] [<ffffffff812f61a3>] apparmor_file_open+0x103/0x170
[ 1920.931062] [<ffffffff812bc555>] security_file_open+0x25/0x80
[ 1920.931064] [<ffffffff811868b7>] do_dentry_open+0x107/0x2b0
[ 1920.931067] [<ffffffff812bb6ec>] ? security_inode_permission+0x1c/0x30
[ 1920.931069] [<ffffffff81188168>] nameidata_to_filp+0xa8/0xf0
[ 1920.931072] [<ffffffff81194435>] ? may_open+0x75/0x120
[ 1920.931074] [<ffffffff81195bcc>] do_last+0x4bc/0x850
[ 1920.931077] [<ffffffff81198296>] path_openat+0xd6/0x440
[ 1920.931080] [<ffffffff81198722>] do_filp_open+0x42/0xa0
[ 1920.931083] [<ffffffff8169f09e>] ? _raw_spin_lock+0xe/0x20
[ 1920.931085] [<ffffffff811a5e61>] ? alloc_fd+0xd1/0x120
[ 1920.931088] [<ffffffff811882ac>] do_sys_open+0xfc/0x240
[ 1920.931090] [<ffffffff81188411>] sys_open+0x21/0x30
[ 1920.931093] [<ffffffff816a78e9>] system_call_fastpath+0x16/0x1b
[ 1920.931095] Code: 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89
[ 1920.950918] BUG: soft lockup - CPU#8 stuck for 23s! [node:18941]
[ 1920.950919] Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1920.950939] CPU 8 Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1920.950958]
[ 1920.950960] Pid: 18941, comm: node Not tainted 3.5.0-45-generic #68~precise1-Ubuntu Dell Inc. PowerEdge R710/0YDJK3
[ 1920.950962] RIP: 0010:[<ffffffff81040a52>] [<ffffffff81040a52>] __ticket_spin_lock+0x22/0x30
[ 1920.950966] RSP: 0018:ffff8806061979f8 EFLAGS: 00000297
[ 1920.950967] RAX: 000000000000ab71 RBX: 0000000000000000 RCX: 0000000000000000
[ 1920.950968] RDX: 000000000000ab7e RSI: ffffffff8181b7d1 RDI: ffffffff81c043c4
[ 1920.950969] RBP: ffff8806061979f8 R08: ffff880606197a40 R09: ffffffff8181b7d0
[ 1920.950970] R10: 0000000000000000 R11: ffffffff8b9e8b8c R12: 0000000000000008
[ 1920.950970] R13: 0000000000000000 R14: ffffffff8149e015 R15: ffff880606197978
[ 1920.950972] FS: 00007f580c152740(0000) GS:ffff88063fc80000(0000) knlGS:0000000000000000
[ 1920.950973] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1920.950973] CR2: 00007fda83549220 CR3: 00000003283be000 CR4: 00000000000007e0
[ 1920.950974] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1920.950975] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1920.950976] Process node (pid: 18941, threadinfo ffff880606196000, task ffff88062e368000)
[ 1920.950977] Stack:
[ 1920.950977] ffff880606197a08 ffffffff8169f09e ffff880606197a58 ffffffff811a11a0
[ 1920.950980] ffff88032325b000 000000ff0000000e ffff88032325b0e8 0000000000000000
[ 1920.950983] 0000000000000000 ffff88062afd35ff ffff880606197a68 ffff88062afd3f10
[ 1920.950986] Call Trace:
[ 1920.950987] [<ffffffff8169f09e>] _raw_spin_lock+0xe/0x20
[ 1920.950989] [<ffffffff811a11a0>] d_absolute_path+0x60/0xd0
[ 1920.950992] [<ffffffff812f077c>] d_namespace_path+0x4c/0x2b0
[ 1920.950994] [<ffffffff810911c2>] ? update_sd_lb_stats+0x3e2/0x5f0
[ 1920.950997] [<ffffffff812f0a24>] get_name_to_buffer+0x44/0xe0
[ 1920.950999] [<ffffffff812f0b1f>] aa_path_name+0x5f/0xe0
[ 1920.951001] [<ffffffff81574a0b>] ? skb_release_data.part.47+0xcb/0x110
[ 1920.951004] [<ffffffff812f6b19>] aa_path_perm+0x89/0x180
[ 1920.951006] [<ffffffff812bb6ec>] ? security_inode_permission+0x1c/0x30
[ 1920.951009] [<ffffffff812f61a3>] apparmor_file_open+0x103/0x170
[ 1920.951012] [<ffffffff812bc555>] security_file_open+0x25/0x80
[ 1920.951014] [<ffffffff811868b7>] do_dentry_open+0x107/0x2b0
[ 1920.951017] [<ffffffff812bb6ec>] ? security_inode_permission+0x1c/0x30
[ 1920.951019] [<ffffffff81188168>] nameidata_to_filp+0xa8/0xf0
[ 1920.951021] [<ffffffff81194435>] ? may_open+0x75/0x120
[ 1920.951024] [<ffffffff81195bcc>] do_last+0x4bc/0x850
[ 1920.951027] [<ffffffff81198296>] path_openat+0xd6/0x440
[ 1920.951030] [<ffffffff81198722>] do_filp_open+0x42/0xa0
[ 1920.951033] [<ffffffff8169f09e>] ? _raw_spin_lock+0xe/0x20
[ 1920.951035] [<ffffffff811a5e61>] ? alloc_fd+0xd1/0x120
[ 1920.951038] [<ffffffff811882ac>] do_sys_open+0xfc/0x240
[ 1920.951040] [<ffffffff81188411>] sys_open+0x21/0x30
[ 1920.951042] [<ffffffff816a78e9>] system_call_fastpath+0x16/0x1b
[ 1920.951045] Code: 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89
[ 1920.966888] BUG: soft lockup - CPU#9 stuck for 23s! [docker:20560]
[ 1920.966889] Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1920.966911] CPU 9 Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1920.966933]
[ 1920.966935] Pid: 20560, comm: docker Not tainted 3.5.0-45-generic #68~precise1-Ubuntu Dell Inc. PowerEdge R710/0YDJK3
[ 1920.966937] RIP: 0010:[<ffffffff81040a52>] [<ffffffff81040a52>] __ticket_spin_lock+0x22/0x30
[ 1920.966941] RSP: 0018:ffff8803141e9c08 EFLAGS: 00000293
[ 1920.966942] RAX: 000000000000ab71 RBX: 00000000000284d0 RCX: 0000000000000000
[ 1920.966943] RDX: 000000000000ab76 RSI: ffffffff8181b7d1 RDI: ffffffff81c043c4
[ 1920.966944] RBP: ffff8803141e9c08 R08: ffff8803141e9c50 R09: ffffffff8181b7d0
[ 1920.966945] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88033fffcd00
[ 1920.966946] R13: ffffea000c8394c0 R14: ffff880333c96e70 R15: ffff88033fffcd00
[ 1920.966947] FS: 00007fd2a1fbd700(0000) GS:ffff880333c80000(0000) knlGS:0000000000000000
[ 1920.966948] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1920.966949] CR2: 00007f5fbc213420 CR3: 000000031dc4e000 CR4: 00000000000007e0
[ 1920.966950] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1920.966951] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1920.966952] Process docker (pid: 20560, threadinfo ffff8803141e8000, task ffff8803262e8000)
[ 1920.966953] Stack:
[ 1920.966953] ffff8803141e9c18 ffffffff8169f09e ffff8803141e9c68 ffffffff811a11a0
[ 1920.966957] ffff8803141e9cb4 000000ff00000014 ffff8803141e9d66 0000000000000000
[ 1920.966960] 0000000000000000 ffff88031dc48dff ffff8803141e9c98 ffff8803233d3710
[ 1920.966964] Call Trace:
[ 1920.966965] [<ffffffff8169f09e>] _raw_spin_lock+0xe/0x20
[ 1920.966967] [<ffffffff811a11a0>] d_absolute_path+0x60/0xd0
[ 1920.966970] [<ffffffff812f077c>] d_namespace_path+0x4c/0x2b0
[ 1920.966972] [<ffffffff8169f09e>] ? _raw_spin_lock+0xe/0x20
[ 1920.966975] [<ffffffff8119fc7e>] ? dput+0x1ee/0x290
[ 1920.966977] [<ffffffff812f0a24>] get_name_to_buffer+0x44/0xe0
[ 1920.966980] [<ffffffff812f0b1f>] aa_path_name+0x5f/0xe0
[ 1920.966982] [<ffffffff812f1227>] apparmor_bprm_set_creds+0x107/0x780
[ 1920.966985] [<ffffffff8115515f>] ? vma_link+0xcf/0xe0
[ 1920.966987] [<ffffffff812bbda3>] security_bprm_set_creds+0x13/0x20
[ 1920.966990] [<ffffffff8118ec48>] prepare_binprm+0xb8/0x1e0
[ 1920.966992] [<ffffffff81190c63>] do_execve_common.isra.30+0x243/0x350
[ 1920.966994] [<ffffffff81190d8b>] do_execve+0x1b/0x20
[ 1920.966997] [<ffffffff8101d6c7>] sys_execve+0x47/0x70
[ 1920.966999] [<ffffffff816a7d2c>] stub_execve+0x6c/0xc0
[ 1920.967002] Code: 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89
[ 1920.986849] BUG: soft lockup - CPU#10 stuck for 23s! [node:18931]
[ 1920.986850] Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1920.986873] CPU 10 Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1920.986895]
[ 1920.986897] Pid: 18931, comm: node Not tainted 3.5.0-45-generic #68~precise1-Ubuntu Dell Inc. PowerEdge R710/0YDJK3
[ 1920.986899] RIP: 0010:[<ffffffff81040a55>] [<ffffffff81040a55>] __ticket_spin_lock+0x25/0x30
[ 1920.986903] RSP: 0018:ffff88062a3319f8 EFLAGS: 00000293
[ 1920.986904] RAX: 000000000000ab71 RBX: 0000000000000000 RCX: 0000000000000000
[ 1920.986905] RDX: 000000000000ab74 RSI: ffffffff8181b7d1 RDI: ffffffff81c043c4
[ 1920.986906] RBP: ffff88062a3319f8 R08: ffff88062a331a40 R09: ffffffff8181b7d0
[ 1920.986906] R10: 0000000000000000 R11: 251ebaff8b9e8b8c R12: 0000000000000008
[ 1920.986907] R13: 0000000000000000 R14: ffffffff8149e015 R15: ffff88062a331978
[ 1920.986909] FS: 00007fcb42cd4740(0000) GS:ffff88063fca0000(0000) knlGS:0000000000000000
[ 1920.986910] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1920.986911] CR2: 00007f74b6692420 CR3: 00000003126cb000 CR4: 00000000000007e0
[ 1920.986912] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1920.986913] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1920.986915] Process node (pid: 18931, threadinfo ffff88062a330000, task ffff88062a3dae00)
[ 1920.986915] Stack:
[ 1920.986916] ffff88062a331a08 ffffffff8169f09e ffff88062a331a58 ffffffff811a11a0
[ 1920.986919] ffff88032325b000 000000ff0000000e ffff88032325b0e8 0000000000000000
[ 1920.986923] 0000000000000000 ffff8803125c70ff ffff88062a331a68 ffff8803125c6510
[ 1920.986926] Call Trace:
[ 1920.986927] [<ffffffff8169f09e>] _raw_spin_lock+0xe/0x20
[ 1920.986930] [<ffffffff811a11a0>] d_absolute_path+0x60/0xd0
[ 1920.986932] [<ffffffff812f077c>] d_namespace_path+0x4c/0x2b0
[ 1920.986935] [<ffffffff810911c2>] ? update_sd_lb_stats+0x3e2/0x5f0
[ 1920.986938] [<ffffffff812f0a24>] get_name_to_buffer+0x44/0xe0
[ 1920.986941] [<ffffffff812f0b1f>] aa_path_name+0x5f/0xe0
[ 1920.986943] [<ffffffff81574a0b>] ? skb_release_data.part.47+0xcb/0x110
[ 1920.986946] [<ffffffff812f6b19>] aa_path_perm+0x89/0x180
[ 1920.986948] [<ffffffff812bb6ec>] ? security_inode_permission+0x1c/0x30
[ 1920.986951] [<ffffffff812f61a3>] apparmor_file_open+0x103/0x170
[ 1920.986954] [<ffffffff812bc555>] security_file_open+0x25/0x80
[ 1920.986957] [<ffffffff811868b7>] do_dentry_open+0x107/0x2b0
[ 1920.986959] [<ffffffff812bb6ec>] ? security_inode_permission+0x1c/0x30
[ 1920.986962] [<ffffffff81188168>] nameidata_to_filp+0xa8/0xf0
[ 1920.986964] [<ffffffff81194435>] ? may_open+0x75/0x120
[ 1920.986967] [<ffffffff81195bcc>] do_last+0x4bc/0x850
[ 1920.986970] [<ffffffff81198296>] path_openat+0xd6/0x440
[ 1920.986973] [<ffffffff81198722>] do_filp_open+0x42/0xa0
[ 1920.986976] [<ffffffff8169f09e>] ? _raw_spin_lock+0xe/0x20
[ 1920.986978] [<ffffffff811a5e61>] ? alloc_fd+0xd1/0x120
[ 1920.986981] [<ffffffff811882ac>] do_sys_open+0xfc/0x240
[ 1920.986984] [<ffffffff81188411>] sys_open+0x21/0x30
[ 1920.986986] [<ffffffff816a78e9>] system_call_fastpath+0x16/0x1b
[ 1920.986989] Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1
[ 1921.002819] BUG: soft lockup - CPU#11 stuck for 23s! [node:18947]
[ 1921.002820] Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1921.002842] CPU 11 Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1921.002864]
[ 1921.002865] Pid: 18947, comm: node Not tainted 3.5.0-45-generic #68~precise1-Ubuntu Dell Inc. PowerEdge R710/0YDJK3
[ 1921.002868] RIP: 0010:[<ffffffff81040a52>] [<ffffffff81040a52>] __ticket_spin_lock+0x22/0x30
[ 1921.002871] RSP: 0018:ffff880625ccf9f8 EFLAGS: 00000293
[ 1921.002872] RAX: 000000000000ab71 RBX: 0000000000000000 RCX: 0000000000000000
[ 1921.002873] RDX: 000000000000ab7f RSI: ffffffff8181b7d1 RDI: ffffffff81c043c4
[ 1921.002874] RBP: ffff880625ccf9f8 R08: ffff880625ccfa40 R09: ffffffff8181b7d0
[ 1921.002875] R10: 0000000000000000 R11: ffff90ff8b9e8b8c R12: 0000000000000008
[ 1921.002875] R13: 0000000000000000 R14: ffffffff8149e015 R15: ffff880625ccf978
[ 1921.002877] FS: 00007fcf30eee740(0000) GS:ffff880333ca0000(0000) knlGS:0000000000000000
[ 1921.002878] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1921.002879] CR2: 00000000006040e0 CR3: 0000000625d12000 CR4: 00000000000007e0
[ 1921.002880] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1921.002881] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1921.002882] Process node (pid: 18947, threadinfo ffff880625cce000, task ffff88062c522e00)
[ 1921.002883] Stack:
[ 1921.002883] ffff880625ccfa08 ffffffff8169f09e ffff880625ccfa58 ffffffff811a11a0
[ 1921.002887] ffffffffffffffff 000000ff2f23c180 ffff880625ccfa68 0000000000000000
[ 1921.002890] 0000000000000000 ffff88030e8dcbff ffff880625ccfa68 ffff88030e8dd710
[ 1921.002893] Call Trace:
[ 1921.002894] [<ffffffff8169f09e>] _raw_spin_lock+0xe/0x20
[ 1921.002897] [<ffffffff811a11a0>] d_absolute_path+0x60/0xd0
[ 1921.002900] [<ffffffff812f077c>] d_namespace_path+0x4c/0x2b0
[ 1921.002902] [<ffffffff810911c2>] ? update_sd_lb_stats+0x3e2/0x5f0
[ 1921.002905] [<ffffffff812f0a24>] get_name_to_buffer+0x44/0xe0
[ 1921.002908] [<ffffffff812f0b1f>] aa_path_name+0x5f/0xe0
[ 1921.002910] [<ffffffff81574a0b>] ? skb_release_data.part.47+0xcb/0x110
[ 1921.002913] [<ffffffff812f6b19>] aa_path_perm+0x89/0x180
[ 1921.002915] [<ffffffff812bb6ec>] ? security_inode_permission+0x1c/0x30
[ 1921.002918] [<ffffffff812f61a3>] apparmor_file_open+0x103/0x170
[ 1921.002920] [<ffffffff812bc555>] security_file_open+0x25/0x80
[ 1921.002923] [<ffffffff811868b7>] do_dentry_open+0x107/0x2b0
[ 1921.002925] [<ffffffff812bb6ec>] ? security_inode_permission+0x1c/0x30
[ 1921.002928] [<ffffffff81188168>] nameidata_to_filp+0xa8/0xf0
[ 1921.002930] [<ffffffff81194435>] ? may_open+0x75/0x120
[ 1921.002933] [<ffffffff81195bcc>] do_last+0x4bc/0x850
[ 1921.002935] [<ffffffff81198296>] path_openat+0xd6/0x440
[ 1921.002938] [<ffffffff81198722>] do_filp_open+0x42/0xa0
[ 1921.002941] [<ffffffff8169f09e>] ? _raw_spin_lock+0xe/0x20
[ 1921.002943] [<ffffffff811a5e61>] ? alloc_fd+0xd1/0x120
[ 1921.002946] [<ffffffff811882ac>] do_sys_open+0xfc/0x240
[ 1921.002949] [<ffffffff81188411>] sys_open+0x21/0x30
[ 1921.002951] [<ffffffff816a78e9>] system_call_fastpath+0x16/0x1b
[ 1921.002954] Code: 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89
[ 1921.022780] BUG: soft lockup - CPU#12 stuck for 23s! [docker:20567]
[ 1921.022781] Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1921.022804] CPU 12 Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1921.022825]
[ 1921.022827] Pid: 20567, comm: docker Not tainted 3.5.0-45-generic #68~precise1-Ubuntu Dell Inc. PowerEdge R710/0YDJK3
[ 1921.022829] RIP: 0010:[<ffffffff81040a52>] [<ffffffff81040a52>] __ticket_spin_lock+0x22/0x30
[ 1921.022832] RSP: 0018:ffff880625d95c08 EFLAGS: 00000287
[ 1921.022834] RAX: 000000000000ab71 RBX: 00000000000284d0 RCX: 0000000000000000
[ 1921.022835] RDX: 000000000000ab81 RSI: ffffffff8181b7d1 RDI: ffffffff81c043c4
[ 1921.022835] RBP: ffff880625d95c08 R08: ffff880625d95c50 R09: ffffffff8181b7d0
[ 1921.022836] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88063fffcd00
[ 1921.022837] R13: ffffea0018573e40 R14: ffff88063fcd6d38 R15: ffff88063fffcd00
[ 1921.022839] FS: 00007fa030c8f700(0000) GS:ffff88063fcc0000(0000) knlGS:0000000000000000
[ 1921.022840] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1921.022841] CR2: 00000000006040e0 CR3: 000000062a525000 CR4: 00000000000007e0
[ 1921.022842] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1921.022843] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1921.022844] Process docker (pid: 20567, threadinfo ffff880625d94000, task ffff88062acadc00)
[ 1921.022844] Stack:
[ 1921.022845] ffff880625d95c18 ffffffff8169f09e ffff880625d95c68 ffffffff811a11a0
[ 1921.022848] ffff880625d95cb4 000000ff00000014 ffff880625d95d66 0000000000000000
[ 1921.022852] 0000000000000000 ffff8803281b0eff ffff880625d95c98 ffff8803281b1110
[ 1921.022855] Call Trace:
[ 1921.022856] [<ffffffff8169f09e>] _raw_spin_lock+0xe/0x20
[ 1921.022858] [<ffffffff811a11a0>] d_absolute_path+0x60/0xd0
[ 1921.022861] [<ffffffff812f077c>] d_namespace_path+0x4c/0x2b0
[ 1921.022864] [<ffffffff8169f09e>] ? _raw_spin_lock+0xe/0x20
[ 1921.022866] [<ffffffff8119fc7e>] ? dput+0x1ee/0x290
[ 1921.022869] [<ffffffff812f0a24>] get_name_to_buffer+0x44/0xe0
[ 1921.022871] [<ffffffff812f0b1f>] aa_path_name+0x5f/0xe0
[ 1921.022874] [<ffffffff812f1227>] apparmor_bprm_set_creds+0x107/0x780
[ 1921.022876] [<ffffffff8115515f>] ? vma_link+0xcf/0xe0
[ 1921.022880] [<ffffffff812bbda3>] security_bprm_set_creds+0x13/0x20
[ 1921.022883] [<ffffffff8118ec48>] prepare_binprm+0xb8/0x1e0
[ 1921.022885] [<ffffffff81190c63>] do_execve_common.isra.30+0x243/0x350
[ 1921.022888] [<ffffffff81190d8b>] do_execve+0x1b/0x20
[ 1921.022890] [<ffffffff8101d6c7>] sys_execve+0x47/0x70
[ 1921.022894] [<ffffffff816a7d2c>] stub_execve+0x6c/0xc0
[ 1921.022897] Code: 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89
[ 1921.038750] BUG: soft lockup - CPU#13 stuck for 23s! [docker:20561]
[ 1921.038750] Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1921.038773] CPU 13 Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1921.038795]
[ 1921.038797] Pid: 20561, comm: docker Not tainted 3.5.0-45-generic #68~precise1-Ubuntu Dell Inc. PowerEdge R710/0YDJK3
[ 1921.038799] RIP: 0010:[<ffffffff81040a52>] [<ffffffff81040a52>] __ticket_spin_lock+0x22/0x30
[ 1921.038803] RSP: 0018:ffff880312687c08 EFLAGS: 00000297
[ 1921.038804] RAX: 000000000000ab71 RBX: 00000000000284d0 RCX: 0000000000000000
[ 1921.038805] RDX: 000000000000ab78 RSI: ffffffff8181b7d1 RDI: ffffffff81c043c4
[ 1921.038806] RBP: ffff880312687c08 R08: ffff880312687c50 R09: ffffffff8181b7d0
[ 1921.038807] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88033fffcd00
[ 1921.038808] R13: ffffea000c498e80 R14: ffff880333cd6e70 R15: ffff88033fffcd00
[ 1921.038809] FS: 0000000001f08880(0063) GS:ffff880333cc0000(0000) knlGS:0000000000000000
[ 1921.038810] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1921.038811] CR2: 00000000011fc050 CR3: 00000003088df000 CR4: 00000000000007e0
[ 1921.038812] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1921.038813] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1921.038814] Process docker (pid: 20561, threadinfo ffff880312686000, task ffff88032d5d5c00)
[ 1921.038815] Stack:
[ 1921.038815] ffff880312687c18 ffffffff8169f09e ffff880312687c68 ffffffff811a11a0
[ 1921.038819] ffff880312687cb4 000000ff00000014 ffff880312687d66 0000000000000000
[ 1921.038822] 0000000000000000 ffff880325badeff ffff880312687c98 ffff88030e8dc510
[ 1921.038826] Call Trace:
[ 1921.038827] [<ffffffff8169f09e>] _raw_spin_lock+0xe/0x20
[ 1921.038829] [<ffffffff811a11a0>] d_absolute_path+0x60/0xd0
[ 1921.038832] [<ffffffff812f077c>] d_namespace_path+0x4c/0x2b0
[ 1921.038834] [<ffffffff8169f09e>] ? _raw_spin_lock+0xe/0x20
[ 1921.038837] [<ffffffff8119fc7e>] ? dput+0x1ee/0x290
[ 1921.038839] [<ffffffff812f0a24>] get_name_to_buffer+0x44/0xe0
[ 1921.038842] [<ffffffff812f0b1f>] aa_path_name+0x5f/0xe0
[ 1921.038844] [<ffffffff812f1227>] apparmor_bprm_set_creds+0x107/0x780
[ 1921.038847] [<ffffffff8115515f>] ? vma_link+0xcf/0xe0
[ 1921.038850] [<ffffffff812bbda3>] security_bprm_set_creds+0x13/0x20
[ 1921.038852] [<ffffffff8118ec48>] prepare_binprm+0xb8/0x1e0
[ 1921.038854] [<ffffffff81190c63>] do_execve_common.isra.30+0x243/0x350
[ 1921.038857] [<ffffffff81190d8b>] do_execve+0x1b/0x20
[ 1921.038859] [<ffffffff8101d6c7>] sys_execve+0x47/0x70
[ 1921.038862] [<ffffffff816a7d2c>] stub_execve+0x6c/0xc0
[ 1921.038865] Code: 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89
[ 1921.058711] BUG: soft lockup - CPU#14 stuck for 23s! [node:18938]
[ 1921.058712] Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1921.058735] CPU 14 Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1921.058757]
[ 1921.058759] Pid: 18938, comm: node Not tainted 3.5.0-45-generic #68~precise1-Ubuntu Dell Inc. PowerEdge R710/0YDJK3
[ 1921.058762] RIP: 0010:[<ffffffff81040a55>] [<ffffffff81040a55>] __ticket_spin_lock+0x25/0x30
[ 1921.058765] RSP: 0018:ffff8806120c99f8 EFLAGS: 00000293
[ 1921.058766] RAX: 000000000000ab71 RBX: 0000000000000000 RCX: 0000000000000000
[ 1921.058767] RDX: 000000000000ab7a RSI: ffffffff8181b7d1 RDI: ffffffff81c043c4
[ 1921.058768] RBP: ffff8806120c99f8 R08: ffff8806120c9a40 R09: ffffffff8181b7d0
[ 1921.058769] R10: 0000000000000000 R11: 8b9e8bff8b9e8b8c R12: 0000000000000008
[ 1921.058769] R13: 0000000000000000 R14: ffffffff8149e015 R15: ffff8806120c9978
[ 1921.058771] FS: 00007f5a2ec40740(0000) GS:ffff88063fce0000(0000) knlGS:0000000000000000
[ 1921.058772] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1921.058773] CR2: 00007f86a642b420 CR3: 000000062bb47000 CR4: 00000000000007e0
[ 1921.058774] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1921.058775] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1921.058777] Process node (pid: 18938, threadinfo ffff8806120c8000, task ffff88062d88c500)
[ 1921.058777] Stack:
[ 1921.058778] ffff8806120c9a08 ffffffff8169f09e ffff8806120c9a58 ffffffff811a11a0
[ 1921.058781] ffffffffffffffff 000000ff2e702200 ffff8806120c9a68 0000000000000000
[ 1921.058785] 0000000000000000 ffff8803282423ff ffff8806120c9a68 ffff880328243b10
[ 1921.058788] Call Trace:
[ 1921.058789] [<ffffffff8169f09e>] _raw_spin_lock+0xe/0x20
[ 1921.058792] [<ffffffff811a11a0>] d_absolute_path+0x60/0xd0
[ 1921.058794] [<ffffffff812f077c>] d_namespace_path+0x4c/0x2b0
[ 1921.058797] [<ffffffff810911c2>] ? update_sd_lb_stats+0x3e2/0x5f0
[ 1921.058800] [<ffffffff812f0a24>] get_name_to_buffer+0x44/0xe0
[ 1921.058802] [<ffffffff812f0b1f>] aa_path_name+0x5f/0xe0
[ 1921.058805] [<ffffffff81574a0b>] ? skb_release_data.part.47+0xcb/0x110
[ 1921.058808] [<ffffffff812f6b19>] aa_path_perm+0x89/0x180
[ 1921.058810] [<ffffffff812bb6ec>] ? security_inode_permission+0x1c/0x30
[ 1921.058813] [<ffffffff812f61a3>] apparmor_file_open+0x103/0x170
[ 1921.058816] [<ffffffff812bc555>] security_file_open+0x25/0x80
[ 1921.058819] [<ffffffff811868b7>] do_dentry_open+0x107/0x2b0
[ 1921.058821] [<ffffffff812bb6ec>] ? security_inode_permission+0x1c/0x30
[ 1921.058824] [<ffffffff81188168>] nameidata_to_filp+0xa8/0xf0
[ 1921.058827] [<ffffffff81194435>] ? may_open+0x75/0x120
[ 1921.058829] [<ffffffff81195bcc>] do_last+0x4bc/0x850
[ 1921.058832] [<ffffffff81198296>] path_openat+0xd6/0x440
[ 1921.058835] [<ffffffff81198722>] do_filp_open+0x42/0xa0
[ 1921.058838] [<ffffffff8169f09e>] ? _raw_spin_lock+0xe/0x20
[ 1921.058840] [<ffffffff811a5e61>] ? alloc_fd+0xd1/0x120
[ 1921.058843] [<ffffffff811882ac>] do_sys_open+0xfc/0x240
[ 1921.058846] [<ffffffff81188411>] sys_open+0x21/0x30
[ 1921.058848] [<ffffffff816a78e9>] system_call_fastpath+0x16/0x1b
[ 1921.058851] Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1
[ 1921.074681] BUG: soft lockup - CPU#15 stuck for 23s! [docker:20562]
[ 1921.074681] Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1921.074704] CPU 15 Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1921.074726]
[ 1921.074728] Pid: 20562, comm: docker Not tainted 3.5.0-45-generic #68~precise1-Ubuntu Dell Inc. PowerEdge R710/0YDJK3
[ 1921.074730] RIP: 0010:[<ffffffff81040a50>] [<ffffffff81040a50>] __ticket_spin_lock+0x20/0x30
[ 1921.074733] RSP: 0018:ffff880325b8dc08 EFLAGS: 00000293
[ 1921.074734] RAX: 000000000000ab71 RBX: 00000000000284d0 RCX: 0000000000000000
[ 1921.074735] RDX: 000000000000ab79 RSI: ffffffff8181b7d1 RDI: ffffffff81c043c4
[ 1921.074736] RBP: ffff880325b8dc08 R08: ffff880325b8dc50 R09: ffffffff8181b7d0
[ 1921.074737] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88033fffcd00
[ 1921.074738] R13: ffffea000c491180 R14: ffff880333cf6e70 R15: ffff88033fffcd00
[ 1921.074740] FS: 00007fd28bfff700(0000) GS:ffff880333ce0000(0000) knlGS:0000000000000000
[ 1921.074741] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1921.074742] CR2: 00000000006040e0 CR3: 00000003088e3000 CR4: 00000000000007e0
[ 1921.074743] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1921.074744] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1921.074745] Process docker (pid: 20562, threadinfo ffff880325b8c000, task ffff88032d5d4500)
[ 1921.074745] Stack:
[ 1921.074746] ffff880325b8dc18 ffffffff8169f09e ffff880325b8dc68 ffffffff811a11a0
[ 1921.074750] ffff880325b8dcb4 000000ff00000014 ffff880325b8dd66 0000000000000000
[ 1921.074753] 0000000000000000 ffff8803233d2dff ffff880325b8dc98 ffff88030e8dd910
[ 1921.074756] Call Trace:
[ 1921.074757] [<ffffffff8169f09e>] _raw_spin_lock+0xe/0x20
[ 1921.074760] [<ffffffff811a11a0>] d_absolute_path+0x60/0xd0
[ 1921.074763] [<ffffffff812f077c>] d_namespace_path+0x4c/0x2b0
[ 1921.074765] [<ffffffff8169f09e>] ? _raw_spin_lock+0xe/0x20
[ 1921.074768] [<ffffffff8119fc7e>] ? dput+0x1ee/0x290
[ 1921.074770] [<ffffffff812f0a24>] get_name_to_buffer+0x44/0xe0
[ 1921.074772] [<ffffffff812f0b1f>] aa_path_name+0x5f/0xe0
[ 1921.074775] [<ffffffff812f1227>] apparmor_bprm_set_creds+0x107/0x780
[ 1921.074778] [<ffffffff8115515f>] ? vma_link+0xcf/0xe0
[ 1921.074780] [<ffffffff812bbda3>] security_bprm_set_creds+0x13/0x20
[ 1921.074783] [<ffffffff8118ec48>] prepare_binprm+0xb8/0x1e0
[ 1921.074785] [<ffffffff81190c63>] do_execve_common.isra.30+0x243/0x350
[ 1921.074788] [<ffffffff81190d8b>] do_execve+0x1b/0x20
[ 1921.074790] [<ffffffff8101d6c7>] sys_execve+0x47/0x70
[ 1921.074792] [<ffffffff816a7d2c>] stub_execve+0x6c/0xc0
[ 1921.074795] Code: 90 90 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 <f3> 90 0f b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0
[ 1920.827158] Modules linked in: veth xt_tcpudp xt_addrtype iptable_filter xt_conntrack aufs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp llc vesafb coretemp kvm_intel psmouse kvm dcdbas gpio_ich serio_raw lp joydev mac_hid i7core_edac edac_core wmi microcode parport lpc_ich acpi_power_meter ext2 hid_generic usbhid hid ses enclosure megaraid_sas bnx2
[ 1933.723143] [<ffffffff812f077c>] d_namespace_path+0x4c/0x2b0
[ 1933.822372] [<ffffffff810911c2>] ? update_sd_lb_stats+0x3e2/0x5f0
[ 1933.836544] [<ffffffff812f0a24>] get_name_to_buffer+0x44/0xe0
[ 1933.850280] [<ffffffff812f0b1f>] aa_path_name+0x5f/0xe0
[ 1933.863323] [<ffffffff81574a0b>] ? skb_release_data.part.47+0xcb/0x110
[ 1933.877846] [<ffffffff812f6b19>] aa_path_perm+0x89/0x180
[ 1933.891054] [<ffffffff812bb6ec>] ? security_inode_permission+0x1c/0x30
[ 1933.905412] [<ffffffff812f61a3>] apparmor_file_open+0x103/0x170
[ 1933.919269] [<ffffffff812bc555>] security_file_open+0x25/0x80
[ 1933.933734] [<ffffffff811868b7>] do_dentry_open+0x107/0x2b0
[ 1933.947434] [<ffffffff812bb6ec>] ? security_inode_permission+0x1c/0x30
[ 1933.962214] [<ffffffff81188168>] nameidata_to_filp+0xa8/0xf0
[ 1933.975985] [<ffffffff81194435>] ? may_open+0x75/0x120
[ 1933.989290] [<ffffffff81195bcc>] do_last+0x4bc/0x850
[ 1934.002409] [<ffffffff81198296>] path_openat+0xd6/0x440
[ 1934.015565] [<ffffffff81198722>] do_filp_open+0x42/0xa0
[ 1934.028858] [<ffffffff8169f09e>] ? _raw_spin_lock+0xe/0x20
[ 1934.042435] [<ffffffff811a5e61>] ? alloc_fd+0xd1/0x120
[ 1934.055468] [<ffffffff811882ac>] do_sys_open+0xfc/0x240
[ 1934.068820] [<ffffffff81188411>] sys_open+0x21/0x30
[ 1934.081762] [<ffffffff816a78e9>] system_call_fastpath+0x16/0x1b
[ 1935.147116] Shutting down cpus with NMI
Raw
3.8.0-35-generic
[ 268.303722] BUG: soft lockup - CPU#1 stuck for 22s! [redis-server:4493]
[ 268.307714] BUG: soft lockup - CPU#0 stuck for 22s! [rs:main Q:Reg:4502]
[ 268.307740] Modules linked in: veth(F) xt_nat(F) xt_tcpudp(F) xt_addrtype(F) iptable_filter(F) xt_conntrack(F) aufs(F) ipt_MASQUERADE(F) iptable_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack(F) ip_tables(F) x_tables(F) bridge(F) stp(F) llc(F) ext2(F) vesafb(F) coretemp(F) kvm_intel(F) kvm(F) gpio_ich(F) dcdbas(F) lp(F) psmouse(F) microcode(F) joydev(F) serio_raw(F) mac_hid(F) acpi_power_meter(F) wmi(F) i7core_edac(F) parport(F) edac_core(F) lpc_ich(F) hid_generic(F) usbhid(F) hid(F) ses(F) enclosure(F) megaraid_sas(F) bnx2(F)
[ 268.307744] CPU 0
[ 268.307745] Pid: 4502, comm: rs:main Q:Reg Tainted: GF I 3.8.0-35-generic #52~precise1-Ubuntu Dell Inc. PowerEdge R710/0YDJK3
[ 268.307753] RIP: 0010:[<ffffffff81045c42>] [<ffffffff81045c42>] __ticket_spin_lock+0x22/0x30
[ 268.307755] RSP: 0018:ffff880314629c38 EFLAGS: 00000293
[ 268.307756] RAX: 0000000000000d42 RBX: ffffffff81626d47 RCX: 0000000000000000
[ 268.307757] RDX: 0000000000000d4a RSI: ffffffff8181cc21 RDI: ffffffff81c063c4
[ 268.307758] RBP: ffff880314629c38 R08: ffff880314629c80 R09: ffffffff8181cc20
[ 268.307759] R10: 0000000000000000 R11: 0000000000000293 R12: ffff88062dfd1c00
[ 268.307760] R13: ffff88032d8a5000 R14: ffffffff815d20ba R15: ffff880314629ba8
[ 268.307761] FS: 00007fbdefd39700(0000) GS:ffff88063fc00000(0000) knlGS:0000000000000000
[ 268.307763] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 268.307764] CR2: 000000c21000f020 CR3: 000000032be51000 CR4: 00000000000007f0
[ 268.307765] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 268.307766] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 268.307767] Process rs:main Q:Reg (pid: 4502, threadinfo ffff880314628000, task ffff880327872e80)
[ 268.307768] Stack:
[ 268.307772] ffff880314629c48 ffffffff816f50fe ffff880314629c98 ffffffff811b4180
[ 268.307775] ffff880314629cd8 000000ff14629cd8 ffff88060b386400 0000000000000000
[ 268.307777] 0000000000000000 ffff88032dfd15ff ffff880314629cc8 ffff88062bab7510
[ 268.307778] Call Trace:
[ 268.307785] [<ffffffff816f50fe>] _raw_spin_lock+0xe/0x20
[ 268.307790] [<ffffffff811b4180>] d_absolute_path+0x60/0xd0
[ 268.307795] [<ffffffff8130f0dc>] d_namespace_path+0x4c/0x2b0
[ 268.307799] [<ffffffff813138f7>] ? apparmor_socket_sendmsg+0x17/0x20
[ 268.307804] [<ffffffff815cb282>] ? sock_sendmsg+0xd2/0xf0
[ 268.307806] [<ffffffff8130f384>] get_name_to_buffer+0x44/0xe0
[ 268.307808] [<ffffffff8130f47f>] aa_path_name+0x5f/0xe0
[ 268.307812] [<ffffffff81315479>] aa_path_perm+0x89/0x180
[ 268.307815] [<ffffffff81315867>] aa_file_perm+0x37/0x40
[ 268.307817] [<ffffffff81314734>] common_file_perm+0xb4/0xd0
[ 268.307820] [<ffffffff81314838>] apparmor_file_permission+0x18/0x20
[ 268.307824] [<ffffffff812da29c>] security_file_permission+0x2c/0xb0
[ 268.307829] [<ffffffff8119bbe1>] rw_verify_area+0x61/0xf0
[ 268.307832] [<ffffffff8119bcf8>] vfs_write+0x88/0x180
[ 268.307834] [<ffffffff8119c062>] sys_write+0x52/0xa0
[ 268.307839] [<ffffffff816fdf5d>] system_call_fastpath+0x1a/0x1f
[ 268.307859] Code: 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89
[ 268.307860] Kernel panic - not syncing: softlockup: hung tasks
[ 268.307862] Pid: 4502, comm: rs:main Q:Reg Tainted: GF I 3.8.0-35-generic #52~precise1-Ubuntu
[ 268.307862] Call Trace:
[ 268.307869] <IRQ> [<ffffffff816dcaa3>] panic+0xc1/0x1d7
[ 268.307874] [<ffffffff810ed6a7>] watchdog_timer_fn+0x177/0x180
[ 268.307878] [<ffffffff810839e8>] __run_hrtimer+0x78/0x1f0
[ 268.307880] [<ffffffff810ed530>] ? watchdog+0x30/0x30
[ 268.307884] [<ffffffff810b00dc>] ? ktime_get_update_offsets+0x4c/0xd0
[ 268.307886] [<ffffffff81084277>] hrtimer_interrupt+0xf7/0x240
[ 268.307889] [<ffffffff816ff3dc>] ? call_softirq+0x1c/0x30
[ 268.307892] [<ffffffff816ffd59>] smp_apic_timer_interrupt+0x69/0x99
[ 268.307894] [<ffffffff816fec9d>] apic_timer_interrupt+0x6d/0x80
[ 268.307900] <EOI> [<ffffffff81626d47>] ? tcp_sendmsg+0x137/0xd90
[ 268.307902] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30
[ 268.307904] [<ffffffff816f50fe>] _raw_spin_lock+0xe/0x20
[ 268.307907] [<ffffffff811b4180>] d_absolute_path+0x60/0xd0
[ 268.307909] [<ffffffff8130f0dc>] d_namespace_path+0x4c/0x2b0
[ 268.307911] [<ffffffff813138f7>] ? apparmor_socket_sendmsg+0x17/0x20
[ 268.307914] [<ffffffff815cb282>] ? sock_sendmsg+0xd2/0xf0
[ 268.307916] [<ffffffff8130f384>] get_name_to_buffer+0x44/0xe0
[ 268.307918] [<ffffffff8130f47f>] aa_path_name+0x5f/0xe0
[ 268.307920] [<ffffffff81315479>] aa_path_perm+0x89/0x180
[ 268.307923] [<ffffffff81315867>] aa_file_perm+0x37/0x40
[ 268.307925] [<ffffffff81314734>] common_file_perm+0xb4/0xd0
[ 268.307928] [<ffffffff81314838>] apparmor_file_permission+0x18/0x20
[ 268.307930] [<ffffffff812da29c>] security_file_permission+0x2c/0xb0
[ 268.307933] [<ffffffff8119bbe1>] rw_verify_area+0x61/0xf0
Raw
break-docker.sh
#!/bin/sh
# Make sure to stop docker before running this
while true
do
/usr/bin/docker -d &
PID=$!
echo -- Waiting for docker
while ! docker ps; do sleep 1; done
echo -- Restarting container
docker ps -q |xargs docker restart
kill $PID
echo -- Waiting for docker to be gone
while docker ps; do sleep 1; done
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment