Skip to content

Instantly share code, notes, and snippets.

@jessfraz

jessfraz/stack

Created Sep 21, 2015
Embed
What would you like to do?
server weirdness
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.023170] BUG: unable to handle kernel NULL pointer dereference at 0000000000000016
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.023273] IP: [<ffffffff81217490>] __detach_mounts+0x40/0x90
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.023365] PGD 2cd17b067 PUD 40bd44067 PMD 0
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.023425] Oops: 0000 [#1] SMP
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.023468] Modules linked in: dummy xt_nat xt_tcpudp dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables x_tables br_netfilter nf_nat nf_conntrack bridge stp llc overlay iosf_mbi ppdev kvm_intel kvm cirrus crct10dif_pclmul crc32_pclmul ttm ghash_clmulni_intel drm_kms_helper aesni_intel aes_x86_64 lrw drm gf128mul glue_helper ablk_helper cryptd syscopyarea serio_raw sysfillrect sysimgblt i2c_piix4 parport_pc pvpanic 8250_fintek parport mac_hid autofs4 psmouse floppy pata_acpi
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.024314] CPU: 1 PID: 3657 Comm: docker Not tainted 3.19.0-22-generic #22-Ubuntu
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.024399] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.024463] task: ffff880409891d70 ti: ffff88032103c000 task.ti: ffff88032103c000
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.024545] RIP: 0010:[<ffffffff81217490>] [<ffffffff81217490>] __detach_mounts+0x40/0x90
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.024641] RSP: 0018:ffff88032103fdf8 EFLAGS: 00010202
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.024701] RAX: 00000000cc46cc46 RBX: fffffffffffffffe RCX: 000000000000000f
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.024780] RDX: 000000000000cc46 RSI: ffffffffffffffff RDI: ffffffff81c0c4c4
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.024859] RBP: ffff88032103fe08 R08: 0000000000000000 R09: 0000000000000000
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.024938] R10: ffff88041fc5b490 R11: ffffea001025b880 R12: ffff8803410f6598
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.025017] R13: ffff88023f990238 R14: ffff88032103fe90 R15: ffff88023f9902e0
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.025101] FS: 00007f4bc22d8700(0000) GS:ffff88041fc40000(0000) knlGS:0000000000000000
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.025191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.025256] CR2: 0000000000000016 CR3: 000000040cb64000 CR4: 00000000001407e0
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.025346] Stack:
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.025371] ffff88032103fe08 ffff8803410f6540 ffff88032103fe58 ffffffff81203612
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.025465] ffff88032103fea8 0000800000000000 ffff880247539a60 0000000000000000
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.025900] ffff8800daba4000 0000000000000000 ffff88032103fea8 ffff88023f990238
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.025994] Call Trace:
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.026030] [<ffffffff81203612>] vfs_unlink+0x172/0x180
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.026095] [<ffffffff81206288>] do_unlinkat+0x2b8/0x330
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.026163] [<ffffffff810640d5>] ? trace_do_page_fault+0x45/0x100
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.026235] [<ffffffff8105be4e>] ? do_async_page_fault+0x1e/0xd0
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.026305] [<ffffffff812073e6>] SyS_unlink+0x16/0x20
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.026366] [<ffffffff817cb24d>] system_call_fastpath+0x16/0x1b
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.026431] Code: 83 ec 08 e8 b3 1d 5b 00 48 89 df e8 db d3 ff ff 48 85 c0 48 89 c3 74 55 48 c7 c7 c4 c4 c0 81 e8 97 39 5b 00 83 05 30 50 9f 00 01 <48> 8b 53 18 48 85 d2 74 1e 0f 1f 80 00 00 00 00 48 8d ba 10 ff
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.026831] RIP [<ffffffff81217490>] __detach_mounts+0x40/0x90
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.026903] RSP <ffff88032103fdf8>
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.026910] CR2: 0000000000000016
Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.034978] ---[ end trace 41068904ecfc4da0 ]---
@jessfraz

This comment has been minimized.

Copy link
Owner Author

@jessfraz jessfraz commented Sep 21, 2015

overlay ubuntu stock 3.19

@jessfraz

This comment has been minimized.

Copy link
Owner Author

@jessfraz jessfraz commented Sep 21, 2015

aufs same kernel running devampper unit tests:

[  657.470092] aufs au_opts_verify:1612:aufs.test[8379]: dirperm1 breaks the protection by the permission bits on the lower branch
[  657.470139] aufs au_opts_verify:1612:aufs.test[8379]: dirperm1 breaks the protection by the permission bits on the lower branch
[  657.470185] aufs au_opts_verify:1612:aufs.test[8379]: dirperm1 breaks the protection by the permission bits on the lower branch
[  660.196524] aufs au_warn_loopback:122:loop0[8690]: you may want to try another patch for loopback file on ext4(0xef53) branch
[  660.224182] divide error: 0000 [#1] SMP 
[  660.224244] Modules linked in: dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables x_tables br_netfilter nf_nat nf_conntrack bridge stp llc aufs iosf_mbi kvm_intel kvm ppdev crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul cirrus glue_helper ablk_helper ttm cryptd drm_kms_helper drm serio_raw syscopyarea parport_pc sysfillrect sysimgblt parport i2c_piix4 8250_fintek pvpanic mac_hid autofs4 psmouse floppy pata_acpi
[  660.225030] CPU: 0 PID: 8689 Comm: devmapper.test Not tainted 3.19.0-22-generic #22-Ubuntu
[  660.225120] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[  660.225186] task: ffff8800365e44b0 ti: ffff880409084000 task.ti: ffff880409084000
[  660.225269] RIP: 0010:[<ffffffffc05b3366>]  [<ffffffffc05b3366>] pool_io_hints+0x196/0x230 [dm_thin_pool]
[  660.225382] RSP: 0018:ffff880409087b28  EFLAGS: 00010246
[  660.225442] RAX: 0000000000010000 RBX: ffff880409087b88 RCX: ffff8800afc40000
[  660.225521] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880409460000
[  660.225600] RBP: ffff880409087b78 R08: 00000000000000ff R09: 0000000000000000
[  660.225679] R10: 0000000000000000 R11: 0000000000000080 R12: ffffffffc05bc5d8
[  660.225758] R13: ffff880364095f60 R14: ffff880409087c28 R15: ffffc90001b82040
[  660.225839] FS:  00007f4b00988700(0000) GS:ffff88041fc00000(0000) knlGS:0000000000000000
[  660.225928] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  660.225992] CR2: 00007fffb443a1d8 CR3: 0000000409940000 CR4: 00000000001407f0
[  660.226082] Stack:
[  660.226107]  ffff880409087b78 000000009a352228 ffffffff09087be8 0000000000000000
[  660.226201]  ffff88040990d600 000000009a352228 ffff880409087c28 0000000000000000
[  660.226294]  ffff88040990d600 0000000000000001 ffff880409087c08 ffffffff816514c7
[  660.226386] Call Trace:
[  660.226423]  [<ffffffff816514c7>] dm_calculate_queue_limits+0x97/0x140
[  660.226508]  [<ffffffff8164ed7c>] dm_swap_table+0x7c/0x360
[  660.226572]  [<ffffffff8164d059>] ? __dm_suspend+0x89/0x1f0
[  660.226638]  [<ffffffff81640101>] ? super_1_load+0x361/0x570
[  660.226705]  [<ffffffff8165440a>] dev_suspend+0x9a/0x260
[  660.226768]  [<ffffffff81654370>] ? table_load+0x340/0x340
[  660.226838]  [<ffffffff81654d7e>] ctl_ioctl+0x22e/0x510
[  660.226906]  [<ffffffff81655073>] dm_ctl_ioctl+0x13/0x20
[  660.226969]  [<ffffffff81209250>] do_vfs_ioctl+0x2e0/0x4e0
[  660.227035]  [<ffffffff810f3c06>] ? SyS_futex+0x76/0x170
[  660.227097]  [<ffffffff812094d1>] SyS_ioctl+0x81/0xa0
[  660.227162]  [<ffffffff817cb24d>] system_call_fastpath+0x16/0x1b
[  660.227229] Code: 00 f6 c4 40 74 35 3b b2 08 06 00 00 49 c7 c4 d8 c5 5b c0 77 2d c1 e6 09 89 f0 8b b2 10 06 00 00 48 39 f0 0f 82 8a 00 00 00 31 d2 <48> f7 f6 85 d2 74 3b 49 c7 c4 80 c5 5b c0 eb 07 49 c7 c4 c6 c1 
[  660.227655] RIP  [<ffffffffc05b3366>] pool_io_hints+0x196/0x230 [dm_thin_pool]
[  660.227740]  RSP <ffff880409087b28>
[  660.233478] ---[ end trace 133ea183dafe1025 ]---

@vbatts

This comment has been minimized.

Copy link

@vbatts vbatts commented Sep 22, 2015

I would say, try another kernel. If updating to 3.19.0-28-generic didn't fix it, see if there is another supported version.
On the overlay+vfs, there looks to be something racy in the cleaning up of unlinking/deleting a file, and how that dentry is cleaned up regarding namespaces. IIUC a lazy clean-up is double cleaning the same dentry, hence the null pointer. The mountpoint's hlist_head seems like it's null.
Nothing to say on the aufs + devmapper tests yet

@vbatts

This comment has been minimized.

Copy link

@vbatts vbatts commented Sep 22, 2015

looking further on the first one, i think this commit for the 4.0 kernel, torvalds/linux@f53e579

@vbatts

This comment has been minimized.

Copy link

@vbatts vbatts commented Sep 22, 2015

and for the devicemapper tests on the AUFS driver, that is bit curios. Just asked @snitm and he's not familiar where a divide by 0 is every reported. I'm inclined to think it is a peculiarity of AUFS :-\

@jessfraz

This comment has been minimized.

Copy link
Owner Author

@jessfraz jessfraz commented Sep 23, 2015

ok lets run all tests w overlay in overlay and see if things improve

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment