Created
September 21, 2015 20:09
-
-
Save jessfraz/df0667df1be407ef96c2 to your computer and use it in GitHub Desktop.
server weirdness
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.023170] BUG: unable to handle kernel NULL pointer dereference at 0000000000000016 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.023273] IP: [<ffffffff81217490>] __detach_mounts+0x40/0x90 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.023365] PGD 2cd17b067 PUD 40bd44067 PMD 0 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.023425] Oops: 0000 [#1] SMP | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.023468] Modules linked in: dummy xt_nat xt_tcpudp dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables x_tables br_netfilter nf_nat nf_conntrack bridge stp llc overlay iosf_mbi ppdev kvm_intel kvm cirrus crct10dif_pclmul crc32_pclmul ttm ghash_clmulni_intel drm_kms_helper aesni_intel aes_x86_64 lrw drm gf128mul glue_helper ablk_helper cryptd syscopyarea serio_raw sysfillrect sysimgblt i2c_piix4 parport_pc pvpanic 8250_fintek parport mac_hid autofs4 psmouse floppy pata_acpi | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.024314] CPU: 1 PID: 3657 Comm: docker Not tainted 3.19.0-22-generic #22-Ubuntu | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.024399] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.024463] task: ffff880409891d70 ti: ffff88032103c000 task.ti: ffff88032103c000 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.024545] RIP: 0010:[<ffffffff81217490>] [<ffffffff81217490>] __detach_mounts+0x40/0x90 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.024641] RSP: 0018:ffff88032103fdf8 EFLAGS: 00010202 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.024701] RAX: 00000000cc46cc46 RBX: fffffffffffffffe RCX: 000000000000000f | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.024780] RDX: 000000000000cc46 RSI: ffffffffffffffff RDI: ffffffff81c0c4c4 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.024859] RBP: ffff88032103fe08 R08: 0000000000000000 R09: 0000000000000000 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.024938] R10: ffff88041fc5b490 R11: ffffea001025b880 R12: ffff8803410f6598 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.025017] R13: ffff88023f990238 R14: ffff88032103fe90 R15: ffff88023f9902e0 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.025101] FS: 00007f4bc22d8700(0000) GS:ffff88041fc40000(0000) knlGS:0000000000000000 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.025191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.025256] CR2: 0000000000000016 CR3: 000000040cb64000 CR4: 00000000001407e0 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.025346] Stack: | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.025371] ffff88032103fe08 ffff8803410f6540 ffff88032103fe58 ffffffff81203612 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.025465] ffff88032103fea8 0000800000000000 ffff880247539a60 0000000000000000 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.025900] ffff8800daba4000 0000000000000000 ffff88032103fea8 ffff88023f990238 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.025994] Call Trace: | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.026030] [<ffffffff81203612>] vfs_unlink+0x172/0x180 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.026095] [<ffffffff81206288>] do_unlinkat+0x2b8/0x330 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.026163] [<ffffffff810640d5>] ? trace_do_page_fault+0x45/0x100 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.026235] [<ffffffff8105be4e>] ? do_async_page_fault+0x1e/0xd0 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.026305] [<ffffffff812073e6>] SyS_unlink+0x16/0x20 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.026366] [<ffffffff817cb24d>] system_call_fastpath+0x16/0x1b | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.026431] Code: 83 ec 08 e8 b3 1d 5b 00 48 89 df e8 db d3 ff ff 48 85 c0 48 89 c3 74 55 48 c7 c7 c4 c4 c0 81 e8 97 39 5b 00 83 05 30 50 9f 00 01 <48> 8b 53 18 48 85 d2 74 1e 0f 1f 80 00 00 00 00 48 8d ba 10 ff | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.026831] RIP [<ffffffff81217490>] __detach_mounts+0x40/0x90 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.026903] RSP <ffff88032103fdf8> | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.026910] CR2: 0000000000000016 | |
| Sep 21 15:10:47 pr-builder-11 kernel: [ 6071.034978] ---[ end trace 41068904ecfc4da0 ]--- |
aufs same kernel running devampper unit tests:
[ 657.470092] aufs au_opts_verify:1612:aufs.test[8379]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 657.470139] aufs au_opts_verify:1612:aufs.test[8379]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 657.470185] aufs au_opts_verify:1612:aufs.test[8379]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 660.196524] aufs au_warn_loopback:122:loop0[8690]: you may want to try another patch for loopback file on ext4(0xef53) branch
[ 660.224182] divide error: 0000 [#1] SMP
[ 660.224244] Modules linked in: dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables x_tables br_netfilter nf_nat nf_conntrack bridge stp llc aufs iosf_mbi kvm_intel kvm ppdev crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul cirrus glue_helper ablk_helper ttm cryptd drm_kms_helper drm serio_raw syscopyarea parport_pc sysfillrect sysimgblt parport i2c_piix4 8250_fintek pvpanic mac_hid autofs4 psmouse floppy pata_acpi
[ 660.225030] CPU: 0 PID: 8689 Comm: devmapper.test Not tainted 3.19.0-22-generic #22-Ubuntu
[ 660.225120] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 660.225186] task: ffff8800365e44b0 ti: ffff880409084000 task.ti: ffff880409084000
[ 660.225269] RIP: 0010:[<ffffffffc05b3366>] [<ffffffffc05b3366>] pool_io_hints+0x196/0x230 [dm_thin_pool]
[ 660.225382] RSP: 0018:ffff880409087b28 EFLAGS: 00010246
[ 660.225442] RAX: 0000000000010000 RBX: ffff880409087b88 RCX: ffff8800afc40000
[ 660.225521] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880409460000
[ 660.225600] RBP: ffff880409087b78 R08: 00000000000000ff R09: 0000000000000000
[ 660.225679] R10: 0000000000000000 R11: 0000000000000080 R12: ffffffffc05bc5d8
[ 660.225758] R13: ffff880364095f60 R14: ffff880409087c28 R15: ffffc90001b82040
[ 660.225839] FS: 00007f4b00988700(0000) GS:ffff88041fc00000(0000) knlGS:0000000000000000
[ 660.225928] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 660.225992] CR2: 00007fffb443a1d8 CR3: 0000000409940000 CR4: 00000000001407f0
[ 660.226082] Stack:
[ 660.226107] ffff880409087b78 000000009a352228 ffffffff09087be8 0000000000000000
[ 660.226201] ffff88040990d600 000000009a352228 ffff880409087c28 0000000000000000
[ 660.226294] ffff88040990d600 0000000000000001 ffff880409087c08 ffffffff816514c7
[ 660.226386] Call Trace:
[ 660.226423] [<ffffffff816514c7>] dm_calculate_queue_limits+0x97/0x140
[ 660.226508] [<ffffffff8164ed7c>] dm_swap_table+0x7c/0x360
[ 660.226572] [<ffffffff8164d059>] ? __dm_suspend+0x89/0x1f0
[ 660.226638] [<ffffffff81640101>] ? super_1_load+0x361/0x570
[ 660.226705] [<ffffffff8165440a>] dev_suspend+0x9a/0x260
[ 660.226768] [<ffffffff81654370>] ? table_load+0x340/0x340
[ 660.226838] [<ffffffff81654d7e>] ctl_ioctl+0x22e/0x510
[ 660.226906] [<ffffffff81655073>] dm_ctl_ioctl+0x13/0x20
[ 660.226969] [<ffffffff81209250>] do_vfs_ioctl+0x2e0/0x4e0
[ 660.227035] [<ffffffff810f3c06>] ? SyS_futex+0x76/0x170
[ 660.227097] [<ffffffff812094d1>] SyS_ioctl+0x81/0xa0
[ 660.227162] [<ffffffff817cb24d>] system_call_fastpath+0x16/0x1b
[ 660.227229] Code: 00 f6 c4 40 74 35 3b b2 08 06 00 00 49 c7 c4 d8 c5 5b c0 77 2d c1 e6 09 89 f0 8b b2 10 06 00 00 48 39 f0 0f 82 8a 00 00 00 31 d2 <48> f7 f6 85 d2 74 3b 49 c7 c4 80 c5 5b c0 eb 07 49 c7 c4 c6 c1
[ 660.227655] RIP [<ffffffffc05b3366>] pool_io_hints+0x196/0x230 [dm_thin_pool]
[ 660.227740] RSP <ffff880409087b28>
[ 660.233478] ---[ end trace 133ea183dafe1025 ]---
I would say, try another kernel. If updating to 3.19.0-28-generic didn't fix it, see if there is another supported version.
On the overlay+vfs, there looks to be something racy in the cleaning up of unlinking/deleting a file, and how that dentry is cleaned up regarding namespaces. IIUC a lazy clean-up is double cleaning the same dentry, hence the null pointer. The mountpoint's hlist_head seems like it's null.
Nothing to say on the aufs + devmapper tests yet
looking further on the first one, i think this commit for the 4.0 kernel, torvalds/linux@f53e579
and for the devicemapper tests on the AUFS driver, that is bit curios. Just asked @snitm and he's not familiar where a divide by 0 is every reported. I'm inclined to think it is a peculiarity of AUFS :-\
ok lets run all tests w overlay in overlay and see if things improve
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
overlay ubuntu stock 3.19