-
-
Save bleuchien/3d2b87985ecb944c60347d5e8660e36a to your computer and use it in GitHub Desktop.
glusterfs oom kill
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: ------------[ cut here ]------------ | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: WARNING: CPU: 1 PID: 85 at fs/xfs/xfs_aops.c:1115 xfs_vm_releasepage+0x136/0x150 [xfs] | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: Modules linked in: fuse nfsv4 dns_resolver nfs fscache cts rpcsec_gss_krb5 iptable_filter ip_tables ip_set_hash_ip bonding ip_set nfnetlink bridge stp llc sb_ | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: CPU: 1 PID: 85 Comm: kswapd0 Kdump: loaded Not tainted 3.10.0-862.9.1.el7.x86_64 #1 | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.05.0004.051120151007 05/11/2015 | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: Call Trace: | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff9050e84e>] dump_stack+0x19/0x1b | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8fe91e18>] __warn+0xd8/0x100 | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8fe91f5d>] warn_slowpath_null+0x1d/0x20 | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffffc0393f76>] xfs_vm_releasepage+0x136/0x150 [xfs] | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8ff94405>] try_to_release_page+0x35/0x50 | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8ffa8d06>] shrink_page_list+0x8d6/0xaf0 | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8ffa954d>] shrink_inactive_list+0x1cd/0x5d0 | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8ffaa045>] shrink_lruvec+0x385/0x730 | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8ffaa466>] shrink_zone+0x76/0x1a0 | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8ffab75c>] balance_pgdat+0x48c/0x5e0 | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8ffaba23>] kswapd+0x173/0x440 | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8febc610>] ? wake_up_atomic_t+0x30/0x30 | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8ffab8b0>] ? balance_pgdat+0x5e0/0x5e0 | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8febb621>] kthread+0xd1/0xe0 | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8febb550>] ? insert_kthread_work+0x40/0x40 | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff905205f7>] ret_from_fork_nospec_begin+0x21/0x21 | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8febb550>] ? insert_kthread_work+0x40/0x40 | |
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: ---[ end trace 8df055437b574dda ]--- | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: nfsd invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0 | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: nfsd cpuset=/ mems_allowed=0 | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: CPU: 7 PID: 1632 Comm: nfsd Kdump: loaded Tainted: G W ------------ 3.10.0-862.9.1.el7.x86_64 #1 | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.05.0004.051120151007 05/11/2015 | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Call Trace: | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff9050e84e>] dump_stack+0x19/0x1b | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff9050a1d0>] dump_header+0x90/0x229 | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff900d865b>] ? cred_has_capability+0x6b/0x120 | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff8ff980c4>] oom_kill_process+0x254/0x3d0 | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff900d872c>] ? selinux_capable+0x1c/0x40 | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff8ff98906>] out_of_memory+0x4b6/0x4f0 | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff8ff9f224>] __alloc_pages_nodemask+0xaa4/0xbb0 | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff8ffe8ce8>] alloc_pages_current+0x98/0x110 | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffffc02205b4>] svc_recv+0xb4/0xb90 [sunrpc] | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffffc0443edd>] nfsd+0xcd/0x150 [nfsd] | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffffc0443e10>] ? nfsd_destroy+0x80/0x80 [nfsd] | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff8febb621>] kthread+0xd1/0xe0 | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff8febb550>] ? insert_kthread_work+0x40/0x40 | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff905205f7>] ret_from_fork_nospec_begin+0x21/0x21 | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff8febb550>] ? insert_kthread_work+0x40/0x40 | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Mem-Info: | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: active_anon:31068872 inactive_anon:1038913 isolated_anon:0 | |
active_file:405 inactive_file:1541 isolated_file:0 | |
unevictable:0 dirty:0 writeback:266 unstable:0 | |
slab_reclaimable:346686 slab_unreclaimable:49588 | |
mapped:11353 shmem:72249 pagetables:67090 bounce:0 | |
free:146717 free_pcp:3139 free_cma:0 | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Node 0 DMA free:15896kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isola | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: lowmem_reserve[]: 0 2745 128645 128645 | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Node 0 DMA32 free:504920kB min:1440kB low:1800kB high:2160kB active_anon:815072kB inactive_anon:524364kB active_file:0kB inactive_file:0kB unevictable:0kB iso | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: lowmem_reserve[]: 0 0 125900 125900 | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Node 0 Normal free:66052kB min:66132kB low:82664kB high:99196kB active_anon:123460416kB inactive_anon:3631288kB active_file:1620kB inactive_file:6192kB unevic | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: lowmem_reserve[]: 0 0 0 0 | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 2*32kB (U) 3*64kB (U) 0*128kB 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15896kB | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Node 0 DMA32: 16732*4kB (UEM) 13237*8kB (UEM) 7868*16kB (UEM) 5332*32kB (UEM) 554*64kB (UEM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 504920kB | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Node 0 Normal: 16445*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 65780kB | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: 131034 total pagecache pages | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: 56166 pages in swap cache | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Swap cache stats: add 6742892, delete 6686711, find 75897171/76349576 | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Free swap = 0kB | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Total swap = 4194300kB | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: 33534254 pages RAM | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: 0 pages HighMem/MovableOnly | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: 596292 pages reserved | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 593] 0 593 33430 17717 71 46 0 systemd-journal | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 636] 0 636 50278 0 32 909 0 lvmetad | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 640] 0 640 12025 41 26 672 -1000 systemd-udevd | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 832] 0 832 13877 31 28 80 -1000 auditd | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 836] 0 836 11489 49 27 61 0 rpc.idmapd | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 853] 0 853 70724 73 87 283 0 sssd | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 854] 998 854 137866 645 61 1388 0 polkitd | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 856] 81 856 19702 94 36 106 -900 dbus-daemon | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 861] 32 861 17305 42 37 106 0 rpcbind | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 865] 0 865 5416 51 16 34 0 irqbalance | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 874] 0 874 71928 244 49 42 0 gssproxy | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 881] 38 881 14871 48 30 131 0 ntpd | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 882] 0 882 98530 578 132 265 0 sssd_be | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 886] 0 886 49029 327 33 214 0 rpc.gssd | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 893] 0 893 79333 6216 109 782 0 sssd_nss | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 894] 0 894 65972 59 83 250 0 sssd_sudo | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 895] 0 895 67606 159 87 202 0 sssd_pam | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 896] 0 896 65969 115 82 196 0 sssd_ssh | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 897] 0 897 76052 56 97 354 0 sssd_pac | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 902] 0 902 119754 278 85 477 0 NetworkManager | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 1465] 0 1465 28203 28 58 228 -1000 sshd | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 1466] 0 1466 143454 106 99 2645 0 tuned | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 1475] 0 1475 114771 10340 111 94 0 rsyslogd | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 1482] 29 1482 11698 101 29 166 0 rpc.statd | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 1496] 0 1496 18128 398 39 127 0 rpc.mountd | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 1694] 27 1694 28329 32 12 46 0 mysqld_safe | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 2259] 0 2259 50314 477 33 269 0 EventMonitor | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 2273] 0 2273 38841 831 33 965 0 munin-node | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 2696] 0 2696 22929 21 43 241 0 master | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 2751] 89 2751 25540 37 47 235 0 qmgr | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [11708] 0 11708 6594 45 18 38 0 systemd-logind | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [11710] 0 11710 31571 29 18 132 0 crond | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [11722] 0 11722 27522 2 10 31 0 agetty | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [28766] 27 28766 18779456 13935876 29304 1020180 0 mysqld | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [95596] 0 95596 42240 344 81 2 0 sshd | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [95602] 0 95602 28892 137 12 0 0 bash | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [210420] 0 210420 42240 343 80 2 0 sshd | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [210426] 0 210426 28892 130 13 0 0 bash | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [212153] 0 212153 33159 98 22 0 0 ping | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [214824] 0 214824 42240 344 81 0 0 sshd | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [214830] 0 214830 28892 118 14 0 0 bash | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [214845] 0 214845 33159 98 21 0 0 ping | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [214846] 0 214846 42240 345 82 0 0 sshd | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [214852] 0 214852 28892 118 13 0 0 bash | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [214867] 0 214867 33159 98 21 0 0 ping | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [203318] 0 203318 17965680 17819421 34873 0 0 glusterfs | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [203332] 0 203332 141056 90716 230 0 0 rsync | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [203333] 0 203333 107188 51295 163 0 0 rsync | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [203334] 0 203334 148948 67973 243 0 0 rsync | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [104716] 89 104716 25496 254 47 0 0 pickup | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Out of memory: Kill process 203318 (glusterfs) score 509 or sacrifice child | |
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Killed process 203318 (glusterfs) total-vm:71862720kB, anon-rss:71276088kB, file-rss:1596kB, shmem-rss:0kB |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment