Skip to content

Instantly share code, notes, and snippets.

@bleuchien

bleuchien/kernel.log Secret

Last active September 11, 2018 08:11
Show Gist options
  • Save bleuchien/3d2b87985ecb944c60347d5e8660e36a to your computer and use it in GitHub Desktop.
Save bleuchien/3d2b87985ecb944c60347d5e8660e36a to your computer and use it in GitHub Desktop.
Download ZIP
glusterfs oom kill
Raw
kernel.log
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: ------------[ cut here ]------------
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: WARNING: CPU: 1 PID: 85 at fs/xfs/xfs_aops.c:1115 xfs_vm_releasepage+0x136/0x150 [xfs]
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: Modules linked in: fuse nfsv4 dns_resolver nfs fscache cts rpcsec_gss_krb5 iptable_filter ip_tables ip_set_hash_ip bonding ip_set nfnetlink bridge stp llc sb_
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel:
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: CPU: 1 PID: 85 Comm: kswapd0 Kdump: loaded Not tainted 3.10.0-862.9.1.el7.x86_64 #1
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.05.0004.051120151007 05/11/2015
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: Call Trace:
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff9050e84e>] dump_stack+0x19/0x1b
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8fe91e18>] __warn+0xd8/0x100
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8fe91f5d>] warn_slowpath_null+0x1d/0x20
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffffc0393f76>] xfs_vm_releasepage+0x136/0x150 [xfs]
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8ff94405>] try_to_release_page+0x35/0x50
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8ffa8d06>] shrink_page_list+0x8d6/0xaf0
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8ffa954d>] shrink_inactive_list+0x1cd/0x5d0
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8ffaa045>] shrink_lruvec+0x385/0x730
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8ffaa466>] shrink_zone+0x76/0x1a0
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8ffab75c>] balance_pgdat+0x48c/0x5e0
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8ffaba23>] kswapd+0x173/0x440
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8febc610>] ? wake_up_atomic_t+0x30/0x30
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8ffab8b0>] ? balance_pgdat+0x5e0/0x5e0
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8febb621>] kthread+0xd1/0xe0
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8febb550>] ? insert_kthread_work+0x40/0x40
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff905205f7>] ret_from_fork_nospec_begin+0x21/0x21
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: [<ffffffff8febb550>] ? insert_kthread_work+0x40/0x40
Sep 10 13:44:14 brax.tbi.univie.ac.at kernel: ---[ end trace 8df055437b574dda ]---
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: nfsd invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: nfsd cpuset=/ mems_allowed=0
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: CPU: 7 PID: 1632 Comm: nfsd Kdump: loaded Tainted: G W ------------ 3.10.0-862.9.1.el7.x86_64 #1
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.05.0004.051120151007 05/11/2015
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Call Trace:
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff9050e84e>] dump_stack+0x19/0x1b
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff9050a1d0>] dump_header+0x90/0x229
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff900d865b>] ? cred_has_capability+0x6b/0x120
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff8ff980c4>] oom_kill_process+0x254/0x3d0
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff900d872c>] ? selinux_capable+0x1c/0x40
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff8ff98906>] out_of_memory+0x4b6/0x4f0
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff8ff9f224>] __alloc_pages_nodemask+0xaa4/0xbb0
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff8ffe8ce8>] alloc_pages_current+0x98/0x110
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffffc02205b4>] svc_recv+0xb4/0xb90 [sunrpc]
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffffc0443edd>] nfsd+0xcd/0x150 [nfsd]
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffffc0443e10>] ? nfsd_destroy+0x80/0x80 [nfsd]
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff8febb621>] kthread+0xd1/0xe0
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff8febb550>] ? insert_kthread_work+0x40/0x40
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff905205f7>] ret_from_fork_nospec_begin+0x21/0x21
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [<ffffffff8febb550>] ? insert_kthread_work+0x40/0x40
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Mem-Info:
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: active_anon:31068872 inactive_anon:1038913 isolated_anon:0
active_file:405 inactive_file:1541 isolated_file:0
unevictable:0 dirty:0 writeback:266 unstable:0
slab_reclaimable:346686 slab_unreclaimable:49588
mapped:11353 shmem:72249 pagetables:67090 bounce:0
free:146717 free_pcp:3139 free_cma:0
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Node 0 DMA free:15896kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isola
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: lowmem_reserve[]: 0 2745 128645 128645
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Node 0 DMA32 free:504920kB min:1440kB low:1800kB high:2160kB active_anon:815072kB inactive_anon:524364kB active_file:0kB inactive_file:0kB unevictable:0kB iso
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: lowmem_reserve[]: 0 0 125900 125900
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Node 0 Normal free:66052kB min:66132kB low:82664kB high:99196kB active_anon:123460416kB inactive_anon:3631288kB active_file:1620kB inactive_file:6192kB unevic
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: lowmem_reserve[]: 0 0 0 0
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 2*32kB (U) 3*64kB (U) 0*128kB 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15896kB
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Node 0 DMA32: 16732*4kB (UEM) 13237*8kB (UEM) 7868*16kB (UEM) 5332*32kB (UEM) 554*64kB (UEM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 504920kB
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Node 0 Normal: 16445*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 65780kB
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: 131034 total pagecache pages
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: 56166 pages in swap cache
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Swap cache stats: add 6742892, delete 6686711, find 75897171/76349576
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Free swap = 0kB
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Total swap = 4194300kB
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: 33534254 pages RAM
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: 0 pages HighMem/MovableOnly
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: 596292 pages reserved
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 593] 0 593 33430 17717 71 46 0 systemd-journal
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 636] 0 636 50278 0 32 909 0 lvmetad
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 640] 0 640 12025 41 26 672 -1000 systemd-udevd
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 832] 0 832 13877 31 28 80 -1000 auditd
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 836] 0 836 11489 49 27 61 0 rpc.idmapd
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 853] 0 853 70724 73 87 283 0 sssd
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 854] 998 854 137866 645 61 1388 0 polkitd
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 856] 81 856 19702 94 36 106 -900 dbus-daemon
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 861] 32 861 17305 42 37 106 0 rpcbind
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 865] 0 865 5416 51 16 34 0 irqbalance
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 874] 0 874 71928 244 49 42 0 gssproxy
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 881] 38 881 14871 48 30 131 0 ntpd
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 882] 0 882 98530 578 132 265 0 sssd_be
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 886] 0 886 49029 327 33 214 0 rpc.gssd
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 893] 0 893 79333 6216 109 782 0 sssd_nss
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 894] 0 894 65972 59 83 250 0 sssd_sudo
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 895] 0 895 67606 159 87 202 0 sssd_pam
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 896] 0 896 65969 115 82 196 0 sssd_ssh
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 897] 0 897 76052 56 97 354 0 sssd_pac
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 902] 0 902 119754 278 85 477 0 NetworkManager
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 1465] 0 1465 28203 28 58 228 -1000 sshd
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 1466] 0 1466 143454 106 99 2645 0 tuned
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 1475] 0 1475 114771 10340 111 94 0 rsyslogd
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 1482] 29 1482 11698 101 29 166 0 rpc.statd
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 1496] 0 1496 18128 398 39 127 0 rpc.mountd
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 1694] 27 1694 28329 32 12 46 0 mysqld_safe
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 2259] 0 2259 50314 477 33 269 0 EventMonitor
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 2273] 0 2273 38841 831 33 965 0 munin-node
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 2696] 0 2696 22929 21 43 241 0 master
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [ 2751] 89 2751 25540 37 47 235 0 qmgr
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [11708] 0 11708 6594 45 18 38 0 systemd-logind
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [11710] 0 11710 31571 29 18 132 0 crond
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [11722] 0 11722 27522 2 10 31 0 agetty
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [28766] 27 28766 18779456 13935876 29304 1020180 0 mysqld
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [95596] 0 95596 42240 344 81 2 0 sshd
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [95602] 0 95602 28892 137 12 0 0 bash
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [210420] 0 210420 42240 343 80 2 0 sshd
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [210426] 0 210426 28892 130 13 0 0 bash
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [212153] 0 212153 33159 98 22 0 0 ping
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [214824] 0 214824 42240 344 81 0 0 sshd
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [214830] 0 214830 28892 118 14 0 0 bash
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [214845] 0 214845 33159 98 21 0 0 ping
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [214846] 0 214846 42240 345 82 0 0 sshd
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [214852] 0 214852 28892 118 13 0 0 bash
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [214867] 0 214867 33159 98 21 0 0 ping
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [203318] 0 203318 17965680 17819421 34873 0 0 glusterfs
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [203332] 0 203332 141056 90716 230 0 0 rsync
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [203333] 0 203333 107188 51295 163 0 0 rsync
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [203334] 0 203334 148948 67973 243 0 0 rsync
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: [104716] 89 104716 25496 254 47 0 0 pickup
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Out of memory: Kill process 203318 (glusterfs) score 509 or sacrifice child
Sep 10 13:46:44 brax.tbi.univie.ac.at kernel: Killed process 203318 (glusterfs) total-vm:71862720kB, anon-rss:71276088kB, file-rss:1596kB, shmem-rss:0kB
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment