Skip to content

Instantly share code, notes, and snippets.

@infinitydon

infinitydon/gist:d2d83baa27822e42732f3665869b739b

Created April 4, 2019 14:34
Show Gist options
  • Save infinitydon/d2d83baa27822e42732f3665869b739b to your computer and use it in GitHub Desktop.
Save infinitydon/d2d83baa27822e42732f3665869b739b to your computer and use it in GitHub Desktop.
Download ZIP
slub_unable To Allocate Memory
Raw
gistfile1.txt
[Wed Apr 3 16:51:45 2019] [<ffffffff8106c512>] mm_fault_error+0x82/0x160
[Wed Apr 3 16:51:45 2019] [<ffffffff8106c9d9>] __do_page_fault+0x3e9/0x410
[Wed Apr 3 16:51:45 2019] [<ffffffff8106ca22>] do_page_fault+0x22/0x30
[Wed Apr 3 16:51:45 2019] [<ffffffff81826858>] page_fault+0x28/0x30
[Wed Apr 3 16:51:45 2019] Task in /kubepods/pod70849c83-5081-11e9-a4f4-0606b708822a/f757a788d85b1b04b1bc5b7d0ed3596a928dcf3cb8c56242112fdb16849d537f killed as a result of limit of /kubepods/pod70849c83-5081-11e9-a4f4-0606b708822a
[Wed Apr 3 16:51:45 2019] memory: usage 1024000kB, limit 1024000kB, failcnt 4662
[Wed Apr 3 16:51:45 2019] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
[Wed Apr 3 16:51:45 2019] kmem: usage 25824kB, limit 9007199254740988kB, failcnt 0
[Wed Apr 3 16:51:45 2019] Memory cgroup stats for /kubepods/pod70849c83-5081-11e9-a4f4-0606b708822a: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_fi
le:0KB unevictable:0KB
[Wed Apr 3 16:51:45 2019] Memory cgroup stats for /kubepods/pod70849c83-5081-11e9-a4f4-0606b708822a/f757a788d85b1b04b1bc5b7d0ed3596a928dcf3cb8c56242112fdb16849d537f: cache:0KB rss:36KB rss_huge:0KB mapped_file:0KB dirty:0KB writeback
:0KB inactive_anon:0KB active_anon:36KB inactive_file:0KB active_file:0KB unevictable:0KB
[Wed Apr 3 16:51:45 2019] Memory cgroup stats for /kubepods/pod70849c83-5081-11e9-a4f4-0606b708822a/b3f1519a8be9ff055106f274e74f7b563fd21125bc43c03f71f9d99e46df036f: cache:120KB rss:998020KB rss_huge:98304KB mapped_file:0KB dirty:0KB
writeback:0KB inactive_anon:0KB active_anon:998020KB inactive_file:120KB active_file:0KB unevictable:0KB
[Wed Apr 3 16:51:45 2019] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[Wed Apr 3 16:51:45 2019] [121908] 0 121908 257 1 4 2 0 -998 pause
[Wed Apr 3 16:51:45 2019] [122062] 1000 122062 541103 251352 1435 847 0 -998 node
[Wed Apr 3 16:51:45 2019] Memory cgroup out of memory: Kill process 121908 (pause) score 0 or sacrifice child
[Wed Apr 3 16:51:45 2019] Killed process 121908 (pause) total-vm:1028kB, anon-rss:4kB, file-rss:0kB
[Wed Apr 3 16:51:46 2019] SLUB: Unable to allocate memory on node -1 (gfp=0x2408040)
[Wed Apr 3 16:51:46 2019] cache: buffer_head(39782:b3f1519a8be9ff055106f274e74f7b563fd21125bc43c03f71f9d99e46df036f), object size: 104, buffer size: 104, default order: 0, min order: 0
[Wed Apr 3 16:51:46 2019] node 0: slabs: 3, objs: 117, free: 0
[Wed Apr 3 16:51:46 2019] SLUB: Unable to allocate memory on node -1 (gfp=0x2408040)
[Wed Apr 3 16:51:46 2019] cache: buffer_head(39782:b3f1519a8be9ff055106f274e74f7b563fd21125bc43c03f71f9d99e46df036f), object size: 104, buffer size: 104, default order: 0, min order: 0
[Wed Apr 3 16:51:46 2019] node 0: slabs: 3, objs: 117, free: 0
[Wed Apr 3 16:51:46 2019] SLUB: Unable to allocate memory on node -1 (gfp=0x2408040)
[Wed Apr 3 16:51:46 2019] cache: buffer_head(39782:b3f1519a8be9ff055106f274e74f7b563fd21125bc43c03f71f9d99e46df036f), object size: 104, buffer size: 104, default order: 0, min order: 0
[Wed Apr 3 16:51:46 2019] node 0: slabs: 3, objs: 117, free: 0
[Wed Apr 3 16:51:52 2019] SLUB: Unable to allocate memory on node -1 (gfp=0x24000c0)
[Wed Apr 3 16:51:52 2019] cache: proc_inode_cache(39782:b3f1519a8be9ff055106f274e74f7b563fd21125bc43c03f71f9d99e46df036f), object size: 616, buffer size: 624, default order: 3, min order: 0
[Wed Apr 3 16:51:52 2019] node 0: slabs: 72, objs: 3744, free: 0
[Wed Apr 3 16:51:52 2019] SLUB: Unable to allocate memory on node -1 (gfp=0x24000c0)
[Wed Apr 3 16:51:52 2019] cache: proc_inode_cache(39782:b3f1519a8be9ff055106f274e74f7b563fd21125bc43c03f71f9d99e46df036f), object size: 616, buffer size: 624, default order: 3, min order: 0
[Wed Apr 3 16:51:52 2019] node 0: slabs: 72, objs: 3744, free: 0
[Wed Apr 3 16:51:52 2019] SLUB: Unable to allocate memory on node -1 (gfp=0x24000c0)
[Wed Apr 3 16:51:52 2019] cache: proc_inode_cache(39782:b3f1519a8be9ff055106f274e74f7b563fd21125bc43c03f71f9d99e46df036f), object size: 616, buffer size: 624, default order: 3, min order: 0
[Wed Apr 3 16:51:52 2019] node 0: slabs: 72, objs: 3744, free: 0
[Wed Apr 3 16:51:52 2019] SLUB: Unable to allocate memory on node -1 (gfp=0x24000c0)
[Wed Apr 3 16:51:52 2019] cache: proc_inode_cache(39782:b3f1519a8be9ff055106f274e74f7b563fd21125bc43c03f71f9d99e46df036f), object size: 616, buffer size: 624, default order: 3, min order: 0
[Wed Apr 3 16:51:52 2019] node 0: slabs: 72, objs: 3744, free: 0
[Wed Apr 3 16:51:52 2019] SLUB: Unable to allocate memory on node -1 (gfp=0x24000c0)
[Wed Apr 3 16:51:52 2019] cache: proc_inode_cache(39782:b3f1519a8be9ff055106f274e74f7b563fd21125bc43c03f71f9d99e46df036f), object size: 616, buffer size: 624, default order: 3, min order: 0
[Wed Apr 3 16:51:52 2019] node 0: slabs: 72, objs: 3744, free: 0
[Wed Apr 3 16:51:52 2019] SLUB: Unable to allocate memory on node -1 (gfp=0x24000c0)
[Wed Apr 3 16:51:52 2019] cache: proc_inode_cache(39782:b3f1519a8be9ff055106f274e74f7b563fd21125bc43c03f71f9d99e46df036f), object size: 616, buffer size: 624, default order: 3, min order: 0
[Wed Apr 3 16:51:52 2019] node 0: slabs: 72, objs: 3744, free: 0
[Wed Apr 3 16:51:52 2019] SLUB: Unable to allocate memory on node -1 (gfp=0x24000c0)
[Wed Apr 3 16:51:52 2019] cache: proc_inode_cache(39782:b3f1519a8be9ff055106f274e74f7b563fd21125bc43c03f71f9d99e46df036f), object size: 616, buffer size: 624, default order: 3, min order: 0
[Wed Apr 3 16:51:52 2019] node 0: slabs: 72, objs: 3744, free: 0
[Wed Apr 3 16:51:52 2019] SLUB: Unable to allocate memory on node -1 (gfp=0x24000c0)
[Wed Apr 3 16:51:52 2019] cache: proc_inode_cache(39782:b3f1519a8be9ff055106f274e74f7b563fd21125bc43c03f71f9d99e46df036f), object size: 616, buffer size: 624, default order: 3, min order: 0
[Wed Apr 3 16:51:52 2019] node 0: slabs: 72, objs: 3744, free: 0
[Wed Apr 3 16:51:52 2019] SLUB: Unable to allocate memory on node -1 (gfp=0x24000c0)
[Wed Apr 3 16:51:52 2019] cache: proc_inode_cache(39782:b3f1519a8be9ff055106f274e74f7b563fd21125bc43c03f71f9d99e46df036f), object size: 616, buffer size: 624, default order: 3, min order: 0
[Wed Apr 3 16:51:52 2019] node 0: slabs: 72, objs: 3744, free: 0
[Wed Apr 3 16:51:52 2019] SLUB: Unable to allocate memory on node -1 (gfp=0x24000c0)
[Wed Apr 3 16:51:52 2019] cache: proc_inode_cache(39782:b3f1519a8be9ff055106f274e74f7b563fd21125bc43c03f71f9d99e46df036f), object size: 616, buffer size: 624, default order: 3, min order: 0
[Wed Apr 3 16:51:52 2019] node 0: slabs: 72, objs: 3744, free: 0
[Wed Apr 3 16:51:53 2019] node invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=-998
[Wed Apr 3 16:51:53 2019] node cpuset=b3f1519a8be9ff055106f274e74f7b563fd21125bc43c03f71f9d99e46df036f mems_allowed=0
[Wed Apr 3 16:51:53 2019] CPU: 8 PID: 122117 Comm: node Tainted: G OE 4.4.0-1055-aws #64-Ubuntu
[Wed Apr 3 16:51:53 2019] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
[Wed Apr 3 16:51:53 2019] 0000000000000286 614007a17272cf61 ffff880111193c88 ffffffff813fe183
[Wed Apr 3 16:51:53 2019] ffff880111193d68 ffff880eb9728000 ffff880111193cf8 ffffffff8120e34a
[Wed Apr 3 16:51:53 2019] ffffffff81cc87c6 ffff880c927a6c00 ffffffff81e6b120 0000000000000206
[Wed Apr 3 16:51:53 2019] Call Trace:
[Wed Apr 3 16:51:53 2019] [<ffffffff813fe183>] dump_stack+0x63/0x90
[Wed Apr 3 16:51:53 2019] [<ffffffff8120e34a>] dump_header+0x5a/0x1c5
[Wed Apr 3 16:51:53 2019] [<ffffffff81194ed2>] oom_kill_process+0x202/0x3c0
[Wed Apr 3 16:51:53 2019] [<ffffffff81202384>] ? mem_cgroup_iter+0x204/0x3a0
[Wed Apr 3 16:51:53 2019] [<ffffffff812043f3>] mem_cgroup_out_of_memory+0x2b3/0x300
[Wed Apr 3 16:51:53 2019] [<ffffffff812051dd>] mem_cgroup_oom_synchronize+0x33d/0x350
[Wed Apr 3 16:51:53 2019] [<ffffffff81200230>] ? kzalloc_node.constprop.49+0x20/0x20
[Wed Apr 3 16:51:53 2019] [<ffffffff81195584>] pagefault_out_of_memory+0x44/0xc0
[Wed Apr 3 16:51:53 2019] [<ffffffff8106c512>] mm_fault_error+0x82/0x160
[Wed Apr 3 16:51:53 2019] [<ffffffff8106c9d9>] __do_page_fault+0x3e9/0x410
[Wed Apr 3 16:51:53 2019] [<ffffffff8106ca22>] do_page_fault+0x22/0x30
[Wed Apr 3 16:51:53 2019] [<ffffffff81826858>] page_fault+0x28/0x30
[Wed Apr 3 16:51:53 2019] Task in /kubepods/pod70849c83-5081-11e9-a4f4-0606b708822a/b3f1519a8be9ff055106f274e74f7b563fd21125bc43c03f71f9d99e46df036f killed as a result of limit of /kubepods/pod70849c83-5081-11e9-a4f4-0606b708822a
[Wed Apr 3 16:51:53 2019] memory: usage 1024000kB, limit 1024000kB, failcnt 7097
[Wed Apr 3 16:51:53 2019] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
[Wed Apr 3 16:51:53 2019] kmem: usage 25356kB, limit 9007199254740988kB, failcnt 0
[Wed Apr 3 16:51:53 2019] Memory cgroup stats for /kubepods/pod70849c83-5081-11e9-a4f4-0606b708822a: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_fi
le:0KB unevictable:0KB
[Wed Apr 3 16:51:53 2019] Memory cgroup stats for /kubepods/pod70849c83-5081-11e9-a4f4-0606b708822a/f757a788d85b1b04b1bc5b7d0ed3596a928dcf3cb8c56242112fdb16849d537f: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB dirty:0KB writeback:
0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
[Wed Apr 3 16:51:53 2019] Memory cgroup stats for /kubepods/pod70849c83-5081-11e9-a4f4-0606b708822a/b3f1519a8be9ff055106f274e74f7b563fd21125bc43c03f71f9d99e46df036f: cache:184KB rss:998460KB rss_huge:98304KB mapped_file:0KB dirty:0KB
writeback:0KB inactive_anon:0KB active_anon:998460KB inactive_file:184KB active_file:0KB unevictable:0KB
[Wed Apr 3 16:51:53 2019] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[Wed Apr 3 16:51:53 2019] [122062] 1000 122062 541103 251453 1435 847 0 -998 node
[Wed Apr 3 16:51:53 2019] Memory cgroup out of memory: Kill process 122062 (node) score 0 or sacrifice child
[Wed Apr 3 16:51:53 2019] Killed process 122062 (node) total-vm:2164412kB, anon-rss:998248kB, file-rss:7564kB
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment