Skip to content

Instantly share code, notes, and snippets.

@pencilcheck
Created August 24, 2015 22:42
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save pencilcheck/4d0700869fdd435e4593 to your computer and use it in GitHub Desktop.
Save pencilcheck/4d0700869fdd435e4593 to your computer and use it in GitHub Desktop.
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.828647] ffffffff815423e7 ffff880100000000 ffff88000837bc98 ffff8800bb642ef8
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.829538] 0000000000000001 0000000000000001 ffff88012fc12ec0 ffff8800bb6429e0
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.830676] Call Trace:
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.831078] [<ffffffff81544e1f>] ? dump_stack+0x41/0x51
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.831764] [<ffffffff815423e7>] ? dump_header+0x76/0x1ec
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.832577] [<ffffffff8115084c>] ? find_lock_task_mm+0x4c/0xa0
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.833420] [<ffffffff81150d5a>] ? oom_kill_process+0x28a/0x3e0
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.834177] [<ffffffff810f154c>] ? css_next_descendant_pre+0x1c/0x60
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.835091] [<ffffffff811b109b>] ? mem_cgroup_iter+0x13b/0x2f0
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.835829] [<ffffffff811b4f6f>] ? mem_cgroup_oom_synchronize+0x50f/0x580
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.836788] [<ffffffff811b4450>] ? mem_cgroup_try_charge_mm+0xb0/0xb0
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.837628] [<ffffffff811515e0>] ? pagefault_out_of_memory+0x10/0x80
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.838566] [<ffffffff8105d1a6>] ? __do_page_fault+0x536/0x540
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.839283] [<ffffffff811bdd24>] ? vfs_read+0x104/0x180
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.839916] [<ffffffff8154d428>] ? page_fault+0x28/0x30
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.840678] Task in /71ee1f503cec5f0baf4fa0508a2baa39513af7b4baba62a5dc2cb7a14424268c killed as a result of limit of /71ee1f503cec5f0baf4fa0508a2baa39513af7b4baba62a5dc2cb7a14424268c
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.842847] memory: usage 204800kB, limit 204800kB, failcnt 107693117
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.843725] memory+swap: usage 0kB, limit 18014398509481983kB, failcnt 0
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.844699] kmem: usage 0kB, limit 18014398509481983kB, failcnt 0
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.845701] Memory cgroup stats for /71ee1f503cec5f0baf4fa0508a2baa39513af7b4baba62a5dc2cb7a14424268c: cache:108KB rss:204692KB rss_huge:0KB mapped_file:0KB writeback:0KB inactive_anon:0KB active_anon:204792KB inactive_file:0KB active_file:0KB unevictable:0KB
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.859817] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.860764] [19624] 0 19624 1112 196 7 0 0 sh
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.861869] [19632] 0 19632 22910 6040 47 0 0 google-fluentd
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.862929] [29159] 0 29159 103689 49468 144 0 0 google-fluentd
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.864019] Memory cgroup out of memory: Kill process 29159 (google-fluentd) score 968 or sacrifice child
Aug 24 22:40:22 gke-demo-7140bbcf-node-xzxu kernel: [1646426.865143] Killed process 29159 (google-fluentd) total-vm:414756kB, anon-rss:192300kB, file-rss:5572kB
[1646435.236356] google-fluentd invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
[1646435.237640] google-fluentd cpuset=71ee1f503cec5f0baf4fa0508a2baa39513af7b4baba62a5dc2cb7a14424268c mems_allowed=0
[1646435.239051] CPU: 0 PID: 29412 Comm: google-fluentd Tainted: G C 3.16.0-0.bpo.4-amd64 #1 Debian 3.16.7-ckt9-3~deb8u1~bpo70+1
[1646435.240643] Hardware name: Google Google, BIOS Google 01/01/2011
[1646435.241423] 0000000000000000 ffff8801271bf800 ffffffff81544e1f ffff880008004150
[1646435.242568] ffffffff815423e7 ffff880100000000 ffff88000916bc98 ffff880008004668
[1646435.243545] 0000000000000001 0000000000000001 ffff88012fc12ec0 ffff880008004150
[1646435.244518] Call Trace:
[1646435.244842] [<ffffffff81544e1f>] ? dump_stack+0x41/0x51
[1646435.245488] [<ffffffff815423e7>] ? dump_header+0x76/0x1ec
[1646435.246247] [<ffffffff8115084c>] ? find_lock_task_mm+0x4c/0xa0
[1646435.246956] [<ffffffff81150d5a>] ? oom_kill_process+0x28a/0x3e0
[1646435.247808] [<ffffffff810f154c>] ? css_next_descendant_pre+0x1c/0x60
[1646435.248633] [<ffffffff811b109b>] ? mem_cgroup_iter+0x13b/0x2f0
[1646435.249430] [<ffffffff811b4f6f>] ? mem_cgroup_oom_synchronize+0x50f/0x580
[1646435.250274] [<ffffffff811c9339>] ? path_lookupat+0xe9/0x770
[1646435.251060] [<ffffffff811b4450>] ? mem_cgroup_try_charge_mm+0xb0/0xb0
[1646435.251840] [<ffffffff811515e0>] ? pagefault_out_of_memory+0x10/0x80
[1646435.252610] [<ffffffff8105d1a6>] ? __do_page_fault+0x536/0x540
[1646435.253357] [<ffffffff812d7b10>] ? lockref_put_or_lock+0x50/0x80
[1646435.254228] [<ffffffff811d3ce1>] ? dput+0x31/0x1a0
[1646435.254905] [<ffffffff811fcbba>] ? SyS_inotify_add_watch+0x1ca/0x380
[1646435.255735] [<ffffffff8154d428>] ? page_fault+0x28/0x30
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.236356] google-fluentd invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.237640] google-fluentd cpuset=71ee1f503cec5f0baf4fa0508a2baa39513af7b4baba62a5dc2cb7a14424268c mems_allowed=0
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.239051] CPU: 0 PID: 29412 Comm: google-fluentd Tainted: G C 3.16.0-0.bpo.4-amd64 #1 Debian 3.16.7-ckt9-3~deb8u1~bpo70+1
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.240643] Hardware name[1646435.265270] Task in : Google Google, BIOS Google 01/01/2011
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.241423] 0000000000000000 ffff8801271bf800 ffffffff81544e1f ffff880008004150
Aug 24 22:40/71ee1f503cec5f0baf4fa0508a2baa39513af7b4baba62a5dc2cb7a14424268c:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.242568] ffffffff815423e7 ffff880100000000 ffff88000916bc98 ffff880008004668
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.243545] 0000000000000001 0000000000000001 ffff88012fc12ec0 ffff880008004150
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.244518] Call Trace:
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.244842] [<ffffffff81544e1f>] ? dump_stack+0x41/0x51
A killed as a result of limit of ug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.245488] [<ffffffff815423e7>] ? dump_header+0x76/0x1ec
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435/71ee1f503cec5f0baf4fa0508a2baa39513af7b4baba62a5dc2cb7a14424268c.246247] [<ffffffff8115084c>] ?
find_lock_task_mm+0x4c/0xa0
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.246956] [<ffffffff81150d5a>] ? oom_kill_process+0x28a/0x3e0
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.247808] [<ffffffff810f154c>] ? css_next_descendant_pre+0x1c/0x60
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.248633] [<ffffffff811b109b>] ? mem_cgroup_iter+0x13b/0x2f0
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.249430] [<ffffffff811b4f6f>] ? mem_cgroup_oom_synchronize+0x50f/0x580
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.250274] [<ffffffff811c9339>] ? path_lookupat+0xe9/0x770
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.251060] [<ffffffff811b4450>] ? mem_cgroup_try_charge_mm+0xb0/0xb0
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.251840] [<ffffffff811515e0>] ? pagefault_out_of_memory+0x10/0x80
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.252610] [<ffffffff8105d1a6>] ? __do_page_fault+0x536/0x540
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.253357] [<ffffffff812d7b10>] ? lockref_put_or_lock+0x50/0x80
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.254228] [<ffffffff811d3ce1>] ? dput+0x31/0x1a0[1646435.293178] memory: usage 204800kB, limit 204800kB, failcnt 107693348
[1646435.293989] memory+swap: usage 0kB, limit 18014398509481983kB, failcnt 0
[1646435.294844] kmem: usage 0kB, limit 18014398509481983kB, failcnt 0
[1646435.295649] Memory cgroup stats for /71ee1f503cec5f0baf4fa0508a2baa39513af7b4baba62a5dc2cb7a14424268c:
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.254905] [<ffffffff811fcbba>] ? SyS_inotify_add_watch+0x1ca/0x380
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.255735] [<ffffffff8154d428>] ? page_fault+0x28/0x30
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.265270] Task in /71ee1f503cec5f0baf4fa0508a2baa39513af7b4baba62a5dc2cb7a14424268c killed as a result of limit of /71ee1f503cec5f0baf4fa0508a2baa39513af7b4baba62a5dc2cb7a14424268c
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.293178] memory: usage 204800kB, limit 204800kB, failcnt 107693348
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.293989] memory+swap: usage 0kB, limit 18014398509481983kB, failcnt 0
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.294844] kmem: usage 0kB, limit 18014398509481983kB, failcnt 0
cache:8904KB rss:195896KB rss_huge:0KB mapped_file:0KB writeback:3080KB inactive_anon:0KB active_anon:195996KB inactive_file:4476KB active_file:4328KB unevictable:0KB
[1646435.313664] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[1646435.314703] [19624] 0 19624 1112 196 7 0 0 sh
[1646435.315749] [19632] 0 19632 22910 6040 47 0 0 google-fluentd
[1646435.316976] [29412] 0 29412 101513 47342 140 0 0 google-fluentd
[1646435.318200] Memory cgroup out of memory: Kill process 29412 (google-fluentd) score 927 or sacrifice child
[1646435.319501] Killed process 29412 (google-fluentd) total-vm:406052kB, anon-rss:183720kB, file-rss:5648kB
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.295649] Memory cgroup stats for /71ee1f503cec5f0baf4fa0508a2baa39513af7b4baba62a5dc2cb7a14424268c: cache:8904KB rss:195896KB rss_huge:0KB mapped_file:0KB writeback:3080KB inactive_anon:0KB active_anon:195996KB inactive_file:4476KB active_file:4328KB unevictable:0KB
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.313664] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.314703] [19624] 0 19624 1112 196 7 0 0 sh
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.315749] [19632] 0 19632 22910 6040 47 0 0 google-fluentd
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.316976] [29412] 0 29412 101513 47342 140 0 0 google-fluentd
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.318200] Memory cgroup out of memory: Kill process 29412 (google-fluentd) score 927 or sacrifice child
Aug 24 22:40:31 gke-demo-7140bbcf-node-xzxu kernel: [1646435.319501] Killed process 29412 (google-fluentd) total-vm:406052kB, anon-rss:183720kB, file-rss:5648kB
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment