dcos api task killed

gistfile1.txt

ep 20 12:46:56 ip-10-0-2-214 kernel: java invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Sep 20 12:46:56 ip-10-0-2-214 kernel: java cpuset=6437d40658683c578e42a3603496173478dfed37ab042331d89f4bf8a31a1338 mems_allowed=0
Sep 20 12:46:56 ip-10-0-2-214 kernel: CPU: 2 PID: 25620 Comm: java Kdump: loaded Tainted: G               ------------ T 3.10.0-862.11.6.el7.x86_64 #1
Sep 20 12:46:56 ip-10-0-2-214 kernel: Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
Sep 20 12:46:56 ip-10-0-2-214 kernel: Call Trace:
Sep 20 12:46:56 ip-10-0-2-214 kernel:  [<ffffffffaef135d4>] dump_stack+0x19/0x1b
Sep 20 12:46:56 ip-10-0-2-214 kernel:  [<ffffffffaef0e79f>] dump_header+0x90/0x229
Sep 20 12:46:56 ip-10-0-2-214 kernel:  [<ffffffffae99a7b6>] ? find_lock_task_mm+0x56/0xc0
Sep 20 12:46:56 ip-10-0-2-214 kernel:  [<ffffffffae99ac64>] oom_kill_process+0x254/0x3d0
Sep 20 12:46:56 ip-10-0-2-214 kernel:  [<ffffffffaea133c6>] mem_cgroup_oom_synchronize+0x546/0x570
Sep 20 12:46:56 ip-10-0-2-214 kernel:  [<ffffffffaea12840>] ? mem_cgroup_charge_common+0xc0/0xc0
Sep 20 12:46:56 ip-10-0-2-214 dcos-metrics[12075]: time="2018-09-20T12:46:56Z" level=info msg="Finished polling DC/OS host 10.0.2.214, took 0.006299 seconds." collector=node
Sep 20 12:46:56 ip-10-0-2-214 kernel:  [<ffffffffae99b4f4>] pagefault_out_of_memory+0x14/0x90
Sep 20 12:46:56 ip-10-0-2-214 kernel:  [<ffffffffaef0c941>] mm_fault_error+0x6a/0x157
Sep 20 12:46:56 ip-10-0-2-214 kernel:  [<ffffffffaef20856>] __do_page_fault+0x496/0x4f0
Sep 20 12:46:56 ip-10-0-2-214 kernel:  [<ffffffffaef208e5>] do_page_fault+0x35/0x90
Sep 20 12:46:56 ip-10-0-2-214 kernel:  [<ffffffffaef1c758>] page_fault+0x28/0x30
Sep 20 12:46:56 ip-10-0-2-214 kernel: Task in /docker/6437d40658683c578e42a3603496173478dfed37ab042331d89f4bf8a31a1338 killed as a result of limit of /docker/6437d40658683c578e42a3603496173478dfed37ab042331d89f4bf8a31a1338
Sep 20 12:46:56 ip-10-0-2-214 kernel: memory: usage 3145728kB, limit 3145728kB, failcnt 2398
Sep 20 12:46:56 ip-10-0-2-214 kernel: memory+swap: usage 3145728kB, limit 6291456kB, failcnt 0
Sep 20 12:46:56 ip-10-0-2-214 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
Sep 20 12:46:56 ip-10-0-2-214 kernel: Memory cgroup stats for /docker/6437d40658683c578e42a3603496173478dfed37ab042331d89f4bf8a31a1338: cache:1576KB rss:3144152KB rss_huge:2951168KB mapped_file:1572KB swap:0KB inactive_anon:1572KB active_anon:3144152KB inactive_file:4KB active_file:0KB unevictable:0KB
Sep 20 12:46:56 ip-10-0-2-214 kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
Sep 20 12:46:56 ip-10-0-2-214 kernel: [ 7402]     0  7402     1109      164       8        0             0 run-uberjar
Sep 20 12:46:56 ip-10-0-2-214 kernel: [ 7421]     0  7421  1731048   788633    1743        0             0 java
Sep 20 12:46:56 ip-10-0-2-214 kernel: [ 7422]     0  7422     1089       90       8        0             0 cat
Sep 20 12:46:56 ip-10-0-2-214 kernel: Memory cgroup out of memory: Kill process 25620 (java) score 1005 or sacrifice child
Sep 20 12:46:56 ip-10-0-2-214 kernel: Killed process 7421 (java) total-vm:6924192kB, anon-rss:3129444kB, file-rss:23516kB, shmem-rss:1572kB
Sep 20 12:46:56 ip-10-0-2-214 mesos-agent[12078]: I0920 12:46:56.496966 12102 slave.cpp:4392] Handling status update TASK_FINISHED (UUID: 5fce164c-48e6-4595-9e91-f911511ba665) for task yapster-2-3-0-api.aee56cf3-bc40-11e8-8373-c29f32394cba of framework d2d19cf0-832e-4d92-af13-fe1d3919811d-0001 from executor(1)@10.0.2.214:39865
Sep 20 12:46:56 ip-10-0-2-214 mesos-agent[12078]: E0920 12:46:56.497354 12097 slave.cpp:4714] Failed to update resources for container 0e6d1257-da40-4528-9716-ba41154b8de1 of executor 'yapster-2-3-0-api.aee56cf3-bc40-11e8-8373-c29f32394cba' running task yapster-2-3-0-api.aee56cf3-bc40-11e8-8373-c29f32394cba on status update for terminal task, destroying container: Failed to determine cgroup for the 'cpu' subsystem: Failed to read /proc/7402/cgroup: Failed to open file: No such file or directory
Sep 20 12:46:56 ip-10-0-2-214 mesos-agent[12078]: I0920 12:46:56.497467 12097 status_update_manager.cpp:323] Received status update TASK_FINISHED (UUID: 5fce164c-48e6-4595-9e91-f911511ba665) for task yapster-2-3-0-api.aee56cf3-bc40-11e8-8373-c29f32394cba of framework d2d19cf0-832e-4d92-af13-fe1d3919811d-0001
Sep 20 12:46:56 ip-10-0-2-214 mesos-agent[12078]: I0920 12:46:56.497505 12097 status_update_manager.cpp:834] Checkpointing UPDATE for status update TASK_FINISHED (UUID: 5fce164c-48e6-4595-9e91-f911511ba665) for task yapster-2-3-0-api.aee56cf3-bc40-11e8-8373-c29f32394cba of framework d2d19cf0-832e-4d92-af13-fe1d3919811d-0001
Sep 20 12:46:56 ip-10-0-2-214 mesos-agent[12078]: I0920 12:46:56.497676 12097 docker.cpp:2218] Destroying container 0e6d1257-da40-4528-9716-ba41154b8de1 in RUNNING state
Sep 20 12:46:56 ip-10-0-2-214 mesos-agent[12078]: I0920 12:46:56.497714 12103 slave.cpp:4873] Forwarding the update TASK_FINISHED (UUID: 5fce164c-48e6-4595-9e91-f911511ba665) for task yapster-2-3-0-api.aee56cf3-bc40-11e8-8373-c29f32394cba of framework d2d19cf0-832e-4d92-af13-fe1d3919811d-0001 to master@10.0.1.133:5050
Sep 20 12:46:56 ip-10-0-2-214 mesos-agent[12078]: I0920 12:46:56.497850 12103 slave.cpp:4783] Sending acknowledgement for status update TASK_FINISHED (UUID: 5fce164c-48e6-4595-9e91-f911511ba665) for task yapster-2-3-0-api.aee56cf3-bc40-11e8-8373-c29f32394cba of framework d2d19cf0-832e-4d92-af13-fe1d3919811d-0001 to executor(1)@10.0.2.214:39865
Sep 20 12:46:56 ip-10-0-2-214 mesos-agent[12078]: I0920 12:46:56.497732 12097 docker.cpp:2223] Sending SIGTERM to executor with pid: 7343
Sep 20 12:46:56 ip-10-0-2-214 mesos-agent[12078]: I0920 12:46:56.509413 12102 status_update_manager.cpp:395] Received status update acknowledgement (UUID: 5fce164c-48e6-4595-9e91-f911511ba665) for task yapster-2-3-0-api.aee56cf3-bc40-11e8-8373-c29f32394cba of framework d2d19cf0-832e-4d92-af13-fe1d3919811d-0001
Sep 20 12:46:56 ip-10-0-2-214 mesos-agent[12078]: I0920 12:46:56.509485 12102 status_update_manager.cpp:834] Checkpointing ACK for status update TASK_FINISHED (UUID: 5fce164c-48e6-4595-9e91-f911511ba665) for task yapster-2-3-0-api.aee56cf3-bc40-11e8-8373-c29f32394cba of framework d2d19cf0-832e-4d92-af13-fe1d3919811d-0001
Sep 20 12:46:56 ip-10-0-2-214 mesos-agent[12078]: I0920 12:46:56.510433 12097 docker.cpp:2268] Running docker stop on container 0e6d1257-da40-4528-9716-ba41154b8de1
Sep 20 12:46:56 ip-10-0-2-214 mesos-agent[12078]: I0920 12:46:56.514318 12101 slave.cpp:5005] Got exited event for executor(1)@10.0.2.214:39865
Sep 20 12:46:56 ip-10-0-2-214 mesos-agent[12078]: I0920 12:46:56.536550 12097 docker.cpp:2447] Executor for container 0e6d1257-da40-4528-9716-ba41154b8de1 has exited
Sep 20 12:46:56 ip-10-0-2-214 mesos-agent[12078]: I0920 12:46:56.537230 12096 slave.cpp:5405] Executor 'yapster-2-3-0-api.aee56cf3-bc40-11e8-8373-c29f32394cba' of framework d2d19cf0-832e-4d92-af13-fe1d3919811d-0001 terminated with signal Terminated
Sep 20 12:46:56 ip-10-0-2-214 mesos-agent[12078]: I0920 12:46:56.537277 12096 slave.cpp:5509] Cleaning up executor 'yapster-2-3-0-api.aee56cf3-bc40-11e8-8373-c29f32394cba' of framework d2d19cf0-832e-4d92-af13-fe1d3919811d-0001 at executor(1)@10.0.2.214:39865
Sep 20 12:46:56 ip-10-0-2-214 mesos-agent[12078]: I0920 12:46:56.537503 12102 gc.cpp:58] Scheduling '/var/lib/mesos/slave/slaves/d2d19cf0-832e-4d92-af13-fe1d3919811d-S11/frameworks/d2d19cf0-832e-4d92-af13-fe1d3919811d-0001/executors/yapster-2-3-0-api.aee56cf3-bc40-11e8-8373-c29f32394cba/runs/0e6d1257-da40-4528-9716-ba41154b8de1' for gc 1.99999377973333days in the future
Sep 20 12:46:56 ip-10-0-2-214 mesos-agent[12078]: I0920 12:46:56.537559 12102 gc.cpp:58] Scheduling '/var/lib/mesos/slave/slaves/d2d19cf0-832e-4d92-af13-fe1d3919811d-S11/frameworks/d2d19cf0-832e-4d92-af13-fe1d3919811d-0001/executors/yapster-2-3-0-api.aee56cf3-bc40-11e8-8373-c29f32394cba' for gc 1.99999377887704days in the future
Sep 20 12:46:56 ip-10-0-2-214 mesos-agent[12078]: I0920 12:46:56.537595 12102 gc.cpp:58] Scheduling '/var/lib/mesos/slave/meta/slaves/d2d19cf0-832e-4d92-af13-fe1d3919811d-S11/frameworks/d2d19cf0-832e-4d92-af13-fe1d3919811d-0001/executors/yapster-2-3-0-api.aee56cf3-bc40-11e8-8373-c29f32394cba/runs/0e6d1257-da40-4528-9716-ba41154b8de1' for gc 1.99999377847111days in the future
Sep 20 12:46:56 ip-10-0-2-214 mesos-agent[12078]: I0920 12:46:56.537627 12102 gc.cpp:58] Scheduling '/var/lib/mesos/slave/meta/slaves/d2d19cf0-832e-4d92-af13-fe1d3919811d-S11/frameworks/d2d19cf0-832e-4d92-af13-fe1d3919811d-0001/executors/yapster-2-3-0-api.aee56cf3-bc40-11e8-8373-c29f32394cba' for gc 1.99999377817185days in the future