Skip to content

Instantly share code, notes, and snippets.

@kirrie

kirrie/oom killed

Created January 24, 2019 01:14
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save kirrie/1be66e3580be4e4cde0d1866c39a29cc to your computer and use it in GitHub Desktop.
Save kirrie/1be66e3580be4e4cde0d1866c39a29cc to your computer and use it in GitHub Desktop.
Download ZIP
Raw
oom killed
Jan 24 06:09:00 ip-172-31-6-229 systemd[1]: Starting Clean php session files...
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540764] php7.1 invoked oom-killer: gfp_mask=0x24200ca, order=0, oom_score_adj=0
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540767] php7.1 cpuset=/ mems_allowed=0
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540773] CPU: 0 PID: 32751 Comm: php7.1 Not tainted 4.4.0-1060-aws #69-Ubuntu
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540774] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540776] 0000000000000286 9bfaa3ce8896ec31 ffff88001a403848 ffffffff813fd3f3
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540779] ffff88001a403a00 ffff880000810dc0 ffff88001a4038b8 ffffffff8120d4fa
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540781] ffffffff81cc63b9 0000000000000000 ffffffff81e6b120 0000000000000206
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540783] Call Trace:
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540790] [<ffffffff813fd3f3>] dump_stack+0x63/0x90
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540795] [<ffffffff8120d4fa>] dump_header+0x5a/0x1c5
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540798] [<ffffffff81194022>] oom_kill_process+0x202/0x3c0
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540800] [<ffffffff81194449>] out_of_memory+0x219/0x460
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540804] [<ffffffff8119a495>] __alloc_pages_slowpath.constprop.88+0x965/0xb00
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540806] [<ffffffff8119a8b8>] __alloc_pages_nodemask+0x288/0x2a0
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540809] [<ffffffff811e5d2d>] alloc_pages_vma+0xad/0x250
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540813] [<ffffffff811a9ef0>] shmem_alloc_page+0x70/0xc0
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540817] [<ffffffff810cd821>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540820] [<ffffffff814030d2>] ? radix_tree_lookup_slot+0x22/0x50
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540823] [<ffffffff811ac9f5>] shmem_getpage_gfp+0x2c5/0x830
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540825] [<ffffffff811ae4ce>] shmem_fault+0x6e/0x1e0
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540830] [<ffffffff81337c34>] shm_fault+0x24/0x30
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540832] [<ffffffff811bf7d6>] __do_fault+0x56/0xf0
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540834] [<ffffffff811c2a3e>] handle_mm_fault+0x41e/0x1820
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540836] [<ffffffff810cd821>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540840] [<ffffffff810ac2c6>] ? finish_task_switch+0x76/0x230
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540844] [<ffffffff8181fc71>] ? __schedule+0x341/0x7f0
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540847] [<ffffffff8106c891>] __do_page_fault+0x1a1/0x410
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540849] [<ffffffff8106cb22>] do_page_fault+0x22/0x30
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540852] [<ffffffff81826cd8>] page_fault+0x28/0x30
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540853] Mem-Info:
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540859] active_anon:98329 inactive_anon:8007 isolated_anon:0
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540859] active_file:59 inactive_file:57 isolated_file:0
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540859] unevictable:913 dirty:1 writeback:0 unstable:0
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540859] slab_reclaimable:4332 slab_unreclaimable:5772
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540859] mapped:14858 shmem:21006 pagetables:1711 bounce:0
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540859] free:1126 free_pcp:30 free_cma:0
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540863] Node 0 DMA free:1900kB min:88kB low:108kB high:132kB active_anon:9376kB inactive_anon:1348kB active_file:24kB inactive_file:16kB unevictable:108kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15900kB mlocked:108kB dirty:0kB writeback:0kB mapped:2268kB shmem:2692kB slab_reclaimable:1224kB slab_unreclaimable:1016kB kernel_stack:128kB pagetables:228kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:268 all_unreclaimable? yes
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540868] lowmem_reserve[]: 0 455 455 455 455
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540871] Node 0 DMA32 free:2604kB min:2684kB low:3352kB high:4024kB active_anon:383940kB inactive_anon:30680kB active_file:212kB inactive_file:212kB unevictable:3544kB isolated(anon):0kB isolated(file):0kB present:507904kB managed:482536kB mlocked:3544kB dirty:4kB writeback:0kB mapped:57164kB shmem:81332kB slab_reclaimable:16104kB slab_unreclaimable:22072kB kernel_stack:2960kB pagetables:6616kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:2548 all_unreclaimable? yes
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540876] lowmem_reserve[]: 0 0 0 0 0
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540878] Node 0 DMA: 47*4kB (UME) 65*8kB (ME) 39*16kB (UME) 18*32kB (UM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1908kB
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540886] Node 0 DMA32: 115*4kB (UE) 118*8kB (UME) 69*16kB (UME) 3*32kB (UME) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2604kB
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540894] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540895] 21735 total pagecache pages
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540896] 0 pages in swap cache
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540898] Swap cache stats: add 0, delete 0, find 0/0
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540899] Free swap = 0kB
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540900] Total swap = 0kB
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540901] 130973 pages RAM
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540902] 0 pages HighMem/MovableOnly
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540902] 6364 pages reserved
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540903] 0 pages cma reserved
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540904] 0 pages hwpoisoned
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540905] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540909] [ 392] 0 392 9080 647 19 3 0 0 systemd-journal
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540912] [ 430] 0 430 25743 58 16 3 0 0 lvmetad
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540914] [ 890] 0 890 4031 215 12 3 0 0 dhclient
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540916] [ 1093] 0 1093 6512 52 19 3 0 0 atd
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540918] [ 1108] 0 1108 55572 2306 68 3 0 0 php-fpm5.6
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540920] [ 1115] 0 1115 7156 84 18 3 0 0 systemd-logind
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540922] [ 1120] 0 1120 68623 193 37 3 0 0 accounts-daemon
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540924] [ 1123] 104 1123 65159 439 29 4 0 0 rsyslogd
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540925] [ 1126] 0 1126 6933 71 19 3 0 0 cron
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540928] [ 1145] 107 1145 10754 155 25 3 0 -900 dbus-daemon
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540929] [ 1165] 0 1165 71558 5098 75 4 0 0 php-fpm7.1
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540931] [ 1168] 0 1168 1101 23 8 3 0 0 acpid
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540933] [ 1172] 0 1172 161035 296 33 4 0 0 lxcfs
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540935] [ 1180] 0 1180 6012 91 17 3 0 0 vsftpd
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540937] [ 1295] 33 1295 71652 1156 64 4 0 0 php-fpm7.1
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540939] [ 1296] 33 1296 71558 764 58 4 0 0 php-fpm7.1
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540941] [ 1307] 0 1307 3666 34 12 3 0 0 agetty
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540943] [ 1310] 0 1310 3620 37 12 3 0 0 agetty
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540945] [29238] 0 29238 3343 36 11 3 0 0 mdadm
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540947] [30145] 108 30145 6720 48 18 3 0 0 uuidd
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540949] [31020] 114 31020 27508 163 25 3 0 0 ntpd
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540951] [31165] 0 31165 1305 28 8 3 0 0 iscsid
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540953] [31166] 0 31166 1430 878 8 3 0 -17 iscsid
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540955] [31498] 0 31498 57357 2926 36 6 0 -900 snapd
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540957] [25564] 0 25564 16378 286 34 3 0 -1000 sshd
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540959] [ 6872] 0 6872 4973 487 14 3 0 0 mysqld_safe
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540961] [16895] 0 16895 41037 553 60 4 0 0 nginx
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540963] [16896] 33 16896 41282 904 62 4 0 0 nginx
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540965] [ 2466] 112 2466 149677 24069 107 3 0 0 mysqld
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540967] [ 2467] 0 2467 6203 95 16 4 0 0 logger
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540969] [ 3471] 0 3471 10530 381 23 3 0 -1000 systemd-udevd
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540971] [17747] 33 17747 66487 18759 111 3 0 0 php-fpm5.6
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540973] [23294] 33 23294 65432 17117 107 3 0 0 php-fpm5.6
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540975] [23439] 33 23439 65203 17175 106 3 0 0 php-fpm5.6
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540977] [ 7954] 0 7954 69273 173 38 3 0 0 polkitd
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540979] [31796] 0 31796 1126 174 8 3 0 0 apt.systemd.dai
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540981] [31803] 0 31803 1126 384 8 3 0 0 apt.systemd.dai
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540983] [31834] 0 31834 34399 8552 70 3 0 0 unattended-upgr
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540984] [31839] 0 31839 34399 8271 70 3 0 0 unattended-upgr
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540986] [32560] 0 32560 15967 10522 34 3 0 0 dpkg
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540988] [32593] 0 32593 1126 152 9 3 0 0 unattended-upgr
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540990] [32594] 0 32594 1126 397 8 3 0 0 invoke-rc.d
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540992] [32633] 0 32633 6223 67 15 3 0 0 systemctl
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540994] [32634] 0 32634 14649 2048 34 3 0 0 unattended-upgr
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540996] [32736] 0 32736 1126 177 8 4 0 0 sessionclean
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.540998] [32740] 0 32740 1126 24 8 4 0 0 sessionclean
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.541000] [32743] 0 32743 3632 43 10 4 0 0 sort
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.541001] [32744] 0 32744 3632 40 10 3 0 0 sort
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.541003] [32745] 0 32745 1126 24 7 4 0 0 sessionclean
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.541005] [32751] 0 32751 66445 2648 60 3 0 0 php7.1
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.541007] Out of memory: Kill process 2466 (mysqld) score 194 or sacrifice child
Jan 24 06:09:00 ip-172-31-6-229 kernel: [11712453.547784] Killed process 2466 (mysqld) total-vm:598708kB, anon-rss:96276kB, file-rss:0kB
Jan 24 06:09:00 ip-172-31-6-229 mysqld_safe: Number of processes running now: 0
Jan 24 06:09:00 ip-172-31-6-229 mysqld_safe: mysqld restarted
Jan 24 06:09:00 ip-172-31-6-229 mysqld: 190124 6:09:00 [Note] /usr/sbin/mysqld (mysqld 10.0.36-MariaDB-0ubuntu0.16.04.1) starting as process 311 ...
Jan 24 06:09:01 ip-172-31-6-229 mysqld: 190124 6:09:01 [ERROR] mysqld: Out of memory (Needed 128909312 bytes)
Jan 24 06:09:01 ip-172-31-6-229 mysqld: 190124 6:09:01 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.
Jan 24 06:09:01 ip-172-31-6-229 mysqld:
Jan 24 06:09:01 ip-172-31-6-229 mysqld: 190124 6:09:01 [Note] InnoDB: Using mutexes to ref count buffer pool pages
Jan 24 06:09:01 ip-172-31-6-229 mysqld: 190124 6:09:01 [Note] InnoDB: The InnoDB memory heap is disabled
Jan 24 06:09:01 ip-172-31-6-229 mysqld: 190124 6:09:01 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
Jan 24 06:09:01 ip-172-31-6-229 mysqld: 190124 6:09:01 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
Jan 24 06:09:01 ip-172-31-6-229 mysqld: 190124 6:09:01 [Note] InnoDB: Compressed tables use zlib 1.2.8
Jan 24 06:09:01 ip-172-31-6-229 mysqld: 190124 6:09:01 [Note] InnoDB: Using Linux native AIO
Jan 24 06:09:01 ip-172-31-6-229 mysqld: 190124 6:09:01 [Note] InnoDB: Using CPU crc32 instructions
Jan 24 06:09:01 ip-172-31-6-229 mysqld: 190124 6:09:01 [Note] InnoDB: Initializing buffer pool, size = 128.0M
Jan 24 06:09:01 ip-172-31-6-229 mysqld: InnoDB: mmap(139722752 bytes) failed; errno 12
Jan 24 06:09:01 ip-172-31-6-229 mysqld: 190124 6:09:01 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
Jan 24 06:09:01 ip-172-31-6-229 mysqld: 190124 6:09:01 [ERROR] Plugin 'InnoDB' init function returned error.
Jan 24 06:09:01 ip-172-31-6-229 mysqld: 190124 6:09:01 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
Jan 24 06:09:01 ip-172-31-6-229 mysqld: 190124 6:09:01 [Note] Plugin 'FEEDBACK' is disabled.
Jan 24 06:09:01 ip-172-31-6-229 mysqld: 190124 6:09:01 [ERROR] Unknown/unsupported storage engine: InnoDB
Jan 24 06:09:01 ip-172-31-6-229 mysqld: 190124 6:09:01 [ERROR] Aborting
Jan 24 06:09:01 ip-172-31-6-229 mysqld:
Jan 24 06:09:01 ip-172-31-6-229 systemd[1]: Started Clean php session files.
Jan 24 06:09:01 ip-172-31-6-229 mysqld: 190124 6:09:01 [Note] /usr/sbin/mysqld: Shutdown complete
Jan 24 06:09:01 ip-172-31-6-229 mysqld:
Jan 24 06:09:01 ip-172-31-6-229 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
Jan 24 06:09:01 ip-172-31-6-229 CRON[374]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jan 24 06:12:07 ip-172-31-6-229 systemd[1]: unattended-upgrades.service: Control process exited, code=exited status=1
Jan 24 06:12:07 ip-172-31-6-229 systemd[1]: Stopped Unattended Upgrades Shutdown.
Jan 24 06:12:07 ip-172-31-6-229 systemd[1]: unattended-upgrades.service: Unit entered failed state.
Jan 24 06:12:07 ip-172-31-6-229 systemd[1]: unattended-upgrades.service: Failed with result 'exit-code'.
Jan 24 06:12:07 ip-172-31-6-229 systemd[1]: Reloading.
Jan 24 06:12:07 ip-172-31-6-229 systemd[1]: Started ACPI event daemon.
Jan 24 06:12:08 ip-172-31-6-229 systemd[1]: Reloading.
Jan 24 06:12:08 ip-172-31-6-229 systemd[1]: Started ACPI event daemon.
Jan 24 06:12:09 ip-172-31-6-229 systemd[1]: Reloading.
Jan 24 06:12:09 ip-172-31-6-229 systemd[1]: Started ACPI event daemon.
Jan 24 06:12:09 ip-172-31-6-229 systemd[1]: Reloading.
Jan 24 06:12:09 ip-172-31-6-229 systemd[1]: Started ACPI event daemon.
Jan 24 06:12:09 ip-172-31-6-229 systemd[1]: Started Unattended Upgrades Shutdown.
Jan 24 06:12:13 ip-172-31-6-229 systemd[1]: Started Daily apt upgrade and clean activities.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment