Skip to content

Instantly share code, notes, and snippets.

/oom.log

Created March 7, 2016 21:02
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save anonymous/6b9ee07c1b8c0b5ebeca to your computer and use it in GitHub Desktop.
Save anonymous/6b9ee07c1b8c0b5ebeca to your computer and use it in GitHub Desktop.
Download ZIP
goofys-oom-killer-log
Raw
oom.log
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289878] slurmctld invoked oom-killer: gfp_mask=0x3000d0, order=1, oom_score_adj=0
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289882] slurmctld cpuset=/ mems_allowed=0
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289885] CPU: 2 PID: 9686 Comm: slurmctld Not tainted 3.13.0-79-generic #123-Ubuntu
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289886] Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/07/2015
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289887] 0000000000000000 ffff8803ee641ba8 ffffffff817257b8 ffff8800eb453000
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289891] ffff8803ee641c30 ffffffff8171fdbf 0000000000000000 0000000000000000
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289893] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289895] Call Trace:
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289902] [<ffffffff817257b8>] dump_stack+0x45/0x56
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289906] [<ffffffff8171fdbf>] dump_header+0x7f/0x1f1
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289910] [<ffffffff8115316e>] oom_kill_process+0x1ce/0x330
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289915] [<ffffffff812d88b5>] ? security_capable_noaudit+0x15/0x20
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289917] [<ffffffff811538a4>] out_of_memory+0x414/0x450
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289919] [<ffffffff81159ba0>] __alloc_pages_nodemask+0xa60/0xb80
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289923] [<ffffffff81065333>] copy_process.part.26+0x143/0x16b0
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289925] [<ffffffff811dc337>] ? __fd_install+0x47/0x60
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289929] [<ffffffff8160f0dd>] ? SYSC_accept4+0x1ad/0x220
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289931] [<ffffffff81066a75>] do_fork+0xd5/0x340
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289933] [<ffffffff81066d66>] SyS_clone+0x16/0x20
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289936] [<ffffffff81736739>] stub_clone+0x69/0x90
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289938] [<ffffffff817363dd>] ? system_call_fastpath+0x1a/0x1f
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289940] Mem-Info:
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289941] Node 0 DMA per-cpu:
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289943] CPU 0: hi: 0, btch: 1 usd: 0
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289944] CPU 1: hi: 0, btch: 1 usd: 0
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289945] CPU 2: hi: 0, btch: 1 usd: 0
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289946] CPU 3: hi: 0, btch: 1 usd: 0
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289947] Node 0 DMA32 per-cpu:
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289948] CPU 0: hi: 186, btch: 31 usd: 0
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289949] CPU 1: hi: 186, btch: 31 usd: 0
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289950] CPU 2: hi: 186, btch: 31 usd: 0
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289951] CPU 3: hi: 186, btch: 31 usd: 0
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289952] Node 0 Normal per-cpu:
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289953] CPU 0: hi: 186, btch: 31 usd: 0
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289954] CPU 1: hi: 186, btch: 31 usd: 0
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289955] CPU 2: hi: 186, btch: 31 usd: 0
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289956] CPU 3: hi: 186, btch: 31 usd: 0
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289959] active_anon:3992559 inactive_anon:84 isolated_anon:0
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289959] active_file:235 inactive_file:215 isolated_file:0
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289959] unevictable:0 dirty:0 writeback:60 unstable:0
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289959] free:36886 slab_reclaimable:5727 slab_unreclaimable:9930
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289959] mapped:53 shmem:115 pagetables:9499 bounce:0
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289959] free_cma:0
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289961] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289965] lowmem_reserve[]: 0 3744 16029 16029
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289968] Node 0 DMA32 free:62140kB min:15768kB low:19708kB high:23652kB active_anon:3677568kB inactive_anon:44kB active_file:184kB inactive_file:220kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3915776kB managed:3836744kB mlocked:0kB dirty:4kB writeback:0kB mapped:4kB shmem:64kB slab_reclaimable:6436kB slab_unreclaimable:12132kB kernel_stack:432kB pagetables:4788kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:612 all_unreclaimable? yes
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289971] lowmem_reserve[]: 0 0 12285 12285
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289973] Node 0 Normal free:69500kB min:51748kB low:64684kB high:77620kB active_anon:12292668kB inactive_anon:292kB active_file:756kB inactive_file:640kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:12845056kB managed:12580484kB mlocked:0kB dirty:0kB writeback:240kB mapped:208kB shmem:396kB slab_reclaimable:16472kB slab_unreclaimable:27588kB kernel_stack:1880kB pagetables:33208kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2148 all_unreclaimable? yes
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289977] lowmem_reserve[]: 0 0 0 0
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289979] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15904kB
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289988] Node 0 DMA32: 744*4kB (UEM) 493*8kB (UEM) 165*16kB (UEM) 154*32kB (UEM) 104*64kB (UEM) 114*128kB (UEM) 59*256kB (UEM) 20*512kB (EM) 1*1024kB (E) 0*2048kB 0*4096kB = 62104kB
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289996] Node 0 Normal: 16255*4kB (UEM) 62*8kB (UEM) 9*16kB (UEM) 9*32kB (EM) 6*64kB (M) 5*128kB (M) 3*256kB (M) 3*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 69276kB
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290005] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290006] 660 total pagecache pages
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290007] 0 pages in swap cache
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290009] Swap cache stats: add 0, delete 0, find 0/0
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290009] Free swap = 0kB
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290010] Total swap = 0kB
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290011] 4194205 pages RAM
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290012] 0 pages HighMem/MovableOnly
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290013] 66143 pages reserved
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290013] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290018] [ 435] 0 435 4868 60 13 0 0 upstart-udev-br
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290020] [ 438] 0 438 12467 146 28 0 -1000 systemd-udevd
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290021] [ 585] 0 585 3814 55 12 0 0 upstart-socket-
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290023] [ 623] 0 623 2555 580 8 0 0 dhclient
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290024] [ 909] 0 909 3818 55 12 0 0 upstart-file-br
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290026] [ 917] 102 917 9813 120 24 0 0 dbus-daemon
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290028] [ 931] 0 931 10862 90 27 0 0 systemd-logind
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290029] [ 1036] 0 1036 3634 40 12 0 0 getty
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290031] [ 1062] 0 1062 3634 41 12 0 0 getty
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290032] [ 1066] 0 1066 3634 39 12 0 0 getty
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290033] [ 1067] 0 1067 3634 40 12 0 0 getty
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290035] [ 1069] 0 1069 3634 41 12 0 0 getty
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290036] [ 1081] 101 1081 65115 5272 38 0 0 rsyslogd
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290038] [ 1111] 0 1111 1091 36 8 0 0 acpid
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290039] [ 1128] 0 1128 5913 65 17 0 0 cron
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290041] [ 1130] 0 1130 4784 42 12 0 0 atd
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290042] [ 1151] 0 1151 4821 72 13 0 0 irqbalance
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290044] [ 1221] 0 1221 3634 40 12 0 0 getty
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290045] [ 1222] 0 1222 3196 39 12 0 0 getty
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290047] [ 8323] 0 8323 71789 1701 27 0 0 amazon-ssm-agen
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290048] [16728] 0 16728 5879 83 16 0 0 rpcbind
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290050] [17134] 108 17134 5385 115 16 0 0 rpc.statd
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290052] [17218] 0 17218 6926 58 17 0 0 rpc.idmapd
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290054] [17503] 0 17503 7057 164 19 0 0 rpc.mountd
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290055] [ 9657] 64030 9657 534494 14055 131 0 0 slurmctld
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290057] [ 9827] 13 9827 41312 15773 72 0 0 squid3
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290058] [ 9838] 13 9838 4753 54 15 0 0 log_file_daemon
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290060] [ 9873] 0 9873 2557 581 9 0 0 dhclient
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290061] [10035] 0 10035 2557 583 6 0 0 dhclient
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290063] [10103] 0 10103 15871 173 34 0 -1000 sshd
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290064] [11362] 0 11362 3653986 3494764 6877 0 0 goofys
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290066] [11372] 0 11372 423714 1007 90 0 0 s3fs
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290067] [11420] 106 11420 7862 152 19 0 0 ntpd
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290069] [30983] 0 30983 27461 250 55 0 0 sshd
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290070] [31092] 1000 31092 27499 287 53 0 0 sshd
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290072] [31098] 0 31098 2670 34 10 0 0 autossh
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290073] [31099] 0 31099 11597 207 26 0 0 ssh
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290075] [18701] 0 18701 15432 117 35 0 0 cron
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290076] [18703] 0 18703 2695 34 10 0 0 sh
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290078] [18704] 0 18704 2668 38 11 0 0 run-parts
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290079] [20956] 0 20956 3848 60 12 0 0 mlocate
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290080] [20961] 0 20961 3059 32 12 0 0 flock
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290082] [20962] 0 20962 3083 80 11 0 0 updatedb.mlocat
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290084] [ 9616] 555 9616 1556726 426139 1038 0 0 java
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290085] [27838] 0 27838 27461 255 56 0 0 sshd
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290087] [27939] 1000 27939 27461 259 54 0 0 sshd
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290088] [27943] 1000 27943 6439 552 18 0 0 bash
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290090] [28025] 0 28025 18049 130 39 0 0 sudo
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290091] [28026] 555 28026 6422 540 18 0 0 bash
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290093] [28794] 0 28789 144444 4172 48 0 0 python
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290094] [13803] 109 13803 58074 370 26 0 0 munged
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290096] [13850] 13 13850 4784 62 13 0 0 pinger
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290097] [13996] 0 13996 2695 33 10 0 0 awslogs-agent-l
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290099] [13998] 0 13998 317541 9639 85 0 0 aws
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290100] [15039] 0 15039 18048 130 39 0 0 sudo
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290101] [15040] 0 15040 144207 3721 49 0 0 python
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290103] [20628] 0 20628 23919 4142 48 0 0 codedeploy-agen
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290105] [20632] 0 20632 44783 6211 59 0 0 codedeploy-agen
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290106] Out of memory: Kill process 11362 (goofys) score 826 or sacrifice child
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.295024] Killed process 11362 (goofys) total-vm:14615944kB, anon-rss:13979056kB, file-rss:0kB
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment