Skip to content

Instantly share code, notes, and snippets.

@jakubgs

jakubgs/sourcecred_dmesg_oom_killer.log

Created August 6, 2020 07:36
Show Gist options
  • Save jakubgs/b8a12bc6659a2241b8bbccbf90540461 to your computer and use it in GitHub Desktop.
Save jakubgs/b8a12bc6659a2241b8bbccbf90540461 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
sourcecred_dmesg_oom_killer.log
[2905711.003834] systemd-journal invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=-250
[2905711.003838] CPU: 1 PID: 426 Comm: systemd-journal Tainted: G W 5.4.0-40-generic #44-Ubuntu
[2905711.003839] Hardware name: DigitalOcean Droplet, BIOS 20171212 12/12/2017
[2905711.003839] Call Trace:
[2905711.003861] dump_stack+0x6d/0x9a
[2905711.003863] dump_header+0x4f/0x1eb
[2905711.003865] oom_kill_process.cold+0xb/0x10
[2905711.003866] out_of_memory.part.0+0x1df/0x3d0
[2905711.003867] out_of_memory+0x6d/0xd0
[2905711.003870] __alloc_pages_slowpath+0xd5e/0xe50
[2905711.003874] ? ext4_inode_csum.isra.0+0x180/0x1f0
[2905711.003876] __alloc_pages_nodemask+0x2d0/0x320
[2905711.003878] alloc_pages_current+0x87/0xe0
[2905711.003885] __page_cache_alloc+0x72/0x90
[2905711.003887] pagecache_get_page+0xbf/0x300
[2905711.003888] filemap_fault+0x69a/0xa40
[2905711.003891] ? __switch_to_asm+0x40/0x70
[2905711.003892] ? __switch_to_asm+0x34/0x70
[2905711.003893] ? __switch_to_asm+0x40/0x70
[2905711.003897] ? __switch_to_xtra+0x1c5/0x5c0
[2905711.003898] ? __switch_to_asm+0x40/0x70
[2905711.003899] ? xas_load+0xd/0x80
[2905711.003900] ? xas_find+0x17f/0x1c0
[2905711.003901] ? filemap_map_pages+0x24c/0x380
[2905711.003903] ext4_filemap_fault+0x32/0x46
[2905711.003905] __do_fault+0x3c/0x130
[2905711.003906] do_fault+0x24b/0x640
[2905711.003909] ? ep_send_events_proc+0xe4/0x1d0
[2905711.003910] __handle_mm_fault+0x4c5/0x7a0
[2905711.003911] handle_mm_fault+0xca/0x200
[2905711.003915] do_user_addr_fault+0x1f9/0x450
[2905711.003916] __do_page_fault+0x58/0x90
[2905711.003918] do_page_fault+0x2c/0xe0
[2905711.003921] do_async_page_fault+0x39/0x70
[2905711.003923] async_page_fault+0x34/0x40
[2905711.003929] RIP: 0033:0x7f805fd31fe9
[2905711.003934] Code: Bad RIP value.
[2905711.003935] RSP: 002b:00007fffd2a6b150 EFLAGS: 00010206
[2905711.003936] RAX: 0000000000000001 RBX: 0000564098073f10 RCX: 00007f806003243e
[2905711.003936] RDX: 0000000000000025 RSI: 00005640980b8580 RDI: 0000000000000000
[2905711.003937] RBP: ffffffffffffffff R08: 0000000000000000 R09: 00007fffd2a6b248
[2905711.003938] R10: 00000000ffffffff R11: 0000000000000000 R12: 0000000000000001
[2905711.003938] R13: 0000000000000025 R14: 0000000000000000 R15: 0000000000000000
[2905711.003939] Mem-Info:
[2905711.003943] active_anon:1708706 inactive_anon:230362 isolated_anon:32
active_file:78 inactive_file:8 isolated_file:0
unevictable:4612 dirty:0 writeback:20 unstable:0
slab_reclaimable:13025 slab_unreclaimable:27502
mapped:2380 shmem:308 pagetables:13735 bounce:0
free:25164 free_pcp:0 free_cma:0
[2905711.003945] Node 0 active_anon:6834824kB inactive_anon:921448kB active_file:312kB inactive_file:32kB unevictable:18448kB isolated(anon):128kB isolated(file):0kB mapped:9520kB dirty:0kB writeback:80kB shmem:1232kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[2905711.003946] Node 0 DMA free:15908kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[2905711.003949] lowmem_reserve[]: 0 3426 7860 7860 7860
[2905711.003950] Node 0 DMA32 free:46796kB min:29400kB low:36748kB high:44096kB active_anon:2939528kB inactive_anon:548500kB active_file:336kB inactive_file:0kB unevictable:0kB writepending:80kB present:3653612kB managed:3588076kB mlocked:0kB kernel_stack:496kB pagetables:21436kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[2905711.003952] lowmem_reserve[]: 0 0 4434 4434 4434
[2905711.003953] Node 0 Normal free:37952kB min:38048kB low:47560kB high:57072kB active_anon:3895852kB inactive_anon:370420kB active_file:648kB inactive_file:160kB unevictable:18448kB writepending:0kB present:4718592kB managed:4548932kB mlocked:18448kB kernel_stack:4560kB pagetables:33504kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[2905711.003955] lowmem_reserve[]: 0 0 0 0 0
[2905711.003956] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
[2905711.003972] Node 0 DMA32: 148*4kB (UME) 227*8kB (UME) 212*16kB (UME) 175*32kB (UME) 130*64kB (UME) 84*128kB (UME) 47*256kB (ME) 10*512kB (UME) 0*1024kB 0*2048kB 0*4096kB = 47624kB
[2905711.003976] Node 0 Normal: 576*4kB (UME) 428*8kB (UME) 452*16kB (UME) 254*32kB (UME) 110*64kB (UME) 34*128kB (UME) 26*256kB (UME) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 39136kB
[2905711.003980] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[2905711.003981] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[2905711.003982] 5156 total pagecache pages
[2905711.003987] 2643 pages in swap cache
[2905711.003987] Swap cache stats: add 5248557, delete 5245245, find 1432629/2044758
[2905711.003988] Free swap = 0kB
[2905711.003988] Total swap = 8388604kB
[2905711.003989] 2097049 pages RAM
[2905711.003989] 0 pages HighMem/MovableOnly
[2905711.003989] 58820 pages reserved
[2905711.003990] 0 pages cma reserved
[2905711.003990] 0 pages hwpoisoned
[2905711.003991] Tasks state (memory values in pages):
[2905711.003991] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[2905711.003997] [ 426] 0 426 39590 901 303104 206 -250 systemd-journal
[2905711.003999] [ 583] 0 583 70029 4481 86016 0 -1000 multipathd
[2905711.004000] [ 625] 0 625 622 130 45056 21 0 none
[2905711.004002] [ 675] 100 675 4667 625 69632 232 0 systemd-network
[2905711.004003] [ 678] 101 678 6078 784 90112 1081 0 systemd-resolve
[2905711.004005] [ 712] 0 712 5361 590 65536 277 -1000 systemd-udevd
[2905711.004006] [ 782] 0 782 59414 508 98304 289 0 accounts-daemon
[2905711.004008] [ 791] 0 791 2190 559 57344 49 0 cron
[2905711.004009] [ 793] 103 793 1965 783 53248 96 -900 dbus-daemon
[2905711.004010] [ 807] 0 807 20470 554 53248 57 0 irqbalance
[2905711.004011] [ 811] 0 811 7426 837 98304 1937 0 networkd-dispat
[2905711.004013] [ 823] 0 823 4386 513 81920 356 0 systemd-logind
[2905711.004014] [ 831] 0 831 1004 474 49152 47 0 atd
[2905711.004016] [ 832] 0 832 352079 2343 335872 3760 0 containerd
[2905711.004017] [ 834] 0 834 3067 526 57344 193 -1000 sshd
[2905711.004018] [ 851] 0 851 1894 417 49152 32 0 agetty
[2905711.004020] [ 853] 0 853 1513 361 45056 30 0 agetty
[2905711.004021] [ 872] 0 872 58232 602 81920 164 0 polkitd
[2905711.004022] [ 924] 0 924 27068 819 102400 1937 0 unattended-upgr
[2905711.004023] [ 935] 106 935 2433 96 61440 43 0 uuidd
[2905711.004025] [ 1391] 1000 1391 4632 571 77824 272 0 systemd
[2905711.004026] [ 1395] 1000 1395 25892 53 86016 817 0 (sd-pam)
[2905711.004028] [ 5722] 107 5722 3358 322 45056 111 0 chronyd
[2905711.004029] [ 5723] 107 5723 1314 412 45056 86 0 chronyd
[2905711.004031] [ 10922] 0 10922 326276 3524 421888 7069 -500 dockerd
[2905711.004032] [ 11046] 0 11046 137261 0 122880 375 -500 docker-proxy
[2905711.004034] [ 11053] 0 11053 27276 97 69632 89 -999 containerd-shim
[2905711.004035] [ 11071] 100070 11071 39357 66 98304 153 0 postgres
[2905711.004036] [ 11150] 100070 11150 39362 18 184320 180 0 postgres
[2905711.004037] [ 11151] 100070 11151 39357 43 86016 146 0 postgres
[2905711.004038] [ 11152] 100070 11152 39357 30 86016 157 0 postgres
[2905711.004039] [ 11153] 100070 11153 39452 96 94208 198 0 postgres
[2905711.004041] [ 11154] 100070 11154 3115 58 65536 140 0 postgres
[2905711.004042] [ 14204] 500 14204 45744 3256 249856 678 0 consul
[2905711.004043] [ 16524] 65534 16524 29062 0 98304 1264 0 watchtower
[2905711.004044] [ 32182] 102 32182 56141 186 77824 413 0 rsyslogd
[2905711.004048] [ 35127] 0 35127 8638 1019 106496 6484 0 tincd
[2905711.004049] [ 35678] 0 35678 14816 479 110592 436 0 nginx
[2905711.004051] [1033475] 0 1033475 476592 14477 589824 13476 0 tsagentd
[2905711.004052] [1033524] 0 1033524 20768 1368 118784 138 0 tsauditd
[2905711.004053] [1033525] 0 1033525 1816 351 49152 18 0 tail
[2905711.004054] [1033527] 0 1033527 1816 361 53248 30 0 tail
[2905711.004056] [1033529] 0 1033529 1816 376 49152 23 0 tail
[2905711.004057] [1515523] 800 1515523 37345 2651 151552 1934 -1000 netdata
[2905711.004058] [1515525] 800 1515525 4352 29 61440 121 -1000 netdata
[2905711.004060] [2605966] 0 2605966 230002 205 180224 484 -500 docker-proxy
[2905711.004061] [2605976] 0 2605976 27276 198 73728 238 -999 containerd-shim
[2905711.004062] [2605997] 100000 2605997 173511 1749 221184 4085 0 fathom
[2905711.004063] [2726278] 33 2726278 16395 723 126976 1916 0 nginx
[2905711.004065] [2726279] 33 2726279 16356 535 126976 2084 0 nginx
[2905711.004066] [2726280] 33 2726280 15320 525 118784 1038 0 nginx
[2905711.004067] [2726281] 33 2726281 15789 530 122880 1495 0 nginx
[2905711.004068] [2933499] 1001 2933499 2154 429 57344 65 0 bash
[2905711.004069] [2933508] 1001 2933508 1508 222 45056 121 0 ssh-agent
[2905711.004071] [2933566] 1001 2933566 4282356 1902487 74878976 2022342 0 node
[2905711.004072] [2960671] 100070 2960671 39765 446 159744 265 0 postgres
[2905711.004073] [2960672] 100070 2960672 39767 453 163840 274 0 postgres
[2905711.004075] [2966923] 0 2966923 3067 594 61440 193 0 sshd
[2905711.004076] [2967028] 0 2967028 5361 174 57344 270 0 systemd-udevd
[2905711.004077] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/sourcecred-gen.service,task=node,pid=2933566,uid=1001
[2905711.004101] Out of memory: Killed process 2933566 (node) total-vm:17129424kB, anon-rss:7607536kB, file-rss:2412kB, shmem-rss:0kB, UID:1001 pgtables:73124kB oom_score_adj:0
[2905711.408199] oom_reaper: reaped process 2933566 (node), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment