oom.log

Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372067] CPU: 0 PID: 9361 Comm: kworker/0:2 Tainted: G        W        4.15.0-55-generic #60-Ubuntu
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372070] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/03/2018
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372162] Workqueue: events_freezable vmballoon_work [vmw_balloon]
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372166] Call Trace:
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372315]  dump_stack+0x63/0x8b
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372376]  dump_header+0x71/0x285
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372419]  ? security_capable_noaudit+0x4b/0x70
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372423]  oom_kill_process+0x220/0x440
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372426]  out_of_memory+0x2d1/0x4f0
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372431]  __alloc_pages_slowpath+0xa61/0xe20
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372454]  ? __switch_to_asm+0x34/0x70
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372459]  __alloc_pages_nodemask+0x263/0x280
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372486]  alloc_pages_current+0x6a/0xe0
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372493]  vmballoon_work+0x454/0x70c [vmw_balloon]
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372535]  process_one_work+0x1de/0x410
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372539]  worker_thread+0x32/0x410
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372544]  kthread+0x121/0x140
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372547]  ? process_one_work+0x410/0x410
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372551]  ? kthread_create_worker_on_cpu+0x70/0x70
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372556]  ret_from_fork+0x35/0x40
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372559] Mem-Info:
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372569] active_anon:3952595 inactive_anon:494656 isolated_anon:0
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372569]  active_file:239 inactive_file:387 isolated_file:0
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372569]  unevictable:0 dirty:0 writeback:4 unstable:0
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372569]  slab_reclaimable:13688 slab_unreclaimable:10716
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372569]  mapped:53 shmem:1 pagetables:10360 bounce:0
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372569]  free:35285 free_pcp:429 free_cma:0
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372589] Node 0 active_anon:7888824kB inactive_anon:987080kB active_file:828kB inactive_file:752kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:12k
B shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 8404992kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372598] Node 1 active_anon:7921556kB inactive_anon:991544kB active_file:128kB inactive_file:796kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:300kB dirty:0kB writeback:4
kB shmem:4kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 8730624kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372600] Node 0 DMA free:15908kB min:56kB low:68kB high:80kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:1
5908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372610] lowmem_reserve[]: 0 2952 11947 11947 11947
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372618] Node 0 DMA32 free:46804kB min:11060kB low:14080kB high:17100kB active_anon:684600kB inactive_anon:0kB active_file:56kB inactive_file:0kB unevictable:0kB writepending:0kB present:
3129216kB managed:3063648kB mlocked:0kB kernel_stack:176kB pagetables:48kB bounce:0kB free_pcp:160kB local_pcp:120kB free_cma:0kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372626] lowmem_reserve[]: 0 0 8995 8995 8995
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372631] Node 0 Normal free:33660kB min:33692kB low:42900kB high:52108kB active_anon:7204224kB inactive_anon:987080kB active_file:844kB inactive_file:532kB unevictable:0kB writepending:12
kB present:9437184kB managed:9217284kB mlocked:0kB kernel_stack:3048kB pagetables:20764kB bounce:0kB free_pcp:1060kB local_pcp:36kB free_cma:0kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372655] lowmem_reserve[]: 0 0 0 0 0
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372661] Node 1 Normal free:44768kB min:45300kB low:57684kB high:70068kB active_anon:7921556kB inactive_anon:991544kB active_file:308kB inactive_file:1316kB unevictable:0kB writepending:4
kB present:12582912kB managed:12384224kB mlocked:0kB kernel_stack:2568kB pagetables:20628kB bounce:0kB free_pcp:496kB local_pcp:120kB free_cma:0kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372666] lowmem_reserve[]: 0 0 0 0 0
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372671] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372699] Node 0 DMA32: 97*4kB (ME) 224*8kB (UME) 161*16kB (ME) 128*32kB (UME) 97*64kB (ME) 60*128kB (UME) 44*256kB (UME) 17*512kB (UM) 2*1024kB (M) 1*2048kB (M) 0*4096kB = 46804kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372727] Node 0 Normal: 290*4kB (UMEH) 763*8kB (MEH) 501*16kB (UMEH) 240*32kB (UMEH) 79*64kB (ME) 25*128kB (MH) 4*256kB (UMH) 2*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 33264kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372746] Node 1 Normal: 1152*4kB (MEH) 1208*8kB (MEH) 810*16kB (UMEH) 364*32kB (UMEH) 93*64kB (UMH) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44832kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372793] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372823] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372826] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372829] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372830] 3131 total pagecache pages
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372833] 2411 pages in swap cache
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372836] Swap cache stats: add 248203, delete 245790, find 1864/3257
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372837] Free swap  = 0kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372838] Total swap = 975868kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372589] Node 0 active_anon:7888824kB inactive_anon:987080kB active_file:828kB inactive_file:752kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:12kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 8404992kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372598] Node 1 active_anon:7921556kB inactive_anon:991544kB active_file:128kB inactive_file:796kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:300kB dirty:0kB writeback:4kB shmem:4kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 8730624kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372600] Node 0 DMA free:15908kB min:56kB low:68kB high:80kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372610] lowmem_reserve[]: 0 2952 11947 11947 11947
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372618] Node 0 DMA32 free:46804kB min:11060kB low:14080kB high:17100kB active_anon:684600kB inactive_anon:0kB active_file:56kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129216kB managed:3063648kB mlocked:0kB kernel_stack:176kB pagetables:48kB bounce:0kB free_pcp:160kB local_pcp:120kB free_cma:0kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372626] lowmem_reserve[]: 0 0 8995 8995 8995
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372631] Node 0 Normal free:33660kB min:33692kB low:42900kB high:52108kB active_anon:7204224kB inactive_anon:987080kB active_file:844kB inactive_file:532kB unevictable:0kB writepending:12kB present:9437184kB managed:9217284kB mlocked:0kB kernel_stack:3048kB pagetables:20764kB bounce:0kB free_pcp:1060kB local_pcp:36kB free_cma:0kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372655] lowmem_reserve[]: 0 0 0 0 0
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372661] Node 1 Normal free:44768kB min:45300kB low:57684kB high:70068kB active_anon:7921556kB inactive_anon:991544kB active_file:308kB inactive_file:1316kB unevictable:0kB writepending:4kB present:12582912kB managed:12384224kB mlocked:0kB kernel_stack:2568kB pagetables:20628kB bounce:0kB free_pcp:496kB local_pcp:120kB free_cma:0kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372666] lowmem_reserve[]: 0 0 0 0 0
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372671] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372699] Node 0 DMA32: 97*4kB (ME) 224*8kB (UME) 161*16kB (ME) 128*32kB (UME) 97*64kB (ME) 60*128kB (UME) 44*256kB (UME) 17*512kB (UM) 2*1024kB (M) 1*2048kB (M) 0*4096kB = 46804kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372727] Node 0 Normal: 290*4kB (UMEH) 763*8kB (MEH) 501*16kB (UMEH) 240*32kB (UMEH) 79*64kB (ME) 25*128kB (MH) 4*256kB (UMH) 2*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 33264kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372746] Node 1 Normal: 1152*4kB (MEH) 1208*8kB (MEH) 810*16kB (UMEH) 364*32kB (UMEH) 93*64kB (UMH) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44832kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372793] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372823] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372826] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372829] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372830] 3131 total pagecache pages
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372833] 2411 pages in swap cache
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372836] Swap cache stats: add 248203, delete 245790, find 1864/3257
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372837] Free swap  = 0kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372838] Total swap = 975868kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372840] 6291326 pages RAM
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372841] 0 pages HighMem/MovableOnly
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372841] 121060 pages reserved
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372842] 0 pages cma reserved
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372843] 0 pages hwpoisoned
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372844] [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372860] [  374]     0   374    61964      147   499712      215             0 systemd-journal
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372863] [  396]     0   396     9015        1   118784      827         -1000 systemd-udevd
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372867] [  404]   100   404    17961        0   176128      177             0 systemd-network
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372870] [  549]     0   549    22056        0   163840      364             0 VGAuthService
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372873] [  550]   101   550    17689        0   172032      167             0 systemd-resolve
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372876] [  552]     0   552    48281       77   241664      295             0 vmtoolsd
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372879] [  554]     0   554    42607        1   229376     1940             0 networkd-dispat
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372882] [  556]     0   556    17650        1   180224      192             0 systemd-logind
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372885] [  557]     0   557     7830        8   110592       65             0 cron
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372888] [  561]     0   561    27638        1   118784      102             0 irqbalance
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372892] [  562]     0   562   108581        0   356352      378             0 ModemManager
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372897] [  563]   103   563    12568        1   147456      208          -900 dbus-daemon
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372903] [  576]     0   576   101690        0   417792      600             0 NetworkManager
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372907] [  577]     0   577    11306        0   122880      143             0 wpa_supplicant
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372911] [  579]     0   579    71907       12   200704      222             0 accounts-daemon
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372915] [  581]   108   581    29478      839   110592     1258             0 node_exporter
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372920] [  609]     0   609    72220        0   204800      219             0 polkitd
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372924] [  657]     0   657    18074        0   188416      187         -1000 sshd
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372948] [  666]   107   666    26944       11   126976      136             0 ntpd
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372952] [  667]     0   667    19692        1   200704      142             0 login
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372956] [  882]     0   882    19158        0   188416      277             0 systemd
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372959] [  883]     0   883    64200        0   258048      570             0 (sd-pam)
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372962] [  893]     0   893     5595        1    90112      335             0 bash
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372965] [ 1702]   102  1702    65759        0   159744      197             0 rsyslogd
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372969] [ 3905]   108  3905 4295081031  4440191 37576704   200964          -950 scylla
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372973] [ 3975]   108  3975   860326     1419   647168    33749             0 scylla-jmx
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.372977] Out of memory: Kill process 3975 (scylla-jmx) score 5 or sacrifice child
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.373953] Killed process 3975 (scylla-jmx) total-vm:3441304kB, anon-rss:5788kB, file-rss:0kB, shmem-rss:0kB
Aug 12 04:59:39 cdw-prod-scylladb-3 kernel: [237297.397761] oom_reaper: reaped process 3975 (scylla-jmx), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Aug 12 04:59:42 cdw-prod-scylladb-3 kernel: [237299.920163] reactor-11 invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=-950
Aug 12 04:59:42 cdw-prod-scylladb-3 kernel: [237299.920167] reactor-11 cpuset=/ mems_allowed=0-1
Aug 12 04:59:42 cdw-prod-scylladb-3 kernel: [237299.920183] CPU: 11 PID: 3919 Comm: reactor-11 Tainted: G        W        4.15.0-55-generic #60-Ubuntu

system.info

Виртуалка на vmware, 24 гига оперативы, убунта.

root@cdw-prod-scylladb-3:~# uname -a
Linux cdw-prod-scylladb-3 4.15.0-55-generic #60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

root@cdw-prod-scylladb-3:~# cat /etc/lsb-release 
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"

sysctl:
vm.swappiness = 60
vm.nr_overcommit_hugepages = 0
vm.overcommit_kbytes = 0
vm.overcommit_memory = 0
vm.overcommit_ratio = 50

free after reboot:
root@cdw-prod-scylladb-3:~# free -m 
              total        used        free      shared  buff/cache   available
Mem:          24102       18487        5164           0         450        5290
Swap:           952           4         948