Skip to content

Instantly share code, notes, and snippets.

@zyriljamez

zyriljamez/dmesg.txt Secret

Last active Aug 2, 2019
Embed
What would you like to do?
[273209.600919] IPVS: Creating netns size=2048 id=177
[273219.680526] IPVS: Creating netns size=2048 id=178
[273228.181556] IPVS: Creating netns size=2048 id=179
[273239.879241] IPVS: Creating netns size=2048 id=180
[273255.001859] IPVS: Creating netns size=2048 id=181
[273262.708951] IPVS: Creating netns size=2048 id=182
[273275.210795] IPVS: Creating netns size=2048 id=183
[273290.348690] IPVS: Creating netns size=2048 id=184
[273297.991296] IPVS: Creating netns size=2048 id=185
[273310.493459] IPVS: Creating netns size=2048 id=186
[273325.619355] IPVS: Creating netns size=2048 id=187
[273333.057316] IPVS: Creating netns size=2048 id=188
[273360.943379] IPVS: Creating netns size=2048 id=189
[273380.308775] IPVS: Creating netns size=2048 id=190
[273396.214391] IPVS: Creating netns size=2048 id=191
[273405.204701] IPVS: Creating netns size=2048 id=192
[273416.417774] IPVS: Creating netns size=2048 id=193
[273424.659549] IPVS: Creating netns size=2048 id=194
[273431.507812] IPVS: Creating netns size=2048 id=195
[273441.560288] IPVS: Creating netns size=2048 id=196
[273449.864207] IPVS: Creating netns size=2048 id=197
[273466.772279] IPVS: Creating netns size=2048 id=198
[273481.884667] IPVS: Creating netns size=2048 id=199
[273490.298978] IPVS: Creating netns size=2048 id=200
[273505.954215] IPVS: Creating netns size=2048 id=201
[273521.441764] IPVS: Creating netns size=2048 id=202
[273532.303459] IPVS: Creating netns size=2048 id=203
[273541.029266] IPVS: Creating netns size=2048 id=204
[273552.409776] IPVS: Creating netns size=2048 id=205
[273562.467554] IPVS: Creating netns size=2048 id=206
[273571.670029] IPVS: Creating netns size=2048 id=207
[273575.898519] IPVS: Creating netns size=2048 id=208
[273582.683929] IPVS: Creating netns size=2048 id=209
[273592.753270] IPVS: Creating netns size=2048 id=210
[273602.807157] IPVS: Creating netns size=2048 id=211
[274483.789549] IPVS: Creating netns size=2048 id=212
[281680.794411] IPVS: Creating netns size=2048 id=213
[288877.767927] IPVS: Creating netns size=2048 id=214
[296084.671317] IPVS: Creating netns size=2048 id=215
[303281.566716] IPVS: Creating netns size=2048 id=216
[310478.456026] IPVS: Creating netns size=2048 id=217
[317625.427962] IPVS: Creating netns size=2048 id=218
[317627.960841] IPVS: Creating netns size=2048 id=219
[317628.399932] IPVS: Creating netns size=2048 id=220
[321284.084491] IPVS: Creating netns size=2048 id=221
[328481.025688] IPVS: Creating netns size=2048 id=222
[332079.463476] IPVS: Creating netns size=2048 id=223
[335678.028042] IPVS: Creating netns size=2048 id=224
[342885.115960] IPVS: Creating netns size=2048 id=225
[350082.032487] IPVS: Creating netns size=2048 id=226
[352167.803870] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000
[352167.803975] pci 0000:00:09.0: reg 0x10: [io 0x0000-0x003f]
[352167.804008] pci 0000:00:09.0: reg 0x14: [mem 0x00000000-0x00000fff]
[352167.804121] pci 0000:00:09.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit pref]
[352167.804806] pci 0000:00:09.0: BAR 4: assigned [mem 0x44000c000-0x44000ffff 64bit pref]
[352167.806908] pci 0000:00:09.0: BAR 1: assigned [mem 0xc0003000-0xc0003fff]
[352167.808530] pci 0000:00:09.0: BAR 0: assigned [io 0x10c0-0x10ff]
[352167.810271] virtio-pci 0000:00:09.0: enabling device (0000 -> 0003)
[352167.849116] virtio-pci 0000:00:09.0: irq 35 for MSI/MSI-X
[352167.849135] virtio-pci 0000:00:09.0: irq 36 for MSI/MSI-X
[352167.849953] virtio_blk virtio6: [vde] 62914560 512-byte logical blocks (32.2 GB/30.0 GiB)
[352178.423716] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000
[352178.423814] pci 0000:00:0a.0: reg 0x10: [io 0x0000-0x003f]
[352178.423847] pci 0000:00:0a.0: reg 0x14: [mem 0x00000000-0x00000fff]
[352178.423966] pci 0000:00:0a.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit pref]
[352178.424612] pci 0000:00:0a.0: BAR 4: assigned [mem 0x440010000-0x440013fff 64bit pref]
[352178.426682] pci 0000:00:0a.0: BAR 1: assigned [mem 0xc0004000-0xc0004fff]
[352178.428407] pci 0000:00:0a.0: BAR 0: assigned [io 0x1400-0x143f]
[352178.430196] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0003)
[352178.467236] virtio-pci 0000:00:0a.0: irq 37 for MSI/MSI-X
[352178.467256] virtio-pci 0000:00:0a.0: irq 38 for MSI/MSI-X
[352178.468045] virtio_blk virtio7: [vdf] 62914560 512-byte logical blocks (32.2 GB/30.0 GiB)
[352184.557955] ata1: soft resetting link
[352184.710766] ata1: EH complete
[352184.724768] ata2: soft resetting link
[352184.876874] ata2: EH complete
[352185.459381] EXT4-fs (vdf): mounted filesystem with ordered data mode. Opts: (null)
[352185.560734] ata1: soft resetting link
[352185.713027] ata1: EH complete
[352185.714540] ata2: soft resetting link
[352185.737394] IPVS: Creating netns size=2048 id=227
[352185.866774] ata2: EH complete
[352186.662276] EXT4-fs (vde): mounted filesystem with ordered data mode. Opts: (null)
[352187.043542] IPVS: Creating netns size=2048 id=228
[361880.652218] SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
[361880.654176] cache: dentry(732:173cfe739439d02d6fd441bf2d446a936e8f64e9c994111862bbf4980d779fd8), object size: 192, buffer size: 192, default order: 0, min order: 0
[361880.658287] node 0: slabs: 679034, objs: 14259714, free: 0
[361880.660406] VM Periodic Tas invoked oom-killer: gfp_mask=0x50, order=0, oom_score_adj=872
[361880.662872] VM Periodic Tas cpuset=173cfe739439d02d6fd441bf2d446a936e8f64e9c994111862bbf4980d779fd8 mems_allowed=0
[361880.666612] CPU: 4 PID: 30742 Comm: VM Periodic Tas Kdump: loaded Tainted: G ------------ T 3.10.0-957.12.2.el7.x86_64 #1
[361880.670843] Hardware name: Fedora Project OpenStack Nova, BIOS 1.10.2-3.el7_4.1.tis.2 04/01/2014
[361880.673620] Call Trace:
[361880.675025] [<ffffffffb8d63041>] dump_stack+0x19/0x1b
[361880.677003] [<ffffffffb8d5da6a>] dump_header+0x90/0x229
[361880.679010] [<ffffffffb87ba306>] ? find_lock_task_mm+0x56/0xc0
[361880.681119] [<ffffffffb87ba7b4>] oom_kill_process+0x254/0x3d0
[361880.683246] [<ffffffffb8835506>] mem_cgroup_oom_synchronize+0x546/0x570
[361880.685528] [<ffffffffb8834980>] ? mem_cgroup_charge_common+0xc0/0xc0
[361880.687783] [<ffffffffb87bb044>] pagefault_out_of_memory+0x14/0x90
[361880.690016] [<ffffffffb8d5bf72>] mm_fault_error+0x6a/0x157
[361880.692123] [<ffffffffb8d707c8>] __do_page_fault+0x3c8/0x4f0
[361880.694267] [<ffffffffb8d709d6>] trace_do_page_fault+0x56/0x150
[361880.696505] [<ffffffffb8d6ff62>] do_async_page_fault+0x22/0xf0
[361880.698750] [<ffffffffb8d6c798>] async_page_fault+0x28/0x30
[361880.700999] Task in /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/173cfe739439d02d6fd441bf2d446a936e8f64e9c994111862bbf4980d779fd8 killed as a result of limit of /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299
[361880.707359] memory: usage 4194304kB, limit 4194304kB, failcnt 2065
[361880.709713] memory+swap: usage 4194304kB, limit 9007199254740988kB, failcnt 0
[361880.712258] kmem: usage 2907928kB, limit 9007199254740988kB, failcnt 0
[361880.714709] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
[361880.721478] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/a1cc34f261f4e6478100777358ad8b952f9ebcadfd0c244f77f225239c286bbf: cache:0KB rss:44KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:44KB inactive_file:0KB active_file:0KB unevictable:0KB
[361880.731066] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/173cfe739439d02d6fd441bf2d446a936e8f64e9c994111862bbf4980d779fd8: cache:0KB rss:1286332KB rss_huge:6144KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:1286332KB inactive_file:0KB active_file:0KB unevictable:0KB
[361880.741196] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[361880.744324] [24821] 0 24821 253 1 4 0 -998 pause
[361880.747456] [29490] 1000 29490 1321721 325819 743 0 872 java
[361880.750396] [30803] 1000 30803 17192 1373 31 0 872 controller
[361880.753503] [26578] 1000 26578 2921 351 11 0 872 sh
[361880.756353] [26583] 1000 26583 21264 1077 46 0 872 curl
[361880.759329] Memory cgroup out of memory: Kill process 716 (elasticsearch[s) score 1182 or sacrifice child
[361880.762885] Killed process 30803 (controller) total-vm:68768kB, anon-rss:688kB, file-rss:4804kB, shmem-rss:0kB
[361880.801109] curl invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=872
[361880.804291] curl cpuset=173cfe739439d02d6fd441bf2d446a936e8f64e9c994111862bbf4980d779fd8 mems_allowed=0
[361880.807730] CPU: 4 PID: 26583 Comm: curl Kdump: loaded Tainted: G ------------ T 3.10.0-957.12.2.el7.x86_64 #1
[361880.811756] Hardware name: Fedora Project OpenStack Nova, BIOS 1.10.2-3.el7_4.1.tis.2 04/01/2014
[361880.815167] Call Trace:
[361880.817282] [<ffffffffb8d63041>] dump_stack+0x19/0x1b
[361880.819730] [<ffffffffb8d5da6a>] dump_header+0x90/0x229
[361880.822268] [<ffffffffb87ba306>] ? find_lock_task_mm+0x56/0xc0
[361880.825642] [<ffffffffb87ba7b4>] oom_kill_process+0x254/0x3d0
[361880.828286] [<ffffffffb8835506>] mem_cgroup_oom_synchronize+0x546/0x570
[361880.830716] [<ffffffffb8834980>] ? mem_cgroup_charge_common+0xc0/0xc0
[361880.833146] [<ffffffffb87bb044>] pagefault_out_of_memory+0x14/0x90
[361880.835763] [<ffffffffb8d5bf72>] mm_fault_error+0x6a/0x157
[361880.837908] [<ffffffffb8d707c8>] __do_page_fault+0x3c8/0x4f0
[361880.840091] [<ffffffffb8d709d6>] trace_do_page_fault+0x56/0x150
[361880.842672] [<ffffffffb8d6ff62>] do_async_page_fault+0x22/0xf0
[361880.845080] [<ffffffffb8d6c798>] async_page_fault+0x28/0x30
[361880.847442] Task in /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/173cfe739439d02d6fd441bf2d446a936e8f64e9c994111862bbf4980d779fd8 killed as a result of limit of /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299
[361880.854097] memory: usage 4194304kB, limit 4194304kB, failcnt 2129
[361880.856760] memory+swap: usage 4194304kB, limit 9007199254740988kB, failcnt 0
[361880.859578] kmem: usage 2908432kB, limit 9007199254740988kB, failcnt 0
[361880.862314] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
[361880.869737] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/a1cc34f261f4e6478100777358ad8b952f9ebcadfd0c244f77f225239c286bbf: cache:0KB rss:44KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:44KB inactive_file:0KB active_file:0KB unevictable:0KB
[361880.880140] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/173cfe739439d02d6fd441bf2d446a936e8f64e9c994111862bbf4980d779fd8: cache:40KB rss:1285788KB rss_huge:6144KB mapped_file:16KB swap:0KB inactive_anon:0KB active_anon:1285788KB inactive_file:4KB active_file:32KB unevictable:0KB
[361880.890283] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[361880.893426] [24821] 0 24821 253 1 4 0 -998 pause
[361880.896602] [29490] 1000 29490 1321721 325834 743 0 872 java
[361880.899700] [26578] 1000 26578 2921 351 11 0 872 sh
[361880.902807] [26583] 1000 26583 21338 1207 46 0 872 curl
[361880.905737] Memory cgroup out of memory: Kill process 716 (elasticsearch[s) score 1182 or sacrifice child
[361880.909230] Killed process 29490 (java) total-vm:5286884kB, anon-rss:1281264kB, file-rss:22072kB, shmem-rss:0kB
[centos@k8s-minion-1 ~]$ ^C
[centos@k8s-minion-1 ~]$ dmesg > b.txt
[centos@k8s-minion-1 ~]$ tail -n150 b.txt
[362440.748075] memory: usage 4194304kB, limit 4194304kB, failcnt 6025
[362440.750765] memory+swap: usage 4194304kB, limit 9007199254740988kB, failcnt 0
[362440.753695] kmem: usage 2903532kB, limit 9007199254740988kB, failcnt 0
[362440.756102] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
[362440.763801] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/a1cc34f261f4e6478100777358ad8b952f9ebcadfd0c244f77f225239c286bbf: cache:0KB rss:44KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:44KB inactive_file:0KB active_file:0KB unevictable:0KB
[362440.774567] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/7c9a2a997da7122aedf1b215e30d9d5ae160423d8edf84ff7349a9c414f86d30: cache:0KB rss:1290728KB rss_huge:2048KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:1290680KB inactive_file:0KB active_file:0KB unevictable:0KB
[362440.785169] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[362440.788535] [24821] 0 24821 253 1 4 0 -998 pause
[362440.791815] [26998] 1000 26998 1271286 326973 740 0 872 java
[362440.794974] [27851] 1000 27851 17192 1374 31 0 872 controller
[362440.798323] [22798] 1000 22798 2921 351 11 0 872 sh
[362440.800850] [22804] 1000 22804 21338 1209 46 0 872 curl
[362440.803779] Memory cgroup out of memory: Kill process 8700 (elasticsearch[s) score 1184 or sacrifice child
[362440.807059] Killed process 27851 (controller) total-vm:68768kB, anon-rss:692kB, file-rss:4804kB, shmem-rss:0kB
[362440.826209] SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
[362440.828989] cache: dentry(735:7c9a2a997da7122aedf1b215e30d9d5ae160423d8edf84ff7349a9c414f86d30), object size: 192, buffer size: 192, default order: 0, min order: 0
[362440.834410] node 0: slabs: 69201, objs: 1453221, free: 0
[362440.844795] curl invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=872
[362440.847961] curl cpuset=7c9a2a997da7122aedf1b215e30d9d5ae160423d8edf84ff7349a9c414f86d30 mems_allowed=0
[362440.851447] CPU: 4 PID: 22804 Comm: curl Kdump: loaded Tainted: G ------------ T 3.10.0-957.12.2.el7.x86_64 #1
[362440.855292] Hardware name: Fedora Project OpenStack Nova, BIOS 1.10.2-3.el7_4.1.tis.2 04/01/2014
[362440.858663] Call Trace:
[362440.860684] [<ffffffffb8d63041>] dump_stack+0x19/0x1b
[362440.863253] [<ffffffffb8d5da6a>] dump_header+0x90/0x229
[362440.865850] [<ffffffffb87ba306>] ? find_lock_task_mm+0x56/0xc0
[362440.868613] [<ffffffffb87ba7b4>] oom_kill_process+0x254/0x3d0
[362440.871177] [<ffffffffb8835506>] mem_cgroup_oom_synchronize+0x546/0x570
[362440.874019] [<ffffffffb8834980>] ? mem_cgroup_charge_common+0xc0/0xc0
[362440.877581] [<ffffffffb87bb044>] pagefault_out_of_memory+0x14/0x90
[362440.880368] [<ffffffffb8d5bf72>] mm_fault_error+0x6a/0x157
[362440.882955] [<ffffffffb8d707c8>] __do_page_fault+0x3c8/0x4f0
[362440.885579] [<ffffffffb8d709d6>] trace_do_page_fault+0x56/0x150
[362440.888203] [<ffffffffb8d6ff62>] do_async_page_fault+0x22/0xf0
[362440.890832] [<ffffffffb8d6c798>] async_page_fault+0x28/0x30
[362440.893394] Task in /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/7c9a2a997da7122aedf1b215e30d9d5ae160423d8edf84ff7349a9c414f86d30 killed as a result of limit of /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299
[362440.900702] memory: usage 4194304kB, limit 4194304kB, failcnt 6142
[362440.903587] memory+swap: usage 4194304kB, limit 9007199254740988kB, failcnt 0
[362440.906765] kmem: usage 2904140kB, limit 9007199254740988kB, failcnt 0
[362440.909683] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
[362440.917120] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/a1cc34f261f4e6478100777358ad8b952f9ebcadfd0c244f77f225239c286bbf: cache:0KB rss:44KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:44KB inactive_file:0KB active_file:0KB unevictable:0KB
[362440.927463] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/7c9a2a997da7122aedf1b215e30d9d5ae160423d8edf84ff7349a9c414f86d30: cache:40KB rss:1290080KB rss_huge:2048KB mapped_file:16KB swap:0KB inactive_anon:0KB active_anon:1290080KB inactive_file:8KB active_file:32KB unevictable:0KB
[362440.937985] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[362440.941269] [24821] 0 24821 253 1 4 0 -998 pause
[362440.944337] [26998] 1000 26998 1271286 326986 740 0 872 java
[362440.947179] [22798] 1000 22798 2921 351 11 0 872 sh
[362440.950054] [22804] 1000 22804 21892 1347 48 0 872 curl
[362440.952976] Memory cgroup out of memory: Kill process 8700 (elasticsearch[s) score 1184 or sacrifice child
[362440.956103] Killed process 26998 (java) total-vm:5085144kB, anon-rss:1285876kB, file-rss:22068kB, shmem-rss:0kB
[362440.961146] curl invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=872
[362440.964601] curl cpuset=7c9a2a997da7122aedf1b215e30d9d5ae160423d8edf84ff7349a9c414f86d30 mems_allowed=0
[362440.968200] CPU: 5 PID: 22804 Comm: curl Kdump: loaded Tainted: G ------------ T 3.10.0-957.12.2.el7.x86_64 #1
[362440.972502] Hardware name: Fedora Project OpenStack Nova, BIOS 1.10.2-3.el7_4.1.tis.2 04/01/2014
[362440.975911] Call Trace:
[362440.978021] [<ffffffffb8d63041>] dump_stack+0x19/0x1b
[362440.980606] [<ffffffffb8d5da6a>] dump_header+0x90/0x229
[362440.983235] [<ffffffffb87ba306>] ? find_lock_task_mm+0x56/0xc0
[362440.986024] [<ffffffffb87ba7b4>] oom_kill_process+0x254/0x3d0
[362440.988624] [<ffffffffb8835506>] mem_cgroup_oom_synchronize+0x546/0x570
[362440.991403] [<ffffffffb8834980>] ? mem_cgroup_charge_common+0xc0/0xc0
[362440.994234] [<ffffffffb87bb044>] pagefault_out_of_memory+0x14/0x90
[362440.997242] [<ffffffffb8d5bf72>] mm_fault_error+0x6a/0x157
[362440.999699] [<ffffffffb8d707c8>] __do_page_fault+0x3c8/0x4f0
[362441.002036] [<ffffffffb8d709d6>] trace_do_page_fault+0x56/0x150
[362441.004700] [<ffffffffb8d6ff62>] do_async_page_fault+0x22/0xf0
[362441.007305] [<ffffffffb8d6c798>] async_page_fault+0x28/0x30
[362441.009787] Task in /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/7c9a2a997da7122aedf1b215e30d9d5ae160423d8edf84ff7349a9c414f86d30 killed as a result of limit of /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299
[362441.016855] memory: usage 4192256kB, limit 4194304kB, failcnt 6171
[362441.019613] memory+swap: usage 4192256kB, limit 9007199254740988kB, failcnt 0
[362441.022727] kmem: usage 2904144kB, limit 9007199254740988kB, failcnt 0
[362441.025454] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
[362441.033243] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/a1cc34f261f4e6478100777358ad8b952f9ebcadfd0c244f77f225239c286bbf: cache:0KB rss:44KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:44KB inactive_file:0KB active_file:0KB unevictable:0KB
[362441.043710] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/7c9a2a997da7122aedf1b215e30d9d5ae160423d8edf84ff7349a9c414f86d30: cache:0KB rss:567108KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:554804KB inactive_file:0KB active_file:0KB unevictable:0KB
[362441.054645] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[362441.058226] [24821] 0 24821 253 1 4 0 -998 pause
[362441.061816] [22798] 1000 22798 2921 351 11 0 872 sh
[362441.065402] [22804] 1000 22804 21925 1410 48 0 872 curl
[362441.069493] Memory cgroup out of memory: Kill process 22804 (curl) score 872 or sacrifice child
[362441.073248] Killed process 22804 (curl) total-vm:87700kB, anon-rss:1568kB, file-rss:4072kB, shmem-rss:0kB
[362534.537159] SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
[362534.539953] cache: dentry(734:dff61daf575d1abc2b87773b3bc8a99be315ddbe32e82717d3595de6f1bd975f), object size: 192, buffer size: 192, default order: 0, min order: 0
[362534.546160] node 0: slabs: 671470, objs: 14100870, free: 0
[362534.549619] curl invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=872
[362534.552715] curl cpuset=dff61daf575d1abc2b87773b3bc8a99be315ddbe32e82717d3595de6f1bd975f mems_allowed=0
[362534.556437] CPU: 3 PID: 28041 Comm: curl Kdump: loaded Tainted: G ------------ T 3.10.0-957.12.2.el7.x86_64 #1
[362534.560224] Hardware name: Fedora Project OpenStack Nova, BIOS 1.10.2-3.el7_4.1.tis.2 04/01/2014
[362534.563619] Call Trace:
[362534.565729] [<ffffffffb8d63041>] dump_stack+0x19/0x1b
[362534.568654] [<ffffffffb8d5da6a>] dump_header+0x90/0x229
[362534.571745] [<ffffffffb87ba306>] ? find_lock_task_mm+0x56/0xc0
[362534.574491] [<ffffffffb87ba7b4>] oom_kill_process+0x254/0x3d0
[362534.577481] [<ffffffffb8835506>] mem_cgroup_oom_synchronize+0x546/0x570
[362534.580460] [<ffffffffb8834980>] ? mem_cgroup_charge_common+0xc0/0xc0
[362534.583605] [<ffffffffb87bb044>] pagefault_out_of_memory+0x14/0x90
[362534.586477] [<ffffffffb8d5bf72>] mm_fault_error+0x6a/0x157
[362534.589488] [<ffffffffb8d707c8>] __do_page_fault+0x3c8/0x4f0
[362534.592160] [<ffffffffb8d709d6>] trace_do_page_fault+0x56/0x150
[362534.594795] [<ffffffffb8d6ff62>] do_async_page_fault+0x22/0xf0
[362534.597315] [<ffffffffb8d6c798>] async_page_fault+0x28/0x30
[362534.599777] Task in /kubepods/burstable/podd7cfcc82-b508-11e9-b430-fa163e29f299/dff61daf575d1abc2b87773b3bc8a99be315ddbe32e82717d3595de6f1bd975f killed as a result of limit of /kubepods/burstable/podd7cfcc82-b508-11e9-b430-fa163e29f299
[362534.607093] memory: usage 4194304kB, limit 4194304kB, failcnt 2361
[362534.610266] memory+swap: usage 4194304kB, limit 9007199254740988kB, failcnt 0
[362534.613986] kmem: usage 2890512kB, limit 9007199254740988kB, failcnt 0
[362534.616883] Memory cgroup stats for /kubepods/burstable/podd7cfcc82-b508-11e9-b430-fa163e29f299: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
[362534.624404] Memory cgroup stats for /kubepods/burstable/podd7cfcc82-b508-11e9-b430-fa163e29f299/5fec18db38d0c38de2e6990ab274d2399ffefe18806c605757865db5d89e4439: cache:0KB rss:44KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:44KB inactive_file:0KB active_file:0KB unevictable:0KB
[362534.635069] Memory cgroup stats for /kubepods/burstable/podd7cfcc82-b508-11e9-b430-fa163e29f299/dff61daf575d1abc2b87773b3bc8a99be315ddbe32e82717d3595de6f1bd975f: cache:12KB rss:1303736KB rss_huge:86016KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:1303736KB inactive_file:4KB active_file:8KB unevictable:0KB
[362534.645365] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[362534.648922] [25311] 0 25311 253 1 4 0 -998 pause
[362534.652913] [29843] 1000 29843 1355515 330132 755 0 872 java
[362534.656383] [30980] 1000 30980 17192 1374 31 0 872 controller
[362534.659471] [28031] 1000 28031 2921 349 11 0 872 sh
[362534.662608] [28041] 1000 28041 21338 1208 47 0 872 curl
[362534.665587] Memory cgroup out of memory: Kill process 749 (elasticsearch[s) score 1187 or sacrifice child
[362534.668802] Killed process 30980 (controller) total-vm:68768kB, anon-rss:692kB, file-rss:4804kB, shmem-rss:0kB
[362534.690340] SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
[362534.693131] cache: dentry(734:dff61daf575d1abc2b87773b3bc8a99be315ddbe32e82717d3595de6f1bd975f), object size: 192, buffer size: 192, default order: 0, min order: 0
[362534.699060] node 0: slabs: 671549, objs: 14102529, free: 0
[362534.702606] SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
[362534.705325] cache: dentry(734:dff61daf575d1abc2b87773b3bc8a99be315ddbe32e82717d3595de6f1bd975f), object size: 192, buffer size: 192, default order: 0, min order: 0
[362534.711052] node 0: slabs: 671552, objs: 14102592, free: 0
[362534.715700] curl invoked oom-killer: gfp_mask=0x50, order=0, oom_score_adj=872
[362534.718810] curl cpuset=dff61daf575d1abc2b87773b3bc8a99be315ddbe32e82717d3595de6f1bd975f mems_allowed=0
[362534.722279] CPU: 3 PID: 28041 Comm: curl Kdump: loaded Tainted: G ------------ T 3.10.0-957.12.2.el7.x86_64 #1
[362534.726518] Hardware name: Fedora Project OpenStack Nova, BIOS 1.10.2-3.el7_4.1.tis.2 04/01/2014
[362534.730796] Call Trace:
[362534.734105] [<ffffffffb8d63041>] dump_stack+0x19/0x1b
[362534.738216] [<ffffffffb8d5da6a>] dump_header+0x90/0x229
[362534.742079] [<ffffffffb87ba306>] ? find_lock_task_mm+0x56/0xc0
[362534.745907] [<ffffffffb87ba7b4>] oom_kill_process+0x254/0x3d0
[362534.749698] [<ffffffffb8835506>] mem_cgroup_oom_synchronize+0x546/0x570
[362534.753707] [<ffffffffb8834980>] ? mem_cgroup_charge_common+0xc0/0xc0
[362534.757568] [<ffffffffb87bb044>] pagefault_out_of_memory+0x14/0x90
[362534.761166] [<ffffffffb8d5bf72>] mm_fault_error+0x6a/0x157
[362534.763680] [<ffffffffb8d707c8>] __do_page_fault+0x3c8/0x4f0
[362534.766197] [<ffffffffb8d709d6>] trace_do_page_fault+0x56/0x150
[362534.768949] [<ffffffffb8d6ff62>] do_async_page_fault+0x22/0xf0
[362534.771669] [<ffffffffb8d6c798>] async_page_fault+0x28/0x30
[362534.774297] Task in /kubepods/burstable/podd7cfcc82-b508-11e9-b430-fa163e29f299/dff61daf575d1abc2b87773b3bc8a99be315ddbe32e82717d3595de6f1bd975f killed as a result of limit of /kubepods/burstable/podd7cfcc82-b508-11e9-b430-fa163e29f299
[362534.782741] memory: usage 4194008kB, limit 4194304kB, failcnt 2508
[362534.785500] memory+swap: usage 4194008kB, limit 9007199254740988kB, failcnt 0
[362534.788661] kmem: usage 2890764kB, limit 9007199254740988kB, failcnt 0
[362534.791499] Memory cgroup stats for /kubepods/burstable/podd7cfcc82-b508-11e9-b430-fa163e29f299: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
[362534.799125] Memory cgroup stats for /kubepods/burstable/podd7cfcc82-b508-11e9-b430-fa163e29f299/5fec18db38d0c38de2e6990ab274d2399ffefe18806c605757865db5d89e4439: cache:0KB rss:44KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:44KB inactive_file:0KB active_file:0KB unevictable:0KB
[362534.809299] Memory cgroup stats for /kubepods/burstable/podd7cfcc82-b508-11e9-b430-fa163e29f299/dff61daf575d1abc2b87773b3bc8a99be315ddbe32e82717d3595de6f1bd975f: cache:160KB rss:1303040KB rss_huge:86016KB mapped_file:8KB swap:0KB inactive_anon:0KB active_anon:1303040KB inactive_file:100KB active_file:60KB unevictable:0KB
[362534.819807] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[362534.823071] [25311] 0 25311 253 1 4 0 -998 pause
[362534.826170] [29843] 1000 29843 1355515 330141 755 0 872 java
[362534.829152] [28031] 1000 28031 2921 349 11 0 872 sh
[362534.832316] [28041] 1000 28041 21895 1344 47 0 872 curl
[362534.835364] Memory cgroup out of memory: Kill process 749 (elasticsearch[s) score 1187 or sacrifice child
[362534.838772] Killed process 29843 (java) total-vm:5422060kB, anon-rss:1298496kB, file-rss:22068kB, shmem-rss:0kB
[centos@k8s-minion-1 ~]$
[centos@k8s-minion-1 ~]$
[centos@k8s-minion-1 ~]$
[centos@k8s-minion-1 ~]$ tail -n200 b.txt
[361880.807730] CPU: 4 PID: 26583 Comm: curl Kdump: loaded Tainted: G ------------ T 3.10.0-957.12.2.el7.x86_64 #1
[361880.811756] Hardware name: Fedora Project OpenStack Nova, BIOS 1.10.2-3.el7_4.1.tis.2 04/01/2014
[361880.815167] Call Trace:
[361880.817282] [<ffffffffb8d63041>] dump_stack+0x19/0x1b
[361880.819730] [<ffffffffb8d5da6a>] dump_header+0x90/0x229
[361880.822268] [<ffffffffb87ba306>] ? find_lock_task_mm+0x56/0xc0
[361880.825642] [<ffffffffb87ba7b4>] oom_kill_process+0x254/0x3d0
[361880.828286] [<ffffffffb8835506>] mem_cgroup_oom_synchronize+0x546/0x570
[361880.830716] [<ffffffffb8834980>] ? mem_cgroup_charge_common+0xc0/0xc0
[361880.833146] [<ffffffffb87bb044>] pagefault_out_of_memory+0x14/0x90
[361880.835763] [<ffffffffb8d5bf72>] mm_fault_error+0x6a/0x157
[361880.837908] [<ffffffffb8d707c8>] __do_page_fault+0x3c8/0x4f0
[361880.840091] [<ffffffffb8d709d6>] trace_do_page_fault+0x56/0x150
[361880.842672] [<ffffffffb8d6ff62>] do_async_page_fault+0x22/0xf0
[361880.845080] [<ffffffffb8d6c798>] async_page_fault+0x28/0x30
[361880.847442] Task in /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/173cfe739439d02d6fd441bf2d446a936e8f64e9c994111862bbf4980d779fd8 killed as a result of limit of /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299
[361880.854097] memory: usage 4194304kB, limit 4194304kB, failcnt 2129
[361880.856760] memory+swap: usage 4194304kB, limit 9007199254740988kB, failcnt 0
[361880.859578] kmem: usage 2908432kB, limit 9007199254740988kB, failcnt 0
[361880.862314] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
[361880.869737] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/a1cc34f261f4e6478100777358ad8b952f9ebcadfd0c244f77f225239c286bbf: cache:0KB rss:44KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:44KB inactive_file:0KB active_file:0KB unevictable:0KB
[361880.880140] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/173cfe739439d02d6fd441bf2d446a936e8f64e9c994111862bbf4980d779fd8: cache:40KB rss:1285788KB rss_huge:6144KB mapped_file:16KB swap:0KB inactive_anon:0KB active_anon:1285788KB inactive_file:4KB active_file:32KB unevictable:0KB
[361880.890283] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[361880.893426] [24821] 0 24821 253 1 4 0 -998 pause
[361880.896602] [29490] 1000 29490 1321721 325834 743 0 872 java
[361880.899700] [26578] 1000 26578 2921 351 11 0 872 sh
[361880.902807] [26583] 1000 26583 21338 1207 46 0 872 curl
[361880.905737] Memory cgroup out of memory: Kill process 716 (elasticsearch[s) score 1182 or sacrifice child
[361880.909230] Killed process 29490 (java) total-vm:5286884kB, anon-rss:1281264kB, file-rss:22072kB, shmem-rss:0kB
[362440.682866] SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
[362440.685696] cache: dentry(735:7c9a2a997da7122aedf1b215e30d9d5ae160423d8edf84ff7349a9c414f86d30), object size: 192, buffer size: 192, default order: 0, min order: 0
[362440.691456] node 0: slabs: 69086, objs: 1450806, free: 0
[362440.694407] curl invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=872
[362440.697317] curl cpuset=7c9a2a997da7122aedf1b215e30d9d5ae160423d8edf84ff7349a9c414f86d30 mems_allowed=0
[362440.700643] CPU: 0 PID: 22804 Comm: curl Kdump: loaded Tainted: G ------------ T 3.10.0-957.12.2.el7.x86_64 #1
[362440.704578] Hardware name: Fedora Project OpenStack Nova, BIOS 1.10.2-3.el7_4.1.tis.2 04/01/2014
[362440.708012] Call Trace:
[362440.710105] [<ffffffffb8d63041>] dump_stack+0x19/0x1b
[362440.712768] [<ffffffffb8d5da6a>] dump_header+0x90/0x229
[362440.715473] [<ffffffffb87ba306>] ? find_lock_task_mm+0x56/0xc0
[362440.718189] [<ffffffffb87ba7b4>] oom_kill_process+0x254/0x3d0
[362440.720814] [<ffffffffb8835506>] mem_cgroup_oom_synchronize+0x546/0x570
[362440.723282] [<ffffffffb8834980>] ? mem_cgroup_charge_common+0xc0/0xc0
[362440.725830] [<ffffffffb87bb044>] pagefault_out_of_memory+0x14/0x90
[362440.728524] [<ffffffffb8d5bf72>] mm_fault_error+0x6a/0x157
[362440.731108] [<ffffffffb8d707c8>] __do_page_fault+0x3c8/0x4f0
[362440.733656] [<ffffffffb8d709d6>] trace_do_page_fault+0x56/0x150
[362440.735778] [<ffffffffb8d6ff62>] do_async_page_fault+0x22/0xf0
[362440.738393] [<ffffffffb8d6c798>] async_page_fault+0x28/0x30
[362440.740950] Task in /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/7c9a2a997da7122aedf1b215e30d9d5ae160423d8edf84ff7349a9c414f86d30 killed as a result of limit of /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299
[362440.748075] memory: usage 4194304kB, limit 4194304kB, failcnt 6025
[362440.750765] memory+swap: usage 4194304kB, limit 9007199254740988kB, failcnt 0
[362440.753695] kmem: usage 2903532kB, limit 9007199254740988kB, failcnt 0
[362440.756102] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
[362440.763801] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/a1cc34f261f4e6478100777358ad8b952f9ebcadfd0c244f77f225239c286bbf: cache:0KB rss:44KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:44KB inactive_file:0KB active_file:0KB unevictable:0KB
[362440.774567] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/7c9a2a997da7122aedf1b215e30d9d5ae160423d8edf84ff7349a9c414f86d30: cache:0KB rss:1290728KB rss_huge:2048KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:1290680KB inactive_file:0KB active_file:0KB unevictable:0KB
[362440.785169] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[362440.788535] [24821] 0 24821 253 1 4 0 -998 pause
[362440.791815] [26998] 1000 26998 1271286 326973 740 0 872 java
[362440.794974] [27851] 1000 27851 17192 1374 31 0 872 controller
[362440.798323] [22798] 1000 22798 2921 351 11 0 872 sh
[362440.800850] [22804] 1000 22804 21338 1209 46 0 872 curl
[362440.803779] Memory cgroup out of memory: Kill process 8700 (elasticsearch[s) score 1184 or sacrifice child
[362440.807059] Killed process 27851 (controller) total-vm:68768kB, anon-rss:692kB, file-rss:4804kB, shmem-rss:0kB
[362440.826209] SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
[362440.828989] cache: dentry(735:7c9a2a997da7122aedf1b215e30d9d5ae160423d8edf84ff7349a9c414f86d30), object size: 192, buffer size: 192, default order: 0, min order: 0
[362440.834410] node 0: slabs: 69201, objs: 1453221, free: 0
[362440.844795] curl invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=872
[362440.847961] curl cpuset=7c9a2a997da7122aedf1b215e30d9d5ae160423d8edf84ff7349a9c414f86d30 mems_allowed=0
[362440.851447] CPU: 4 PID: 22804 Comm: curl Kdump: loaded Tainted: G ------------ T 3.10.0-957.12.2.el7.x86_64 #1
[362440.855292] Hardware name: Fedora Project OpenStack Nova, BIOS 1.10.2-3.el7_4.1.tis.2 04/01/2014
[362440.858663] Call Trace:
[362440.860684] [<ffffffffb8d63041>] dump_stack+0x19/0x1b
[362440.863253] [<ffffffffb8d5da6a>] dump_header+0x90/0x229
[362440.865850] [<ffffffffb87ba306>] ? find_lock_task_mm+0x56/0xc0
[362440.868613] [<ffffffffb87ba7b4>] oom_kill_process+0x254/0x3d0
[362440.871177] [<ffffffffb8835506>] mem_cgroup_oom_synchronize+0x546/0x570
[362440.874019] [<ffffffffb8834980>] ? mem_cgroup_charge_common+0xc0/0xc0
[362440.877581] [<ffffffffb87bb044>] pagefault_out_of_memory+0x14/0x90
[362440.880368] [<ffffffffb8d5bf72>] mm_fault_error+0x6a/0x157
[362440.882955] [<ffffffffb8d707c8>] __do_page_fault+0x3c8/0x4f0
[362440.885579] [<ffffffffb8d709d6>] trace_do_page_fault+0x56/0x150
[362440.888203] [<ffffffffb8d6ff62>] do_async_page_fault+0x22/0xf0
[362440.890832] [<ffffffffb8d6c798>] async_page_fault+0x28/0x30
[362440.893394] Task in /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/7c9a2a997da7122aedf1b215e30d9d5ae160423d8edf84ff7349a9c414f86d30 killed as a result of limit of /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299
[362440.900702] memory: usage 4194304kB, limit 4194304kB, failcnt 6142
[362440.903587] memory+swap: usage 4194304kB, limit 9007199254740988kB, failcnt 0
[362440.906765] kmem: usage 2904140kB, limit 9007199254740988kB, failcnt 0
[362440.909683] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
[362440.917120] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/a1cc34f261f4e6478100777358ad8b952f9ebcadfd0c244f77f225239c286bbf: cache:0KB rss:44KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:44KB inactive_file:0KB active_file:0KB unevictable:0KB
[362440.927463] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/7c9a2a997da7122aedf1b215e30d9d5ae160423d8edf84ff7349a9c414f86d30: cache:40KB rss:1290080KB rss_huge:2048KB mapped_file:16KB swap:0KB inactive_anon:0KB active_anon:1290080KB inactive_file:8KB active_file:32KB unevictable:0KB
[362440.937985] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[362440.941269] [24821] 0 24821 253 1 4 0 -998 pause
[362440.944337] [26998] 1000 26998 1271286 326986 740 0 872 java
[362440.947179] [22798] 1000 22798 2921 351 11 0 872 sh
[362440.950054] [22804] 1000 22804 21892 1347 48 0 872 curl
[362440.952976] Memory cgroup out of memory: Kill process 8700 (elasticsearch[s) score 1184 or sacrifice child
[362440.956103] Killed process 26998 (java) total-vm:5085144kB, anon-rss:1285876kB, file-rss:22068kB, shmem-rss:0kB
[362440.961146] curl invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=872
[362440.964601] curl cpuset=7c9a2a997da7122aedf1b215e30d9d5ae160423d8edf84ff7349a9c414f86d30 mems_allowed=0
[362440.968200] CPU: 5 PID: 22804 Comm: curl Kdump: loaded Tainted: G ------------ T 3.10.0-957.12.2.el7.x86_64 #1
[362440.972502] Hardware name: Fedora Project OpenStack Nova, BIOS 1.10.2-3.el7_4.1.tis.2 04/01/2014
[362440.975911] Call Trace:
[362440.978021] [<ffffffffb8d63041>] dump_stack+0x19/0x1b
[362440.980606] [<ffffffffb8d5da6a>] dump_header+0x90/0x229
[362440.983235] [<ffffffffb87ba306>] ? find_lock_task_mm+0x56/0xc0
[362440.986024] [<ffffffffb87ba7b4>] oom_kill_process+0x254/0x3d0
[362440.988624] [<ffffffffb8835506>] mem_cgroup_oom_synchronize+0x546/0x570
[362440.991403] [<ffffffffb8834980>] ? mem_cgroup_charge_common+0xc0/0xc0
[362440.994234] [<ffffffffb87bb044>] pagefault_out_of_memory+0x14/0x90
[362440.997242] [<ffffffffb8d5bf72>] mm_fault_error+0x6a/0x157
[362440.999699] [<ffffffffb8d707c8>] __do_page_fault+0x3c8/0x4f0
[362441.002036] [<ffffffffb8d709d6>] trace_do_page_fault+0x56/0x150
[362441.004700] [<ffffffffb8d6ff62>] do_async_page_fault+0x22/0xf0
[362441.007305] [<ffffffffb8d6c798>] async_page_fault+0x28/0x30
[362441.009787] Task in /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/7c9a2a997da7122aedf1b215e30d9d5ae160423d8edf84ff7349a9c414f86d30 killed as a result of limit of /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299
[362441.016855] memory: usage 4192256kB, limit 4194304kB, failcnt 6171
[362441.019613] memory+swap: usage 4192256kB, limit 9007199254740988kB, failcnt 0
[362441.022727] kmem: usage 2904144kB, limit 9007199254740988kB, failcnt 0
[362441.025454] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
[362441.033243] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/a1cc34f261f4e6478100777358ad8b952f9ebcadfd0c244f77f225239c286bbf: cache:0KB rss:44KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:44KB inactive_file:0KB active_file:0KB unevictable:0KB
[362441.043710] Memory cgroup stats for /kubepods/burstable/podda7d21e3-b508-11e9-b430-fa163e29f299/7c9a2a997da7122aedf1b215e30d9d5ae160423d8edf84ff7349a9c414f86d30: cache:0KB rss:567108KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:554804KB inactive_file:0KB active_file:0KB unevictable:0KB
[362441.054645] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[362441.058226] [24821] 0 24821 253 1 4 0 -998 pause
[362441.061816] [22798] 1000 22798 2921 351 11 0 872 sh
[362441.065402] [22804] 1000 22804 21925 1410 48 0 872 curl
[362441.069493] Memory cgroup out of memory: Kill process 22804 (curl) score 872 or sacrifice child
[362441.073248] Killed process 22804 (curl) total-vm:87700kB, anon-rss:1568kB, file-rss:4072kB, shmem-rss:0kB
[362534.537159] SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
[362534.539953] cache: dentry(734:dff61daf575d1abc2b87773b3bc8a99be315ddbe32e82717d3595de6f1bd975f), object size: 192, buffer size: 192, default order: 0, min order: 0
[362534.546160] node 0: slabs: 671470, objs: 14100870, free: 0
[362534.549619] curl invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=872
[362534.552715] curl cpuset=dff61daf575d1abc2b87773b3bc8a99be315ddbe32e82717d3595de6f1bd975f mems_allowed=0
[362534.556437] CPU: 3 PID: 28041 Comm: curl Kdump: loaded Tainted: G ------------ T 3.10.0-957.12.2.el7.x86_64 #1
[362534.560224] Hardware name: Fedora Project OpenStack Nova, BIOS 1.10.2-3.el7_4.1.tis.2 04/01/2014
[362534.563619] Call Trace:
[362534.565729] [<ffffffffb8d63041>] dump_stack+0x19/0x1b
[362534.568654] [<ffffffffb8d5da6a>] dump_header+0x90/0x229
[362534.571745] [<ffffffffb87ba306>] ? find_lock_task_mm+0x56/0xc0
[362534.574491] [<ffffffffb87ba7b4>] oom_kill_process+0x254/0x3d0
[362534.577481] [<ffffffffb8835506>] mem_cgroup_oom_synchronize+0x546/0x570
[362534.580460] [<ffffffffb8834980>] ? mem_cgroup_charge_common+0xc0/0xc0
[362534.583605] [<ffffffffb87bb044>] pagefault_out_of_memory+0x14/0x90
[362534.586477] [<ffffffffb8d5bf72>] mm_fault_error+0x6a/0x157
[362534.589488] [<ffffffffb8d707c8>] __do_page_fault+0x3c8/0x4f0
[362534.592160] [<ffffffffb8d709d6>] trace_do_page_fault+0x56/0x150
[362534.594795] [<ffffffffb8d6ff62>] do_async_page_fault+0x22/0xf0
[362534.597315] [<ffffffffb8d6c798>] async_page_fault+0x28/0x30
[362534.599777] Task in /kubepods/burstable/podd7cfcc82-b508-11e9-b430-fa163e29f299/dff61daf575d1abc2b87773b3bc8a99be315ddbe32e82717d3595de6f1bd975f killed as a result of limit of /kubepods/burstable/podd7cfcc82-b508-11e9-b430-fa163e29f299
[362534.607093] memory: usage 4194304kB, limit 4194304kB, failcnt 2361
[362534.610266] memory+swap: usage 4194304kB, limit 9007199254740988kB, failcnt 0
[362534.613986] kmem: usage 2890512kB, limit 9007199254740988kB, failcnt 0
[362534.616883] Memory cgroup stats for /kubepods/burstable/podd7cfcc82-b508-11e9-b430-fa163e29f299: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
[362534.624404] Memory cgroup stats for /kubepods/burstable/podd7cfcc82-b508-11e9-b430-fa163e29f299/5fec18db38d0c38de2e6990ab274d2399ffefe18806c605757865db5d89e4439: cache:0KB rss:44KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:44KB inactive_file:0KB active_file:0KB unevictable:0KB
[362534.635069] Memory cgroup stats for /kubepods/burstable/podd7cfcc82-b508-11e9-b430-fa163e29f299/dff61daf575d1abc2b87773b3bc8a99be315ddbe32e82717d3595de6f1bd975f: cache:12KB rss:1303736KB rss_huge:86016KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:1303736KB inactive_file:4KB active_file:8KB unevictable:0KB
[362534.645365] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[362534.648922] [25311] 0 25311 253 1 4 0 -998 pause
[362534.652913] [29843] 1000 29843 1355515 330132 755 0 872 java
[362534.656383] [30980] 1000 30980 17192 1374 31 0 872 controller
[362534.659471] [28031] 1000 28031 2921 349 11 0 872 sh
[362534.662608] [28041] 1000 28041 21338 1208 47 0 872 curl
[362534.665587] Memory cgroup out of memory: Kill process 749 (elasticsearch[s) score 1187 or sacrifice child
[362534.668802] Killed process 30980 (controller) total-vm:68768kB, anon-rss:692kB, file-rss:4804kB, shmem-rss:0kB
[362534.690340] SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
[362534.693131] cache: dentry(734:dff61daf575d1abc2b87773b3bc8a99be315ddbe32e82717d3595de6f1bd975f), object size: 192, buffer size: 192, default order: 0, min order: 0
[362534.699060] node 0: slabs: 671549, objs: 14102529, free: 0
[362534.702606] SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
[362534.705325] cache: dentry(734:dff61daf575d1abc2b87773b3bc8a99be315ddbe32e82717d3595de6f1bd975f), object size: 192, buffer size: 192, default order: 0, min order: 0
[362534.711052] node 0: slabs: 671552, objs: 14102592, free: 0
[362534.715700] curl invoked oom-killer: gfp_mask=0x50, order=0, oom_score_adj=872
[362534.718810] curl cpuset=dff61daf575d1abc2b87773b3bc8a99be315ddbe32e82717d3595de6f1bd975f mems_allowed=0
[362534.722279] CPU: 3 PID: 28041 Comm: curl Kdump: loaded Tainted: G ------------ T 3.10.0-957.12.2.el7.x86_64 #1
[362534.726518] Hardware name: Fedora Project OpenStack Nova, BIOS 1.10.2-3.el7_4.1.tis.2 04/01/2014
[362534.730796] Call Trace:
[362534.734105] [<ffffffffb8d63041>] dump_stack+0x19/0x1b
[362534.738216] [<ffffffffb8d5da6a>] dump_header+0x90/0x229
[362534.742079] [<ffffffffb87ba306>] ? find_lock_task_mm+0x56/0xc0
[362534.745907] [<ffffffffb87ba7b4>] oom_kill_process+0x254/0x3d0
[362534.749698] [<ffffffffb8835506>] mem_cgroup_oom_synchronize+0x546/0x570
[362534.753707] [<ffffffffb8834980>] ? mem_cgroup_charge_common+0xc0/0xc0
[362534.757568] [<ffffffffb87bb044>] pagefault_out_of_memory+0x14/0x90
[362534.761166] [<ffffffffb8d5bf72>] mm_fault_error+0x6a/0x157
[362534.763680] [<ffffffffb8d707c8>] __do_page_fault+0x3c8/0x4f0
[362534.766197] [<ffffffffb8d709d6>] trace_do_page_fault+0x56/0x150
[362534.768949] [<ffffffffb8d6ff62>] do_async_page_fault+0x22/0xf0
[362534.771669] [<ffffffffb8d6c798>] async_page_fault+0x28/0x30
[362534.774297] Task in /kubepods/burstable/podd7cfcc82-b508-11e9-b430-fa163e29f299/dff61daf575d1abc2b87773b3bc8a99be315ddbe32e82717d3595de6f1bd975f killed as a result of limit of /kubepods/burstable/podd7cfcc82-b508-11e9-b430-fa163e29f299
[362534.782741] memory: usage 4194008kB, limit 4194304kB, failcnt 2508
[362534.785500] memory+swap: usage 4194008kB, limit 9007199254740988kB, failcnt 0
[362534.788661] kmem: usage 2890764kB, limit 9007199254740988kB, failcnt 0
[362534.791499] Memory cgroup stats for /kubepods/burstable/podd7cfcc82-b508-11e9-b430-fa163e29f299: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
[362534.799125] Memory cgroup stats for /kubepods/burstable/podd7cfcc82-b508-11e9-b430-fa163e29f299/5fec18db38d0c38de2e6990ab274d2399ffefe18806c605757865db5d89e4439: cache:0KB rss:44KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:44KB inactive_file:0KB active_file:0KB unevictable:0KB
[362534.809299] Memory cgroup stats for /kubepods/burstable/podd7cfcc82-b508-11e9-b430-fa163e29f299/dff61daf575d1abc2b87773b3bc8a99be315ddbe32e82717d3595de6f1bd975f: cache:160KB rss:1303040KB rss_huge:86016KB mapped_file:8KB swap:0KB inactive_anon:0KB active_anon:1303040KB inactive_file:100KB active_file:60KB unevictable:0KB
[362534.819807] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[362534.823071] [25311] 0 25311 253 1 4 0 -998 pause
[362534.826170] [29843] 1000 29843 1355515 330141 755 0 872 java
[362534.829152] [28031] 1000 28031 2921 349 11 0 872 sh
[362534.832316] [28041] 1000 28041 21895 1344 47 0 872 curl
[362534.835364] Memory cgroup out of memory: Kill process 749 (elasticsearch[s) score 1187 or sacrifice child
[362534.838772] Killed process 29843 (java) total-vm:5422060kB, anon-rss:1298496kB, file-rss:22068kB, shmem-rss:0kB
Created elasticsearch keystore in /usr/share/elasticsearch/config
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{"type": "server", "timestamp": "2019-08-02T06:03:39,227+0000", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "security", "node.name": "security-master-1", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/vdf)]], net usable_space [29.3gb], net total_space [29.4gb], types [ext4]" }
{"type": "server", "timestamp": "2019-08-02T06:03:39,230+0000", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "security", "node.name": "security-master-1", "message": "heap size [1015.6mb], compressed ordinary object pointers [true]" }
{"type": "server", "timestamp": "2019-08-02T06:03:39,233+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "security", "node.name": "security-master-1", "message": "node name [security-master-1], node ID [SzH19ZKqRJKAGEg2OqmG2w], cluster name [security]" }
{"type": "server", "timestamp": "2019-08-02T06:03:39,233+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "security", "node.name": "security-master-1", "message": "version[7.2.0], pid[1], build[default/docker/508c38a/2019-06-20T15:54:18.811730Z], OS[Linux/3.10.0-957.12.2.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.1/12.0.1+12]" }
{"type": "server", "timestamp": "2019-08-02T06:03:39,234+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "security", "node.name": "security-master-1", "message": "JVM home [/usr/share/elasticsearch/jdk]" }
{"type": "server", "timestamp": "2019-08-02T06:03:39,234+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "security", "node.name": "security-master-1", "message": "JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-9125738995646204065, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Des.cgroups.hierarchy.override=/, -Xmx1g, -Xms1g, -Dio.netty.allocator.type=unpooled, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,386+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [aggs-matrix-stats]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,386+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [analysis-common]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,387+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [data-frame]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,387+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [ingest-common]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,387+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [ingest-geoip]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,387+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [ingest-user-agent]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,387+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [lang-expression]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,387+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [lang-mustache]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,388+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [lang-painless]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,388+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [mapper-extras]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,388+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [parent-join]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,388+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [percolator]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,388+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [rank-eval]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,388+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [reindex]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,388+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [repository-url]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,389+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [transport-netty4]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,389+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [x-pack-ccr]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,389+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [x-pack-core]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,389+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [x-pack-deprecation]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,389+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [x-pack-graph]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,390+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [x-pack-ilm]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,390+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [x-pack-logstash]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,390+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [x-pack-ml]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,390+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [x-pack-monitoring]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,390+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [x-pack-rollup]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,390+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [x-pack-security]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,391+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [x-pack-sql]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,391+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "loaded module [x-pack-watcher]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,391+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-1", "message": "no plugins loaded" }
{"type": "server", "timestamp": "2019-08-02T06:03:52,496+0000", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "security", "node.name": "security-master-1", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
{"type": "server", "timestamp": "2019-08-02T06:03:54,204+0000", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "security", "node.name": "security-master-1", "message": "[controller/213] [Main.cc@110] controller (64 bit): Version 7.2.0 (Build 65aefcbfce449b) Copyright (c) 2019 Elasticsearch BV" }
{"type": "server", "timestamp": "2019-08-02T06:03:55,186+0000", "level": "DEBUG", "component": "o.e.a.ActionModule", "cluster.name": "security", "node.name": "security-master-1", "message": "Using REST wrapper from plugin org.elasticsearch.xpack.security.Security" }
{"type": "server", "timestamp": "2019-08-02T06:03:55,819+0000", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "security", "node.name": "security-master-1", "message": "using discovery type [zen] and seed hosts providers [settings]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,712+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "security", "node.name": "security-master-1", "message": "initialized" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,712+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "security", "node.name": "security-master-1", "message": "starting ..." }
{"type": "server", "timestamp": "2019-08-02T06:03:57,987+0000", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "security", "node.name": "security-master-1", "message": "publish_address {10.233.92.38:9300}, bound_addresses {0.0.0.0:9300}" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,995+0000", "level": "INFO", "component": "o.e.b.BootstrapChecks", "cluster.name": "security", "node.name": "security-master-1", "message": "bound or publishing to a non-loopback address, enforcing bootstrap checks" }
{"type": "server", "timestamp": "2019-08-02T06:03:59,715+0000", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "security", "node.name": "security-master-1", "message": "setting initial configuration to VotingConfiguration{JIZdkWG_Ql69ZRSu6UtdSw,SzH19ZKqRJKAGEg2OqmG2w,{bootstrap-placeholder}-security-master-0}" }
{"type": "server", "timestamp": "2019-08-02T06:04:00,689+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "security", "node.name": "security-master-1", "message": "failed to join {security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}, targetNode={security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}}]}" ,
"stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [security-master-1][10.233.92.38:9300][internal:cluster/coordination/join]",
"Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}",
"at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:451) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:512) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:299) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:375) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:183) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:130) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:246) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:306) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:317) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:244) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:306) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:703) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
"at java.lang.Thread.run(Thread.java:835) [?:?]"] }
{"type": "server", "timestamp": "2019-08-02T06:04:00,719+0000", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "security", "node.name": "security-master-1", "message": "elected-as-master ([2] nodes joined)[{security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20} elect leader, {security-master-2}{JIZdkWG_Ql69ZRSu6UtdSw}{OZ6o4MT3SYyUVR8xVMPphA}{10.233.89.152}{10.233.89.152:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 2, version: 1, reason: master node changed {previous [], current [{security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}]}, added {{security-master-2}{JIZdkWG_Ql69ZRSu6UtdSw}{OZ6o4MT3SYyUVR8xVMPphA}{10.233.89.152}{10.233.89.152:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}" }
{"type": "server", "timestamp": "2019-08-02T06:04:00,883+0000", "level": "INFO", "component": "o.e.c.c.CoordinationState", "cluster.name": "security", "node.name": "security-master-1", "message": "cluster UUID set to [iGQVix2gRMWa_Kx-yIsisA]" }
{"type": "server", "timestamp": "2019-08-02T06:04:01,002+0000", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "security", "node.name": "security-master-1", "message": "master node changed {previous [], current [{security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}]}, added {{security-master-2}{JIZdkWG_Ql69ZRSu6UtdSw}{OZ6o4MT3SYyUVR8xVMPphA}{10.233.89.152}{10.233.89.152:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 1, reason: Publication{term=2, version=1}" }
{"type": "server", "timestamp": "2019-08-02T06:04:01,089+0000", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "security", "node.name": "security-master-1", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "SzH19ZKqRJKAGEg2OqmG2w", "message": "publish_address {10.233.92.38:9200}, bound_addresses {0.0.0.0:9200}" }
{"type": "server", "timestamp": "2019-08-02T06:04:01,090+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "security", "node.name": "security-master-1", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "SzH19ZKqRJKAGEg2OqmG2w", "message": "started" }
{"type": "server", "timestamp": "2019-08-02T06:04:01,301+0000", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "security", "node.name": "security-master-1", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "SzH19ZKqRJKAGEg2OqmG2w", "message": "recovered [0] indices into cluster_state" }
{"type": "server", "timestamp": "2019-08-02T06:04:02,323+0000", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "security", "node.name": "security-master-1", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "SzH19ZKqRJKAGEg2OqmG2w", "message": "adding template [.watch-history-9] for index patterns [.watcher-history-9*]" }
{"type": "server", "timestamp": "2019-08-02T06:04:02,484+0000", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "security", "node.name": "security-master-1", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "SzH19ZKqRJKAGEg2OqmG2w", "message": "adding template [.triggered_watches] for index patterns [.triggered_watches*]" }
{"type": "server", "timestamp": "2019-08-02T06:04:02,587+0000", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "security", "node.name": "security-master-1", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "SzH19ZKqRJKAGEg2OqmG2w", "message": "adding template [.watches] for index patterns [.watches*]" }
{"type": "server", "timestamp": "2019-08-02T06:04:02,694+0000", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "security", "node.name": "security-master-1", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "SzH19ZKqRJKAGEg2OqmG2w", "message": "adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-7-*]" }
{"type": "server", "timestamp": "2019-08-02T06:04:02,882+0000", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "security", "node.name": "security-master-1", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "SzH19ZKqRJKAGEg2OqmG2w", "message": "adding template [.monitoring-es] for index patterns [.monitoring-es-7-*]" }
{"type": "server", "timestamp": "2019-08-02T06:04:03,008+0000", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "security", "node.name": "security-master-1", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "SzH19ZKqRJKAGEg2OqmG2w", "message": "adding template [.monitoring-beats] for index patterns [.monitoring-beats-7-*]" }
{"type": "server", "timestamp": "2019-08-02T06:04:03,191+0000", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "security", "node.name": "security-master-1", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "SzH19ZKqRJKAGEg2OqmG2w", "message": "adding template [.monitoring-alerts-7] for index patterns [.monitoring-alerts-7]" }
{"type": "server", "timestamp": "2019-08-02T06:04:03,389+0000", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "security", "node.name": "security-master-1", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "SzH19ZKqRJKAGEg2OqmG2w", "message": "adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-7-*]" }
{"type": "server", "timestamp": "2019-08-02T06:04:03,927+0000", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "security", "node.name": "security-master-1", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "SzH19ZKqRJKAGEg2OqmG2w", "message": "license [e909ad74-2edf-4e16-91c1-da3860be9c8e] mode [basic] - valid" }
{"type": "server", "timestamp": "2019-08-02T06:04:03,934+0000", "level": "INFO", "component": "o.e.x.i.a.TransportPutLifecycleAction", "cluster.name": "security", "node.name": "security-master-1", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "SzH19ZKqRJKAGEg2OqmG2w", "message": "adding index lifecycle policy [watch-history-ilm-policy]" }
{"type": "server", "timestamp": "2019-08-02T06:04:10,675+0000", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "security", "node.name": "security-master-1", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "SzH19ZKqRJKAGEg2OqmG2w", "message": "node-join[{security-data-0}{bOra2xonQeid3whGZqnyfw}{78U9d9jLTS-EQaQspcgJmg}{10.233.102.255}{10.233.102.255:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true} join existing leader], term: 2, version: 16, reason: added {{security-data-0}{bOra2xonQeid3whGZqnyfw}{78U9d9jLTS-EQaQspcgJmg}{10.233.102.255}{10.233.102.255:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}" }
{"type": "server", "timestamp": "2019-08-02T06:04:12,659+0000", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "security", "node.name": "security-master-1", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "SzH19ZKqRJKAGEg2OqmG2w", "message": "added {{security-data-0}{bOra2xonQeid3whGZqnyfw}{78U9d9jLTS-EQaQspcgJmg}{10.233.102.255}{10.233.102.255:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 16, reason: Publication{term=2, version=16}" }
{"type": "server", "timestamp": "2019-08-02T06:04:12,687+0000", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "security", "node.name": "security-master-1", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "SzH19ZKqRJKAGEg2OqmG2w", "message": "node-join[{security-master-0}{DoM0e1u4T7-QNqxyOs-fEg}{vNB_4MCkTFqmrE9DZ_1ETg}{10.233.120.99}{10.233.120.99:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true} join existing leader], term: 2, version: 17, reason: added {{security-master-0}{DoM0e1u4T7-QNqxyOs-fEg}{vNB_4MCkTFqmrE9DZ_1ETg}{10.233.120.99}{10.233.120.99:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}" }
{"type": "server", "timestamp": "2019-08-02T06:04:14,024+0000", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "security", "node.name": "security-master-1", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "SzH19ZKqRJKAGEg2OqmG2w", "message": "added {{security-master-0}{DoM0e1u4T7-QNqxyOs-fEg}{vNB_4MCkTFqmrE9DZ_1ETg}{10.233.120.99}{10.233.120.99:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 17, reason: Publication{term=2, version=17}" }
{"type": "server", "timestamp": "2019-08-02T06:04:14,318+0000", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "security", "node.name": "security-master-1", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "SzH19ZKqRJKAGEg2OqmG2w", "message": "node-join[{security-data-1}{XPQAk7NvSVCIhZhZwGmoKQ}{6TFIyNgzTvWTEczIBSH7sw}{10.233.92.86}{10.233.92.86:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true} join existing leader], term: 2, version: 19, reason: added {{security-data-1}{XPQAk7NvSVCIhZhZwGmoKQ}{6TFIyNgzTvWTEczIBSH7sw}{10.233.92.86}{10.233.92.86:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}" }
{"type": "server", "timestamp": "2019-08-02T06:04:16,898+0000", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "security", "node.name": "security-master-1", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "SzH19ZKqRJKAGEg2OqmG2w", "message": "added {{security-data-1}{XPQAk7NvSVCIhZhZwGmoKQ}{6TFIyNgzTvWTEczIBSH7sw}{10.233.92.86}{10.233.92.86:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 19, reason: Publication{term=2, version=19}" }
{"type": "server", "timestamp": "2019-08-02T06:04:25,810+0000", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "security", "node.name": "security-master-1", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "SzH19ZKqRJKAGEg2OqmG2w", "message": "node-join[{security-data-2}{lS0KL3TfRzi3JBDSVSzouA}{07yXpkoRRXCBRFVns2Jjew}{10.233.120.211}{10.233.120.211:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true} join existing leader], term: 2, version: 20, reason: added {{security-data-2}{lS0KL3TfRzi3JBDSVSzouA}{07yXpkoRRXCBRFVns2Jjew}{10.233.120.211}{10.233.120.211:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}" }
{"type": "server", "timestamp": "2019-08-02T06:04:28,327+0000", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "security", "node.name": "security-master-1", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "SzH19ZKqRJKAGEg2OqmG2w", "message": "added {{security-data-2}{lS0KL3TfRzi3JBDSVSzouA}{07yXpkoRRXCBRFVns2Jjew}{10.233.120.211}{10.233.120.211:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 20, reason: Publication{term=2, version=20}" }
Created elasticsearch keystore in /usr/share/elasticsearch/config
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{"type": "server", "timestamp": "2019-08-02T06:03:28,825+0000", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "security", "node.name": "security-master-2", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/vdd)]], net usable_space [29.3gb], net total_space [29.4gb], types [ext4]" }
{"type": "server", "timestamp": "2019-08-02T06:03:28,828+0000", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "security", "node.name": "security-master-2", "message": "heap size [1015.6mb], compressed ordinary object pointers [true]" }
{"type": "server", "timestamp": "2019-08-02T06:03:28,831+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "security", "node.name": "security-master-2", "message": "node name [security-master-2], node ID [JIZdkWG_Ql69ZRSu6UtdSw], cluster name [security]" }
{"type": "server", "timestamp": "2019-08-02T06:03:28,831+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "security", "node.name": "security-master-2", "message": "version[7.2.0], pid[1], build[default/docker/508c38a/2019-06-20T15:54:18.811730Z], OS[Linux/3.10.0-957.12.2.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.1/12.0.1+12]" }
{"type": "server", "timestamp": "2019-08-02T06:03:28,832+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "security", "node.name": "security-master-2", "message": "JVM home [/usr/share/elasticsearch/jdk]" }
{"type": "server", "timestamp": "2019-08-02T06:03:28,832+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "security", "node.name": "security-master-2", "message": "JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-16003449575508533165, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Des.cgroups.hierarchy.override=/, -Xmx1g, -Xms1g, -Dio.netty.allocator.type=unpooled, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,910+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [aggs-matrix-stats]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,911+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [analysis-common]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,911+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [data-frame]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,911+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [ingest-common]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,911+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [ingest-geoip]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,911+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [ingest-user-agent]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,912+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [lang-expression]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,912+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [lang-mustache]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,912+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [lang-painless]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,912+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [mapper-extras]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,912+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [parent-join]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,912+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [percolator]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,912+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [rank-eval]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,913+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [reindex]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,913+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [repository-url]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,913+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [transport-netty4]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,913+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [x-pack-ccr]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,913+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [x-pack-core]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,913+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [x-pack-deprecation]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,914+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [x-pack-graph]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,914+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [x-pack-ilm]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,914+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [x-pack-logstash]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,914+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [x-pack-ml]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,915+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [x-pack-monitoring]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,915+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [x-pack-rollup]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,915+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [x-pack-security]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,916+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [x-pack-sql]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,916+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "loaded module [x-pack-watcher]" }
{"type": "server", "timestamp": "2019-08-02T06:03:33,916+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-2", "message": "no plugins loaded" }
{"type": "server", "timestamp": "2019-08-02T06:03:42,895+0000", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "security", "node.name": "security-master-2", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
{"type": "server", "timestamp": "2019-08-02T06:03:44,814+0000", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "security", "node.name": "security-master-2", "message": "[controller/213] [Main.cc@110] controller (64 bit): Version 7.2.0 (Build 65aefcbfce449b) Copyright (c) 2019 Elasticsearch BV" }
{"type": "server", "timestamp": "2019-08-02T06:03:45,898+0000", "level": "DEBUG", "component": "o.e.a.ActionModule", "cluster.name": "security", "node.name": "security-master-2", "message": "Using REST wrapper from plugin org.elasticsearch.xpack.security.Security" }
{"type": "server", "timestamp": "2019-08-02T06:03:46,609+0000", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "security", "node.name": "security-master-2", "message": "using discovery type [zen] and seed hosts providers [settings]" }
{"type": "server", "timestamp": "2019-08-02T06:03:48,430+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "security", "node.name": "security-master-2", "message": "initialized" }
{"type": "server", "timestamp": "2019-08-02T06:03:48,430+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "security", "node.name": "security-master-2", "message": "starting ..." }
{"type": "server", "timestamp": "2019-08-02T06:03:48,792+0000", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "security", "node.name": "security-master-2", "message": "publish_address {10.233.89.152:9300}, bound_addresses {0.0.0.0:9300}" }
{"type": "server", "timestamp": "2019-08-02T06:03:48,800+0000", "level": "INFO", "component": "o.e.b.BootstrapChecks", "cluster.name": "security", "node.name": "security-master-2", "message": "bound or publishing to a non-loopback address, enforcing bootstrap checks" }
{"type": "server", "timestamp": "2019-08-02T06:03:58,817+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "security", "node.name": "security-master-2", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [security-master-0, security-master-1, security-master-2] to bootstrap a cluster: have discovered []; discovery will continue using [10.233.120.99:9300, 10.233.92.38:9300] from hosts providers and [{security-master-2}{JIZdkWG_Ql69ZRSu6UtdSw}{OZ6o4MT3SYyUVR8xVMPphA}{10.233.89.152}{10.233.89.152:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
{"type": "server", "timestamp": "2019-08-02T06:04:00,633+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "security", "node.name": "security-master-2", "message": "failed to join {security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={security-master-2}{JIZdkWG_Ql69ZRSu6UtdSw}{OZ6o4MT3SYyUVR8xVMPphA}{10.233.89.152}{10.233.89.152:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={security-master-2}{JIZdkWG_Ql69ZRSu6UtdSw}{OZ6o4MT3SYyUVR8xVMPphA}{10.233.89.152}{10.233.89.152:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}, targetNode={security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
"stacktrace": ["org.elasticsearch.transport.NodeNotConnectedException: [security-master-1][10.233.92.38:9300] Node not connected",
"at org.elasticsearch.transport.ConnectionManager.getConnection(ConnectionManager.java:151) ~[elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:568) ~[elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:540) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.cluster.coordination.JoinHelper.sendJoinRequest(JoinHelper.java:278) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.cluster.coordination.JoinHelper.sendJoinRequest(JoinHelper.java:211) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:135) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:299) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:375) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:183) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:130) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:246) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:306) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:317) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:244) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:306) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
"at java.lang.Thread.run(Thread.java:835) [?:?]"] }
{"type": "server", "timestamp": "2019-08-02T06:04:00,693+0000", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "security", "node.name": "security-master-2", "message": "setting initial configuration to VotingConfiguration{JIZdkWG_Ql69ZRSu6UtdSw,SzH19ZKqRJKAGEg2OqmG2w,{bootstrap-placeholder}-security-master-0}" }
{"type": "server", "timestamp": "2019-08-02T06:04:00,692+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "security", "node.name": "security-master-2", "message": "failed to join {security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={security-master-2}{JIZdkWG_Ql69ZRSu6UtdSw}{OZ6o4MT3SYyUVR8xVMPphA}{10.233.89.152}{10.233.89.152:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={security-master-2}{JIZdkWG_Ql69ZRSu6UtdSw}{OZ6o4MT3SYyUVR8xVMPphA}{10.233.89.152}{10.233.89.152:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}, targetNode={security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
"stacktrace": ["org.elasticsearch.transport.NodeNotConnectedException: [security-master-1][10.233.92.38:9300] Node not connected",
"at org.elasticsearch.transport.ConnectionManager.getConnection(ConnectionManager.java:151) ~[elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:568) ~[elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:540) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.cluster.coordination.JoinHelper.sendJoinRequest(JoinHelper.java:278) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.cluster.coordination.JoinHelper.sendJoinRequest(JoinHelper.java:211) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:135) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:299) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:375) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:183) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:130) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:246) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:306) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:317) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:244) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:306) [x-pack-security-7.2.0.jar:7.2.0]",
"at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
"at java.lang.Thread.run(Thread.java:835) [?:?]"] }
{"type": "server", "timestamp": "2019-08-02T06:04:00,890+0000", "level": "INFO", "component": "o.e.c.c.CoordinationState", "cluster.name": "security", "node.name": "security-master-2", "message": "cluster UUID set to [iGQVix2gRMWa_Kx-yIsisA]" }
{"type": "server", "timestamp": "2019-08-02T06:04:00,927+0000", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "security", "node.name": "security-master-2", "message": "master node changed {previous [], current [{security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}]}, added {{security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 1, reason: ApplyCommitRequest{term=2, version=1, sourceNode={security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}}" }
{"type": "server", "timestamp": "2019-08-02T06:04:01,006+0000", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "security", "node.name": "security-master-2", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "JIZdkWG_Ql69ZRSu6UtdSw", "message": "publish_address {10.233.89.152:9200}, bound_addresses {0.0.0.0:9200}" }
{"type": "server", "timestamp": "2019-08-02T06:04:01,007+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "security", "node.name": "security-master-2", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "JIZdkWG_Ql69ZRSu6UtdSw", "message": "started" }
{"type": "server", "timestamp": "2019-08-02T06:04:01,116+0000", "level": "INFO", "component": "o.e.x.m.e.l.LocalExporter", "cluster.name": "security", "node.name": "security-master-2", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "JIZdkWG_Ql69ZRSu6UtdSw", "message": "waiting for elected master node [{security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)" }
{"type": "server", "timestamp": "2019-08-02T06:04:01,390+0000", "level": "INFO", "component": "o.e.x.s.a.TokenService", "cluster.name": "security", "node.name": "security-master-2", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "JIZdkWG_Ql69ZRSu6UtdSw", "message": "refresh keys" }
{"type": "server", "timestamp": "2019-08-02T06:04:02,109+0000", "level": "INFO", "component": "o.e.x.s.a.TokenService", "cluster.name": "security", "node.name": "security-master-2", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "JIZdkWG_Ql69ZRSu6UtdSw", "message": "refreshed keys" }
{"type": "server", "timestamp": "2019-08-02T06:04:02,115+0000", "level": "INFO", "component": "o.e.x.m.e.l.LocalExporter", "cluster.name": "security", "node.name": "security-master-2", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "JIZdkWG_Ql69ZRSu6UtdSw", "message": "waiting for elected master node [{security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)" }
{"type": "server", "timestamp": "2019-08-02T06:04:02,421+0000", "level": "INFO", "component": "o.e.x.m.e.l.LocalExporter", "cluster.name": "security", "node.name": "security-master-2", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "JIZdkWG_Ql69ZRSu6UtdSw", "message": "waiting for elected master node [{security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)" }
{"type": "server", "timestamp": "2019-08-02T06:04:02,523+0000", "level": "INFO", "component": "o.e.x.m.e.l.LocalExporter", "cluster.name": "security", "node.name": "security-master-2", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "JIZdkWG_Ql69ZRSu6UtdSw", "message": "waiting for elected master node [{security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)" }
{"type": "server", "timestamp": "2019-08-02T06:04:02,634+0000", "level": "INFO", "component": "o.e.x.m.e.l.LocalExporter", "cluster.name": "security", "node.name": "security-master-2", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "JIZdkWG_Ql69ZRSu6UtdSw", "message": "waiting for elected master node [{security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)" }
{"type": "server", "timestamp": "2019-08-02T06:04:02,735+0000", "level": "INFO", "component": "o.e.x.m.e.l.LocalExporter", "cluster.name": "security", "node.name": "security-master-2", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "JIZdkWG_Ql69ZRSu6UtdSw", "message": "waiting for elected master node [{security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)" }
{"type": "server", "timestamp": "2019-08-02T06:04:02,989+0000", "level": "INFO", "component": "o.e.x.m.e.l.LocalExporter", "cluster.name": "security", "node.name": "security-master-2", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "JIZdkWG_Ql69ZRSu6UtdSw", "message": "waiting for elected master node [{security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)" }
{"type": "server", "timestamp": "2019-08-02T06:04:03,114+0000", "level": "INFO", "component": "o.e.x.m.e.l.LocalExporter", "cluster.name": "security", "node.name": "security-master-2", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "JIZdkWG_Ql69ZRSu6UtdSw", "message": "waiting for elected master node [{security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)" }
{"type": "server", "timestamp": "2019-08-02T06:04:03,292+0000", "level": "INFO", "component": "o.e.x.m.e.l.LocalExporter", "cluster.name": "security", "node.name": "security-master-2", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "JIZdkWG_Ql69ZRSu6UtdSw", "message": "waiting for elected master node [{security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)" }
{"type": "server", "timestamp": "2019-08-02T06:04:03,495+0000", "level": "INFO", "component": "o.e.x.m.e.l.LocalExporter", "cluster.name": "security", "node.name": "security-master-2", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "JIZdkWG_Ql69ZRSu6UtdSw", "message": "waiting for elected master node [{security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)" }
{"type": "server", "timestamp": "2019-08-02T06:04:03,831+0000", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "security", "node.name": "security-master-2", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "JIZdkWG_Ql69ZRSu6UtdSw", "message": "license [e909ad74-2edf-4e16-91c1-da3860be9c8e] mode [basic] - valid" }
{"type": "server", "timestamp": "2019-08-02T06:04:03,891+0000", "level": "INFO", "component": "o.e.x.m.e.l.LocalExporter", "cluster.name": "security", "node.name": "security-master-2", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "JIZdkWG_Ql69ZRSu6UtdSw", "message": "waiting for elected master node [{security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)" }
{"type": "server", "timestamp": "2019-08-02T06:04:04,026+0000", "level": "INFO", "component": "o.e.x.m.e.l.LocalExporter", "cluster.name": "security", "node.name": "security-master-2", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "JIZdkWG_Ql69ZRSu6UtdSw", "message": "waiting for elected master node [{security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)" }
{"type": "server", "timestamp": "2019-08-02T06:04:04,120+0000", "level": "INFO", "component": "o.e.x.m.e.l.LocalExporter", "cluster.name": "security", "node.name": "security-master-2", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "JIZdkWG_Ql69ZRSu6UtdSw", "message": "waiting for elected master node [{security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)" }
{"type": "server", "timestamp": "2019-08-02T06:04:10,695+0000", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "security", "node.name": "security-master-2", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "JIZdkWG_Ql69ZRSu6UtdSw", "message": "added {{security-data-0}{bOra2xonQeid3whGZqnyfw}{78U9d9jLTS-EQaQspcgJmg}{10.233.102.255}{10.233.102.255:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 16, reason: ApplyCommitRequest{term=2, version=16, sourceNode={security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}}" }
{"type": "server", "timestamp": "2019-08-02T06:04:12,715+0000", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "security", "node.name": "security-master-2", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "JIZdkWG_Ql69ZRSu6UtdSw", "message": "added {{security-master-0}{DoM0e1u4T7-QNqxyOs-fEg}{vNB_4MCkTFqmrE9DZ_1ETg}{10.233.120.99}{10.233.120.99:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 17, reason: ApplyCommitRequest{term=2, version=17, sourceNode={security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}}" }
{"type": "server", "timestamp": "2019-08-02T06:04:14,384+0000", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "security", "node.name": "security-master-2", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "JIZdkWG_Ql69ZRSu6UtdSw", "message": "added {{security-data-1}{XPQAk7NvSVCIhZhZwGmoKQ}{6TFIyNgzTvWTEczIBSH7sw}{10.233.92.86}{10.233.92.86:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 19, reason: ApplyCommitRequest{term=2, version=19, sourceNode={security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}}" }
{"type": "server", "timestamp": "2019-08-02T06:04:25,826+0000", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "security", "node.name": "security-master-2", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "JIZdkWG_Ql69ZRSu6UtdSw", "message": "added {{security-data-2}{lS0KL3TfRzi3JBDSVSzouA}{07yXpkoRRXCBRFVns2Jjew}{10.233.120.211}{10.233.120.211:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 20, reason: ApplyCommitRequest{term=2, version=20, sourceNode={security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}}" }
Created elasticsearch keystore in /usr/share/elasticsearch/config
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{"type": "server", "timestamp": "2019-08-02T06:03:52,992+0000", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "security", "node.name": "security-master-0", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/vdd)]], net usable_space [29.3gb], net total_space [29.4gb], types [ext4]" }
{"type": "server", "timestamp": "2019-08-02T06:03:52,994+0000", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "security", "node.name": "security-master-0", "message": "heap size [1015.6mb], compressed ordinary object pointers [true]" }
{"type": "server", "timestamp": "2019-08-02T06:03:52,996+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "security", "node.name": "security-master-0", "message": "node name [security-master-0], node ID [DoM0e1u4T7-QNqxyOs-fEg], cluster name [security]" }
{"type": "server", "timestamp": "2019-08-02T06:03:52,997+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "security", "node.name": "security-master-0", "message": "version[7.2.0], pid[1], build[default/docker/508c38a/2019-06-20T15:54:18.811730Z], OS[Linux/3.10.0-957.12.2.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.1/12.0.1+12]" }
{"type": "server", "timestamp": "2019-08-02T06:03:52,997+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "security", "node.name": "security-master-0", "message": "JVM home [/usr/share/elasticsearch/jdk]" }
{"type": "server", "timestamp": "2019-08-02T06:03:52,997+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "security", "node.name": "security-master-0", "message": "JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-9650274296716949602, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Des.cgroups.hierarchy.override=/, -Xmx1g, -Xms1g, -Dio.netty.allocator.type=unpooled, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,122+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [aggs-matrix-stats]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,122+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [analysis-common]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,122+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [data-frame]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,122+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [ingest-common]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,123+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [ingest-geoip]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,123+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [ingest-user-agent]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,123+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [lang-expression]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,123+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [lang-mustache]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,123+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [lang-painless]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,124+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [mapper-extras]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,124+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [parent-join]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,124+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [percolator]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,124+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [rank-eval]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,124+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [reindex]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,124+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [repository-url]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,124+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [transport-netty4]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,124+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [x-pack-ccr]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,125+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [x-pack-core]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,125+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [x-pack-deprecation]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,125+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [x-pack-graph]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,125+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [x-pack-ilm]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,125+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [x-pack-logstash]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,126+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [x-pack-ml]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,126+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [x-pack-monitoring]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,126+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [x-pack-rollup]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,126+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [x-pack-security]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,126+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [x-pack-sql]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,126+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "loaded module [x-pack-watcher]" }
{"type": "server", "timestamp": "2019-08-02T06:03:57,127+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "security", "node.name": "security-master-0", "message": "no plugins loaded" }
{"type": "server", "timestamp": "2019-08-02T06:04:04,388+0000", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "security", "node.name": "security-master-0", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
{"type": "server", "timestamp": "2019-08-02T06:04:05,588+0000", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "security", "node.name": "security-master-0", "message": "[controller/213] [Main.cc@110] controller (64 bit): Version 7.2.0 (Build 65aefcbfce449b) Copyright (c) 2019 Elasticsearch BV" }
{"type": "server", "timestamp": "2019-08-02T06:04:06,322+0000", "level": "DEBUG", "component": "o.e.a.ActionModule", "cluster.name": "security", "node.name": "security-master-0", "message": "Using REST wrapper from plugin org.elasticsearch.xpack.security.Security" }
{"type": "server", "timestamp": "2019-08-02T06:04:06,899+0000", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "security", "node.name": "security-master-0", "message": "using discovery type [zen] and seed hosts providers [settings]" }
{"type": "server", "timestamp": "2019-08-02T06:04:08,489+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "security", "node.name": "security-master-0", "message": "initialized" }
{"type": "server", "timestamp": "2019-08-02T06:04:08,489+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "security", "node.name": "security-master-0", "message": "starting ..." }
{"type": "server", "timestamp": "2019-08-02T06:04:08,719+0000", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "security", "node.name": "security-master-0", "message": "publish_address {10.233.120.99:9300}, bound_addresses {0.0.0.0:9300}" }
{"type": "server", "timestamp": "2019-08-02T06:04:08,725+0000", "level": "INFO", "component": "o.e.b.BootstrapChecks", "cluster.name": "security", "node.name": "security-master-0", "message": "bound or publishing to a non-loopback address, enforcing bootstrap checks" }
{"type": "server", "timestamp": "2019-08-02T06:04:10,492+0000", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "security", "node.name": "security-master-0", "message": "setting initial configuration to VotingConfiguration{SzH19ZKqRJKAGEg2OqmG2w,DoM0e1u4T7-QNqxyOs-fEg,{bootstrap-placeholder}-security-master-2}" }
{"type": "server", "timestamp": "2019-08-02T06:04:13,015+0000", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "security", "node.name": "security-master-0", "message": "master node changed {previous [], current [{security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}]}, added {{security-master-2}{JIZdkWG_Ql69ZRSu6UtdSw}{OZ6o4MT3SYyUVR8xVMPphA}{10.233.89.152}{10.233.89.152:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},{security-data-0}{bOra2xonQeid3whGZqnyfw}{78U9d9jLTS-EQaQspcgJmg}{10.233.102.255}{10.233.102.255:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},{security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 17, reason: ApplyCommitRequest{term=2, version=17, sourceNode={security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}}" }
{"type": "server", "timestamp": "2019-08-02T06:04:13,724+0000", "level": "INFO", "component": "o.e.x.s.a.TokenService", "cluster.name": "security", "node.name": "security-master-0", "message": "refresh keys" }
{"type": "server", "timestamp": "2019-08-02T06:04:13,986+0000", "level": "INFO", "component": "o.e.x.s.a.TokenService", "cluster.name": "security", "node.name": "security-master-0", "message": "refreshed keys" }
{"type": "server", "timestamp": "2019-08-02T06:04:14,012+0000", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "security", "node.name": "security-master-0", "message": "license [e909ad74-2edf-4e16-91c1-da3860be9c8e] mode [basic] - valid" }
{"type": "server", "timestamp": "2019-08-02T06:04:14,030+0000", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "security", "node.name": "security-master-0", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "DoM0e1u4T7-QNqxyOs-fEg", "message": "publish_address {10.233.120.99:9200}, bound_addresses {0.0.0.0:9200}" }
{"type": "server", "timestamp": "2019-08-02T06:04:14,030+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "security", "node.name": "security-master-0", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "DoM0e1u4T7-QNqxyOs-fEg", "message": "started" }
{"type": "server", "timestamp": "2019-08-02T06:04:14,335+0000", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "security", "node.name": "security-master-0", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "DoM0e1u4T7-QNqxyOs-fEg", "message": "added {{security-data-1}{XPQAk7NvSVCIhZhZwGmoKQ}{6TFIyNgzTvWTEczIBSH7sw}{10.233.92.86}{10.233.92.86:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 19, reason: ApplyCommitRequest{term=2, version=19, sourceNode={security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}}" }
{"type": "server", "timestamp": "2019-08-02T06:04:25,828+0000", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "security", "node.name": "security-master-0", "cluster.uuid": "iGQVix2gRMWa_Kx-yIsisA", "node.id": "DoM0e1u4T7-QNqxyOs-fEg", "message": "added {{security-data-2}{lS0KL3TfRzi3JBDSVSzouA}{07yXpkoRRXCBRFVns2Jjew}{10.233.120.211}{10.233.120.211:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 20, reason: ApplyCommitRequest{term=2, version=20, sourceNode={security-master-1}{SzH19ZKqRJKAGEg2OqmG2w}{N55hEOE-Tpe2YHtkitNHcQ}{10.233.92.38}{10.233.92.38:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}}" }
@zyriljamez

This comment has been minimized.

Copy link
Owner Author

@zyriljamez zyriljamez commented Aug 2, 2019

elasticsearch cluster logs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.