Skip to content

Instantly share code, notes, and snippets.

@jack-nl

jack-nl/master2.log Secret

Created May 6, 2021 12:00
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jack-nl/ad76a4172508de109e0aaa4882228304 to your computer and use it in GitHub Desktop.
Save jack-nl/ad76a4172508de109e0aaa4882228304 to your computer and use it in GitHub Desktop.
Download ZIP
master2.log
Raw
master2.log
[2021-05-03T13:54:46,162][INFO ][o.e.n.Node ] [master2] stopping ...
[2021-05-03T13:54:46,198][INFO ][o.e.c.c.Coordinator ] [master2] master node [{master0}{xexrc0uXQUWgwFDWEpAouA}{VYK7ojdqQU6DH676OtMWyA}{192.168.0.10}{192.168.0.10:9300}{im}{xpack.installed=true}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [master0][192.168.0.10:9300][disconnected] disconnected
[2021-05-03T13:54:46,210][WARN ][o.e.d.HandshakingTransportAddressConnector] [master2] handshake failed for [connectToRemoteMasterNode[192.168.0.10:9300]]
org.elasticsearch.transport.SendRequestTransportException: [][192.168.0.10:9300][internal:transport/handshake]
at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:704) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:602) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:453) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:431) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.discovery.HandshakingTransportAddressConnector$1$1.onResponse(HandshakingTransportAddressConnector.java:94) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.discovery.HandshakingTransportAddressConnector$1$1.onResponse(HandshakingTransportAddressConnector.java:87) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:146) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.action.support.ThreadedActionListener$1.doRun(ThreadedActionListener.java:98) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: org.elasticsearch.transport.TransportException: TransportService is closed stopped can't send request
at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:686) ~[elasticsearch-7.5.2.jar:7.5.2]
... 13 more
[2021-05-03T13:54:46,211][WARN ][o.e.d.HandshakingTransportAddressConnector] [master2] handshake failed for [connectToRemoteMasterNode[192.168.0.11:9300]]
org.elasticsearch.transport.SendRequestTransportException: [][192.168.0.11:9300][internal:transport/handshake]
at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:704) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:602) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:453) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:431) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.discovery.HandshakingTransportAddressConnector$1$1.onResponse(HandshakingTransportAddressConnector.java:94) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.discovery.HandshakingTransportAddressConnector$1$1.onResponse(HandshakingTransportAddressConnector.java:87) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:146) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.action.support.ThreadedActionListener$1.doRun(ThreadedActionListener.java:98) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: org.elasticsearch.transport.TransportException: TransportService is closed stopped can't send request
at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:686) ~[elasticsearch-7.5.2.jar:7.5.2]
... 13 more
[2021-05-03T13:54:46,234][INFO ][o.e.n.Node ] [master2] stopped
[2021-05-03T13:54:46,234][INFO ][o.e.n.Node ] [master2] closing ...
[2021-05-03T13:54:46,257][INFO ][o.e.n.Node ] [master2] closed
[2021-05-03T14:10:01,367][INFO ][o.e.e.NodeEnvironment ] [master2] using [1] data paths, mounts [[/ (/dev/sda5)]], net usable_space [26.4gb], net total_space [39.7gb], types [ext4]
[2021-05-03T14:10:01,376][INFO ][o.e.e.NodeEnvironment ] [master2] heap size [11.9gb], compressed ordinary object pointers [true]
[2021-05-03T14:10:40,309][INFO ][o.e.n.Node ] [master2] node name [master2], node ID [RBMXwoyITeqpwJlY7VFFhA], cluster name [elasticsearch]
[2021-05-03T14:10:40,310][INFO ][o.e.n.Node ] [master2] version[7.5.2], pid[1270], build[default/deb/8bec50e1e0ad29dad5653712cf3bb580cd1afcdf/2020-01-15T12:11:52.313576Z], OS[Linux/4.15.0-142-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13.0.1/13.0.1+9]
[2021-05-03T14:10:40,310][INFO ][o.e.n.Node ] [master2] JVM home [/usr/share/elasticsearch/jdk]
[2021-05-03T14:10:40,311][INFO ][o.e.n.Node ] [master2] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms12g, -Xmx12g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -XX:MaxDirectMemorySize=6442450944, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2021-05-03T14:10:44,643][INFO ][o.e.p.PluginsService ] [master2] loaded module [aggs-matrix-stats]
[2021-05-03T14:10:44,643][INFO ][o.e.p.PluginsService ] [master2] loaded module [analysis-common]
[2021-05-03T14:10:44,643][INFO ][o.e.p.PluginsService ] [master2] loaded module [flattened]
[2021-05-03T14:10:44,644][INFO ][o.e.p.PluginsService ] [master2] loaded module [frozen-indices]
[2021-05-03T14:10:44,644][INFO ][o.e.p.PluginsService ] [master2] loaded module [ingest-common]
[2021-05-03T14:10:44,644][INFO ][o.e.p.PluginsService ] [master2] loaded module [ingest-geoip]
[2021-05-03T14:10:44,644][INFO ][o.e.p.PluginsService ] [master2] loaded module [ingest-user-agent]
[2021-05-03T14:10:44,645][INFO ][o.e.p.PluginsService ] [master2] loaded module [lang-expression]
[2021-05-03T14:10:44,645][INFO ][o.e.p.PluginsService ] [master2] loaded module [lang-mustache]
[2021-05-03T14:10:44,645][INFO ][o.e.p.PluginsService ] [master2] loaded module [lang-painless]
[2021-05-03T14:10:44,645][INFO ][o.e.p.PluginsService ] [master2] loaded module [mapper-extras]
[2021-05-03T14:10:44,645][INFO ][o.e.p.PluginsService ] [master2] loaded module [parent-join]
[2021-05-03T14:10:44,646][INFO ][o.e.p.PluginsService ] [master2] loaded module [percolator]
[2021-05-03T14:10:44,646][INFO ][o.e.p.PluginsService ] [master2] loaded module [rank-eval]
[2021-05-03T14:10:44,646][INFO ][o.e.p.PluginsService ] [master2] loaded module [reindex]
[2021-05-03T14:10:44,646][INFO ][o.e.p.PluginsService ] [master2] loaded module [repository-url]
[2021-05-03T14:10:44,646][INFO ][o.e.p.PluginsService ] [master2] loaded module [search-business-rules]
[2021-05-03T14:10:44,647][INFO ][o.e.p.PluginsService ] [master2] loaded module [spatial]
[2021-05-03T14:10:44,647][INFO ][o.e.p.PluginsService ] [master2] loaded module [systemd]
[2021-05-03T14:10:44,647][INFO ][o.e.p.PluginsService ] [master2] loaded module [transform]
[2021-05-03T14:10:44,647][INFO ][o.e.p.PluginsService ] [master2] loaded module [transport-netty4]
[2021-05-03T14:10:44,647][INFO ][o.e.p.PluginsService ] [master2] loaded module [vectors]
[2021-05-03T14:10:44,648][INFO ][o.e.p.PluginsService ] [master2] loaded module [x-pack-analytics]
[2021-05-03T14:10:44,648][INFO ][o.e.p.PluginsService ] [master2] loaded module [x-pack-ccr]
[2021-05-03T14:10:44,648][INFO ][o.e.p.PluginsService ] [master2] loaded module [x-pack-core]
[2021-05-03T14:10:44,648][INFO ][o.e.p.PluginsService ] [master2] loaded module [x-pack-deprecation]
[2021-05-03T14:10:44,648][INFO ][o.e.p.PluginsService ] [master2] loaded module [x-pack-enrich]
[2021-05-03T14:10:44,649][INFO ][o.e.p.PluginsService ] [master2] loaded module [x-pack-graph]
[2021-05-03T14:10:44,649][INFO ][o.e.p.PluginsService ] [master2] loaded module [x-pack-ilm]
[2021-05-03T14:10:44,649][INFO ][o.e.p.PluginsService ] [master2] loaded module [x-pack-logstash]
[2021-05-03T14:10:44,649][INFO ][o.e.p.PluginsService ] [master2] loaded module [x-pack-ml]
[2021-05-03T14:10:44,650][INFO ][o.e.p.PluginsService ] [master2] loaded module [x-pack-monitoring]
[2021-05-03T14:10:44,650][INFO ][o.e.p.PluginsService ] [master2] loaded module [x-pack-rollup]
[2021-05-03T14:10:44,650][INFO ][o.e.p.PluginsService ] [master2] loaded module [x-pack-security]
[2021-05-03T14:10:44,650][INFO ][o.e.p.PluginsService ] [master2] loaded module [x-pack-sql]
[2021-05-03T14:10:44,650][INFO ][o.e.p.PluginsService ] [master2] loaded module [x-pack-voting-only-node]
[2021-05-03T14:10:44,651][INFO ][o.e.p.PluginsService ] [master2] loaded module [x-pack-watcher]
[2021-05-03T14:10:44,651][INFO ][o.e.p.PluginsService ] [master2] no plugins loaded
[2021-05-03T14:10:48,677][INFO ][o.e.d.DiscoveryModule ] [master2] using discovery type [zen] and seed hosts providers [settings]
[2021-05-03T14:10:49,322][INFO ][o.e.n.Node ] [master2] initialized
[2021-05-03T14:10:49,323][INFO ][o.e.n.Node ] [master2] starting ...
[2021-05-03T14:10:49,460][INFO ][o.e.t.TransportService ] [master2] publish_address {192.168.0.12:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}, {192.168.0.12:9300}
[2021-05-03T14:10:59,342][ERROR][o.e.x.m.c.i.IndexRecoveryCollector] [master2] collector [index_recovery] failed to collect data
java.lang.NullPointerException: null
at org.elasticsearch.xpack.monitoring.collector.Collector.collect(Collector.java:85) [x-pack-monitoring-7.5.2.jar:7.5.2]
at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:242) [x-pack-monitoring-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
[2021-05-03T14:10:59,351][ERROR][o.e.x.m.c.s.ShardsCollector] [master2] collector [shards] failed to collect data
java.lang.NullPointerException: null
at org.elasticsearch.xpack.monitoring.collector.Collector.collect(Collector.java:85) [x-pack-monitoring-7.5.2.jar:7.5.2]
at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:242) [x-pack-monitoring-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
[2021-05-03T14:10:59,352][ERROR][o.e.x.m.c.m.JobStatsCollector] [master2] collector [job_stats] failed to collect data
java.lang.NullPointerException: null
at org.elasticsearch.xpack.monitoring.collector.Collector.collect(Collector.java:85) [x-pack-monitoring-7.5.2.jar:7.5.2]
at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:242) [x-pack-monitoring-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
[2021-05-03T14:10:59,354][ERROR][o.e.x.m.c.e.EnrichStatsCollector] [master2] collector [enrich_coordinator_stats] failed to collect data
java.lang.NullPointerException: null
at org.elasticsearch.xpack.monitoring.collector.Collector.collect(Collector.java:85) [x-pack-monitoring-7.5.2.jar:7.5.2]
at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:242) [x-pack-monitoring-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
[2021-05-03T14:10:59,354][ERROR][o.e.x.m.c.i.IndexStatsCollector] [master2] collector [index-stats] failed to collect data
java.lang.NullPointerException: null
at org.elasticsearch.xpack.monitoring.collector.Collector.collect(Collector.java:85) [x-pack-monitoring-7.5.2.jar:7.5.2]
at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:242) [x-pack-monitoring-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
[2021-05-03T14:10:59,355][ERROR][o.e.x.m.c.n.NodeStatsCollector] [master2] collector [node_stats] failed to collect data
java.lang.NullPointerException: null
at org.elasticsearch.xpack.monitoring.collector.Collector.collect(Collector.java:85) [x-pack-monitoring-7.5.2.jar:7.5.2]
at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:242) [x-pack-monitoring-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
[2021-05-03T14:10:59,356][ERROR][o.e.x.m.c.c.StatsCollector] [master2] collector [ccr_stats] failed to collect data
java.lang.NullPointerException: null
at org.elasticsearch.xpack.monitoring.collector.Collector.collect(Collector.java:85) [x-pack-monitoring-7.5.2.jar:7.5.2]
at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:242) [x-pack-monitoring-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
[2021-05-03T14:10:59,356][ERROR][o.e.x.m.c.c.ClusterStatsCollector] [master2] collector [cluster_stats] failed to collect data
java.lang.NullPointerException: null
at org.elasticsearch.xpack.monitoring.collector.Collector.collect(Collector.java:85) [x-pack-monitoring-7.5.2.jar:7.5.2]
at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:242) [x-pack-monitoring-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
[2021-05-03T14:11:05,485][INFO ][o.e.b.BootstrapChecks ] [master2] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2021-05-03T14:11:05,488][INFO ][o.e.c.c.Coordinator ] [master2] cluster UUID [9ebN7hmbR3O_-EqkjWDXwg]
[2021-05-03T14:11:07,471][INFO ][o.e.c.s.ClusterApplierService] [master2] master node changed {previous [], current [{master0}{xexrc0uXQUWgwFDWEpAouA}{VYK7ojdqQU6DH676OtMWyA}{192.168.0.10}{192.168.0.10:9300}{im}{xpack.installed=true}]}, added {{data11a}{EI9MEUGwQP6yuwlmVsH-EA}{QRz_oL35TyOvnBqTWqWgEA}{192.168.0.31}{192.168.0.31:9300}{di}{xpack.installed=true},{data09}{dN08oWhmSNObQgeb9xH_-A}{9aLNT59cQZutpH2xd1LzTQ}{192.168.0.29}{192.168.0.29:9300}{di}{xpack.installed=true},{data03}{9MyOkgP3TuSevNQJkEmOnw}{MBDXNqQOR6G05JiS0UQlEw}{192.168.0.23}{192.168.0.23:9300}{di}{xpack.installed=true},{data15b}{bT-KQ6TgQHWYl3pLMjj9MQ}{I9wc6TOxQ2WkqbOZ89g8XQ}{192.168.0.35}{192.168.0.35:9301}{di}{xpack.installed=true},{data06}{NqAsywRoTVSuqTxD8cmq3A}{ffiDh9EhT0OpZ0Suj4hHcQ}{192.168.0.26}{192.168.0.26:9300}{di}{xpack.installed=true},{data17b}{7uaz7rwQRg-UQQ4WLhclBg}{SXb-eNTSTP-9-dO7pgnEfQ}{192.168.0.37}{192.168.0.37:9301}{di}{xpack.installed=true},{data16a}{SzyPNxe6Rm6H9ucP26H6Mw}{PALqrfCdR6CAvYfrioYAZQ}{192.168.0.36}{192.168.0.36:9300}{di}{xpack.installed=true},{master0}{xexrc0uXQUWgwFDWEpAouA}{VYK7ojdqQU6DH676OtMWyA}{192.168.0.10}{192.168.0.10:9300}{im}{xpack.installed=true},{data17a}{x-YWJyTxRoCh8Lf4oS0qWQ}{MqzCX5bVRCa7MWZ3e7CtZg}{192.168.0.37}{192.168.0.37:9300}{di}{xpack.installed=true},{data13a}{vF-2rZ1ZTuG8rcYDTScDbQ}{NY6x69ejT9ehohT-uXKyww}{192.168.0.33}{192.168.0.33:9300}{di}{xpack.installed=true},{data12a}{22tiK4-STyC2XRJ01nWQ3A}{rKiBCqOFTGKTBesQCzI7aw}{192.168.0.32}{192.168.0.32:9300}{di}{xpack.installed=true},{data10b}{WBnLOtGgS6KXAV8FsiFYjg}{D1xTZOswTAiMc3h1YMae3Q}{192.168.0.30}{192.168.0.30:9301}{di}{xpack.installed=true},{data04}{M7yBnQeXTwCohL8IWN4PBg}{HRi5FXR0S_m2YlWM5zAMCw}{192.168.0.24}{192.168.0.24:9300}{di}{xpack.installed=true},{data07}{uYjW78V_QAaAQPHjUSNjjQ}{ZA9njwKuRgOEdIllyccwqQ}{192.168.0.27}{192.168.0.27:9300}{di}{xpack.installed=true},{master1}{Ig33yWUwTv-UNSwIhhfNWg}{GyE6pKvWTfeRglHhD-5oCw}{192.168.0.11}{192.168.0.11:9300}{im}{xpack.installed=true},{data13b}{mAQ5jgAMQjaYzwpVE83kcg}{LBsGSuLJR5Kg8iWCA-SX1A}{192.168.0.33}{192.168.0.33:9301}{di}{xpack.installed=true},{data11b}{6FVMYvwiQD-W6fpHtVXPfg}{dAQSQFZMSjKq6kxq8cc98g}{192.168.0.31}{192.168.0.31:9301}{di}{xpack.installed=true},{data10a}{Cl-VWQCjQ3OStxiPmdIXrQ}{KjVwOpXuQCKLy2GQFBluZg}{192.168.0.30}{192.168.0.30:9300}{di}{xpack.installed=true},{data18b}{5HTiO0Z8RM2tRi0kaNQQ8w}{O-T8IWgXQIekz9QHQyKy4g}{192.168.0.38}{192.168.0.38:9301}{di}{xpack.installed=true},{data15a}{0sFGQdJwSNermm8lSE7v6w}{TU-BKgdwQO2xOnhMx6o-ow}{192.168.0.35}{192.168.0.35:9300}{di}{xpack.installed=true},{data12b}{Fs5fJqYNSLOPK7OVW7qaYQ}{2SeThcYbQ5uUG6c7JyP_Qg}{192.168.0.32}{192.168.0.32:9301}{di}{xpack.installed=true},{data18a}{_MeUtbWCQK6WAcOWs6L4fQ}{qjYEjHO0Qp-pysLXoM8iYA}{192.168.0.38}{192.168.0.38:9300}{di}{xpack.installed=true},{data05}{3sy0vv41TXGy4Yt8ywpXVw}{AoblOEIlSF2ZWQSqEERqwA}{192.168.0.25}{192.168.0.25:9300}{di}{xpack.installed=true},{data08}{GIaGadJ0RhyBUePNzWpBdA}{fz0VawBhT_6hIre0uymIfw}{192.168.0.28}{192.168.0.28:9300}{di}{xpack.installed=true},{data16b}{2afokmtOQGGzfFLz3v4_Bg}{Lo4izLlTQqSMfbrf21aAkw}{192.168.0.36}{192.168.0.36:9301}{di}{xpack.installed=true}}, term: 38, version: 2131737, reason: ApplyCommitRequest{term=38, version=2131737, sourceNode={master0}{xexrc0uXQUWgwFDWEpAouA}{VYK7ojdqQU6DH676OtMWyA}{192.168.0.10}{192.168.0.10:9300}{im}{xpack.installed=true}}
[2021-05-03T14:11:07,764][INFO ][o.e.c.s.ClusterSettings ] [master2] updating [cluster.routing.allocation.cluster_concurrent_rebalance] from [2] to [20]
[2021-05-03T14:11:07,765][INFO ][o.e.c.s.ClusterSettings ] [master2] updating [cluster.routing.allocation.same_shard.host] from [false] to [true]
[2021-05-03T14:11:07,765][INFO ][o.e.c.s.ClusterSettings ] [master2] updating [cluster.routing.allocation.disk.watermark.low] from [85%] to [150gb]
[2021-05-03T14:11:07,765][INFO ][o.e.c.s.ClusterSettings ] [master2] updating [cluster.routing.allocation.disk.watermark.high] from [90%] to [100gb]
[2021-05-03T14:11:07,766][INFO ][o.e.c.s.ClusterSettings ] [master2] updating [cluster.routing.allocation.disk.watermark.flood_stage] from [95%] to [50gb]
[2021-05-03T14:11:07,766][INFO ][o.e.c.s.ClusterSettings ] [master2] updating [cluster.routing.allocation.node_initial_primaries_recoveries] from [4] to [16]
[2021-05-03T14:11:07,766][INFO ][o.e.c.s.ClusterSettings ] [master2] updating [cluster.routing.allocation.node_concurrent_incoming_recoveries] from [2] to [20]
[2021-05-03T14:11:07,766][INFO ][o.e.c.s.ClusterSettings ] [master2] updating [cluster.routing.allocation.node_concurrent_outgoing_recoveries] from [2] to [20]
[2021-05-03T14:11:07,767][INFO ][o.e.c.s.ClusterSettings ] [master2] updating [xpack.monitoring.elasticsearch.collection.enabled] from [true] to [false]
[2021-05-03T14:11:07,767][INFO ][o.e.c.s.ClusterSettings ] [master2] updating [cluster.routing.allocation.disk.watermark.low] from [85%] to [150gb]
[2021-05-03T14:11:07,767][INFO ][o.e.c.s.ClusterSettings ] [master2] updating [cluster.routing.allocation.disk.watermark.high] from [90%] to [100gb]
[2021-05-03T14:11:07,767][INFO ][o.e.c.s.ClusterSettings ] [master2] updating [cluster.routing.allocation.disk.watermark.flood_stage] from [95%] to [50gb]
[2021-05-03T14:11:07,768][INFO ][o.e.c.s.ClusterSettings ] [master2] updating [search.max_buckets] from [10000] to [20000]
[2021-05-03T14:11:07,768][INFO ][o.e.c.s.ClusterSettings ] [master2] updating [indices.recovery.max_bytes_per_sec] from [40mb] to [120mb]
[2021-05-03T14:11:08,103][INFO ][o.e.l.LicenseService ] [master2] license [c01f4c23-89af-4849-841d-675391762534] mode [basic] - valid
[2021-05-03T14:11:08,131][INFO ][o.e.h.AbstractHttpServerTransport] [master2] publish_address {192.168.0.12:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}, {192.168.0.12:9200}
[2021-05-03T14:11:08,132][INFO ][o.e.n.Node ] [master2] started
[2021-05-03T14:29:07,883][INFO ][o.e.c.c.Coordinator ] [master2] master node [{master0}{xexrc0uXQUWgwFDWEpAouA}{VYK7ojdqQU6DH676OtMWyA}{192.168.0.10}{192.168.0.10:9300}{im}{xpack.installed=true}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [master0][192.168.0.10:9300][disconnected] disconnected
[2021-05-03T14:29:07,885][INFO ][o.e.c.s.ClusterApplierService] [master2] master node changed {previous [{master0}{xexrc0uXQUWgwFDWEpAouA}{VYK7ojdqQU6DH676OtMWyA}{192.168.0.10}{192.168.0.10:9300}{im}{xpack.installed=true}], current []}, term: 38, version: 2131738, reason: becoming candidate: onLeaderFailure
[2021-05-03T14:29:07,912][WARN ][o.e.c.NodeConnectionsService] [master2] failed to connect to {master0}{xexrc0uXQUWgwFDWEpAouA}{VYK7ojdqQU6DH676OtMWyA}{192.168.0.10}{192.168.0.10:9300}{im}{xpack.installed=true} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [master0][192.168.0.10:9300] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:989) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$3(ActionListener.java:162) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.5.2.jar:7.5.2]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.5.2.jar:7.5.2]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:68) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:688) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:600) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:554) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 192.168.0.10/192.168.0.10:9300
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
at sun.nio.ch.Net.pollConnectNow(Net.java:579) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:820) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]
... 7 more
[2021-05-03T14:29:08,301][INFO ][o.e.c.s.MasterService ] [master2] elected-as-master ([2] nodes joined)[{master1}{Ig33yWUwTv-UNSwIhhfNWg}{GyE6pKvWTfeRglHhD-5oCw}{192.168.0.11}{192.168.0.11:9300}{im}{xpack.installed=true} elect leader, {master2}{RBMXwoyITeqpwJlY7VFFhA}{a7xhJcj7R46HQAm3OcmNJw}{192.168.0.12}{192.168.0.12:9300}{im}{xpack.installed=true} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 39, version: 2131739, delta: master node changed {previous [], current [{master2}{RBMXwoyITeqpwJlY7VFFhA}{a7xhJcj7R46HQAm3OcmNJw}{192.168.0.12}{192.168.0.12:9300}{im}{xpack.installed=true}]}
[2021-05-03T14:29:08,303][WARN ][o.e.c.s.MasterService ] [master2] failing [elected-as-master ([2] nodes joined)[{master1}{Ig33yWUwTv-UNSwIhhfNWg}{GyE6pKvWTfeRglHhD-5oCw}{192.168.0.11}{192.168.0.11:9300}{im}{xpack.installed=true} elect leader, {master2}{RBMXwoyITeqpwJlY7VFFhA}{a7xhJcj7R46HQAm3OcmNJw}{192.168.0.12}{192.168.0.12:9300}{im}{xpack.installed=true} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_]]: failed to commit cluster state version [2131739]
org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer master for term 39 while handling publication
at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1049) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:268) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:250) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
[2021-05-03T14:29:08,306][INFO ][o.e.c.c.JoinHelper ] [master2] failed to join {master2}{RBMXwoyITeqpwJlY7VFFhA}{a7xhJcj7R46HQAm3OcmNJw}{192.168.0.12}{192.168.0.12:9300}{im}{xpack.installed=true} with JoinRequest{sourceNode={master2}{RBMXwoyITeqpwJlY7VFFhA}{a7xhJcj7R46HQAm3OcmNJw}{192.168.0.12}{192.168.0.12:9300}{im}{xpack.installed=true}, optionalJoin=Optional[Join{term=39, lastAcceptedTerm=38, lastAcceptedVersion=2131738, sourceNode={master2}{RBMXwoyITeqpwJlY7VFFhA}{a7xhJcj7R46HQAm3OcmNJw}{192.168.0.12}{192.168.0.12:9300}{im}{xpack.installed=true}, targetNode={master2}{RBMXwoyITeqpwJlY7VFFhA}{a7xhJcj7R46HQAm3OcmNJw}{192.168.0.12}{192.168.0.12:9300}{im}{xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [master2][192.168.0.12:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer master for term 39 while handling publication
at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1049) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:268) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:250) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
[2021-05-03T14:29:15,737][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:15,737][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:15,738][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:15,737][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:15,737][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:15,739][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:15,739][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:15,739][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:25,739][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:25,740][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:25,740][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:25,739][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:25,739][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:25,741][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:25,741][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:25,742][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:35,741][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:35,743][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:35,744][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:35,743][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:35,746][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:35,747][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:35,747][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:35,747][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T14:29:38,480][INFO ][o.e.c.c.JoinHelper ] [master2] failed to join {master1}{Ig33yWUwTv-UNSwIhhfNWg}{GyE6pKvWTfeRglHhD-5oCw}{192.168.0.11}{192.168.0.11:9300}{im}{xpack.installed=true} with JoinRequest{sourceNode={master2}{RBMXwoyITeqpwJlY7VFFhA}{a7xhJcj7R46HQAm3OcmNJw}{192.168.0.12}{192.168.0.12:9300}{im}{xpack.installed=true}, optionalJoin=Optional[Join{term=40, lastAcceptedTerm=38, lastAcceptedVersion=2131738, sourceNode={master2}{RBMXwoyITeqpwJlY7VFFhA}{a7xhJcj7R46HQAm3OcmNJw}{192.168.0.12}{192.168.0.12:9300}{im}{xpack.installed=true}, targetNode={master1}{Ig33yWUwTv-UNSwIhhfNWg}{GyE6pKvWTfeRglHhD-5oCw}{192.168.0.11}{192.168.0.11:9300}{im}{xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [master1][192.168.0.11:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed
at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$4.onFailure(Coordinator.java:1429) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:88) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:68) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1349) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:125) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.cluster.coordination.Publication.cancel(Publication.java:89) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$2.run(Coordinator.java:1302) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) ~[elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: org.elasticsearch.ElasticsearchException: publication cancelled before committing: timed out after 30s
at org.elasticsearch.cluster.coordination.Publication.cancel(Publication.java:86) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$2.run(Coordinator.java:1302) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) ~[elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:830) ~[?:?]
[2021-05-03T14:29:45,739][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] timed out while retrying [cluster:monitor/state] after failure (timeout [30s])
[2021-05-03T14:29:45,741][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] timed out while retrying [cluster:monitor/state] after failure (timeout [30s])
[2021-05-03T14:29:45,739][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] timed out while retrying [cluster:monitor/state] after failure (timeout [30s])
[2021-05-03T14:29:45,742][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] timed out while retrying [cluster:monitor/state] after failure (timeout [30s])
[2021-05-03T14:29:45,742][WARN ][r.suppressed ] [master2] path: /_cluster/state/master_node, params: {metric=master_node, settings_filter=xpack.security.authc.realms.oidc.*.ssl.keystore.path,transport.profiles.*.xpack.security.ssl.trust_restrictions,reindex.ssl.supported_protocols,xpack.security.authc.realms.ldap.*.ssl.verification_mode,xpack.security.authc.realms.ldap.*.ssl.keystore.key_password,xpack.security.http.ssl.truststore.password,xpack.security.http.ssl.keystore.password,xpack.notification.email.ssl.supported_protocols,xpack.security.authc.realms.ldap.*.ssl.truststore.path,xpack.security.transport.ssl.truststore.password,xpack.notification.email.ssl.truststore.password,xpack.security.authc.realms.active_directory.*.ssl.verification_mode,xpack.http.ssl.truststore.algorithm,xpack.security.http.ssl.supported_protocols,reindex.ssl.truststore.password,xpack.security.authc.realms.oidc.*.ssl.verification_mode,xpack.security.transport.ssl.supported_protocols,xpack.security.authc.realms.saml.*.ssl.keystore.path,xpack.security.http.ssl.truststore.algorithm,xpack.security.transport.ssl.keystore.key_password,xpack.security.http.ssl.key,xpack.security.authc.realms.saml.*.encryption.key,xpack.security.http.ssl.keystore.type,xpack.security.http.ssl.verification_mode,xpack.security.authc.realms.pki.*.truststore.path,xpack.security.authc.realms.saml.*.signing.keystore.path,xpack.security.authc.realms.ldap.*.ssl.key,xpack.security.authc.realms.oidc.*.ssl.keystore.password,transport.profiles.*.xpack.security.ssl.supported_protocols,reindex.ssl.keystore.key_password,xpack.security.authc.realms.active_directory.*.ssl.supported_protocols,xpack.security.authc.realms.active_directory.*.hostname_verification,xpack.security.authc.realms.saml.*.ssl.truststore.algorithm,xpack.security.authc.realms.ldap.*.ssl.certificate_authorities,xpack.security.authc.realms.saml.*.signing.key,xpack.security.authc.realms.oidc.*.ssl.truststore.type,xpack.security.authc.realms.active_directory.*.ssl.keystore.key_password,transport.profiles.*.xpack.security.ssl.truststore.path,xpack.security.authc.realms.oidc.*.ssl.key,reindex.ssl.key_passphrase,xpack.security.authc.realms.ldap.*.ssl.cipher_suites,xpack.security.authc.realms.ldap.*.ssl.keystore.algorithm,xpack.http.ssl.keystore.algorithm,xpack.security.authc.realms.saml.*.ssl.keystore.password,xpack.http.ssl.supported_protocols,xpack.http.ssl.keystore.path,xpack.security.authc.realms.saml.*.encryption.keystore.algorithm,xpack.notification.email.ssl.certificate,xpack.security.http.ssl.truststore.path,xpack.http.ssl.truststore.password,xpack.security.transport.ssl.verification_mode,xpack.security.authc.realms.active_directory.*.ssl.truststore.password,xpack.security.authc.realms.saml.*.ssl.truststore.type,xpack.security.authc.realms.oidc.*.ssl.certificate_authorities,xpack.security.authc.realms.ldap.*.ssl.keystore.password,xpack.security.transport.ssl.certificate,xpack.monitoring.exporters.*.auth.*,xpack.security.authc.realms.oidc.*.ssl.keystore.algorithm,xpack.notification.email.ssl.key,xpack.http.ssl.keystore.key_password,xpack.security.authc.realms.saml.*.ssl.keystore.key_password,xpack.security.authc.realms.oidc.*.ssl.key_passphrase,reindex.ssl.truststore.path,xpack.notification.email.ssl.keystore.algorithm,xpack.notification.email.ssl.verification_mode,transport.profiles.*.xpack.security.ssl.truststore.password,transport.profiles.*.xpack.security.ssl.certificate,transport.profiles.*.xpack.security.*,xpack.security.authc.realms.ldap.*.ssl.truststore.password,xpack.security.authc.realms.pki.*.certificate_authorities,xpack.security.authc.realms.active_directory.*.bind_dn,xpack.http.ssl.truststore.path,xpack.http.ssl.key_passphrase,xpack.security.authc.realms.saml.*.ssl.certificate_authorities,reindex.ssl.verification_mode,xpack.http.ssl.keystore.password,xpack.notification.email.ssl.trust_restrictions.path,xpack.notification.email.ssl.keystore.key_password,xpack.security.transport.ssl.truststore.path,xpack.security.transport.ssl.trust_restrictions.path,xpack.security.http.ssl.truststore.type,xpack.http.ssl.certificate_authorities,xpack.security.authc.realms.active_directory.*.bind_password,xpack.security.transport.ssl.keystore.type,xpack.security.authc.realms.active_directory.*.ssl.keystore.path,xpack.security.http.ssl.keystore.algorithm,reindex.ssl.keystore.path,xpack.security.authc.realms.ldap.*.ssl.trust_restrictions,xpack.security.authc.realms.saml.*.ssl.verification_mode,xpack.security.authc.realms.oidc.*.ssl.cipher_suites,xpack.security.authc.realms.ldap.*.ssl.supported_protocols,transport.profiles.*.xpack.security.ssl.key,xpack.security.transport.ssl.keystore.password,reindex.ssl.cipher_suites,xpack.security.authc.realms.pki.*.truststore.algorithm,xpack.monitoring.exporters.*.auth.password,xpack.security.transport.ssl.keystore.algorithm,transport.profiles.*.xpack.security.ssl.keystore.type,xpack.security.authc.realms.saml.*.encryption.keystore.path,xpack.notification.email.ssl.client_authentication,transport.profiles.*.xpack.security.ssl.client_authentication,xpack.security.authc.realms.ldap.*.ssl.key_passphrase,reindex.ssl.keystore.algorithm,xpack.security.authc.realms.oidc.*.ssl.supported_protocols,xpack.http.ssl.cipher_suites,xpack.security.authc.realms.active_directory.*.ssl.certificate_authorities,xpack.security.authc.realms.saml.*.ssl.client_authentication,xpack.security.authc.realms.oidc.*.ssl.client_authentication,xpack.monitoring.exporters.*.auth.username,xpack.security.authc.realms.active_directory.*.ssl.truststore.type,xpack.security.http.ssl.client_authentication,xpack.security.authc.accept_default_password,xpack.security.authc.realms.ldap.*.ssl.keystore.path,xpack.notification.email.ssl.keystore.path,xpack.security.authc.realms.active_directory.*.ssl.keystore.algorithm,xpack.security.transport.ssl.key,xpack.security.transport.ssl.truststore.algorithm,xpack.security.authc.realms.ldap.*.ssl.truststore.type,xpack.http.ssl.certificate,transport.profiles.*.xpack.security.ssl.certificate_authorities,xpack.security.authc.realms.active_directory.*.ssl.cipher_suites,transport.profiles.*.xpack.security.ssl.keystore.password,xpack.notification.email.ssl.truststore.algorithm,xpack.security.authc.realms.saml.*.ssl.key,xpack.notification.email.ssl.cipher_suites,xpack.security.authc.realms.saml.*.ssl.keystore.type,xpack.security.authc.realms.saml.*.ssl.keystore.algorithm,xpack.security.authc.realms.oidc.*.ssl.trust_restrictions,xpack.security.authc.realms.saml.*.signing.keystore.type,xpack.security.authc.realms.active_directory.*.ssl.certificate,reindex.ssl.truststore.type,xpack.security.transport.ssl.cipher_suites,xpack.security.http.ssl.certificate,xpack.http.ssl.keystore.type,xpack.security.http.ssl.certificate_authorities,xpack.security.authc.realms.oidc.*.ssl.truststore.password,transport.profiles.*.xpack.security.ssl.truststore.type,transport.profiles.*.xpack.security.ssl.cipher_suites,xpack.security.authc.realms.saml.*.ssl.cipher_suites,xpack.security.authc.realms.saml.*.ssl.key_passphrase,xpack.security.authc.realms.active_directory.*.ssl.truststore.algorithm,xpack.security.authc.realms.saml.*.signing.keystore.algorithm,xpack.security.authc.realms.ldap.*.bind_dn,xpack.security.authc.realms.saml.*.encryption.certificate,xpack.security.authc.realms.oidc.*.ssl.keystore.key_password,xpack.http.ssl.key,xpack.notification.email.ssl.truststore.type,xpack.security.authc.realms.saml.*.ssl.truststore.path,xpack.security.authc.realms.active_directory.*.ssl.key,xpack.security.authc.realms.saml.*.ssl.truststore.password,xpack.security.authc.realms.pki.*.truststore.password,reindex.ssl.keystore.password,xpack.security.authc.realms.oidc.*.ssl.certificate,xpack.http.ssl.verification_mode,reindex.ssl.certificate_authorities,transport.profiles.*.xpack.security.ssl.verification_mode,xpack.security.http.ssl.keystore.key_password,transport.profiles.*.xpack.security.ssl.keystore.key_password,xpack.notification.email.ssl.certificate_authorities,xpack.notification.email.ssl.keystore.type,xpack.monitoring.exporters.*.ssl,xpack.security.authc.realms.saml.*.ssl.trust_restrictions,xpack.security.authc.realms.ldap.*.bind_password,xpack.security.http.ssl.cipher_suites,reindex.ssl.certificate,transport.profiles.*.xpack.security.ssl.truststore.algorithm,xpack.security.authc.realms.active_directory.*.ssl.keystore.type,xpack.security.transport.ssl.truststore.type,xpack.security.transport.ssl.certificate_authorities,xpack.security.authc.realms.active_directory.*.ssl.trust_restrictions,xpack.security.authc.realms.saml.*.ssl.supported_protocols,xpack.security.transport.ssl.keystore.path,xpack.notification.email.ssl.truststore.path,xpack.http.ssl.trust_restrictions.path,xpack.security.http.ssl.key_passphrase,xpack.security.http.ssl.trust_restrictions.path,xpack.http.ssl.truststore.type,transport.profiles.*.xpack.security.ssl.keystore.algorithm,xpack.security.transport.ssl.client_authentication,xpack.security.authc.realms.ldap.*.ssl.client_authentication,reindex.ssl.client_authentication,xpack.security.authc.realms.oidc.*.ssl.truststore.path,reindex.ssl.keystore.type,xpack.http.ssl.client_authentication,xpack.security.transport.ssl.key_passphrase,xpack.monitoring.exporters.*.ssl.*,xpack.security.hide_settings,xpack.security.authc.realms.saml.*.signing.certificate,xpack.security.authc.realms.saml.*.encryption.keystore.type,xpack.security.authc.realms.ldap.*.hostname_verification,xpack.security.authc.realms.active_directory.*.ssl.truststore.path,xpack.notification.email.ssl.keystore.password,reindex.ssl.key,xpack.security.authc.realms.active_directory.*.ssl.key_passphrase,transport.profiles.*.xpack.security.ssl.keystore.path,xpack.security.http.ssl.keystore.path,xpack.security.authc.realms.ldap.*.ssl.truststore.algorithm,xpack.security.authc.realms.ldap.*.ssl.keystore.type,xpack.security.authc.realms.active_directory.*.ssl.client_authentication,xpack.security.authc.realms.oidc.*.ssl.keystore.type,xpack.security.authc.realms.saml.*.ssl.certificate,xpack.notification.email.ssl.key_passphrase,xpack.security.authc.realms.active_directory.*.ssl.keystore.password,xpack.security.authc.realms.oidc.*.ssl.truststore.algorithm,transport.profiles.*.xpack.security.ssl.key_passphrase,xpack.security.authc.realms.ldap.*.ssl.certificate}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:218) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:598) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
[ deleted repeated messages ]
[2021-05-03T15:44:28,805][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T15:44:29,215][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T15:44:29,215][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T15:44:29,215][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T15:44:29,216][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T15:44:29,231][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T15:44:29,231][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T15:44:29,231][DEBUG][o.e.a.s.m.TransportMasterNodeAction] [master2] no known master node, scheduling a retry
[2021-05-03T15:44:29,636][INFO ][o.e.c.s.ClusterApplierService] [master2] master node changed {previous [], current [{master2}{RBMXwoyITeqpwJlY7VFFhA}{a7xhJcj7R46HQAm3OcmNJw}{192.168.0.12}{192.168.0.12:9300}{im}{xpack.installed=true}]}, removed {{master1}{Ig33yWUwTv-UNSwIhhfNWg}{GyE6pKvWTfeRglHhD-5oCw}{192.168.0.11}{192.168.0.11:9300}{im}{xpack.installed=true}}, added {{master1}{Ig33yWUwTv-UNSwIhhfNWg}{iUGHa9kFTjK57EFpCcNWbA}{192.168.0.11}{192.168.0.11:9300}{im}{xpack.installed=true}}, term: 49, version: 2131752, reason: Publication{term=49, version=2131752}
[2021-05-03T15:44:30,376][WARN ][o.e.c.c.C.CoordinatorPublication] [master2] after [30.6s] publication of cluster state version [2131752] is still waiting for {master0}{xexrc0uXQUWgwFDWEpAouA}{2uUAYysWQ8-1HDcE4lAqZw}{192.168.0.10}{192.168.0.10:9300}{im}{xpack.installed=true} [SENT_PUBLISH_REQUEST]
[2021-05-03T15:44:35,430][DEBUG][o.e.a.a.c.n.t.c.TransportCancelTasksAction] [master2] Removing ban for the parent [22tiK4-STyC2XRJ01nWQ3A:3325538968] on the node [RBMXwoyITeqpwJlY7VFFhA]
[2021-05-03T15:44:35,672][DEBUG][o.e.a.a.c.n.t.c.TransportCancelTasksAction] [master2] Removing ban for the parent [dN08oWhmSNObQgeb9xH_-A:214934874] on the node [RBMXwoyITeqpwJlY7VFFhA]
[2021-05-03T15:44:40,401][INFO ][o.e.c.c.C.CoordinatorPublication] [master2] after [9.8s] publication of cluster state version [2131753] is still waiting for {master0}{xexrc0uXQUWgwFDWEpAouA}{2uUAYysWQ8-1HDcE4lAqZw}{192.168.0.10}{192.168.0.10:9300}{im}{xpack.installed=true} [SENT_PUBLISH_REQUEST]
[2021-05-03T15:44:58,636][INFO ][o.e.m.j.JvmGcMonitorService] [master2] [gc][5647] overhead, spent [270ms] collecting in the last [1s]
[2021-05-03T15:45:07,639][INFO ][o.e.m.j.JvmGcMonitorService] [master2] [gc][5656] overhead, spent [362ms] collecting in the last [1s]
[2021-05-03T15:45:37,889][INFO ][o.e.m.j.JvmGcMonitorService] [master2] [gc][5686] overhead, spent [436ms] collecting in the last [1s]
[2021-05-03T15:46:18,117][INFO ][o.e.m.j.JvmGcMonitorService] [master2] [gc][5726] overhead, spent [366ms] collecting in the last [1s]
[2021-05-03T15:47:58,449][INFO ][o.e.m.j.JvmGcMonitorService] [master2] [gc][5826] overhead, spent [296ms] collecting in the last [1s]
[2021-05-03T15:48:18,459][INFO ][o.e.m.j.JvmGcMonitorService] [master2] [gc][5846] overhead, spent [267ms] collecting in the last [1s]
[2021-05-03T15:48:24,999][DEBUG][o.e.a.a.c.n.t.c.TransportCancelTasksAction] [master2] Received ban for the parent [Cl-VWQCjQ3OStxiPmdIXrQ:165291719] on the node [RBMXwoyITeqpwJlY7VFFhA], reason: [by user request]
[2021-05-03T15:48:25,014][DEBUG][o.e.a.a.c.n.t.c.TransportCancelTasksAction] [master2] Received ban for the parent [3sy0vv41TXGy4Yt8ywpXVw:218731516] on the node [RBMXwoyITeqpwJlY7VFFhA], reason: [by user request]
[2021-05-03T15:48:28,461][INFO ][o.e.m.j.JvmGcMonitorService] [master2] [gc][5856] overhead, spent [258ms] collecting in the last [1s]
[2021-05-03T15:48:30,407][DEBUG][o.e.a.a.c.n.t.c.TransportCancelTasksAction] [master2] Removing ban for the parent [3sy0vv41TXGy4Yt8ywpXVw:218731516] on the node [RBMXwoyITeqpwJlY7VFFhA]
[2021-05-03T15:48:36,035][DEBUG][o.e.a.a.c.n.t.c.TransportCancelTasksAction] [master2] Removing ban for the parent [Cl-VWQCjQ3OStxiPmdIXrQ:165291719] on the node [RBMXwoyITeqpwJlY7VFFhA]
[2021-05-03T15:48:57,523][INFO ][o.e.m.j.JvmGcMonitorService] [master2] [gc][5885] overhead, spent [308ms] collecting in the last [1s]
[2021-05-03T15:49:17,582][INFO ][o.e.m.j.JvmGcMonitorService] [master2] [gc][5905] overhead, spent [273ms] collecting in the last [1s]
[2021-05-03T15:49:39,780][DEBUG][o.e.a.a.c.n.t.c.TransportCancelTasksAction] [master2] Received ban for the parent [3sy0vv41TXGy4Yt8ywpXVw:218820445] on the node [RBMXwoyITeqpwJlY7VFFhA], reason: [by user request]
[2021-05-03T15:49:45,513][DEBUG][o.e.a.a.c.n.t.c.TransportCancelTasksAction] [master2] Removing ban for the parent [3sy0vv41TXGy4Yt8ywpXVw:218820445] on the node [RBMXwoyITeqpwJlY7VFFhA]
[2021-05-03T15:49:49,369][DEBUG][o.e.a.a.c.n.t.c.TransportCancelTasksAction] [master2] Received ban for the parent [22tiK4-STyC2XRJ01nWQ3A:3325759373] on the node [RBMXwoyITeqpwJlY7VFFhA], reason: [by user request]
[2021-05-03T15:49:55,488][DEBUG][o.e.a.a.c.n.t.c.TransportCancelTasksAction] [master2] Removing ban for the parent [22tiK4-STyC2XRJ01nWQ3A:3325759373] on the node [RBMXwoyITeqpwJlY7VFFhA]
[2021-05-03T15:49:57,657][INFO ][o.e.m.j.JvmGcMonitorService] [master2] [gc][5945] overhead, spent [311ms] collecting in the last [1s]
[2021-05-03T15:50:17,726][INFO ][o.e.m.j.JvmGcMonitorService] [master2] [gc][5965] overhead, spent [260ms] collecting in the last [1s]
[2021-05-03T15:50:27,752][INFO ][o.e.m.j.JvmGcMonitorService] [master2] [gc][5975] overhead, spent [283ms] collecting in the last [1s]
[2021-05-03T15:50:37,769][INFO ][o.e.m.j.JvmGcMonitorService] [master2] [gc][5985] overhead, spent [353ms] collecting in the last [1s]
[2021-05-03T15:50:47,772][INFO ][o.e.m.j.JvmGcMonitorService] [master2] [gc][5995] overhead, spent [260ms] collecting in the last [1s]
[2021-05-03T15:51:07,787][INFO ][o.e.m.j.JvmGcMonitorService] [master2] [gc][6015] overhead, spent [316ms] collecting in the last [1s]
[2021-05-03T15:51:26,798][INFO ][o.e.m.j.JvmGcMonitorService] [master2] [gc][6034] overhead, spent [254ms] collecting in the last [1s]
[2021-05-03T15:51:27,799][INFO ][o.e.m.j.JvmGcMonitorService] [master2] [gc][6035] overhead, spent [296ms] collecting in the last [1s]
[2021-05-03T15:52:18,127][INFO ][o.e.m.j.JvmGcMonitorService] [master2] [gc][6085] overhead, spent [282ms] collecting in the last [1s]
[2021-05-03T15:52:28,132][INFO ][o.e.m.j.JvmGcMonitorService] [master2] [gc][6095] overhead, spent [252ms] collecting in the last [1s]
[2021-05-03T15:52:38,135][INFO ][o.e.m.j.JvmGcMonitorService] [master2] [gc][6105] overhead, spent [407ms] collecting in the last [1s]
[2021-05-03T15:52:48,139][INFO ][o.e.m.j.JvmGcMonitorService] [master2] [gc][6115] overhead, spent [287ms] collecting in the last [1s]
[2021-05-03T15:52:54,454][INFO ][o.e.c.s.MasterService ] [master2] node-left[{master1}{Ig33yWUwTv-UNSwIhhfNWg}{iUGHa9kFTjK57EFpCcNWbA}{192.168.0.11}{192.168.0.11:9300}{im}{xpack.installed=true} reason: disconnected], term: 49, version: 2131757, delta: removed {{master1}{Ig33yWUwTv-UNSwIhhfNWg}{iUGHa9kFTjK57EFpCcNWbA}{192.168.0.11}{192.168.0.11:9300}{im}{xpack.installed=true}}
[2021-05-03T15:52:54,649][INFO ][o.e.c.s.ClusterApplierService] [master2] removed {{master1}{Ig33yWUwTv-UNSwIhhfNWg}{iUGHa9kFTjK57EFpCcNWbA}{192.168.0.11}{192.168.0.11:9300}{im}{xpack.installed=true}}, term: 49, version: 2131757, reason: Publication{term=49, version=2131757}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment