Skip to content

Instantly share code, notes, and snippets.

@elastic-paul

elastic-paul/trace_master_election.log Secret

Created September 3, 2019 15:46
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save elastic-paul/15c8a925a0f5171c19c3b84ca9b6826d to your computer and use it in GitHub Desktop.
Save elastic-paul/15c8a925a0f5171c19c3b84ca9b6826d to your computer and use it in GitHub Desktop.
Download ZIP
Raw
trace_master_election.log
This file has been truncated, but you can view the full file.
[2019-09-03T15:40:47,336][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [hdp13] [controller/15147] [Main.cc@110] controller (64 bit): Version 7.3.1 (Build 1d93901e09ef43) Copyright (c) 2019 Elasticsearch BV
[2019-09-03T15:40:49,509][INFO ][o.e.d.DiscoveryModule ] [hdp13] using discovery type [zen] and seed hosts providers [settings]
[2019-09-03T15:40:50,410][INFO ][o.e.n.Node ] [hdp13] initialized
[2019-09-03T15:40:50,410][INFO ][o.e.n.Node ] [hdp13] starting ...
[2019-09-03T15:40:50,556][INFO ][o.e.t.TransportService ] [hdp13] publish_address {192.168.1.213:9300}, bound_addresses {192.168.1.213:9300}
[2019-09-03T15:40:50,566][INFO ][o.e.b.BootstrapChecks ] [hdp13] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-09-03T15:40:50,576][INFO ][o.e.c.c.Coordinator ] [hdp13] cluster UUID [yElPwPBVT-GPUb_lj3zJZg]
[2019-09-03T15:40:50,579][DEBUG][o.e.c.c.Coordinator ] [hdp13] startInitialJoin: coordinator becoming CANDIDATE in term 5222 (was null, lastKnownLeader was [Optional.empty])
[2019-09-03T15:40:50,584][TRACE][o.e.c.c.LeaderChecker ] [hdp13] setCurrentNodes: nodes:
[2019-09-03T15:40:50,586][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5222, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader=null
[2019-09-03T15:40:50,823][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=0s, thisAttempt=0, maxDelayMillis=100, delayMillis=85, ElectionScheduler{attempt=1, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:40:50,824][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5265
[2019-09-03T15:40:50,829][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5222] due to StartJoinRequest{term=5265,node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:40:50,908][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=0s, thisAttempt=0, maxDelayMillis=100, delayMillis=85, ElectionScheduler{attempt=1, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:40:50,909][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=1, maxDelayMillis=200, delayMillis=662, ElectionScheduler{attempt=2, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:40:51,276][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5265, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader=null
[2019-09-03T15:40:51,277][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5265, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:40:51,279][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5267
[2019-09-03T15:40:51,279][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5265] due to StartJoinRequest{term=5267,node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:40:51,285][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5265, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5265 does not match current term 5267
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:51,293][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5265, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5265 does not match current term 5267
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:51,572][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=1, maxDelayMillis=200, delayMillis=662, ElectionScheduler{attempt=2, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:40:51,572][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=2, maxDelayMillis=300, delayMillis=672, ElectionScheduler{attempt=3, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:40:51,660][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5267, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader=null
[2019-09-03T15:40:51,660][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5267, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:40:51,660][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5267
[2019-09-03T15:40:51,661][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5267,node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5267]
[2019-09-03T15:40:51,664][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5267}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:40:51,667][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5267}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5267, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}
[2019-09-03T15:40:51,667][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5267}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5268, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:40:51,667][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5267}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:40:51,669][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}] with term 5268
[2019-09-03T15:40:51,669][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5267] due to StartJoinRequest{term=5268,node={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:40:51,670][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5267}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5268, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:40:51,670][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5267, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5267 does not match current term 5268
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:51,672][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5267, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5267 does not match current term 5268
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:51,706][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5267}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5268, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:40:51,761][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5267}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5268, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:40:52,069][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5268, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader=null
[2019-09-03T15:40:52,069][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5265
[2019-09-03T15:40:52,069][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5268, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:40:52,069][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5265,node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5268]
[2019-09-03T15:40:52,069][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5268
[2019-09-03T15:40:52,070][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5268,node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5268]
[2019-09-03T15:40:52,070][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5269
[2019-09-03T15:40:52,070][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5268] due to StartJoinRequest{term=5269,node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:40:52,209][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5267}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5269, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:40:52,219][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5267}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5269, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:40:52,244][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=2, maxDelayMillis=300, delayMillis=672, ElectionScheduler{attempt=3, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:40:52,245][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=3, maxDelayMillis=400, delayMillis=810, ElectionScheduler{attempt=4, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:40:52,461][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5269, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader=null
[2019-09-03T15:40:52,461][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5269, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:40:52,461][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5269
[2019-09-03T15:40:52,461][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5269,node={hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5269]
[2019-09-03T15:40:52,462][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5268, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5267}, isClosed=false} ignoring PreVoteResponse{currentTerm=5268, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:52,462][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5268, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5267}, isClosed=false} ignoring PreVoteResponse{currentTerm=5268, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:52,462][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5268, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5267}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5269, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}
[2019-09-03T15:40:52,462][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5268, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5267}, isClosed=true} ignoring PreVoteResponse{currentTerm=5268, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:52,462][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5269}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:40:52,462][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5268, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5267}, isClosed=true} added PreVoteResponse{currentTerm=5268, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, no quorum yet
[2019-09-03T15:40:52,463][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5270
[2019-09-03T15:40:52,464][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5269] due to StartJoinRequest{term=5270,node={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:40:52,463][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5269, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5269 does not match current term 5270
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:52,464][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5269, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5269 does not match current term 5270
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:52,480][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5268, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5267}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5270, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:40:52,480][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5267}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5270, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:40:52,535][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5268, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp15b][192.168.1.225:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5268 does not match current term 5270
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:52,536][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5268, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp15b][192.168.1.225:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5268 does not match current term 5270
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:52,862][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5270, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader=null
[2019-09-03T15:40:52,862][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5268, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5267}, isClosed=true} ignoring PreVoteResponse{currentTerm=5269, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:52,862][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5268, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5267}, isClosed=true} ignoring PreVoteResponse{currentTerm=5269, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:52,862][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5270
[2019-09-03T15:40:52,862][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5270,node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5270]
[2019-09-03T15:40:52,862][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5270, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:40:52,863][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5269}, isClosed=false} ignoring PreVoteResponse{currentTerm=5271, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:52,863][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5272
[2019-09-03T15:40:52,863][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5269}, isClosed=false} ignoring PreVoteResponse{currentTerm=5271, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:52,863][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5270] due to StartJoinRequest{term=5272,node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:40:53,055][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=3, maxDelayMillis=400, delayMillis=810, ElectionScheduler{attempt=4, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:40:53,055][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=4, maxDelayMillis=500, delayMillis=519, ElectionScheduler{attempt=5, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:40:53,262][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5272, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader=null
[2019-09-03T15:40:53,262][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5272, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:40:53,262][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5269}, isClosed=false} ignoring PreVoteResponse{currentTerm=5271, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:53,262][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5272
[2019-09-03T15:40:53,263][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5272,node={hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5272]
[2019-09-03T15:40:53,263][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional.empty}
[2019-09-03T15:40:53,264][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5271
[2019-09-03T15:40:53,264][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5271,node={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5272]
[2019-09-03T15:40:53,264][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}] with term 5271
[2019-09-03T15:40:53,265][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5271,node={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5272]
[2019-09-03T15:40:53,265][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5271
[2019-09-03T15:40:53,265][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5271,node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5272]
[2019-09-03T15:40:53,265][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5270, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5269}, isClosed=false} ignoring PreVoteResponse{currentTerm=5270, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:53,265][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5273
[2019-09-03T15:40:53,265][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5270, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5269}, isClosed=false} added PreVoteResponse{currentTerm=5270, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, no quorum yet
[2019-09-03T15:40:53,265][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5270, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5269}, isClosed=false} ignoring PreVoteResponse{currentTerm=5270, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:53,266][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5272] due to StartJoinRequest{term=5273,node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:40:53,265][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5270, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5269}, isClosed=false} ignoring PreVoteResponse{currentTerm=5270, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:53,404][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5270, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14][192.168.1.214:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5270 does not match current term 5273
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:53,406][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5270, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14][192.168.1.214:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5270 does not match current term 5273
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:53,575][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=4, maxDelayMillis=500, delayMillis=519, ElectionScheduler{attempt=5, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:40:53,575][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=5, maxDelayMillis=600, delayMillis=730, ElectionScheduler{attempt=6, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:40:53,671][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5273, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader=null
[2019-09-03T15:40:53,671][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5273
[2019-09-03T15:40:53,671][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5273,node={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5273]
[2019-09-03T15:40:53,671][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5273, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:40:53,672][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5273}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:40:53,673][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5273}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5273, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}
[2019-09-03T15:40:53,673][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5273}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:40:53,674][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5274
[2019-09-03T15:40:53,674][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5273] due to StartJoinRequest{term=5274,node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:40:53,717][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5273}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5274, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:40:53,790][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5272, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5272 does not match current term 5274
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:53,791][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5272, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5272 does not match current term 5274
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:54,023][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5273}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5275, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:40:54,071][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5274, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader=null
[2019-09-03T15:40:54,071][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}] with term 5274
[2019-09-03T15:40:54,072][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5274,node={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5274]
[2019-09-03T15:40:54,072][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5274, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:40:54,072][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5274
[2019-09-03T15:40:54,072][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5274,node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5274]
[2019-09-03T15:40:54,072][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5275
[2019-09-03T15:40:54,072][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5273}, isClosed=false} ignoring PreVoteResponse{currentTerm=5275, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:54,072][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5274] due to StartJoinRequest{term=5275,node={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:40:54,073][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5270, lastAcceptedTerm=5161, lastAcceptedVersion=152}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5272, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5269}, isClosed=true} added PreVoteResponse{currentTerm=5272, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:40:54,105][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5273}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5275, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:40:54,106][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5273}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5275, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:40:54,107][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5273, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5273 does not match current term 5275
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:54,108][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5273, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5273 does not match current term 5275
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:54,112][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5273}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5275, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:40:54,125][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5273}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5275, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:40:54,213][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5273}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5275, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:40:54,305][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=5, maxDelayMillis=600, delayMillis=730, ElectionScheduler{attempt=6, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:40:54,306][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=6, maxDelayMillis=700, delayMillis=1045, ElectionScheduler{attempt=7, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:40:54,424][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5274, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5274 does not match current term 5276
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:54,425][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5274, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5274 does not match current term 5276
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:54,480][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5275, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader=null
[2019-09-03T15:40:54,480][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5275, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:40:54,480][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5275
[2019-09-03T15:40:54,481][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5275,node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5275]
[2019-09-03T15:40:54,481][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5277
[2019-09-03T15:40:54,481][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5273}, isClosed=false} ignoring PreVoteResponse{currentTerm=5274, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:54,481][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5275] due to StartJoinRequest{term=5277,node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:40:54,707][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5275, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14][192.168.1.214:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5275 does not match current term 5277
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:54,709][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5275, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14][192.168.1.214:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5275 does not match current term 5277
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:54,830][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5277, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader=null
[2019-09-03T15:40:54,830][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5277
[2019-09-03T15:40:54,831][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5277, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:40:54,831][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5277,node={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5277]
[2019-09-03T15:40:54,831][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5277
[2019-09-03T15:40:54,831][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5277,node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5277]
[2019-09-03T15:40:54,831][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}] with term 5277
[2019-09-03T15:40:54,831][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5277,node={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5277]
[2019-09-03T15:40:54,831][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5277
[2019-09-03T15:40:54,832][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5277,node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5277]
[2019-09-03T15:40:54,832][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5277}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:40:54,833][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5275, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5273}, isClosed=true} ignoring PreVoteResponse{currentTerm=5275, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:54,833][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5275, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5273}, isClosed=true} added PreVoteResponse{currentTerm=5275, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, no quorum yet
[2019-09-03T15:40:54,833][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}] with term 5276
[2019-09-03T15:40:54,833][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5276,node={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5277]
[2019-09-03T15:40:54,834][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5275, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5273}, isClosed=true} ignoring PreVoteResponse{currentTerm=5275, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:54,834][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5276
[2019-09-03T15:40:54,834][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5276,node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5277]
[2019-09-03T15:40:54,834][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5275, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5273}, isClosed=true} ignoring PreVoteResponse{currentTerm=5275, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:54,834][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5275, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5273}, isClosed=true} ignoring PreVoteResponse{currentTerm=5275, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:54,834][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5278
[2019-09-03T15:40:54,835][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5277] due to StartJoinRequest{term=5278,node={hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:40:55,009][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5277, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5277 does not match current term 5278
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:55,010][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5277, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5277 does not match current term 5278
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:55,248][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5278, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader=null
[2019-09-03T15:40:55,248][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5278
[2019-09-03T15:40:55,248][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5278, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:40:55,248][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5278,node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5278]
[2019-09-03T15:40:55,249][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5279
[2019-09-03T15:40:55,249][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5278] due to StartJoinRequest{term=5279,node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:40:55,249][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5275, lastAcceptedTerm=5161, lastAcceptedVersion=152}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5275, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5273}, isClosed=true} added PreVoteResponse{currentTerm=5275, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:40:55,351][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=6, maxDelayMillis=700, delayMillis=1045, ElectionScheduler{attempt=7, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:40:55,351][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=7, maxDelayMillis=800, delayMillis=1244, ElectionScheduler{attempt=8, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:40:55,681][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5279, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader=null
[2019-09-03T15:40:55,681][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5277}, isClosed=false} ignoring PreVoteResponse{currentTerm=5278, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:55,681][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5277}, isClosed=false} ignoring PreVoteResponse{currentTerm=5278, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:55,682][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5279, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:40:55,682][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5281
[2019-09-03T15:40:55,682][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5277}, isClosed=false} ignoring PreVoteResponse{currentTerm=5278, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:55,682][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5277, lastAcceptedTerm=5161, lastAcceptedVersion=152}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5277, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5277}, isClosed=false} added PreVoteResponse{currentTerm=5277, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:40:55,682][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5279] due to StartJoinRequest{term=5281,node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:40:55,682][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5277, lastAcceptedTerm=5161, lastAcceptedVersion=152}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5277, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5277}, isClosed=false} added PreVoteResponse{currentTerm=5277, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, no quorum yet
[2019-09-03T15:40:55,778][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5278, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16b][192.168.1.226:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5278 does not match current term 5280
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:55,779][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5278, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16b][192.168.1.226:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5278 does not match current term 5280
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:55,951][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5279, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5279 does not match current term 5281
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:55,952][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5279, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5279 does not match current term 5281
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:56,040][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5281, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader=null
[2019-09-03T15:40:56,040][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5281, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:40:56,041][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5281}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:40:56,041][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5277, lastAcceptedTerm=5161, lastAcceptedVersion=152}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5277, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5277}, isClosed=true} ignoring PreVoteResponse{currentTerm=5279, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:56,041][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5280
[2019-09-03T15:40:56,042][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5280,node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5281]
[2019-09-03T15:40:56,042][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5282
[2019-09-03T15:40:56,042][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5277, lastAcceptedTerm=5161, lastAcceptedVersion=152}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5277, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5277}, isClosed=true} ignoring PreVoteResponse{currentTerm=5279, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:56,042][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5281] due to StartJoinRequest{term=5282,node={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:40:56,196][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5281, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5281 does not match current term 5282
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:56,197][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5281, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5281 does not match current term 5282
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:56,432][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5282, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader=null
[2019-09-03T15:40:56,432][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5282
[2019-09-03T15:40:56,432][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5282, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:40:56,432][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5282,node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5282]
[2019-09-03T15:40:56,433][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5277, lastAcceptedTerm=5161, lastAcceptedVersion=152}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5277, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5277}, isClosed=true} ignoring PreVoteResponse{currentTerm=5280, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:56,433][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5281
[2019-09-03T15:40:56,433][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5281,node={hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5282]
[2019-09-03T15:40:56,433][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}] with term 5283
[2019-09-03T15:40:56,433][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5282] due to StartJoinRequest{term=5283,node={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:40:56,596][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=7, maxDelayMillis=800, delayMillis=1244, ElectionScheduler{attempt=8, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:40:56,596][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=8, maxDelayMillis=900, delayMillis=708, ElectionScheduler{attempt=9, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:40:56,838][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5283, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader=null
[2019-09-03T15:40:56,838][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5283
[2019-09-03T15:40:56,838][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5283, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:40:56,838][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5283,node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5283]
[2019-09-03T15:40:56,838][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5283
[2019-09-03T15:40:56,838][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5283,node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5283]
[2019-09-03T15:40:56,839][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5282, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5281}, isClosed=false} added PreVoteResponse{currentTerm=5282, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, no quorum yet
[2019-09-03T15:40:56,839][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5282, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5281}, isClosed=false} ignoring PreVoteResponse{currentTerm=5282, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:56,839][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5282, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5281}, isClosed=false} ignoring PreVoteResponse{currentTerm=5282, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:56,839][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5282, lastAcceptedTerm=5161, lastAcceptedVersion=152}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5281, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5281}, isClosed=false} added PreVoteResponse{currentTerm=5281, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:40:56,839][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}] with term 5284
[2019-09-03T15:40:56,839][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5282, lastAcceptedTerm=5161, lastAcceptedVersion=152}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5281, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5281}, isClosed=false} ignoring PreVoteResponse{currentTerm=5281, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:56,839][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5282, lastAcceptedTerm=5161, lastAcceptedVersion=152}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5281, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5281}, isClosed=false} ignoring PreVoteResponse{currentTerm=5281, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:56,839][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5282, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5281}, isClosed=false} ignoring PreVoteResponse{currentTerm=5282, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:56,839][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5283] due to StartJoinRequest{term=5284,node={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:40:56,964][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5283, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp15b][192.168.1.225:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5283 does not match current term 5284
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:56,966][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5283, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp15b][192.168.1.225:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5283 does not match current term 5284
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:57,210][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5282, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14][192.168.1.214:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5282 does not match current term 5284
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:57,211][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5282, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14][192.168.1.214:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5282 does not match current term 5284
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:57,263][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5284, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader=null
[2019-09-03T15:40:57,264][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5284
[2019-09-03T15:40:57,264][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5284, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:40:57,264][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5284,node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5284]
[2019-09-03T15:40:57,264][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5284}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:40:57,266][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}] with term 5285
[2019-09-03T15:40:57,266][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5284, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5284}, isClosed=false} added PreVoteResponse{currentTerm=5284, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:40:57,266][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5282, lastAcceptedTerm=5161, lastAcceptedVersion=152}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5281, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5281}, isClosed=true} ignoring PreVoteResponse{currentTerm=5284, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:57,266][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5284] due to StartJoinRequest{term=5285,node={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:40:57,290][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5284, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp15b][192.168.1.225:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5284 does not match current term 5285
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:57,291][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5284, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp15b][192.168.1.225:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5284 does not match current term 5285
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:57,304][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=8, maxDelayMillis=900, delayMillis=708, ElectionScheduler{attempt=9, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:40:57,304][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=9, maxDelayMillis=1000, delayMillis=1026, ElectionScheduler{attempt=10, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:40:57,605][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5285, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader=null
[2019-09-03T15:40:57,605][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5285
[2019-09-03T15:40:57,606][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5285,node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5285]
[2019-09-03T15:40:57,606][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5285, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:40:57,606][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5285
[2019-09-03T15:40:57,606][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5285,node={hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5285]
[2019-09-03T15:40:57,606][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5286
[2019-09-03T15:40:57,606][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5285] due to StartJoinRequest{term=5286,node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:40:57,997][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5286, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader=null
[2019-09-03T15:40:57,998][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5286, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:40:57,998][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5286
[2019-09-03T15:40:57,998][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5286,node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5286]
[2019-09-03T15:40:57,998][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5284, lastAcceptedTerm=5161, lastAcceptedVersion=152}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5285, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5284}, isClosed=true} added PreVoteResponse{currentTerm=5285, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, no quorum yet
[2019-09-03T15:40:57,998][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5286}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:40:57,999][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5284, lastAcceptedTerm=5161, lastAcceptedVersion=152}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5285, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5284}, isClosed=true} ignoring PreVoteResponse{currentTerm=5285, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:57,999][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5284, lastAcceptedTerm=5161, lastAcceptedVersion=152}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5285, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5284}, isClosed=true} ignoring PreVoteResponse{currentTerm=5285, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:57,999][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5284, lastAcceptedTerm=5161, lastAcceptedVersion=152}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5285, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5284}, isClosed=true} ignoring PreVoteResponse{currentTerm=5286, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:57,999][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5284, lastAcceptedTerm=5161, lastAcceptedVersion=152}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5285, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5284}, isClosed=true} ignoring PreVoteResponse{currentTerm=5285, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:57,999][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5287
[2019-09-03T15:40:57,999][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5286, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5286}, isClosed=false} added PreVoteResponse{currentTerm=5286, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:40:58,000][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5286] due to StartJoinRequest{term=5287,node={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:40:58,036][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5286, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5286 does not match current term 5287
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:58,037][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5286, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5286 does not match current term 5287
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:58,081][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5285, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp15b][192.168.1.225:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5285 does not match current term 5287
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:58,082][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5285, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp15b][192.168.1.225:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5285 does not match current term 5287
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:58,112][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional.empty}
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: became follower
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.lambda$close$3(JoinHelper.java:477) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.HashMap$Values.forEach(HashMap.java:976) ~[?:?]
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.close(JoinHelper.java:477) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.becomeFollower(Coordinator.java:606) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.onFollowerCheckRequest(Coordinator.java:243) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.FollowersChecker$2.doRun(FollowersChecker.java:187) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:58,113][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional.empty}
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: became follower
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.lambda$close$3(JoinHelper.java:477) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.HashMap$Values.forEach(HashMap.java:976) ~[?:?]
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.close(JoinHelper.java:477) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.becomeFollower(Coordinator.java:606) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.onFollowerCheckRequest(Coordinator.java:243) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.FollowersChecker$2.doRun(FollowersChecker.java:187) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:40:58,331][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=9, maxDelayMillis=1000, delayMillis=1026, ElectionScheduler{attempt=10, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:40:58,331][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=10, maxDelayMillis=1100, delayMillis=1350, ElectionScheduler{attempt=11, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:40:58,364][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader=null
[2019-09-03T15:40:58,364][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:40:58,365][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5284, lastAcceptedTerm=5161, lastAcceptedVersion=152}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5285, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5284}, isClosed=true} ignoring PreVoteResponse{currentTerm=5286, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:58,365][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:40:58,366][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5286, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5286}, isClosed=true} ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:58,366][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5286, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5286}, isClosed=true} ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:58,366][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5286, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5286}, isClosed=true} ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:58,366][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5286, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5286}, isClosed=true} ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:58,366][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} added PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:40:58,429][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5284, lastAcceptedTerm=5161, lastAcceptedVersion=152}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5285, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5284}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5286, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:40:58,430][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} ignoring PreVoteResponse{currentTerm=5286, lastAcceptedTerm=5220, lastAcceptedVersion=153} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:40:58,595][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:40:59,596][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:40:59,681][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=10, maxDelayMillis=1100, delayMillis=1350, ElectionScheduler{attempt=11, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:40:59,682][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=11, maxDelayMillis=1200, delayMillis=932, ElectionScheduler{attempt=12, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:40:59,702][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:40:59,703][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} added PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:41:00,584][INFO ][o.e.c.c.JoinHelper ] [hdp13] last failed join attempt was 2.4s ago, failed to join {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional.empty}
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: became follower
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.lambda$close$3(JoinHelper.java:477) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.HashMap$Values.forEach(HashMap.java:976) ~[?:?]
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.close(JoinHelper.java:477) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.becomeFollower(Coordinator.java:606) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.onFollowerCheckRequest(Coordinator.java:243) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.FollowersChecker$2.doRun(FollowersChecker.java:187) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:00,587][WARN ][o.e.c.c.ClusterFormationFailureHelper] [hdp13] master not discovered or elected yet, an election requires at least 4 nodes with ids from [3-dBP8WZQZO-Rymo0vC3mQ, EYrLIFzETzSwcUvvabPegg, jguIm07bTXC_8ot6fYN9bw, 1cT_ZkvzTHaSyiqDEi2GSw, 3BP0A2-JTTS-Vp_1YdnJrw, vp_SiTAYQyCE1TQVIUXx2A, bkVjp2mNSk6G0PH9QFBu9Q], have discovered [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [192.168.1.214:9300, 192.168.1.215:9300, 192.168.1.216:9300, 192.168.1.223:9302, 192.168.1.224:9302, 192.168.1.225:9302, 192.168.1.226:9302] from hosts providers and [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 5287, last-accepted version 152 in term 5161
[2019-09-03T15:41:00,598][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:00,634][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=11, maxDelayMillis=1200, delayMillis=932, ElectionScheduler{attempt=12, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:41:00,634][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=12, maxDelayMillis=1300, delayMillis=1332, ElectionScheduler{attempt=13, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:41:00,634][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:41:00,635][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} added PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:41:01,599][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:01,966][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=12, maxDelayMillis=1300, delayMillis=1332, ElectionScheduler{attempt=13, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:41:01,966][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=13, maxDelayMillis=1400, delayMillis=1234, ElectionScheduler{attempt=14, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:41:01,967][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:41:01,968][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} added PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:41:02,600][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:03,201][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=13, maxDelayMillis=1400, delayMillis=1234, ElectionScheduler{attempt=14, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:41:03,201][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=14, maxDelayMillis=1500, delayMillis=645, ElectionScheduler{attempt=15, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:41:03,201][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:41:03,202][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} added PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:41:03,602][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:03,846][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=14, maxDelayMillis=1500, delayMillis=645, ElectionScheduler{attempt=15, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:41:03,847][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=15, maxDelayMillis=1600, delayMillis=887, ElectionScheduler{attempt=16, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:41:03,847][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:41:03,848][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} added PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:41:04,603][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:04,734][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=15, maxDelayMillis=1600, delayMillis=887, ElectionScheduler{attempt=16, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:41:04,734][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=16, maxDelayMillis=1700, delayMillis=1094, ElectionScheduler{attempt=17, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:41:04,734][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:41:04,777][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} added PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:41:05,605][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:05,829][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=16, maxDelayMillis=1700, delayMillis=1094, ElectionScheduler{attempt=17, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:41:05,829][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=17, maxDelayMillis=1800, delayMillis=1893, ElectionScheduler{attempt=18, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:41:05,829][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:41:05,830][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} added PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:41:06,605][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:07,606][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:07,722][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=17, maxDelayMillis=1800, delayMillis=1893, ElectionScheduler{attempt=18, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:41:07,722][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=18, maxDelayMillis=1900, delayMillis=1321, ElectionScheduler{attempt=19, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:41:07,723][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:41:07,723][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} added PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:41:08,608][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:09,044][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=18, maxDelayMillis=1900, delayMillis=1321, ElectionScheduler{attempt=19, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:41:09,044][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=19, maxDelayMillis=2000, delayMillis=1656, ElectionScheduler{attempt=20, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:41:09,044][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:41:09,045][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} added PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:41:09,609][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:10,588][WARN ][o.e.c.c.ClusterFormationFailureHelper] [hdp13] master not discovered or elected yet, an election requires at least 4 nodes with ids from [3-dBP8WZQZO-Rymo0vC3mQ, EYrLIFzETzSwcUvvabPegg, jguIm07bTXC_8ot6fYN9bw, 1cT_ZkvzTHaSyiqDEi2GSw, 3BP0A2-JTTS-Vp_1YdnJrw, vp_SiTAYQyCE1TQVIUXx2A, bkVjp2mNSk6G0PH9QFBu9Q], have discovered [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [192.168.1.214:9300, 192.168.1.215:9300, 192.168.1.216:9300, 192.168.1.223:9302, 192.168.1.224:9302, 192.168.1.225:9302, 192.168.1.226:9302] from hosts providers and [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 5287, last-accepted version 152 in term 5161
[2019-09-03T15:41:10,610][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:10,700][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=19, maxDelayMillis=2000, delayMillis=1656, ElectionScheduler{attempt=20, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:41:10,701][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=20, maxDelayMillis=2100, delayMillis=842, ElectionScheduler{attempt=21, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:41:10,701][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:41:10,702][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} added PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:41:11,543][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=20, maxDelayMillis=2100, delayMillis=842, ElectionScheduler{attempt=21, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:41:11,543][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=21, maxDelayMillis=2200, delayMillis=1643, ElectionScheduler{attempt=22, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:41:11,544][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:41:11,545][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} added PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:41:11,611][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:12,613][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:13,187][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=21, maxDelayMillis=2200, delayMillis=1643, ElectionScheduler{attempt=22, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:41:13,187][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=22, maxDelayMillis=2300, delayMillis=1744, ElectionScheduler{attempt=23, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:41:13,188][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:41:13,189][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} added PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:41:13,613][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:14,615][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:14,932][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=22, maxDelayMillis=2300, delayMillis=1744, ElectionScheduler{attempt=23, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:41:14,932][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=23, maxDelayMillis=2400, delayMillis=2329, ElectionScheduler{attempt=24, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:41:14,932][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:41:14,933][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} added PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:41:15,616][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:16,618][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:17,261][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=23, maxDelayMillis=2400, delayMillis=2329, ElectionScheduler{attempt=24, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:41:17,261][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=24, maxDelayMillis=2500, delayMillis=667, ElectionScheduler{attempt=25, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:41:17,262][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:41:17,262][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} added PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:41:17,618][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:17,929][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=24, maxDelayMillis=2500, delayMillis=667, ElectionScheduler{attempt=25, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:41:17,929][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=25, maxDelayMillis=2600, delayMillis=1141, ElectionScheduler{attempt=26, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:41:17,929][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:41:17,930][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} added PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:41:18,619][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:19,070][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=25, maxDelayMillis=2600, delayMillis=1141, ElectionScheduler{attempt=26, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:41:19,071][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=26, maxDelayMillis=2700, delayMillis=958, ElectionScheduler{attempt=27, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:41:19,071][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:41:19,072][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} added PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:41:19,620][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:20,029][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=26, maxDelayMillis=2700, delayMillis=958, ElectionScheduler{attempt=27, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:41:20,029][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=27, maxDelayMillis=2800, delayMillis=2044, ElectionScheduler{attempt=28, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:41:20,029][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:41:20,030][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} added PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:41:20,589][WARN ][o.e.c.c.ClusterFormationFailureHelper] [hdp13] master not discovered or elected yet, an election requires at least 4 nodes with ids from [3-dBP8WZQZO-Rymo0vC3mQ, EYrLIFzETzSwcUvvabPegg, jguIm07bTXC_8ot6fYN9bw, 1cT_ZkvzTHaSyiqDEi2GSw, 3BP0A2-JTTS-Vp_1YdnJrw, vp_SiTAYQyCE1TQVIUXx2A, bkVjp2mNSk6G0PH9QFBu9Q], have discovered [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [192.168.1.214:9300, 192.168.1.215:9300, 192.168.1.216:9300, 192.168.1.223:9302, 192.168.1.224:9302, 192.168.1.225:9302, 192.168.1.226:9302] from hosts providers and [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 5287, last-accepted version 152 in term 5161
[2019-09-03T15:41:20,597][WARN ][o.e.n.Node ] [hdp13] timed out while waiting for initial discovery state - timeout: 30s
[2019-09-03T15:41:20,606][INFO ][o.e.h.AbstractHttpServerTransport] [hdp13] publish_address {192.168.1.213:9200}, bound_addresses {192.168.1.213:9200}
[2019-09-03T15:41:20,606][INFO ][o.e.n.Node ] [hdp13] started
[2019-09-03T15:41:20,621][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:21,622][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:22,073][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=27, maxDelayMillis=2800, delayMillis=2044, ElectionScheduler{attempt=28, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:41:22,074][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=28, maxDelayMillis=2900, delayMillis=2708, ElectionScheduler{attempt=29, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:41:22,074][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:41:22,075][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} added PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:41:22,623][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:22,971][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:22,970][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:22,970][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5286} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:22,970][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:22,970][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:22,975][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:22,975][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:22,980][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:22,981][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:22,980][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:22,983][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:22,985][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:22,986][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:22,985][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:22,986][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:22,986][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:22,989][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:22,989][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:22,990][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:22,990][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:23,624][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:24,626][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:24,782][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=28, maxDelayMillis=2900, delayMillis=2708, ElectionScheduler{attempt=29, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:41:24,782][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=29, maxDelayMillis=3000, delayMillis=1743, ElectionScheduler{attempt=30, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:41:24,783][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:41:24,784][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} added PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:41:24,784][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:25,626][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:26,526][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=29, maxDelayMillis=3000, delayMillis=1743, ElectionScheduler{attempt=30, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:41:26,526][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=30, maxDelayMillis=3100, delayMillis=1006, ElectionScheduler{attempt=31, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:41:26,526][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:41:26,527][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} added PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:41:26,527][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:26,627][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:27,532][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=30, maxDelayMillis=3100, delayMillis=1006, ElectionScheduler{attempt=31, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:41:27,532][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=31, maxDelayMillis=3200, delayMillis=2601, ElectionScheduler{attempt=32, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:41:27,533][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:41:27,533][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} added PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:41:27,534][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:27,628][DEBUG][o.e.c.c.JoinHelper ] [hdp13] already attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with request JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}, not sending request
[2019-09-03T15:41:28,544][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:28,544][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:41:28,544][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:28,544][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:28,544][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:28,544][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:28,544][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:28,544][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:28,544][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:28,544][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:28,544][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:28,544][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:28,544][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:28,544][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:28,545][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:28,545][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:28,545][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:28,545][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:28,545][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:28,545][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:28,545][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:28,545][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:29,980][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:29,980][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:29,984][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:29,980][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:29,980][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:29,980][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:29,984][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:29,980][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:29,986][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:29,987][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:29,988][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:29,989][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:29,990][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:29,990][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:29,990][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:29,991][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:29,991][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:29,993][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:29,993][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:29,994][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:29,994][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:29,994][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,134][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=31, maxDelayMillis=3200, delayMillis=2601, ElectionScheduler{attempt=32, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:41:30,134][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=32, maxDelayMillis=3300, delayMillis=1752, ElectionScheduler{attempt=33, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:41:30,134][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:41:30,135][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} added PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:41:30,136][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=false} ignoring PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:41:30,136][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,136][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,590][WARN ][o.e.c.c.ClusterFormationFailureHelper] [hdp13] master not discovered or elected yet, an election requires at least 4 nodes with ids from [3-dBP8WZQZO-Rymo0vC3mQ, EYrLIFzETzSwcUvvabPegg, jguIm07bTXC_8ot6fYN9bw, 1cT_ZkvzTHaSyiqDEi2GSw, 3BP0A2-JTTS-Vp_1YdnJrw, vp_SiTAYQyCE1TQVIUXx2A, bkVjp2mNSk6G0PH9QFBu9Q], have discovered [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [192.168.1.214:9300, 192.168.1.215:9300, 192.168.1.216:9300, 192.168.1.223:9302, 192.168.1.224:9302, 192.168.1.225:9302, 192.168.1.226:9302] from hosts providers and [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 5287, last-accepted version 152 in term 5161
[2019-09-03T15:41:30,650][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5288
[2019-09-03T15:41:30,653][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5287] due to StartJoinRequest{term=5288,node={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:41:30,651][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,651][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,651][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,651][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,650][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,650][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,650][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,650][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,650][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,650][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,650][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,650][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,650][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,650][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,650][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,650][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,650][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,651][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,650][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,650][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,651][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,650][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:30,650][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5287} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:31,041][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5288, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader=null
[2019-09-03T15:41:31,041][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5288, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:41:31,113][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5289
[2019-09-03T15:41:31,114][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5288] due to StartJoinRequest{term=5289,node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:41:31,114][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14][192.168.1.214:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:452) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:518) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:31,115][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14][192.168.1.214:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:452) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:518) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:31,516][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5289, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader=null
[2019-09-03T15:41:31,516][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5290
[2019-09-03T15:41:31,516][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5289, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:41:31,516][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5289] due to StartJoinRequest{term=5290,node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:41:31,745][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5289, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5289 does not match current term 5290
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:31,746][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5289, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5289 does not match current term 5290
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:31,833][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1=null, v2=PreVoteResponse{currentTerm=5289, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp16b][192.168.1.226:9302][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5286} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:31,886][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=32, maxDelayMillis=3300, delayMillis=1752, ElectionScheduler{attempt=33, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:41:31,886][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=33, maxDelayMillis=3400, delayMillis=978, ElectionScheduler{attempt=34, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:41:31,892][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader=null
[2019-09-03T15:41:31,892][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5290, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:41:31,892][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5290}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:41:31,893][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5290}, isClosed=false} added PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5161, lastAcceptedVersion=152} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:41:31,893][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5290}, isClosed=false} ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:41:31,893][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5290}, isClosed=false} ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:41:31,893][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5290}, isClosed=false} ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:41:32,237][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:32,237][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:32,238][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:32,238][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:32,238][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:32,239][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:32,239][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:32,239][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:32,239][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:32,239][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:32,240][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:32,240][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:32,240][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:32,240][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:32,240][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:32,241][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:32,241][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:32,241][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:32,241][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:32,241][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:32,241][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:32,241][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:32,243][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5290}, isClosed=false} ignoring PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5287, lastAcceptedVersion=154} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} as it is fresher
[2019-09-03T15:41:32,251][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on slow path
[2019-09-03T15:41:32,251][DEBUG][o.e.c.c.Coordinator ] [hdp13] onFollowerCheckRequest: coordinator becoming FOLLOWER of [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] in term 5290 (was CANDIDATE, lastKnownLeader was [Optional.empty])
[2019-09-03T15:41:32,252][TRACE][o.e.c.c.LeaderChecker ] [hdp13] setCurrentNodes: nodes:
[2019-09-03T15:41:32,253][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5161, lastAcceptedVersion=152}, leader={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:41:32,253][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:32,255][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:32,279][DEBUG][o.e.c.c.PublicationTransportHandler] [hdp13] received full cluster state version [155] with size [199467]
[2019-09-03T15:41:32,309][TRACE][o.e.c.c.Coordinator ] [hdp13] handlePublishRequest: handling [PublishRequest{term=5290, version=155, state=cluster uuid: yElPwPBVT-GPUb_lj3zJZg [committed: true]
version: 155
state uuid: KULiumQoQHa5Sdbsg1-g6g
from_diff: false
meta data version: 15559
coordination_metadata:
term: 5290
last_committed_config: VotingConfiguration{3-dBP8WZQZO-Rymo0vC3mQ,EYrLIFzETzSwcUvvabPegg,jguIm07bTXC_8ot6fYN9bw,1cT_ZkvzTHaSyiqDEi2GSw,3BP0A2-JTTS-Vp_1YdnJrw,vp_SiTAYQyCE1TQVIUXx2A,bkVjp2mNSk6G0PH9QFBu9Q}
last_accepted_config: VotingConfiguration{3-dBP8WZQZO-Rymo0vC3mQ,EYrLIFzETzSwcUvvabPegg,jguIm07bTXC_8ot6fYN9bw,1cT_ZkvzTHaSyiqDEi2GSw,3BP0A2-JTTS-Vp_1YdnJrw,vp_SiTAYQyCE1TQVIUXx2A,bkVjp2mNSk6G0PH9QFBu9Q}
voting tombstones: []
[shodan_ssl-2019.07/pLRXI7egRxOyE7g1XA-cng]: v[234], mv[113], sv[1], av[1]
0: p_term [17], isa_ids [CfaW5L3CRKyoOqL635wtDw]
1: p_term [16], isa_ids [8Mcr93H1RRCHY5T8Nc_qbQ]
2: p_term [15], isa_ids [2nOC5sLATCyedGGfRWUGWw]
3: p_term [12], isa_ids [xhNrrzriQ5OADAgA1i6Urw]
4: p_term [19], isa_ids [bNwMZ317TMWtCcN1wYNBpw]
5: p_term [18], isa_ids [Ci13iju1QA-MKPHL-kry2w]
6: p_term [19], isa_ids [j0x78ARmR2qzxhSOlr4Kig]
7: p_term [17], isa_ids [XvtCrVq5TnKKm3HHiMZ37Q]
8: p_term [17], isa_ids [tXk68xfZQuugvmChniUKOg]
9: p_term [16], isa_ids [zN-90fRDRUOw80QqEV8oJQ]
10: p_term [15], isa_ids [z4mIdy0KSUGmlRgL8-YLtQ]
11: p_term [12], isa_ids [uwQBA4QISmq6MLDi_UU-Eg]
12: p_term [19], isa_ids [yR3fTKfmRvqH5gEHhN3wQw]
13: p_term [18], isa_ids [izvaZHyhQ0itB46J1E-5yg]
14: p_term [19], isa_ids [Rxo14TX0TTK1VxlJ43X83Q]
15: p_term [17], isa_ids [7-KM2MkiS_ebDS-J23MwjQ]
16: p_term [17], isa_ids [uIOFOfZeRsC8RBgLW0rKNw]
17: p_term [17], isa_ids [4JIxLJJwQ2eD3bOb8arM8w]
18: p_term [15], isa_ids [sPqUB5ezQCa7UDursb60cg]
19: p_term [12], isa_ids [Rhqq3BT5Tv-KkzAOYxBlrA]
20: p_term [19], isa_ids [tuOpQ878QYqtYrJGLcR-QA]
21: p_term [18], isa_ids [wp2vcmbbTRCVJ99Uo26ZLw]
22: p_term [19], isa_ids [eUGhsLKhTj6moDE5l1h0qw]
23: p_term [17], isa_ids [_-zSLSryQUeaM6bm4awO_g]
24: p_term [17], isa_ids [F3fl1iJYR-u2Av3j4JiMpw]
25: p_term [17], isa_ids [TmeAxFpXSbKgTx4gZzmRcQ]
26: p_term [15], isa_ids [rOp7_0J1Q6Kxlkh-vcczfg]
27: p_term [12], isa_ids [SxhnWanNSmm_YJQ7o_SFLg]
28: p_term [19], isa_ids [wwYOGp0IS6mfkq-6U5ZFDA]
29: p_term [18], isa_ids [J2Wf8A5mSFWD6xD-LQLwPw]
30: p_term [19], isa_ids [6Ab4H6SYSeuJCT-eGsFNtg]
31: p_term [17], isa_ids [qnLd2CcyTpCqICb8WizxZA]
[new_ssl-2019.01/Y0dv_Rc9TIKO6z4qbAQ8JA]: v[278], mv[1], sv[1], av[1]
0: p_term [19], isa_ids [qYm2qXYlSTCiWli0uYfMvA]
1: p_term [21], isa_ids [_K6qoP0yThaUJzpjuK3GeA]
2: p_term [27], isa_ids [KCHgndezQHSaOZOtBJNNyQ]
3: p_term [27], isa_ids [Y9iuwB-NRQ-6e5lgcQpwBg]
4: p_term [20], isa_ids [NsUpSNsYTsa4dA5dFCIJBQ]
5: p_term [21], isa_ids [im3gw0KOTS6zaozNkvjGZQ]
6: p_term [20], isa_ids [Tqr4keYCSxqtST6FDFzzxw]
7: p_term [19], isa_ids [Oa6uy2Z9RhKhuwixanDsZQ]
8: p_term [19], isa_ids [jqql0ROpTwqBOmD6nYPKNQ]
9: p_term [21], isa_ids [2RKg-r0uT_qM0FiE5gVaBQ]
10: p_term [26], isa_ids [Qp1a0QzuTKyxDo7N97VRRQ]
11: p_term [27], isa_ids [M-61zzUZS2e8n-OB-e-hYg]
12: p_term [20], isa_ids [d73YMsNzSIax2EYWRHD5AA]
13: p_term [21], isa_ids [pDYcOkKvTza2bxzuzSNB_g]
14: p_term [20], isa_ids [I25zP54YS8WTovfEB_hafw]
15: p_term [19], isa_ids [UlS5mZWyTyKUnmbEKOJZCQ]
16: p_term [19], isa_ids [KDd0gAzfQWWsFjWL4eBosQ]
17: p_term [21], isa_ids [Ts1iY3aOQM-rv5wY7bunZw]
18: p_term [26], isa_ids [cBk7cFHbQT6UgLVwcp-FRg]
19: p_term [27], isa_ids [4qK6yV18TrSYZ5jRcQkUiQ]
20: p_term [20], isa_ids [bSbUQTIfTAiFdVelHQ4L-w]
21: p_term [21], isa_ids [Wd-2Wy1lQV2JoZohNZTSNg]
22: p_term [20], isa_ids [0ZgHyqI8SymhdmhxDMmEfA]
23: p_term [19], isa_ids [-KlJFCP2RXyuTWszfvERMg]
24: p_term [19], isa_ids [bg-HZ-39QIqaALzDvUxyRA]
25: p_term [21], isa_ids [wozO-z4IQZiPYvmirr_vmQ]
26: p_term [26], isa_ids [vEfHoAyuTPix4crY_AezNQ]
27: p_term [27], isa_ids [whkRjpJVTXO_RnyuIuQEcA]
28: p_term [20], isa_ids [wFqz5SYbT1SbnhyVkzP2TA]
29: p_term [21], isa_ids [fZYF8ADAT3Gq3DE7SgJMVQ]
30: p_term [20], isa_ids [rfYxdcOWTRGupa_8cbA9dA]
31: p_term [19], isa_ids [GFNDRgr0TPuClK0gh97EHA]
32: p_term [19], isa_ids [EpNAQcHPQamBs8S4lAZwVg]
33: p_term [21], isa_ids [An5a9ZIkTNSBnLRgnkJGGg]
34: p_term [26], isa_ids [qUmVpbiqSgmj7sKDmwJd9g]
35: p_term [27], isa_ids [FdJMk4bEQMORFinPxdbdsA]
36: p_term [20], isa_ids [LrlDPl3eTnSBhk8f74Dvqg]
37: p_term [21], isa_ids [anXhubx-S6uc8qEvH3gW8A]
38: p_term [20], isa_ids [aWosPlE_ROCLDiV1ds9ttA]
39: p_term [19], isa_ids [Pgof8mcJT9eKi9qKR4EonA]
40: p_term [19], isa_ids [nJnnPK7-R52IGM6lhsHe4A]
41: p_term [21], isa_ids [5jToPKakSHeFsCt7VXkCww]
42: p_term [26], isa_ids [Hz763Ro2T3-jC1tbSh8dcQ]
43: p_term [27], isa_ids [EdKNWai_T-WDcYKY1EyE5Q]
44: p_term [20], isa_ids [HpsNL9s4T6uJmoxxOzRePw]
45: p_term [21], isa_ids [7QfOhN5gQ0a2JmzQfb-Xaw]
46: p_term [20], isa_ids [YpgUirdNRW2EWaNtDgM43g]
47: p_term [19], isa_ids [D4hXR4zETYO9dgpSgF3c4Q]
[new_ssl-2018.05/o4jLqUw_TR2-po7F0JYoTw]: v[342], mv[1], sv[1], av[1]
0: p_term [19], isa_ids [TQWWZ20xSmSGPE-hQHE4Gg]
1: p_term [23], isa_ids [ZtEMqa5zTWyjtxHRHEZzCQ]
2: p_term [19], isa_ids [NkIQOPyxRkWuiYQLQFHUBQ]
3: p_term [20], isa_ids [auek9PlCTXOrKy0AcK23Ew]
4: p_term [23], isa_ids [JupFG2DcR12V0hkOelnvjA]
5: p_term [23], isa_ids [Tl1WEt_LSCe0zWZ4hsjarA]
6: p_term [26], isa_ids [llv6iZbuT5KPktKnfqf_Gw]
7: p_term [24], isa_ids [qbWfmRWrQL-AybW4igro6Q]
8: p_term [19], isa_ids [TcrTQLgiTc6wgZsRHtaDlQ]
9: p_term [23], isa_ids [ypgN47FITqSJHixcyCRbsg]
10: p_term [19], isa_ids [-WpmnG-sTCOBrIh8Ts7Y0Q]
11: p_term [20], isa_ids [6GZ0GGOqQ5SJMjtUjDHEBw]
12: p_term [23], isa_ids [syZSQGkZQgi1GbsbBNjYTg]
13: p_term [23], isa_ids [PhT-Kun1QrmZPNFqb8V2tw]
14: p_term [26], isa_ids [aHTsuketRkK27QmUECPfyA]
15: p_term [24], isa_ids [7WwF88vMSCahEaW0hh0OVg]
16: p_term [19], isa_ids [vsXUJRicRZahg-z__FgCGw]
17: p_term [23], isa_ids [WOvfnLvnS6y-f_XLeTDj_Q]
18: p_term [19], isa_ids [9AZTO0HpROWACzRSyb0H_Q]
19: p_term [20], isa_ids [D0GrP5WiT4emKhVQhECX4Q]
20: p_term [24], isa_ids [dVqmLPGJSBGHHRYWzN3_Eg]
21: p_term [23], isa_ids [RFIMWNgHSECOZUOkYlX9Dw]
22: p_term [26], isa_ids [qU00C8qeR6K3OWUrg8Paig]
23: p_term [24], isa_ids [KfG5if82SwihdfpN5ye_AA]
24: p_term [19], isa_ids [lkYlTByZTfu-VntuUX0Ehg]
25: p_term [23], isa_ids [xD-_uRtNSb27slXR-bMJ8g]
26: p_term [19], isa_ids [8-oQHDAFRGWCrrwHakej0Q]
27: p_term [20], isa_ids [gxEP4yfiSni9vJ6UEbvYdA]
28: p_term [23], isa_ids [1tMf5fPHQJSdwagfRMHd7g]
29: p_term [23], isa_ids [O1Bm42LKQ-aUpeH6ApCnfQ]
30: p_term [26], isa_ids [OiGgOnafRf6_l2F5XZNvFg]
31: p_term [24], isa_ids [RwWYQzqeQo2YnqA7fjbAlQ]
32: p_term [19], isa_ids [rvt5kNKLQye6yLg6LlR5ww]
33: p_term [23], isa_ids [XVGELT9kRqaBUgYgM4KIww]
34: p_term [19], isa_ids [e3VCcAb3Rx2qusF37S6AhA]
35: p_term [20], isa_ids [bXjorUKsSKy3bIrvcAJuyg]
36: p_term [23], isa_ids [cJtDDJjkTu2t1ckNXCS4HQ]
37: p_term [23], isa_ids [_OVGaCkYQsOd1W8bA-M6UQ]
38: p_term [26], isa_ids [i2HPjbY2QwOGL1deUH57dg]
39: p_term [24], isa_ids [ngRjTc8HSc2leQBuDV-pxw]
40: p_term [19], isa_ids [oo-CdMK5RJGcMv8mJDNCnw]
41: p_term [23], isa_ids [A7Iai_1wTzWGnlZF3KOz_g]
42: p_term [19], isa_ids [w2JSEfXXRLCylpOmGZIYdA]
43: p_term [20], isa_ids [bloaGnddQ5G8xAapN_YxtA]
44: p_term [23], isa_ids [AKrTvhGMT0Wa6PtANoLQSw]
45: p_term [23], isa_ids [a6Nl8XuTSk2rLtqh09E5YQ]
46: p_term [26], isa_ids [DiVzkWS6RwiSqxFdk2hNzg]
47: p_term [24], isa_ids [ZroG3A-nTduiH6gOmujvEg]
[ssh-2018.12/LwQsi9YdTzq8hNbdUYXCxw]: v[194], mv[18], sv[1], av[1]
0: p_term [18], isa_ids [ssBg1DjBSbmUTp_RkpZBZw]
1: p_term [21], isa_ids [_XvzeundRXyDVpAUTv11ZA]
2: p_term [19], isa_ids [8jK4S__iQlawsIn_FK-c5Q]
3: p_term [26], isa_ids [5YUbolOqSIih-Cd0AqGxAA]
4: p_term [19], isa_ids [Mc4pEatLTDyk0jPF6UJeAA]
5: p_term [28], isa_ids [WA3Zn2DYT92hGsjtfCiB1Q]
6: p_term [21], isa_ids [du5vOvghSFKV-v1k7cKtWQ]
7: p_term [20], isa_ids [pyZbbMF0RxyIc0wZxQFbig]
8: p_term [18], isa_ids [-QwcvJtTT0mB5IVTAfIwvg]
9: p_term [21], isa_ids [wgQpdq7mTmy7rRGcKkPe8g]
10: p_term [18], isa_ids [1RTVBGeuQjKele9D99A51w]
11: p_term [26], isa_ids [Cs5OmpHFRv-5ug4s-f_vRQ]
12: p_term [19], isa_ids [1fynZODoTe2FqCl9pXNW5Q]
13: p_term [28], isa_ids [FZ3Z41LTStK5DvyMzpd8CQ]
14: p_term [21], isa_ids [_gSxYUPbRPyKnY-Z9v5mRw]
15: p_term [20], isa_ids [zWjiOxytQguFhvDu21kNYw]
[new_ssl-2018.08/LVDmXE4DSoezlZV1seOOXQ]: v[311], mv[1], sv[1], av[1]
0: p_term [20], isa_ids [b79ST7G3S6OXZfO7A2ZKNQ]
1: p_term [19], isa_ids [M8mogPCPTIqwVy4gWBEHRQ]
2: p_term [24], isa_ids [Caux2LPsT7u9U07MefqP9g]
3: p_term [21], isa_ids [EhSAFAuUQwG4AdNefXghYw]
4: p_term [18], isa_ids [47RpcQflRG6Y1FoyO1E2YQ]
5: p_term [16], isa_ids [1dT6SF5BRSeCcwyfisYsiA]
6: p_term [32], isa_ids [zVc9GD-yRtiOZXqZcowRZA]
7: p_term [20], isa_ids [RJ5ElAxXQ_mwawk96KxQmw]
8: p_term [20], isa_ids [e92v3p3dQIedERNaKoO4vg]
9: p_term [19], isa_ids [9J-ho8ppSru-tFH2SrV06Q]
10: p_term [24], isa_ids [8JDVIFzlSHePnvEHGdC7Iw]
11: p_term [21], isa_ids [0SeX6in3Q-SfHtEMXZil1Q]
12: p_term [18], isa_ids [cmanVzL3S4idFazKRrHRsA]
13: p_term [16], isa_ids [o-tqLZ1XRTWfi88e8mbIuQ]
14: p_term [32], isa_ids [ZTcQQ7cVTRigVPWs6STEJA]
15: p_term [21], isa_ids [b4WMBfv6SIa9IxlV7vGdFg]
16:
[2019-09-03T15:41:32,310][TRACE][o.e.c.c.CoordinationState] [hdp13] handlePublishRequest: accepting publish request for version [155] and term [5290]
[2019-09-03T15:41:32,865][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=33, maxDelayMillis=3400, delayMillis=978, ElectionScheduler{attempt=34, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} not starting election
[2019-09-03T15:41:32,899][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, v2=PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp15][192.168.1.215:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5286} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:41:33,255][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:33,256][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:33,256][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:34,256][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:34,257][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:34,257][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:35,257][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:35,257][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:35,258][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:36,258][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:36,258][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:36,312][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:37,259][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:37,313][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:37,314][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:38,259][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:38,314][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:38,315][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:39,260][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:39,315][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:39,316][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:40,261][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:40,316][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:40,317][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:41,262][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:41,317][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:41,318][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:42,263][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:42,318][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:42,319][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:43,264][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:43,319][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:43,320][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:44,265][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:44,320][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:44,321][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:45,266][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:45,321][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:45,321][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:46,266][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:46,322][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:46,322][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:47,267][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:47,323][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:47,324][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:48,268][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:48,324][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:48,325][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:49,269][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:49,325][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:49,326][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:50,270][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:50,326][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:50,327][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:51,270][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:51,327][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:51,328][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:52,271][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:52,328][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:52,329][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:53,272][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:53,329][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:53,330][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:54,273][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:54,330][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:54,331][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:55,273][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:55,331][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:55,332][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:56,274][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:56,332][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:56,333][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:57,275][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:57,333][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:57,334][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:58,276][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:58,334][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:58,335][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:41:59,276][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:41:59,335][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:41:59,336][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:42:00,277][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:42:00,336][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:42:00,337][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:42:01,278][TRACE][o.e.c.c.FollowersChecker ] [hdp13] responding to FollowerCheckRequest{term=5290, sender={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} on fast path
[2019-09-03T15:42:01,337][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:42:01,338][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:42:02,253][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5290, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed
at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$3.onFailure(Coordinator.java:1378) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:101) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:54) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1302) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:124) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Publication.cancel(Publication.java:88) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$2.run(Coordinator.java:1269) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: org.elasticsearch.ElasticsearchException: publication cancelled before committing: timed out after 30s
at org.elasticsearch.cluster.coordination.Publication.cancel(Publication.java:85) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$2.run(Coordinator.java:1269) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) ~[?:?]
[2019-09-03T15:42:02,257][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5290, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed
at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$3.onFailure(Coordinator.java:1378) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:101) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:54) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1302) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:124) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Publication.cancel(Publication.java:88) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$2.run(Coordinator.java:1269) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: org.elasticsearch.ElasticsearchException: publication cancelled before committing: timed out after 30s
at org.elasticsearch.cluster.coordination.Publication.cancel(Publication.java:85) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$2.run(Coordinator.java:1269) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) ~[?:?]
[2019-09-03T15:42:02,388][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:42:02,389][DEBUG][o.e.c.c.LeaderChecker ] [hdp13] 1 consecutive failures (limit [cluster.fault_detection.leader_check.retry_count] is 3) with leader [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: non-leader rejecting leader check
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:178) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:105) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:188) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:121) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:105) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:660) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62) [transport-netty4-client-7.3.1.jar:7.3.1]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [netty-codec-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.36.Final.jar:4.1.36.Final]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:02,390][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:42:03,390][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:42:03,391][DEBUG][o.e.c.c.LeaderChecker ] [hdp13] 2 consecutive failures (limit [cluster.fault_detection.leader_check.retry_count] is 3) with leader [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: non-leader rejecting leader check
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:178) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:105) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:188) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:121) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:105) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:660) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62) [transport-netty4-client-7.3.1.jar:7.3.1]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [netty-codec-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.36.Final.jar:4.1.36.Final]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:03,392][TRACE][o.e.c.c.LeaderChecker ] [hdp13] scheduling next check of {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} for [cluster.fault_detection.leader_check.interval] = 1s
[2019-09-03T15:42:04,392][TRACE][o.e.c.c.LeaderChecker ] [hdp13] checking {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with [cluster.fault_detection.leader_check.timeout] = 10s
[2019-09-03T15:42:04,393][DEBUG][o.e.c.c.LeaderChecker ] [hdp13] leader [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] has failed 3 consecutive checks (limit [cluster.fault_detection.leader_check.retry_count] is 3); last failure was:
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: non-leader rejecting leader check
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:178) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:105) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:188) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:121) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:105) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:660) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62) [transport-netty4-client-7.3.1.jar:7.3.1]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [netty-codec-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.36.Final.jar:4.1.36.Final]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:05,156][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] TransportResponseHandler{PreVoteCollector{state=Tuple [v1={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, v2=PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5161, lastAcceptedVersion=152}]}, node={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} failed
org.elasticsearch.transport.RemoteTransportException: [hdp14][192.168.1.214:9300][internal:cluster/request_pre_vote]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, currentTerm=5290} as there is already a leader
at org.elasticsearch.cluster.coordination.PreVoteCollector.handlePreVoteRequest(PreVoteCollector.java:123) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PreVoteCollector.lambda$new$0(PreVoteCollector.java:69) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:05,156][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5288, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14][192.168.1.214:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: became follower
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.lambda$close$3(JoinHelper.java:477) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.HashMap$Values.forEach(HashMap.java:976) ~[?:?]
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.close(JoinHelper.java:477) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.becomeFollower(Coordinator.java:606) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handlePublishRequest(Coordinator.java:328) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PublicationTransportHandler.acceptState(PublicationTransportHandler.java:452) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PublicationTransportHandler.handleIncomingPublishRequest(PublicationTransportHandler.java:408) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PublicationTransportHandler.lambda$new$0(PublicationTransportHandler.java:100) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:05,157][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5288, lastAcceptedTerm=5161, lastAcceptedVersion=152, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14][192.168.1.214:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: became follower
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.lambda$close$3(JoinHelper.java:477) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.HashMap$Values.forEach(HashMap.java:976) ~[?:?]
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.close(JoinHelper.java:477) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.becomeFollower(Coordinator.java:606) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handlePublishRequest(Coordinator.java:328) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PublicationTransportHandler.acceptState(PublicationTransportHandler.java:452) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PublicationTransportHandler.handleIncomingPublishRequest(PublicationTransportHandler.java:408) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.PublicationTransportHandler.lambda$new$0(PublicationTransportHandler.java:100) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:08,779][TRACE][o.e.c.c.Coordinator ] [hdp13] handlePublishRequest: coordinator remaining FOLLOWER of [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] in term 5290
[2019-09-03T15:42:08,780][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5290, lastAcceptedVersion=155}, leader={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:08,780][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5294
[2019-09-03T15:42:08,780][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5290] due to StartJoinRequest{term=5294,node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:42:09,356][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: coordinator becoming CANDIDATE in term 5294 (was FOLLOWER, lastKnownLeader was [Optional[{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]])
[2019-09-03T15:42:09,365][TRACE][o.e.c.c.LeaderChecker ] [hdp13] setCurrentNodes: nodes:
[2019-09-03T15:42:09,365][TRACE][o.e.c.c.LeaderChecker ] [hdp13] already closed, doing nothing
[2019-09-03T15:42:09,365][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5294, lastAcceptedTerm=5290, lastAcceptedVersion=155}, leader=null
[2019-09-03T15:42:09,366][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5294, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:42:09,366][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=0s, thisAttempt=0, maxDelayMillis=100, delayMillis=3, ElectionScheduler{attempt=1, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:42:09,367][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5293
[2019-09-03T15:42:09,367][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5293,node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5294]
[2019-09-03T15:42:09,367][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5292
[2019-09-03T15:42:09,367][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5292,node={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5294]
[2019-09-03T15:42:09,367][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5292
[2019-09-03T15:42:09,367][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5292,node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5294]
[2019-09-03T15:42:09,367][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5291
[2019-09-03T15:42:09,367][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5291,node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5294]
[2019-09-03T15:42:09,368][DEBUG][o.e.c.c.Coordinator ] [hdp13] onLeaderFailure: coordinator becoming CANDIDATE in term 5294 (was CANDIDATE, lastKnownLeader was [Optional[{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]])
[2019-09-03T15:42:09,368][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5294, lastAcceptedTerm=5290, lastAcceptedVersion=155}, leader=null
[2019-09-03T15:42:09,368][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5295
[2019-09-03T15:42:09,368][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5294] due to StartJoinRequest{term=5295,node={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:42:09,370][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=0s, thisAttempt=0, maxDelayMillis=100, delayMillis=3, ElectionScheduler{attempt=1, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:42:09,370][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=1, maxDelayMillis=200, delayMillis=665, ElectionScheduler{attempt=2, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:42:09,491][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5290}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5294, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:09,623][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5294, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5294 does not match current term 5296
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:09,624][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5294, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5294 does not match current term 5296
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:09,672][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5295, lastAcceptedTerm=5290, lastAcceptedVersion=155}, leader=null
[2019-09-03T15:42:09,672][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5295, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:42:09,676][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5296
[2019-09-03T15:42:09,677][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5295] due to StartJoinRequest{term=5296,node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:42:09,968][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5295, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14][192.168.1.214:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5295 does not match current term 5297
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:09,968][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5295, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14][192.168.1.214:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5295 does not match current term 5297
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:10,035][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=1, maxDelayMillis=200, delayMillis=665, ElectionScheduler{attempt=2, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:42:10,035][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=2, maxDelayMillis=300, delayMillis=777, ElectionScheduler{attempt=3, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:42:10,098][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5296, lastAcceptedTerm=5290, lastAcceptedVersion=155}, leader=null
[2019-09-03T15:42:10,098][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5296, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:42:10,098][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5296
[2019-09-03T15:42:10,098][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5296,node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5296]
[2019-09-03T15:42:10,098][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5295
[2019-09-03T15:42:10,098][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5295,node={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5296]
[2019-09-03T15:42:10,099][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5298
[2019-09-03T15:42:10,099][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5296] due to StartJoinRequest{term=5298,node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:42:10,158][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5296, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,160][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5290, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5290}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,160][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,160][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,160][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,160][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,160][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,160][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,160][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,160][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,160][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,160][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,160][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,160][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,160][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,160][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,160][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,160][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,161][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,160][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,160][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,160][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,160][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,161][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,160][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5297, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,291][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5296, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5296 does not match current term 5298
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:10,292][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5296, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5296 does not match current term 5298
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:10,456][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5298, lastAcceptedTerm=5290, lastAcceptedVersion=155}, leader=null
[2019-09-03T15:42:10,456][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5298, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:42:10,457][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5298}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:42:10,457][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5297
[2019-09-03T15:42:10,457][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5297,node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5298]
[2019-09-03T15:42:10,459][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5298}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:42:10,460][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5299
[2019-09-03T15:42:10,460][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5298}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5298, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}
[2019-09-03T15:42:10,460][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5298] due to StartJoinRequest{term=5299,node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:42:10,562][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5298}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5298, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,615][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5298}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5299, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,813][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=2, maxDelayMillis=300, delayMillis=777, ElectionScheduler{attempt=3, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:42:10,813][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=3, maxDelayMillis=400, delayMillis=776, ElectionScheduler{attempt=4, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:42:10,826][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5299, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,857][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5299, lastAcceptedTerm=5290, lastAcceptedVersion=155}, leader=null
[2019-09-03T15:42:10,857][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5299, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:42:10,857][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5299}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:42:10,858][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5300
[2019-09-03T15:42:10,858][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5299] due to StartJoinRequest{term=5300,node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:42:10,951][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5298}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,951][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5298}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,965][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5298}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:10,965][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5298}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:11,008][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5298}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:11,008][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5298}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:11,008][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5298, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5298 does not match current term 5300
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:11,009][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5298, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5298 does not match current term 5300
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:11,038][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5298}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:11,038][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5298}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:11,159][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:11,266][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155}, leader=null
[2019-09-03T15:42:11,266][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:42:11,266][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5302
[2019-09-03T15:42:11,266][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5299, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5298, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5298, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5298}, isClosed=true} added PreVoteResponse{currentTerm=5299, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, no quorum yet
[2019-09-03T15:42:11,266][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5299, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5298, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5298, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5298}, isClosed=true} added PreVoteResponse{currentTerm=5298, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:42:11,266][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5299, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5298, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5298, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5298}, isClosed=true} added PreVoteResponse{currentTerm=5298, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, no quorum yet
[2019-09-03T15:42:11,266][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5300] due to StartJoinRequest{term=5302,node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:42:11,333][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5299, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5299 does not match current term 5301
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:11,333][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5299, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5299 does not match current term 5301
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:11,493][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5301, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:11,509][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5300 does not match current term 5302
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:11,509][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5300 does not match current term 5302
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:11,589][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=3, maxDelayMillis=400, delayMillis=776, ElectionScheduler{attempt=4, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:42:11,589][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=4, maxDelayMillis=500, delayMillis=773, ElectionScheduler{attempt=5, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:42:11,600][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, leader=null
[2019-09-03T15:42:11,600][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5301, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5299}, isClosed=false} added PreVoteResponse{currentTerm=5301, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, no quorum yet
[2019-09-03T15:42:11,600][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:42:11,600][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5301
[2019-09-03T15:42:11,600][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5301,node={hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5302]
[2019-09-03T15:42:11,600][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5301
[2019-09-03T15:42:11,600][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5301,node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5302]
[2019-09-03T15:42:11,601][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5301
[2019-09-03T15:42:11,601][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5301,node={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5302]
[2019-09-03T15:42:11,601][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5301, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5299}, isClosed=true} added PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, no quorum yet
[2019-09-03T15:42:11,601][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5301, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5299}, isClosed=true} added PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, no quorum yet
[2019-09-03T15:42:11,601][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5301, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5299}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}
[2019-09-03T15:42:11,601][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5302}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:42:11,602][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5301, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5301, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5301, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=true, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5299}, isClosed=true} added PreVoteResponse{currentTerm=5301, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, starting election
[2019-09-03T15:42:11,602][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5302}, isClosed=false} added PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:42:11,602][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5302}, isClosed=false} added PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, no quorum yet
[2019-09-03T15:42:11,602][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5302}, isClosed=false} added PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, no quorum yet
[2019-09-03T15:42:11,602][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5301, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5301, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5301, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=true, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5299}, isClosed=true} added PreVoteResponse{currentTerm=5301, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} but election has already started
[2019-09-03T15:42:11,602][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5302}, isClosed=false} added PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, no quorum yet
[2019-09-03T15:42:11,602][DEBUG][o.e.c.c.Coordinator ] [hdp13] starting election with StartJoinRequest{term=5303,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}}
[2019-09-03T15:42:11,604][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}] with term 5303
[2019-09-03T15:42:11,604][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5302] due to StartJoinRequest{term=5303,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}}
[2019-09-03T15:42:11,759][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5301, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5301, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5301, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=true, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5299}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:11,844][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:11,859][DEBUG][o.e.c.c.JoinHelper ] [hdp13] successful response to StartJoinRequest{term=5303,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:11,860][TRACE][o.e.c.c.Coordinator ] [hdp13] handleJoinRequest: as CANDIDATE, handling JoinRequest{sourceNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, optionalJoin=Optional[Join{term=5303, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:42:11,899][TRACE][o.e.c.c.Coordinator ] [hdp13] handleJoinRequest: as CANDIDATE, handling JoinRequest{sourceNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, optionalJoin=Optional[Join{term=5303, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:42:11,899][DEBUG][o.e.c.c.JoinHelper ] [hdp13] successful response to StartJoinRequest{term=5303,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:11,903][DEBUG][o.e.c.c.JoinHelper ] [hdp13] successful response to StartJoinRequest{term=5303,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:11,903][TRACE][o.e.c.c.Coordinator ] [hdp13] handleJoinRequest: as CANDIDATE, handling JoinRequest{sourceNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, optionalJoin=Optional[Join{term=5303, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:42:11,906][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failure in response to StartJoinRequest{term=5303,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
org.elasticsearch.transport.RemoteTransportException: [hdp16b][192.168.1.226:9302][internal:cluster/coordination/start_join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5303 not greater than current term 5303
at org.elasticsearch.cluster.coordination.CoordinationState.handleStartJoin(CoordinationState.java:179) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:444) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:136) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:11,916][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5303, lastAcceptedTerm=5290, lastAcceptedVersion=155}, leader=null
[2019-09-03T15:42:11,917][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5303, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=true, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5302}, isClosed=false} added PreVoteResponse{currentTerm=5303, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, starting election
[2019-09-03T15:42:11,917][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5303, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}}]}
[2019-09-03T15:42:11,917][DEBUG][o.e.c.c.JoinHelper ] [hdp13] successful response to StartJoinRequest{term=5303,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}
[2019-09-03T15:42:11,917][TRACE][o.e.c.c.Coordinator ] [hdp13] handleJoinRequest: as CANDIDATE, handling JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5303, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}}]}
[2019-09-03T15:42:11,919][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleJoin: added join Join{term=5303, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} from [{hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] for election, electionWon=false lastAcceptedTerm=5290 lastAcceptedVersion=155
[2019-09-03T15:42:11,919][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleJoin: added join Join{term=5303, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}} from [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] for election, electionWon=false lastAcceptedTerm=5290 lastAcceptedVersion=155
[2019-09-03T15:42:11,919][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5304
[2019-09-03T15:42:11,920][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5303] due to StartJoinRequest{term=5304,node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:42:11,920][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: discarding VoteCollection{votes=[1cT_ZkvzTHaSyiqDEi2GSw, 3BP0A2-JTTS-Vp_1YdnJrw], joins=[Join{term=5303, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}, Join{term=5303, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}: failed election
[2019-09-03T15:42:11,975][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failure in response to StartJoinRequest{term=5303,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/coordination/start_join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5303 not greater than current term 5303
at org.elasticsearch.cluster.coordination.CoordinationState.handleStartJoin(CoordinationState.java:179) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:444) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:136) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:12,084][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failure in response to StartJoinRequest{term=5303,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
org.elasticsearch.transport.RemoteTransportException: [hdp15b][192.168.1.225:9302][internal:cluster/coordination/start_join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5303 not greater than current term 5303
at org.elasticsearch.cluster.coordination.CoordinationState.handleStartJoin(CoordinationState.java:179) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:444) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:136) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:12,177][DEBUG][o.e.c.c.JoinHelper ] [hdp13] successful response to StartJoinRequest{term=5303,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:12,178][TRACE][o.e.c.c.Coordinator ] [hdp13] handleJoinRequest: as CANDIDATE, handling JoinRequest{sourceNode={hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, optionalJoin=Optional[Join{term=5303, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:42:12,217][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5304, lastAcceptedTerm=5290, lastAcceptedVersion=155}, leader=null
[2019-09-03T15:42:12,217][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5304, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:42:12,217][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5303
[2019-09-03T15:42:12,217][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5303, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=true, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5302}, isClosed=false} added PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true} but election has already started
[2019-09-03T15:42:12,217][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5303,node={hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5304]
[2019-09-03T15:42:12,218][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5305
[2019-09-03T15:42:12,218][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5304] due to StartJoinRequest{term=5305,node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:42:12,362][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=4, maxDelayMillis=500, delayMillis=773, ElectionScheduler{attempt=5, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:42:12,363][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=5, maxDelayMillis=600, delayMillis=905, ElectionScheduler{attempt=6, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:42:12,494][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5304, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5304 does not match current term 5305
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:12,494][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5304, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5304 does not match current term 5305
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:12,503][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5304, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:12,542][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5305, lastAcceptedTerm=5290, lastAcceptedVersion=155}, leader=null
[2019-09-03T15:42:12,543][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5305
[2019-09-03T15:42:12,543][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5305, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:42:12,543][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5305,node={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5305]
[2019-09-03T15:42:12,543][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5305
[2019-09-03T15:42:12,543][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5305,node={hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5305]
[2019-09-03T15:42:12,544][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleJoin: ignored join due to term mismatch (expected: [5305], actual: [5303])
[2019-09-03T15:42:12,544][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5304
[2019-09-03T15:42:12,544][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5304,node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5305]
[2019-09-03T15:42:12,544][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5303, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5303, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=true, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5302}, isClosed=false} added PreVoteResponse{currentTerm=5303, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true} but election has already started
[2019-09-03T15:42:12,544][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleJoin: ignored join due to term mismatch (expected: [5305], actual: [5303])
[2019-09-03T15:42:12,544][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5303, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5303, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5303, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=true, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5302}, isClosed=false} added PreVoteResponse{currentTerm=5303, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} but election has already started
[2019-09-03T15:42:12,545][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleJoin: ignored join due to term mismatch (expected: [5305], actual: [5303])
[2019-09-03T15:42:12,545][DEBUG][o.e.c.c.Coordinator ] [hdp13] starting election with StartJoinRequest{term=5306,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}}
[2019-09-03T15:42:12,545][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5303, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13][192.168.1.213:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5303 does not match current term 5305
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:703) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:12,546][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}] with term 5306
[2019-09-03T15:42:12,546][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5305] due to StartJoinRequest{term=5306,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}}
[2019-09-03T15:42:12,546][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5303, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13][192.168.1.213:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5303 does not match current term 5305
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:703) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:12,744][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failure in response to StartJoinRequest{term=5306,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/start_join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5306 not greater than current term 5306
at org.elasticsearch.cluster.coordination.CoordinationState.handleStartJoin(CoordinationState.java:179) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:444) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:136) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:12,771][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failure in response to StartJoinRequest{term=5306,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
org.elasticsearch.transport.RemoteTransportException: [hdp14][192.168.1.214:9300][internal:cluster/coordination/start_join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5306 not greater than current term 5306
at org.elasticsearch.cluster.coordination.CoordinationState.handleStartJoin(CoordinationState.java:179) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:444) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:136) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:12,824][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failure in response to StartJoinRequest{term=5306,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
org.elasticsearch.transport.RemoteTransportException: [hdp16b][192.168.1.226:9302][internal:cluster/coordination/start_join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5306 not greater than current term 5306
at org.elasticsearch.cluster.coordination.CoordinationState.handleStartJoin(CoordinationState.java:179) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:444) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:136) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:12,836][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5305, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:12,836][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5305, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:12,868][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5306, lastAcceptedTerm=5290, lastAcceptedVersion=155}, leader=null
[2019-09-03T15:42:12,868][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5306, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}}]}
[2019-09-03T15:42:12,868][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5306
[2019-09-03T15:42:12,868][DEBUG][o.e.c.c.JoinHelper ] [hdp13] successful response to StartJoinRequest{term=5306,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}
[2019-09-03T15:42:12,868][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5306,node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5306]
[2019-09-03T15:42:12,868][TRACE][o.e.c.c.Coordinator ] [hdp13] handleJoinRequest: as CANDIDATE, handling JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5306, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}}]}
[2019-09-03T15:42:12,869][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleJoin: added join Join{term=5306, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}] for election, electionWon=false lastAcceptedTerm=5290 lastAcceptedVersion=155
[2019-09-03T15:42:12,869][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5306}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:42:12,870][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5307
[2019-09-03T15:42:12,870][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5306] due to StartJoinRequest{term=5307,node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:42:12,870][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: discarding VoteCollection{votes=[SaA74YaMSNim17koypZiFQ], joins=[Join{term=5306, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}}]}: failed election
[2019-09-03T15:42:12,943][DEBUG][o.e.c.c.JoinHelper ] [hdp13] successful response to StartJoinRequest{term=5306,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:12,943][TRACE][o.e.c.c.Coordinator ] [hdp13] handleJoinRequest: as CANDIDATE, handling JoinRequest{sourceNode={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, optionalJoin=Optional[Join{term=5306, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:42:13,102][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failure in response to StartJoinRequest{term=5306,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
org.elasticsearch.transport.RemoteTransportException: [hdp15b][192.168.1.225:9302][internal:cluster/coordination/start_join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5306 not greater than current term 5306
at org.elasticsearch.cluster.coordination.CoordinationState.handleStartJoin(CoordinationState.java:179) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:444) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:136) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:13,179][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5306, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:13,179][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5306, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:13,189][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failure in response to StartJoinRequest{term=5306,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/coordination/start_join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5306 not greater than current term 5307
at org.elasticsearch.cluster.coordination.CoordinationState.handleStartJoin(CoordinationState.java:179) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:444) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:136) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:13,190][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5305, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5305 does not match current term 5307
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:13,190][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5305, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5305 does not match current term 5307
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:13,202][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}, leader=null
[2019-09-03T15:42:13,202][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5307
[2019-09-03T15:42:13,202][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:42:13,202][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5307,node={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5307]
[2019-09-03T15:42:13,202][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5307
[2019-09-03T15:42:13,202][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5307,node={hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5307]
[2019-09-03T15:42:13,202][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}] with term 5306
[2019-09-03T15:42:13,203][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5306,node={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5307]
[2019-09-03T15:42:13,203][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5308
[2019-09-03T15:42:13,203][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5306}, isClosed=false} added PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, no quorum yet
[2019-09-03T15:42:13,203][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5307] due to StartJoinRequest{term=5308,node={hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:42:13,268][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=5, maxDelayMillis=600, delayMillis=905, ElectionScheduler{attempt=6, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:42:13,268][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=6, maxDelayMillis=700, delayMillis=904, ElectionScheduler{attempt=7, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:42:13,511][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5307 does not match current term 5308
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:13,512][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:13,512][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5307 does not match current term 5308
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:13,552][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5308, lastAcceptedTerm=5290, lastAcceptedVersion=155}, leader=null
[2019-09-03T15:42:13,552][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5308, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:42:13,552][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}] with term 5307
[2019-09-03T15:42:13,552][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5306}, isClosed=false} added PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, no quorum yet
[2019-09-03T15:42:13,552][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5307,node={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5308]
[2019-09-03T15:42:13,553][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5307
[2019-09-03T15:42:13,553][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5306, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=true, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5306}, isClosed=false} added PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, no quorum yet
[2019-09-03T15:42:13,553][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5306, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=true, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5306}, isClosed=false} added PreVoteResponse{currentTerm=5306, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, starting election
[2019-09-03T15:42:13,553][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5307,node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5308]
[2019-09-03T15:42:13,553][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleJoin: ignored join due to term mismatch (expected: [5308], actual: [5306])
[2019-09-03T15:42:13,554][DEBUG][o.e.c.c.Coordinator ] [hdp13] starting election with StartJoinRequest{term=5309,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}}
[2019-09-03T15:42:13,554][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5306, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5306, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=true, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5306}, isClosed=false} added PreVoteResponse{currentTerm=5306, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20} but election has already started
[2019-09-03T15:42:13,554][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5309
[2019-09-03T15:42:13,555][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5308] due to StartJoinRequest{term=5309,node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:42:13,823][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failure in response to StartJoinRequest{term=5309,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
org.elasticsearch.transport.RemoteTransportException: [hdp14][192.168.1.214:9300][internal:cluster/coordination/start_join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5309 not greater than current term 5309
at org.elasticsearch.cluster.coordination.CoordinationState.handleStartJoin(CoordinationState.java:179) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:444) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:136) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:13,853][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failure in response to StartJoinRequest{term=5309,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/coordination/start_join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5309 not greater than current term 5309
at org.elasticsearch.cluster.coordination.CoordinationState.handleStartJoin(CoordinationState.java:179) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:444) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:136) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:13,869][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155}, leader=null
[2019-09-03T15:42:13,869][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:42:13,869][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5309
[2019-09-03T15:42:13,869][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5309,node={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5309]
[2019-09-03T15:42:13,869][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5306, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5306, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5308, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=true, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5306}, isClosed=false} added PreVoteResponse{currentTerm=5308, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} but election has already started
[2019-09-03T15:42:13,869][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5306, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5306, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5308, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=true, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5306}, isClosed=true} added PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true} but election has already started
[2019-09-03T15:42:13,869][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5309}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:42:13,870][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}] with term 5309
[2019-09-03T15:42:13,870][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5309,node={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5309]
[2019-09-03T15:42:13,870][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5309
[2019-09-03T15:42:13,870][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5309,node={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5309]
[2019-09-03T15:42:13,870][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}] with term 5309
[2019-09-03T15:42:13,870][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:452) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:518) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:13,870][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5309,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}}] as term provided is not greater than current term [5309]
[2019-09-03T15:42:13,871][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5309}, isClosed=false} added PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, no quorum yet
[2019-09-03T15:42:13,871][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5309}, isClosed=false} added PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, no quorum yet
[2019-09-03T15:42:13,871][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5309}, isClosed=false} added PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, no quorum yet
[2019-09-03T15:42:13,871][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5302, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:452) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:518) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:13,871][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failure in response to StartJoinRequest{term=5309,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}
org.elasticsearch.transport.RemoteTransportException: [hdp13][192.168.1.213:9300][internal:cluster/coordination/start_join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5309 not greater than current term 5309
at org.elasticsearch.cluster.coordination.CoordinationState.handleStartJoin(CoordinationState.java:179) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:444) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:136) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:703) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:13,900][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5308, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5309}, isClosed=false} added PreVoteResponse{currentTerm=5308, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, no quorum yet
[2019-09-03T15:42:13,944][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failure in response to StartJoinRequest{term=5309,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/coordination/start_join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5309 not greater than current term 5309
at org.elasticsearch.cluster.coordination.CoordinationState.handleStartJoin(CoordinationState.java:179) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:444) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:136) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:13,946][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5308, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=true, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5309}, isClosed=false} added PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, starting election
[2019-09-03T15:42:13,946][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5310
[2019-09-03T15:42:13,946][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5309] due to StartJoinRequest{term=5310,node={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:42:14,128][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failure in response to StartJoinRequest{term=5309,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
org.elasticsearch.transport.RemoteTransportException: [hdp15b][192.168.1.225:9302][internal:cluster/coordination/start_join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5309 not greater than current term 5309
at org.elasticsearch.cluster.coordination.CoordinationState.handleStartJoin(CoordinationState.java:179) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:444) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:136) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:14,172][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduleNextElection{gracePeriod=500ms, thisAttempt=6, maxDelayMillis=700, delayMillis=904, ElectionScheduler{attempt=7, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election
[2019-09-03T15:42:14,172][DEBUG][o.e.c.c.ElectionSchedulerFactory] [hdp13] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=7, maxDelayMillis=800, delayMillis=933, ElectionScheduler{attempt=8, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}}
[2019-09-03T15:42:14,196][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:14,196][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:14,197][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:14,197][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:14,196][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failure in response to StartJoinRequest{term=5309,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
org.elasticsearch.transport.RemoteTransportException: [hdp15][192.168.1.215:9300][internal:cluster/coordination/start_join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5309 not greater than current term 5309
at org.elasticsearch.cluster.coordination.CoordinationState.handleStartJoin(CoordinationState.java:179) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:444) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:136) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:14,197][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:14,198][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:14,198][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:14,198][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:14,198][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:14,198][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:14,198][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:14,198][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5287, lastAcceptedTerm=5161, lastAcceptedVersion=152}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5287}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5309, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:14,244][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5310, lastAcceptedTerm=5290, lastAcceptedVersion=155}, leader=null
[2019-09-03T15:42:14,244][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5310, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:42:14,244][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5311
[2019-09-03T15:42:14,245][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5310] due to StartJoinRequest{term=5311,node={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}
[2019-09-03T15:42:14,367][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failure in response to StartJoinRequest{term=5309,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
org.elasticsearch.transport.RemoteTransportException: [hdp16b][192.168.1.226:9302][internal:cluster/coordination/start_join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5309 not greater than current term 5309
at org.elasticsearch.cluster.coordination.CoordinationState.handleStartJoin(CoordinationState.java:179) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:444) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:136) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:14,367][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5308, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16b][192.168.1.226:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5308 does not match current term 5309
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:14,368][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5308, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16b][192.168.1.226:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5308 does not match current term 5309
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:14,533][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5310, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14][192.168.1.214:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5310 does not match current term 5311
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:14,534][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5310, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14][192.168.1.214:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5310 does not match current term 5311
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:14,555][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5306, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5306, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5307, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5308, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=true, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5306}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5310, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:14,556][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failure in response to StartJoinRequest{term=5306,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
org.elasticsearch.transport.RemoteTransportException: [hdp15][192.168.1.215:9300][internal:cluster/coordination/start_join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5306 not greater than current term 5310
at org.elasticsearch.cluster.coordination.CoordinationState.handleStartJoin(CoordinationState.java:179) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:444) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:136) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:14,556][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5301, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5300, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5301, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5301, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=true, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5299}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5310, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:14,556][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5298}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5310, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:14,556][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5299, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}=PreVoteResponse{currentTerm=5298, lastAcceptedTerm=5290, lastAcceptedVersion=155}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}=PreVoteResponse{currentTerm=5298, lastAcceptedTerm=5290, lastAcceptedVersion=155}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5298}, isClosed=true} is closed, ignoring PreVoteResponse{currentTerm=5310, lastAcceptedTerm=5290, lastAcceptedVersion=155} from {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}
[2019-09-03T15:42:14,604][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failure in response to StartJoinRequest{term=5309,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/start_join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5309 not greater than current term 5311
at org.elasticsearch.cluster.coordination.CoordinationState.handleStartJoin(CoordinationState.java:179) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:444) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:136) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:14,620][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5311, lastAcceptedTerm=5290, lastAcceptedVersion=155}, leader=null
[2019-09-03T15:42:14,620][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5311, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
[2019-09-03T15:42:14,621][DEBUG][o.e.c.c.PreVoteCollector ] [hdp13] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, currentTerm=5311}, isClosed=false} requesting pre-votes from [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}]
[2019-09-03T15:42:14,621][DEBUG][o.e.c.c.Coordinator ] [hdp13] starting election with StartJoinRequest{term=5312,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}}
[2019-09-03T15:42:14,622][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}] with term 5312
[2019-09-03T15:42:14,622][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: leaving term [5311] due to StartJoinRequest{term=5312,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}}
[2019-09-03T15:42:14,912][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failure in response to StartJoinRequest{term=5312,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/coordination/start_join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5312 not greater than current term 5312
at org.elasticsearch.cluster.coordination.CoordinationState.handleStartJoin(CoordinationState.java:179) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:444) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:136) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:14,938][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failure in response to StartJoinRequest{term=5312,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/start_join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5312 not greater than current term 5312
at org.elasticsearch.cluster.coordination.CoordinationState.handleStartJoin(CoordinationState.java:179) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:444) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:136) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:14,946][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failure in response to StartJoinRequest{term=5312,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/coordination/start_join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5312 not greater than current term 5312
at org.elasticsearch.cluster.coordination.CoordinationState.handleStartJoin(CoordinationState.java:179) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:444) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:136) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:14,946][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5311, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5311 does not match current term 5312
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:14,947][DEBUG][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5311, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13b][192.168.1.223:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 5311 does not match current term 5312
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:42:14,954][TRACE][o.e.c.c.PreVoteCollector ] [hdp13] updating with preVoteResponse=PreVoteResponse{currentTerm=5312, lastAcceptedTerm=5290, lastAcceptedVersion=155}, leader=null
[2019-09-03T15:42:14,954][DEBUG][o.e.c.c.JoinHelper ] [hdp13] attempting to join {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5312, lastAcceptedTerm=5290, lastAcceptedVersion=155, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}}]}
[2019-09-03T15:42:14,954][DEBUG][o.e.c.c.Coordinator ] [hdp13] joinLeaderInTerm: for [{hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}] with term 5312
[2019-09-03T15:42:14,954][DEBUG][o.e.c.c.JoinHelper ] [hdp13] successful response to StartJoinRequest{term=5312,node={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}} from {hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}
[2019-09-03T15:42:14,954][DEBUG][o.e.c.c.CoordinationState] [hdp13] handleStartJoin: ignoring [StartJoinRequest{term=5312,node={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}] as term provided is not greater than current term [5312]
[2019-09-03T15:42:14,954][TRACE][o.e.c.c.Coordinator ] [hdp13] handleJoinRequest: as CANDIDATE, handling JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{XWHGUvleT32R_oq0ee4ZHA}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5312, lastAccept
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment