Skip to content

Instantly share code, notes, and snippets.

@elastic-paul

elastic-paul/master_election.logs Secret

Created September 3, 2019 15:23
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save elastic-paul/f06102bae617550baf17f11431dfc8de to your computer and use it in GitHub Desktop.
Save elastic-paul/f06102bae617550baf17f11431dfc8de to your computer and use it in GitHub Desktop.
Download ZIP
Raw
master_election.logs
[2019-09-03T15:20:23,767][INFO ][o.e.c.c.JoinHelper ] [hdp13] last failed join attempt was 11s ago, failed to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3871, lastAcceptedTerm=3795, lastAcceptedVersion=129, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14][192.168.1.214:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.NotMasterException: Node [{hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}] not master for join request
[2019-09-03T15:20:47,626][WARN ][o.e.c.c.ClusterFormationFailureHelper] [hdp13] master not discovered or elected yet, an election requires at least 4 nodes with ids from [3-dBP8WZQZO-Rymo0vC3mQ, EYrLIFzETzSwcUvvabPegg, jguIm07bTXC_8ot6fYN9bw, 1cT_ZkvzTHaSyiqDEi2GSw, 3BP0A2-JTTS-Vp_1YdnJrw, vp_SiTAYQyCE1TQVIUXx2A, bkVjp2mNSk6G0PH9QFBu9Q], have discovered [{hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}] which is not a quorum; discovery will continue using [192.168.1.214:9300, 192.168.1.215:9300, 192.168.1.216:9300, 192.168.1.223:9302, 192.168.1.224:9302, 192.168.1.225:9302, 192.168.1.226:9302] from hosts providers and [{hdp15}{vp_SiTAYQyCE1TQVIUXx2A}{g-8hPRKETaqoDRVASeOhHQ}{192.168.1.215}{192.168.1.215:9300}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13b}{1cT_ZkvzTHaSyiqDEi2GSw}{AZT-6Qu-QMC-pXj6VHLdLA}{192.168.1.223}{192.168.1.223:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}, {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}, {hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 3873, last-accepted version 131 in term 3873
[2019-09-03T15:20:48,497][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3873, lastAcceptedTerm=3871, lastAcceptedVersion=130, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:452) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:518) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:20:48,837][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3874, lastAcceptedTerm=3873, lastAcceptedVersion=131, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer master for term 3874 while handling publication
at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1018) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:252) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:238) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:20:50,464][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3879, lastAcceptedTerm=3873, lastAcceptedVersion=131, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 3879 does not match current term 3880
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:20:50,500][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3875, lastAcceptedTerm=3873, lastAcceptedVersion=131, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13][192.168.1.213:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:452) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:518) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:703) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:20:50,650][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3876, lastAcceptedTerm=3873, lastAcceptedVersion=131, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14b}{3-dBP8WZQZO-Rymo0vC3mQ}{GJqINNhtSYq702zOW7QkJA}{192.168.1.224}{192.168.1.224:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14b][192.168.1.224:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 3876 does not match current term 3880
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:20:51,134][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3877, lastAcceptedTerm=3873, lastAcceptedVersion=131, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:452) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:518) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:20:51,442][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3880, lastAcceptedTerm=3873, lastAcceptedVersion=131, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13][192.168.1.213:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:452) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:518) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:703) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:20:52,144][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3882, lastAcceptedTerm=3873, lastAcceptedVersion=131, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:452) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:518) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:20:52,200][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3884, lastAcceptedTerm=3873, lastAcceptedVersion=131, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16b][192.168.1.226:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 3884 does not match current term 3885
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:20:52,769][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3885, lastAcceptedTerm=3873, lastAcceptedVersion=131, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:452) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:518) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:20:52,771][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3886, lastAcceptedTerm=3873, lastAcceptedVersion=131, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13][192.168.1.213:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 3886 does not match current term 3887
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:703) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:20:53,479][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3878, lastAcceptedTerm=3873, lastAcceptedVersion=131, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14][192.168.1.214:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:452) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:518) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:20:54,037][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3890, lastAcceptedTerm=3873, lastAcceptedVersion=131, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 3890 does not match current term 3891
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:20:54,096][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3883, lastAcceptedTerm=3873, lastAcceptedVersion=131, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13][192.168.1.213:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:452) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:518) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:703) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:20:55,547][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3892, lastAcceptedTerm=3873, lastAcceptedVersion=131, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 3892 does not match current term 3896
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:20:56,038][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3896, lastAcceptedTerm=3873, lastAcceptedVersion=131, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 3896 does not match current term 3897
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:20:56,281][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3891, lastAcceptedTerm=3873, lastAcceptedVersion=131, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13][192.168.1.213:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:452) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:518) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:703) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:20:56,327][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3894, lastAcceptedTerm=3873, lastAcceptedVersion=131, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp15b}{EYrLIFzETzSwcUvvabPegg}{rEIiQUnkQM-teByxHqtceg}{192.168.1.225}{192.168.1.225:9302}{dim}{ml.machine_memory=135445331968, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp15b][192.168.1.225:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 3894 does not match current term 3897
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:20:56,631][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3898, lastAcceptedTerm=3873, lastAcceptedVersion=131, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16}{3BP0A2-JTTS-Vp_1YdnJrw}{fdSyGsDPTvCGSjYK5kSIlw}{192.168.1.216}{192.168.1.216:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16][192.168.1.216:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 3898 does not match current term 3899
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:20:57,308][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3899, lastAcceptedTerm=3873, lastAcceptedVersion=131, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp14}{bkVjp2mNSk6G0PH9QFBu9Q}{tfqd4mUsR-26x9ymImqa5w}{192.168.1.214}{192.168.1.214:9300}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp14][192.168.1.214:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 3899 does not match current term 3901
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:20:57,807][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3900, lastAcceptedTerm=3873, lastAcceptedVersion=131, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16b][192.168.1.226:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 3900 does not match current term 3902
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:20:57,875][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3897, lastAcceptedTerm=3873, lastAcceptedVersion=131, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp13][192.168.1.213:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:452) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:518) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:703) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:20:58,425][INFO ][o.e.c.c.JoinHelper ] [hdp13] failed to join {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3903, lastAcceptedTerm=3873, lastAcceptedVersion=131, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16b][192.168.1.226:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 3903 does not match current term 3904
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-09-03T15:20:58,496][INFO ][o.e.c.c.JoinHelper ] [hdp13] last failed join attempt was 71ms ago, failed to join {hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=3903, lastAcceptedTerm=3873, lastAcceptedVersion=131, sourceNode={hdp13}{SaA74YaMSNim17koypZiFQ}{4dRaECP_RpqlvzhqIQg02A}{192.168.1.213}{192.168.1.213:9300}{dim}{ml.machine_memory=135446564864, xpack.installed=true, ml.max_open_jobs=20}, targetNode={hdp16b}{jguIm07bTXC_8ot6fYN9bw}{tdWA14Q9QPWNPgC59mfc0w}{192.168.1.226}{192.168.1.226:9302}{dim}{ml.machine_memory=135446564864, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [hdp16b][192.168.1.226:9302][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 3903 does not match current term 3904
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:223) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:948) ~[elasticsearch-7.3.1.jar:7.3.1]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:517) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:484) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment