Skip to content

Instantly share code, notes, and snippets.

@barakseri1

barakseri1/data_node

Created August 17, 2020 08:21
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save barakseri1/3f107c3a38b5e5549ce90ac66117769f to your computer and use it in GitHub Desktop.
Save barakseri1/3f107c3a38b5e5549ce90ac66117769f to your computer and use it in GitHub Desktop.
Download ZIP
data node cant re-join the cluster
Raw
data_node
[2020-08-17T08:08:14,479][INFO ][o.e.e.NodeEnvironment ] [ip-172-30-1-61.ec2.internal] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/mapper/vg_elastic-lv_elastic)]], net usable_space [886.4gb], net total_space [984.1gb], types [ext4]
[2020-08-17T08:08:14,482][INFO ][o.e.e.NodeEnvironment ] [ip-172-30-1-61.ec2.internal] heap size [31gb], compressed ordinary object pointers [true]
[2020-08-17T08:08:14,622][INFO ][o.e.n.Node ] [ip-172-30-1-61.ec2.internal] node name [ip-172-30-1-61.ec2.internal], node ID [heumpdUlSRezT69xwljSgg], cluster name [elasticsearch-prod]
[2020-08-17T08:08:14,624][INFO ][o.e.n.Node ] [ip-172-30-1-61.ec2.internal] version[7.7.1], pid[24785], build[default/rpm/ad56dce891c901a492bb1ee393f12dfff473a423/2020-05-28T16:30:01.040088Z], OS[Linux/4.14.97-90.72.amzn2.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/14.0.1/14.0.1+7]
[2020-08-17T08:08:14,625][INFO ][o.e.n.Node ] [ip-172-30-1-61.ec2.internal] JVM home [/usr/share/elasticsearch/jdk]
[2020-08-17T08:08:14,625][INFO ][o.e.n.Node ] [ip-172-30-1-61.ec2.internal] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms31g, -Xmx31g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-14489621377398879393, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -XX:MaxDirectMemorySize=16642998272, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2020-08-17T08:08:16,526][WARN ][c.a.a.p.i.BasicProfileConfigFileLoader] [ip-172-30-1-61.ec2.internal] Unable to load config file null
java.security.AccessControlException: access denied ("java.io.FilePermission" "/nonexistent/.aws/config" "read")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
at java.security.AccessController.checkPermission(AccessController.java:1036) ~[?:?]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:408) ~[?:?]
at java.lang.SecurityManager.checkRead(SecurityManager.java:747) ~[?:?]
at java.io.File.exists(File.java:818) ~[?:?]
at com.amazonaws.profile.path.config.SharedConfigDefaultLocationProvider.getLocation(SharedConfigDefaultLocationProvider.java:36) ~[aws-java-sdk-core-1.11.749.jar:?]
at com.amazonaws.profile.path.AwsProfileFileLocationProviderChain.getLocation(AwsProfileFileLocationProviderChain.java:41) ~[aws-java-sdk-core-1.11.749.jar:?]
at com.amazonaws.auth.profile.internal.BasicProfileConfigFileLoader.getProfilesConfigFile(BasicProfileConfigFileLoader.java:69) [aws-java-sdk-core-1.11.749.jar:?]
at com.amazonaws.auth.profile.internal.BasicProfileConfigFileLoader.getProfile(BasicProfileConfigFileLoader.java:55) [aws-java-sdk-core-1.11.749.jar:?]
at com.amazonaws.retry.internal.RetryModeResolver.profile(RetryModeResolver.java:92) [aws-java-sdk-core-1.11.749.jar:?]
at com.amazonaws.retry.internal.RetryModeResolver.resolveRetryMode(RetryModeResolver.java:83) [aws-java-sdk-core-1.11.749.jar:?]
at com.amazonaws.retry.internal.RetryModeResolver.<init>(RetryModeResolver.java:46) [aws-java-sdk-core-1.11.749.jar:?]
at com.amazonaws.retry.RetryPolicy.<clinit>(RetryPolicy.java:35) [aws-java-sdk-core-1.11.749.jar:?]
at com.amazonaws.retry.PredefinedRetryPolicies.<clinit>(PredefinedRetryPolicies.java:30) [aws-java-sdk-core-1.11.749.jar:?]
at com.amazonaws.ClientConfiguration.<clinit>(ClientConfiguration.java:89) [aws-java-sdk-core-1.11.749.jar:?]
at java.lang.Class.forName0(Native Method) [?:?]
at java.lang.Class.forName(Class.java:340) [?:?]
at org.elasticsearch.discovery.ec2.Ec2DiscoveryPlugin.lambda$static$0(Ec2DiscoveryPlugin.java:69) [discovery-ec2-7.7.1.jar:7.7.1]
at java.security.AccessController.doPrivileged(AccessController.java:312) [?:?]
at org.elasticsearch.discovery.ec2.Ec2DiscoveryPlugin.<clinit>(Ec2DiscoveryPlugin.java:63) [discovery-ec2-7.7.1.jar:7.7.1]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.node.Node.<init>(Node.java:321) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.node.Node.<init>(Node.java:264) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.7.1.jar:7.7.1]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.7.1.jar:7.7.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.7.1.jar:7.7.1]
[2020-08-17T08:08:16,720][WARN ][c.a.a.p.i.BasicProfileConfigFileLoader] [ip-172-30-1-61.ec2.internal] Unable to load config file null
java.security.AccessControlException: access denied ("java.io.FilePermission" "/nonexistent/.aws/config" "read")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
at java.security.AccessController.checkPermission(AccessController.java:1036) ~[?:?]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:408) ~[?:?]
at java.lang.SecurityManager.checkRead(SecurityManager.java:747) ~[?:?]
at java.io.File.exists(File.java:818) ~[?:?]
at com.amazonaws.profile.path.config.SharedConfigDefaultLocationProvider.getLocation(SharedConfigDefaultLocationProvider.java:36) ~[aws-java-sdk-core-1.11.749.jar:?]
at com.amazonaws.profile.path.AwsProfileFileLocationProviderChain.getLocation(AwsProfileFileLocationProviderChain.java:41) ~[aws-java-sdk-core-1.11.749.jar:?]
at com.amazonaws.auth.profile.internal.BasicProfileConfigFileLoader.getProfilesConfigFile(BasicProfileConfigFileLoader.java:69) [aws-java-sdk-core-1.11.749.jar:?]
at com.amazonaws.auth.profile.internal.BasicProfileConfigFileLoader.getProfile(BasicProfileConfigFileLoader.java:55) [aws-java-sdk-core-1.11.749.jar:?]
at com.amazonaws.retry.internal.RetryModeResolver.profile(RetryModeResolver.java:92) [aws-java-sdk-core-1.11.749.jar:?]
at com.amazonaws.retry.internal.RetryModeResolver.resolveRetryMode(RetryModeResolver.java:83) [aws-java-sdk-core-1.11.749.jar:?]
at com.amazonaws.retry.internal.RetryModeResolver.<init>(RetryModeResolver.java:46) [aws-java-sdk-core-1.11.749.jar:?]
at com.amazonaws.retry.RetryPolicy.<clinit>(RetryPolicy.java:35) [aws-java-sdk-core-1.11.749.jar:?]
at com.amazonaws.retry.PredefinedRetryPolicies.<clinit>(PredefinedRetryPolicies.java:30) [aws-java-sdk-core-1.11.749.jar:?]
at com.amazonaws.ClientConfiguration.<clinit>(ClientConfiguration.java:89) [aws-java-sdk-core-1.11.749.jar:?]
at java.lang.Class.forName0(Native Method) [?:?]
at java.lang.Class.forName(Class.java:340) [?:?]
at org.elasticsearch.repositories.s3.S3RepositoryPlugin.lambda$static$0(S3RepositoryPlugin.java:57) [repository-s3-7.7.1.jar:7.7.1]
at java.security.AccessController.doPrivileged(AccessController.java:312) [?:?]
at org.elasticsearch.repositories.s3.S3RepositoryPlugin.<clinit>(S3RepositoryPlugin.java:51) [repository-s3-7.7.1.jar:7.7.1]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.node.Node.<init>(Node.java:321) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.node.Node.<init>(Node.java:264) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.7.1.jar:7.7.1]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.7.1.jar:7.7.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.7.1.jar:7.7.1]
[2020-08-17T08:08:16,735][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [aggs-matrix-stats]
[2020-08-17T08:08:16,735][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [analysis-common]
[2020-08-17T08:08:16,735][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [constant-keyword]
[2020-08-17T08:08:16,736][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [flattened]
[2020-08-17T08:08:16,736][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [frozen-indices]
[2020-08-17T08:08:16,736][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [ingest-common]
[2020-08-17T08:08:16,736][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [ingest-geoip]
[2020-08-17T08:08:16,736][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [ingest-user-agent]
[2020-08-17T08:08:16,737][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [lang-expression]
[2020-08-17T08:08:16,737][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [lang-mustache]
[2020-08-17T08:08:16,737][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [lang-painless]
[2020-08-17T08:08:16,737][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [mapper-extras]
[2020-08-17T08:08:16,737][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [parent-join]
[2020-08-17T08:08:16,737][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [percolator]
[2020-08-17T08:08:16,738][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [rank-eval]
[2020-08-17T08:08:16,738][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [reindex]
[2020-08-17T08:08:16,738][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [repository-url]
[2020-08-17T08:08:16,738][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [search-business-rules]
[2020-08-17T08:08:16,738][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [spatial]
[2020-08-17T08:08:16,739][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [systemd]
[2020-08-17T08:08:16,739][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [tasks]
[2020-08-17T08:08:16,739][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [transform]
[2020-08-17T08:08:16,739][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [transport-netty4]
[2020-08-17T08:08:16,739][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [vectors]
[2020-08-17T08:08:16,739][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [x-pack-analytics]
[2020-08-17T08:08:16,740][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [x-pack-async-search]
[2020-08-17T08:08:16,740][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [x-pack-autoscaling]
[2020-08-17T08:08:16,740][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [x-pack-ccr]
[2020-08-17T08:08:16,740][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [x-pack-core]
[2020-08-17T08:08:16,740][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [x-pack-deprecation]
[2020-08-17T08:08:16,741][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [x-pack-enrich]
[2020-08-17T08:08:16,741][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [x-pack-eql]
[2020-08-17T08:08:16,741][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [x-pack-graph]
[2020-08-17T08:08:16,741][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [x-pack-identity-provider]
[2020-08-17T08:08:16,741][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [x-pack-ilm]
[2020-08-17T08:08:16,741][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [x-pack-logstash]
[2020-08-17T08:08:16,742][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [x-pack-ml]
[2020-08-17T08:08:16,742][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [x-pack-monitoring]
[2020-08-17T08:08:16,742][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [x-pack-ql]
[2020-08-17T08:08:16,742][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [x-pack-rollup]
[2020-08-17T08:08:16,742][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [x-pack-security]
[2020-08-17T08:08:16,742][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [x-pack-sql]
[2020-08-17T08:08:16,743][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [x-pack-voting-only-node]
[2020-08-17T08:08:16,743][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded module [x-pack-watcher]
[2020-08-17T08:08:16,743][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded plugin [discovery-ec2]
[2020-08-17T08:08:16,743][INFO ][o.e.p.PluginsService ] [ip-172-30-1-61.ec2.internal] loaded plugin [repository-s3]
[2020-08-17T08:08:20,134][INFO ][o.e.x.s.a.s.FileRolesStore] [ip-172-30-1-61.ec2.internal] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2020-08-17T08:08:20,601][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [ip-172-30-1-61.ec2.internal] [controller/25056] [Main.cc@110] controller (64 bit): Version 7.7.1 (Build e5ebdcb44d66cd) Copyright (c) 2020 Elasticsearch BV
[2020-08-17T08:08:21,219][INFO ][o.e.d.DiscoveryModule ] [ip-172-30-1-61.ec2.internal] using discovery type [zen] and seed hosts providers [settings, ec2]
[2020-08-17T08:08:22,003][INFO ][o.e.n.Node ] [ip-172-30-1-61.ec2.internal] initialized
[2020-08-17T08:08:22,003][INFO ][o.e.n.Node ] [ip-172-30-1-61.ec2.internal] starting ...
[2020-08-17T08:08:22,126][INFO ][o.e.t.TransportService ] [ip-172-30-1-61.ec2.internal] publish_address {172.17.0.1:9300}, bound_addresses {[::]:9300}
[2020-08-17T08:08:22,649][INFO ][o.e.b.BootstrapChecks ] [ip-172-30-1-61.ec2.internal] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-08-17T08:08:22,671][INFO ][o.e.c.c.Coordinator ] [ip-172-30-1-61.ec2.internal] cluster UUID [vd63AAWCTc6rZ1RiIRSK4g]
[2020-08-17T08:08:32,680][WARN ][o.e.c.c.ClusterFormationFailureHelper] [ip-172-30-1-61.ec2.internal] master not discovered yet: have discovered [{ip-172-30-1-61.ec2.internal}{heumpdUlSRezT69xwljSgg}{3NbMrFIKTRylx4c9z1xnQg}{172.17.0.1}{172.17.0.1:9300}{dilrt}{aws_availability_zone=us-east-1b, ml.machine_memory=67534430208, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, {ip-172-30-0-123.ec2.internal}{vPn-iGVfQdWH_PCebTHUiQ}{a_Zb72LGSdiT92cjpwPaeg}{172.30.0.123}{172.30.0.123:9300}{lm}{aws_availability_zone=us-east-1a, ml.machine_memory=16820563968, ml.max_open_jobs=20, xpack.installed=true, transform.node=false}, {ip-172-30-2-62.ec2.internal}{sMuj5JCHQNKcNfUZdovHfA}{Sj6ihWVYRh6gpDx9yvXwhw}{172.30.2.62}{172.30.2.62:9300}{lm}{aws_availability_zone=us-east-1c, ml.machine_memory=16820563968, ml.max_open_jobs=20, xpack.installed=true, transform.node=false}, {ip-172-30-1-9.ec2.internal}{I7bTPetXQxmLz_a7maEWFw}{EGbpYlR7TvucszK5IaauYw}{172.30.1.9}{172.30.1.9:9300}{lm}{aws_availability_zone=us-east-1b, ml.machine_memory=16820563968, ml.max_open_jobs=20, xpack.installed=true, transform.node=false}, {ip-172-30-2-153.ec2.internal}{5M30QMotQhWEER6NEm_wiw}{NdlEK_tURk6VzwPdUnKSOQ}{172.30.2.153}{172.30.2.153:9300}{lm}{aws_availability_zone=us-east-1c, ml.machine_memory=16626966528, ml.max_open_jobs=20, xpack.installed=true, transform.node=false}, {ip-172-30-0-170.ec2.internal}{3uWMMAZRRO2e_0aOYpELHA}{OCck7fwqSXuZ6kC4amioZw}{172.30.0.170}{172.30.0.170:9300}{lmr}{aws_availability_zone=us-east-1a, ml.machine_memory=16626966528, ml.max_open_jobs=20, xpack.installed=true, transform.node=false}]; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, [::1]:9305, 172.30.0.123:9300, 172.30.2.62:9300, 172.30.1.9:9300, 172.30.2.153:9300, 172.30.2.214:9300, 172.30.2.243:9300, 172.30.0.4:9300, 172.30.0.170:9300] from hosts providers and [] from last-known cluster state; node term 83, last-accepted version 1092773 in term 83
[2020-08-17T08:08:42,682][WARN ][o.e.c.c.ClusterFormationFailureHelper] [ip-172-30-1-61.ec2.internal] master not discovered yet: have discovered [{ip-172-30-1-61.ec2.internal}{heumpdUlSRezT69xwljSgg}{3NbMrFIKTRylx4c9z1xnQg}{172.17.0.1}{172.17.0.1:9300}{dilrt}{aws_availability_zone=us-east-1b, ml.machine_memory=67534430208, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, {ip-172-30-0-123.ec2.internal}{vPn-iGVfQdWH_PCebTHUiQ}{a_Zb72LGSdiT92cjpwPaeg}{172.30.0.123}{172.30.0.123:9300}{lm}{aws_availability_zone=us-east-1a, ml.machine_memory=16820563968, ml.max_open_jobs=20, xpack.installed=true, transform.node=false}, {ip-172-30-2-62.ec2.internal}{sMuj5JCHQNKcNfUZdovHfA}{Sj6ihWVYRh6gpDx9yvXwhw}{172.30.2.62}{172.30.2.62:9300}{lm}{aws_availability_zone=us-east-1c, ml.machine_memory=16820563968, ml.max_open_jobs=20, xpack.installed=true, transform.node=false}, {ip-172-30-1-9.ec2.internal}{I7bTPetXQxmLz_a7maEWFw}{EGbpYlR7TvucszK5IaauYw}{172.30.1.9}{172.30.1.9:9300}{lm}{aws_availability_zone=us-east-1b, ml.machine_memory=16820563968, ml.max_open_jobs=20, xpack.installed=true, transform.node=false}, {ip-172-30-2-153.ec2.internal}{5M30QMotQhWEER6NEm_wiw}{NdlEK_tURk6VzwPdUnKSOQ}{172.30.2.153}{172.30.2.153:9300}{lm}{aws_availability_zone=us-east-1c, ml.machine_memory=16626966528, ml.max_open_jobs=20, xpack.installed=true, transform.node=false}, {ip-172-30-0-170.ec2.internal}{3uWMMAZRRO2e_0aOYpELHA}{OCck7fwqSXuZ6kC4amioZw}{172.30.0.170}{172.30.0.170:9300}{lmr}{aws_availability_zone=us-east-1a, ml.machine_memory=16626966528, ml.max_open_jobs=20, xpack.installed=true, transform.node=false}]; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, [::1]:9305, 172.30.0.123:9300, 172.30.2.62:9300, 172.30.1.9:9300, 172.30.2.153:9300, 172.30.2.214:9300, 172.30.2.243:9300, 172.30.0.4:9300, 172.30.0.170:9300] from hosts providers and [] from last-known cluster state; node term 83, last-accepted version 1092773 in term 83
[2020-08-17T08:08:52,684][WARN ][o.e.n.Node ] [ip-172-30-1-61.ec2.internal] timed out while waiting for initial discovery state - timeout: 30s
[2020-08-17T08:08:52,684][WARN ][o.e.c.c.ClusterFormationFailureHelper] [ip-172-30-1-61.ec2.internal] master not discovered yet: have discovered [{ip-172-30-1-61.ec2.internal}{heumpdUlSRezT69xwljSgg}{3NbMrFIKTRylx4c9z1xnQg}{172.17.0.1}{172.17.0.1:9300}{dilrt}{aws_availability_zone=us-east-1b, ml.machine_memory=67534430208, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, {ip-172-30-0-123.ec2.internal}{vPn-iGVfQdWH_PCebTHUiQ}{a_Zb72LGSdiT92cjpwPaeg}{172.30.0.123}{172.30.0.123:9300}{lm}{aws_availability_zone=us-east-1a, ml.machine_memory=16820563968, ml.max_open_jobs=20, xpack.installed=true, transform.node=false}, {ip-172-30-2-62.ec2.internal}{sMuj5JCHQNKcNfUZdovHfA}{Sj6ihWVYRh6gpDx9yvXwhw}{172.30.2.62}{172.30.2.62:9300}{lm}{aws_availability_zone=us-east-1c, ml.machine_memory=16820563968, ml.max_open_jobs=20, xpack.installed=true, transform.node=false}, {ip-172-30-1-9.ec2.internal}{I7bTPetXQxmLz_a7maEWFw}{EGbpYlR7TvucszK5IaauYw}{172.30.1.9}{172.30.1.9:9300}{lm}{aws_availability_zone=us-east-1b, ml.machine_memory=16820563968, ml.max_open_jobs=20, xpack.installed=true, transform.node=false}, {ip-172-30-2-153.ec2.internal}{5M30QMotQhWEER6NEm_wiw}{NdlEK_tURk6VzwPdUnKSOQ}{172.30.2.153}{172.30.2.153:9300}{lm}{aws_availability_zone=us-east-1c, ml.machine_memory=16626966528, ml.max_open_jobs=20, xpack.installed=true, transform.node=false}, {ip-172-30-0-170.ec2.internal}{3uWMMAZRRO2e_0aOYpELHA}{OCck7fwqSXuZ6kC4amioZw}{172.30.0.170}{172.30.0.170:9300}{lmr}{aws_availability_zone=us-east-1a, ml.machine_memory=16626966528, ml.max_open_jobs=20, xpack.installed=true, transform.node=false}]; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, [::1]:9305, 172.30.0.123:9300, 172.30.2.62:9300, 172.30.1.9:9300, 172.30.2.153:9300, 172.30.2.214:9300, 172.30.2.243:9300, 172.30.0.4:9300, 172.30.0.170:9300] from hosts providers and [] from last-known cluster state; node term 83, last-accepted version 1092773 in term 83
[2020-08-17T08:08:52,692][INFO ][o.e.h.AbstractHttpServerTransport] [ip-172-30-1-61.ec2.internal] publish_address {172.17.0.1:9200}, bound_addresses {[::]:9200}
[2020-08-17T08:08:52,693][INFO ][o.e.n.Node ] [ip-172-30-1-61.ec2.internal] started
[2020-08-17T08:08:53,617][INFO ][o.e.c.c.JoinHelper ] [ip-172-30-1-61.ec2.internal] failed to join {ip-172-30-0-170.ec2.internal}{3uWMMAZRRO2e_0aOYpELHA}{OCck7fwqSXuZ6kC4amioZw}{172.30.0.170}{172.30.0.170:9300}{lmr}{aws_availability_zone=us-east-1a, ml.machine_memory=16626966528, ml.max_open_jobs=20, xpack.installed=true, transform.node=false} with JoinRequest{sourceNode={ip-172-30-1-61.ec2.internal}{heumpdUlSRezT69xwljSgg}{3NbMrFIKTRylx4c9z1xnQg}{172.17.0.1}{172.17.0.1:9300}{dilrt}{aws_availability_zone=us-east-1b, ml.machine_memory=67534430208, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, minimumTerm=83, optionalJoin=Optional.empty}
org.elasticsearch.transport.RemoteTransportException: [ip-172-30-0-170.ec2.internal][172.30.0.170:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.transport.ConnectTransportException: [ip-172-30-1-61.ec2.internal][172.17.0.1:9300] connect_timeout[30s]
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onTimeout(TcpTransport.java:1004) ~[elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) ~[elasticsearch-7.7.1.jar:7.7.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2020-08-17T08:08:53,620][INFO ][o.e.c.c.JoinHelper ] [ip-172-30-1-61.ec2.internal] failed to join {ip-172-30-0-170.ec2.internal}{3uWMMAZRRO2e_0aOYpELHA}{OCck7fwqSXuZ6kC4amioZw}{172.30.0.170}{172.30.0.170:9300}{lmr}{aws_availability_zone=us-east-1a, ml.machine_memory=16626966528, ml.max_open_jobs=20, xpack.installed=true, transform.node=false} with JoinRequest{sourceNode={ip-172-30-1-61.ec2.internal}{heumpdUlSRezT69xwljSgg}{3NbMrFIKTRylx4c9z1xnQg}{172.17.0.1}{172.17.0.1:9300}{dilrt}{aws_availability_zone=us-east-1b, ml.machine_memory=67534430208, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, minimumTerm=83, optionalJoin=Optional.empty}
org.elasticsearch.transport.RemoteTransportException: [ip-172-30-0-170.ec2.internal][172.30.0.170:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.transport.ConnectTransportException: [ip-172-30-1-61.ec2.internal][172.17.0.1:9300] connect_timeout[30s]
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onTimeout(TcpTransport.java:1004) ~[elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) ~[elasticsearch-7.7.1.jar:7.7.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2020-08-17T08:09:02,686][INFO ][o.e.c.c.JoinHelper ] [ip-172-30-1-61.ec2.internal] last failed join attempt was 9s ago, failed to join {ip-172-30-0-170.ec2.internal}{3uWMMAZRRO2e_0aOYpELHA}{OCck7fwqSXuZ6kC4amioZw}{172.30.0.170}{172.30.0.170:9300}{lmr}{aws_availability_zone=us-east-1a, ml.machine_memory=16626966528, ml.max_open_jobs=20, xpack.installed=true, transform.node=false} with JoinRequest{sourceNode={ip-172-30-1-61.ec2.internal}{heumpdUlSRezT69xwljSgg}{3NbMrFIKTRylx4c9z1xnQg}{172.17.0.1}{172.17.0.1:9300}{dilrt}{aws_availability_zone=us-east-1b, ml.machine_memory=67534430208, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, minimumTerm=83, optionalJoin=Optional.empty}
org.elasticsearch.transport.RemoteTransportException: [ip-172-30-0-170.ec2.internal][172.30.0.170:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.transport.ConnectTransportException: [ip-172-30-1-61.ec2.internal][172.17.0.1:9300] connect_timeout[30s]
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onTimeout(TcpTransport.java:1004) ~[elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) ~[elasticsearch-7.7.1.jar:7.7.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
Raw
master_node
[2020-08-17T08:07:51,552][INFO ][o.e.c.s.MasterService ] [ip-172-30-0-170.ec2.internal] node-left[{ip-172-30-1-61.ec2.internal}{heumpdUlSRezT69xwljSgg}{a5VszEDvT7uXNy2aCzcXIg}{172.30.1.61}{172.30.1.61:9300}{dilrt}{aws_availability_zone=us-east-1b, ml.machine_memory=67534430208, ml.max_open_jobs=20, xpack.installed=true, transform.node=true} reason: disconnected], term: 83, version: 1092774, delta: removed {{ip-172-30-1-61.ec2.internal}{heumpdUlSRezT69xwljSgg}{a5VszEDvT7uXNy2aCzcXIg}{172.30.1.61}{172.30.1.61:9300}{dilrt}{aws_availability_zone=us-east-1b, ml.machine_memory=67534430208, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}}
[2020-08-17T08:07:51,602][INFO ][o.e.c.s.ClusterApplierService] [ip-172-30-0-170.ec2.internal] removed {{ip-172-30-1-61.ec2.internal}{heumpdUlSRezT69xwljSgg}{a5VszEDvT7uXNy2aCzcXIg}{172.30.1.61}{172.30.1.61:9300}{dilrt}{aws_availability_zone=us-east-1b, ml.machine_memory=67534430208, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}}, term: 83, version: 1092774, reason: Publication{term=83, version=1092774}
[2020-08-17T08:07:51,604][INFO ][o.e.c.r.DelayedAllocationService] [ip-172-30-0-170.ec2.internal] scheduling reroute for delayed shards in [945.8ms] (8 delayed shards)
[2020-08-17T08:07:52,550][INFO ][o.e.c.r.DelayedAllocationService] [ip-172-30-0-170.ec2.internal] scheduling reroute for delayed shards in [0s] (8 delayed shards)
[2020-08-17T08:07:52,583][INFO ][o.e.c.r.DelayedAllocationService] [ip-172-30-0-170.ec2.internal] scheduling reroute for delayed shards in [58.9s] (7 delayed shards)
[2020-08-17T08:07:53,933][WARN ][o.e.c.r.a.AllocationService] [ip-172-30-0-170.ec2.internal] [images_restored_index][5] marking unavailable shards as stale: [JV-KNH8TTDmzZPjGmNHlcg]
[2020-08-17T08:07:59,524][WARN ][o.e.c.r.a.AllocationService] [ip-172-30-0-170.ec2.internal] [challenges_prod][0] marking unavailable shards as stale: [ee5wXBXrTuCyqb5dhE-iAw]
[2020-08-17T08:08:06,155][WARN ][o.e.c.r.a.AllocationService] [ip-172-30-0-170.ec2.internal] [images_restored_index][0] marking unavailable shards as stale: [MaRevXhyRNO3OCBW8nkyfA]
[2020-08-17T08:08:51,582][INFO ][o.e.c.r.DelayedAllocationService] [ip-172-30-0-170.ec2.internal] scheduling reroute for delayed shards in [10.9m] (2 delayed shards)
[2020-08-17T08:08:51,629][WARN ][o.e.c.r.a.AllocationService] [ip-172-30-0-170.ec2.internal] [elasticsearch_metrics-2020.06.30][0] marking unavailable shards as stale: [wUwSEcyYT1eJS19eg7rWtg]
[2020-08-17T08:08:51,706][WARN ][o.e.c.r.a.AllocationService] [ip-172-30-0-170.ec2.internal] [teams_prod][0] marking unavailable shards as stale: [fAoIyWVSQwCrnUgOpcoLzQ]
[2020-08-17T08:08:52,106][WARN ][o.e.c.r.a.AllocationService] [ip-172-30-0-170.ec2.internal] [ilm-history-2-000002][0] marking unavailable shards as stale: [nfL_K8NWR7CMk_aUhvG47Q]
[2020-08-17T08:08:52,523][WARN ][o.e.c.r.a.AllocationService] [ip-172-30-0-170.ec2.internal] [.monitoring-kibana-7-2020.08.14][0] marking unavailable shards as stale: [kdLTyII1Q_C_iUWrn9azcg]
[2020-08-17T08:09:00,457][WARN ][o.e.c.r.a.AllocationService] [ip-172-30-0-170.ec2.internal] [.monitoring-es-7-2020.08.12][0] marking unavailable shards as stale: [Al5qNluYSRWrW3bT97UT2A]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment