Skip to content

Instantly share code, notes, and snippets.

@xavi-

xavi-/es-node-doctor-spectrum.log Secret

Last active June 7, 2016 23:54
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save xavi-/6ecc4ba16b39680fb28c8fb25307bcc7 to your computer and use it in GitHub Desktop.
Save xavi-/6ecc4ba16b39680fb28c8fb25307bcc7 to your computer and use it in GitHub Desktop.
Download ZIP
es logs
Raw
es-node-doctor-spectrum.log
[2016-06-07 23:35:58,213][INFO ][node ] [Doctor Spectrum] version[2.3.3], pid[1], build[218bdf1/2016-05-17T15:40:04Z]
[2016-06-07 23:35:58,215][INFO ][node ] [Doctor Spectrum] initializing ...
[2016-06-07 23:35:59,520][INFO ][plugins ] [Doctor Spectrum] modules [reindex, lang-expression, lang-groovy], plugins [cloud-aws], sites []
[2016-06-07 23:35:59,589][INFO ][env ] [Doctor Spectrum] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/xvda1)]], net usable_space [23.3gb], net total_space [29.4gb], spins? [possibly], types [ext4]
[2016-06-07 23:35:59,590][INFO ][env ] [Doctor Spectrum] heap size [1007.3mb], compressed ordinary object pointers [true]
[2016-06-07 23:35:59,590][WARN ][env ] [Doctor Spectrum] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at least [65536]
[2016-06-07 23:36:01,867][DEBUG][discovery.zen.elect ] [Doctor Spectrum] using minimum_master_nodes [-1]
[2016-06-07 23:36:02,509][DEBUG][discovery.ec2 ] [Doctor Spectrum] using host_type [PRIVATE_IP], tags [{}], groups [[sg-20d26c44]] with any_group [false], availability_zones [[]]
[2016-06-07 23:36:02,514][DEBUG][discovery.zen.ping.unicast] [Doctor Spectrum] using initial hosts [127.0.0.1, [::1]], with concurrent_connects [10]
[2016-06-07 23:36:02,530][DEBUG][discovery.ec2 ] [Doctor Spectrum] using ping.timeout [3s], join.timeout [1m], master_election.filter_client [true], master_election.filter_data [false]
[2016-06-07 23:36:02,532][DEBUG][discovery.zen.fd ] [Doctor Spectrum] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2016-06-07 23:36:02,535][DEBUG][discovery.zen.fd ] [Doctor Spectrum] [node ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2016-06-07 23:36:03,317][DEBUG][discovery.ec2 ] [Doctor Spectrum] using host_type [PRIVATE_IP], tags [{}], groups [[sg-20d26c44]] with any_group [false], availability_zones [[]]
[2016-06-07 23:36:03,317][DEBUG][discovery.ec2 ] [Doctor Spectrum] using host_type [PRIVATE_IP], tags [{}], groups [[sg-20d26c44]] with any_group [false], availability_zones [[]]
[2016-06-07 23:36:03,326][INFO ][node ] [Doctor Spectrum] initialized
[2016-06-07 23:36:03,326][INFO ][node ] [Doctor Spectrum] starting ...
[2016-06-07 23:36:03,446][INFO ][transport ] [Doctor Spectrum] publish_address {10.0.151.127:32111}, bound_addresses {[::]:9300}
[2016-06-07 23:36:03,452][INFO ][discovery ] [Doctor Spectrum] hihi/T326441uQgiodwOXCsoYMQ
[2016-06-07 23:36:03,459][TRACE][discovery.zen ] [Doctor Spectrum] starting to accumulate joins
[2016-06-07 23:36:03,459][TRACE][discovery.ec2 ] [Doctor Spectrum] starting to ping
[2016-06-07 23:36:03,482][WARN ][com.amazonaws.jmx.SdkMBeanRegistrySupport]
java.security.AccessControlException: access denied ("javax.management.MBeanServerPermission" "findMBeanServer")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at java.security.AccessController.checkPermission(AccessController.java:884)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at javax.management.MBeanServerFactory.checkPermission(MBeanServerFactory.java:413)
at javax.management.MBeanServerFactory.findMBeanServer(MBeanServerFactory.java:361)
at com.amazonaws.jmx.MBeans.getMBeanServer(MBeans.java:111)
at com.amazonaws.jmx.MBeans.registerMBean(MBeans.java:50)
at com.amazonaws.jmx.SdkMBeanRegistrySupport.registerMetricAdminMBean(SdkMBeanRegistrySupport.java:27)
at com.amazonaws.metrics.AwsSdkMetrics.registerMetricAdminMBean(AwsSdkMetrics.java:355)
at com.amazonaws.metrics.AwsSdkMetrics.<clinit>(AwsSdkMetrics.java:316)
at com.amazonaws.AmazonWebServiceClient.requestMetricCollector(AmazonWebServiceClient.java:563)
at com.amazonaws.AmazonWebServiceClient.isRMCEnabledAtClientOrSdkLevel(AmazonWebServiceClient.java:504)
at com.amazonaws.AmazonWebServiceClient.isRequestMetricsEnabled(AmazonWebServiceClient.java:496)
at com.amazonaws.AmazonWebServiceClient.createExecutionContext(AmazonWebServiceClient.java:457)
at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5924)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider.fetchDynamicNodes(AwsEc2UnicastHostsProvider.java:118)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider$DiscoNodesCache.refresh(AwsEc2UnicastHostsProvider.java:230)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider$DiscoNodesCache.refresh(AwsEc2UnicastHostsProvider.java:215)
at org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:55)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider.buildDynamicNodes(AwsEc2UnicastHostsProvider.java:104)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.sendPings(UnicastZenPing.java:335)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.ping(UnicastZenPing.java:240)
at org.elasticsearch.discovery.zen.ping.ZenPingService.ping(ZenPingService.java:106)
at org.elasticsearch.discovery.zen.ping.ZenPingService.pingAndWait(ZenPingService.java:84)
at org.elasticsearch.discovery.zen.ZenDiscovery.findMaster(ZenDiscovery.java:886)
at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:350)
at org.elasticsearch.discovery.zen.ZenDiscovery.access$4800(ZenDiscovery.java:91)
at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1237)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-06-07 23:36:03,488][WARN ][com.amazonaws.metrics.AwsSdkMetrics]
java.security.AccessControlException: access denied ("javax.management.MBeanServerPermission" "findMBeanServer")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at java.security.AccessController.checkPermission(AccessController.java:884)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at javax.management.MBeanServerFactory.checkPermission(MBeanServerFactory.java:413)
at javax.management.MBeanServerFactory.findMBeanServer(MBeanServerFactory.java:361)
at com.amazonaws.jmx.MBeans.getMBeanServer(MBeans.java:111)
at com.amazonaws.jmx.MBeans.isRegistered(MBeans.java:98)
at com.amazonaws.jmx.SdkMBeanRegistrySupport.isMBeanRegistered(SdkMBeanRegistrySupport.java:46)
at com.amazonaws.metrics.AwsSdkMetrics.registerMetricAdminMBean(AwsSdkMetrics.java:361)
at com.amazonaws.metrics.AwsSdkMetrics.<clinit>(AwsSdkMetrics.java:316)
at com.amazonaws.AmazonWebServiceClient.requestMetricCollector(AmazonWebServiceClient.java:563)
at com.amazonaws.AmazonWebServiceClient.isRMCEnabledAtClientOrSdkLevel(AmazonWebServiceClient.java:504)
at com.amazonaws.AmazonWebServiceClient.isRequestMetricsEnabled(AmazonWebServiceClient.java:496)
at com.amazonaws.AmazonWebServiceClient.createExecutionContext(AmazonWebServiceClient.java:457)
at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5924)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider.fetchDynamicNodes(AwsEc2UnicastHostsProvider.java:118)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider$DiscoNodesCache.refresh(AwsEc2UnicastHostsProvider.java:230)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider$DiscoNodesCache.refresh(AwsEc2UnicastHostsProvider.java:215)
at org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:55)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider.buildDynamicNodes(AwsEc2UnicastHostsProvider.java:104)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.sendPings(UnicastZenPing.java:335)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.ping(UnicastZenPing.java:240)
at org.elasticsearch.discovery.zen.ping.ZenPingService.ping(ZenPingService.java:106)
at org.elasticsearch.discovery.zen.ping.ZenPingService.pingAndWait(ZenPingService.java:84)
at org.elasticsearch.discovery.zen.ZenDiscovery.findMaster(ZenDiscovery.java:886)
at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:350)
at org.elasticsearch.discovery.zen.ZenDiscovery.access$4800(ZenDiscovery.java:91)
at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1237)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-06-07 23:36:05,250][TRACE][discovery.ec2 ] [Doctor Spectrum] building dynamic unicast discovery nodes...
[2016-06-07 23:36:05,251][TRACE][discovery.ec2 ] [Doctor Spectrum] filtering out instance i-2d049498 based on groups [{GroupName: ecs-production-ECSNodeSG-K9ZR984A5E5Q,GroupId: sg-868a27e2}, {GroupName: RedisClientSG,GroupId: sg-795fbd16}, {GroupName: PostgresClientSG,GroupId: sg-a82af8cd}, {GroupName: GearmanClientSG,GroupId: sg-4f140b23}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:05,251][TRACE][discovery.ec2 ] [Doctor Spectrum] filtering out instance i-2c049499 based on groups [{GroupName: ecs-production-ECSNodeSG-K9ZR984A5E5Q,GroupId: sg-868a27e2}, {GroupName: RedisClientSG,GroupId: sg-795fbd16}, {GroupName: PostgresClientSG,GroupId: sg-a82af8cd}, {GroupName: GearmanClientSG,GroupId: sg-4f140b23}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:05,251][TRACE][discovery.ec2 ] [Doctor Spectrum] filtering out instance i-2e04949b based on groups [{GroupName: ecs-production-ECSNodeSG-K9ZR984A5E5Q,GroupId: sg-868a27e2}, {GroupName: RedisClientSG,GroupId: sg-795fbd16}, {GroupName: PostgresClientSG,GroupId: sg-a82af8cd}, {GroupName: GearmanClientSG,GroupId: sg-4f140b23}], does not include all of [sg-20d26c44]
.... snipped filtering out a bunch of machines ....
[2016-06-07 23:36:05,299][TRACE][discovery.ec2 ] [Doctor Spectrum] filtering out instance i-29feb59b based on groups [{GroupName: DBServerSG,GroupId: sg-f5a5b999}, {GroupName: SaltMinionSG,GroupId: sg-6ad0cf06}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:05,299][TRACE][discovery.ec2 ] [Doctor Spectrum] filtering out instance i-0ce772b9 based on groups [{GroupName: ecs-production-ECSNodeSG-K9ZR984A5E5Q,GroupId: sg-868a27e2}, {GroupName: RedisClientSG,GroupId: sg-795fbd16}, {GroupName: PostgresClientSG,GroupId: sg-a82af8cd}, {GroupName: GearmanClientSG,GroupId: sg-4f140b23}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:05,300][TRACE][discovery.ec2 ] [Doctor Spectrum] filtering out instance i-0fe772ba based on groups [{GroupName: ecs-production-ECSNodeSG-K9ZR984A5E5Q,GroupId: sg-868a27e2}, {GroupName: RedisClientSG,GroupId: sg-795fbd16}, {GroupName: PostgresClientSG,GroupId: sg-a82af8cd}, {GroupName: GearmanClientSG,GroupId: sg-4f140b23}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:05,304][TRACE][discovery.ec2 ] [Doctor Spectrum] adding i-5889eded, address 10.0.150.235, transport_address 10.0.150.235:9300
[2016-06-07 23:36:05,304][TRACE][discovery.ec2 ] [Doctor Spectrum] filtering out instance i-de52631c based on groups [{GroupName: LoggingTestSG,GroupId: sg-be435ad2}, {GroupName: SaltMinionSG,GroupId: sg-6ad0cf06}, {GroupName: Staging,GroupId: sg-78b5571d}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:05,304][TRACE][discovery.ec2 ] [Doctor Spectrum] filtering out instance i-6b830231 based on groups [{GroupName: SaltMasterSG,GroupId: sg-6bd0cf07}, {GroupName: CarbonClientSG,GroupId: sg-4ef3ec22}, {GroupName: SaltMinionSG,GroupId: sg-6ad0cf06}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:05,305][TRACE][discovery.ec2 ] [Doctor Spectrum] filtering out instance i-5b32f3e9 based on groups [{GroupName: SupervisorSG,GroupId: sg-9732fff2}, {GroupName: HaproxyHealthCheck,GroupId: sg-b394fdd6}, {GroupName: SaltMinionSG,GroupId: sg-6ad0cf06}, {GroupName: Staging,GroupId: sg-78b5571d}], does not include all of [sg-20d26c44]
.... snipped filtering out a bunch of machines ....
[2016-06-07 23:36:05,328][TRACE][discovery.ec2 ] [Doctor Spectrum] filtering out instance i-1010ebd3 based on groups [{GroupName: RedisServerSG,GroupId: sg-775fbd18}, {GroupName: RedisClientSG,GroupId: sg-795fbd16}, {GroupName: SaltMinionSG,GroupId: sg-6ad0cf06}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:05,329][TRACE][discovery.ec2 ] [Doctor Spectrum] filtering out instance i-c1b66d74 based on groups [{GroupName: LoggingTestSG,GroupId: sg-be435ad2}, {GroupName: SaltMinionSG,GroupId: sg-6ad0cf06}, {GroupName: Staging,GroupId: sg-78b5571d}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:05,329][TRACE][discovery.ec2 ] [Doctor Spectrum] filtering out instance i-9d90895d based on groups [{GroupName: DBServerSG,GroupId: sg-f5a5b999}, {GroupName: SaltMinionSG,GroupId: sg-6ad0cf06}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:05,329][TRACE][discovery.ec2 ] [Doctor Spectrum] adding i-e70acb70, address 10.0.151.127, transport_address 10.0.151.127:9300
[2016-06-07 23:36:05,330][TRACE][discovery.ec2 ] [Doctor Spectrum] filtering out instance i-f9e7276e based on groups [{GroupName: DBServerDevSG,GroupId: sg-4b233d27}, {GroupName: SaltMinionSG,GroupId: sg-6ad0cf06}, {GroupName: Staging,GroupId: sg-78b5571d}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:05,330][TRACE][discovery.ec2 ] [Doctor Spectrum] filtering out instance i-b6fdb604 based on groups [{GroupName: DBServerSG,GroupId: sg-f5a5b999}, {GroupName: SaltMinionSG,GroupId: sg-6ad0cf06}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:05,330][TRACE][discovery.ec2 ] [Doctor Spectrum] filtering out instance i-867aff33 based on groups [{GroupName: Docker,GroupId: sg-fb8b8599}, {GroupName: SaltMinionSG,GroupId: sg-6ad0cf06}], does not include all of [sg-20d26c44]
.... snipped filtering out a bunch of machines ....
[2016-06-07 23:36:05,334][TRACE][discovery.ec2 ] [Doctor Spectrum] filtering out instance i-27a6c492 based on groups [{GroupName: ecs-loadtest-ECSNodeSG-1I91YTXN1CZQC,GroupId: sg-898726ed}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:05,334][TRACE][discovery.ec2 ] [Doctor Spectrum] filtering out instance i-2ba6c49e based on groups [{GroupName: ecs-loadtest-ECSNodeSG-1I91YTXN1CZQC,GroupId: sg-898726ed}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:05,335][TRACE][discovery.ec2 ] [Doctor Spectrum] filtering out instance i-2aa6c49f based on groups [{GroupName: ecs-loadtest-ECSNodeSG-1I91YTXN1CZQC,GroupId: sg-898726ed}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:05,335][DEBUG][discovery.ec2 ] [Doctor Spectrum] using dynamic discovery nodes [{#cloud-i-5889eded-0}{10.0.150.235}{10.0.150.235:9300}, {#cloud-i-e70acb70-0}{10.0.151.127}{10.0.151.127:9300}]
[2016-06-07 23:36:05,339][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] connecting (light) to {#zen_unicast_2#}{::1}{[::1]:9300}
[2016-06-07 23:36:05,339][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] replacing {#cloud-i-5889eded-0}{10.0.150.235}{10.0.150.235:9300} with temp node {#zen_unicast_3_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
[2016-06-07 23:36:05,338][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] connecting (light) to {#zen_unicast_1#}{127.0.0.1}{127.0.0.1:9300}
[2016-06-07 23:36:05,342][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] replacing {#cloud-i-e70acb70-0}{10.0.151.127}{10.0.151.127:9300} with temp node {#zen_unicast_4_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
[2016-06-07 23:36:05,343][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] sending to {Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}
[2016-06-07 23:36:05,351][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] connecting (light) to {#zen_unicast_3_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
[2016-06-07 23:36:05,357][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] received response from {Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}: [ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[2], master [null], hasJoinedOnce [false], cluster_name[hihi]}]
[2016-06-07 23:36:05,358][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] connecting (light) to {#zen_unicast_4_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
[2016-06-07 23:36:05,391][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] failed to connect to {#zen_unicast_3_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
ConnectTransportException[[][10.0.150.235:9300] connect_timeout[30s]]; nested: ConnectException[Connection refused: /10.0.150.235:9300];
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:952)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:916)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:888)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:267)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:395)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: /10.0.150.235:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2016-06-07 23:36:05,394][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] failed to connect to {#zen_unicast_4_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
ConnectTransportException[[][10.0.151.127:9300] connect_timeout[30s]]; nested: ConnectException[Connection refused: /10.0.151.127:9300];
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:952)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:916)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:888)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:267)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:395)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: /10.0.151.127:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2016-06-07 23:36:05,402][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] connected to {#zen_unicast_1#}{127.0.0.1}{127.0.0.1:9300}
[2016-06-07 23:36:05,402][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] connected to {#zen_unicast_2#}{::1}{[::1]:9300}
[2016-06-07 23:36:05,402][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] sending to {#zen_unicast_2#}{::1}{[::1]:9300}
[2016-06-07 23:36:05,403][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] sending to {#zen_unicast_1#}{127.0.0.1}{127.0.0.1:9300}
[2016-06-07 23:36:05,435][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] received response from {#zen_unicast_1#}{127.0.0.1}{127.0.0.1:9300}: [ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[3], master [null], hasJoinedOnce [false], cluster_name[hihi]}]
[2016-06-07 23:36:05,439][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] received response from {#zen_unicast_2#}{::1}{[::1]:9300}: [ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[4], master [null], hasJoinedOnce [false], cluster_name[hihi]}]
[2016-06-07 23:36:06,866][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] sending to {#zen_unicast_1#}{127.0.0.1}{127.0.0.1:9300}
[2016-06-07 23:36:06,867][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] sending to {#zen_unicast_2#}{::1}{[::1]:9300}
[2016-06-07 23:36:06,868][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] replacing {#cloud-i-5889eded-0}{10.0.150.235}{10.0.150.235:9300} with temp node {#zen_unicast_5_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
[2016-06-07 23:36:06,868][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] connecting (light) to {#zen_unicast_5_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
[2016-06-07 23:36:06,868][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] replacing {#cloud-i-e70acb70-0}{10.0.151.127}{10.0.151.127:9300} with temp node {#zen_unicast_6_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
[2016-06-07 23:36:06,869][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] sending to {Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}
[2016-06-07 23:36:06,869][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] connecting (light) to {#zen_unicast_6_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
[2016-06-07 23:36:06,873][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] received response from {Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}: [ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[8], master [null], hasJoinedOnce [false], cluster_name[hihi]}]
[2016-06-07 23:36:06,874][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] failed to connect to {#zen_unicast_5_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
ConnectTransportException[[][10.0.150.235:9300] connect_timeout[30s]]; nested: ConnectException[Connection refused: /10.0.150.235:9300];
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:952)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:916)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:888)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:267)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:395)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: /10.0.150.235:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2016-06-07 23:36:06,875][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] failed to connect to {#zen_unicast_6_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
ConnectTransportException[[][10.0.151.127:9300] connect_timeout[30s]]; nested: ConnectException[Connection refused: /10.0.151.127:9300];
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:952)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:916)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:888)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:267)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:395)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: /10.0.151.127:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2016-06-07 23:36:06,876][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] received response from {#zen_unicast_1#}{127.0.0.1}{127.0.0.1:9300}: [ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[6], master [null], hasJoinedOnce [false], cluster_name[hihi]}]
[2016-06-07 23:36:06,877][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] received response from {#zen_unicast_2#}{::1}{[::1]:9300}: [ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[7], master [null], hasJoinedOnce [false], cluster_name[hihi]}]
[2016-06-07 23:36:08,375][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] sending to {#zen_unicast_1#}{127.0.0.1}{127.0.0.1:9300}
[2016-06-07 23:36:08,376][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] sending to {#zen_unicast_2#}{::1}{[::1]:9300}
[2016-06-07 23:36:08,377][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] replacing {#cloud-i-5889eded-0}{10.0.150.235}{10.0.150.235:9300} with temp node {#zen_unicast_7_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
[2016-06-07 23:36:08,377][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] replacing {#cloud-i-e70acb70-0}{10.0.151.127}{10.0.151.127:9300} with temp node {#zen_unicast_8_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
[2016-06-07 23:36:08,377][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] sending to {Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}
[2016-06-07 23:36:08,377][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] connecting (light) to {#zen_unicast_7_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
[2016-06-07 23:36:08,380][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] received response from {#zen_unicast_1#}{127.0.0.1}{127.0.0.1:9300}: [ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[9], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[9], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[11], master [null], hasJoinedOnce [false], cluster_name[hihi]}]
[2016-06-07 23:36:08,379][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] received response from {Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}: [ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[9], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[10], master [null], hasJoinedOnce [false], cluster_name[hihi]}]
[2016-06-07 23:36:08,377][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] connecting (light) to {#zen_unicast_8_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
[2016-06-07 23:36:08,382][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] failed to connect to {#zen_unicast_8_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
ConnectTransportException[[][10.0.151.127:9300] connect_timeout[30s]]; nested: ConnectException[Connection refused: /10.0.151.127:9300];
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:952)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:916)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:888)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:267)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:395)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: /10.0.151.127:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2016-06-07 23:36:08,384][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] failed to connect to {#zen_unicast_7_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
ConnectTransportException[[][10.0.150.235:9300] connect_timeout[30s]]; nested: ConnectException[Connection refused: /10.0.150.235:9300];
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:952)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:916)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:888)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:267)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:395)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: /10.0.150.235:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2016-06-07 23:36:08,383][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] received response from {#zen_unicast_2#}{::1}{[::1]:9300}: [ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[9], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[9], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[9], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}], id[12], master [null], hasJoinedOnce [false], cluster_name[hihi]}]
[2016-06-07 23:36:08,390][TRACE][discovery.ec2 ] [Doctor Spectrum] full ping responses: {none}
[2016-06-07 23:36:08,390][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] disconnecting from {#zen_unicast_3_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
[2016-06-07 23:36:08,390][DEBUG][discovery.ec2 ] [Doctor Spectrum] filtered ping responses: (filter_client[true], filter_data[false]) {none}
[2016-06-07 23:36:08,390][DEBUG][discovery.ec2 ] [Doctor Spectrum] elected as master, waiting for incoming joins ([0] needed)
[2016-06-07 23:36:08,392][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] disconnecting from {#zen_unicast_2#}{::1}{[::1]:9300}
[2016-06-07 23:36:08,394][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] disconnecting from {#zen_unicast_1#}{127.0.0.1}{127.0.0.1:9300}
[2016-06-07 23:36:08,395][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] disconnecting from {#zen_unicast_6_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
[2016-06-07 23:36:08,397][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] disconnecting from {#zen_unicast_7_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
[2016-06-07 23:36:08,400][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] disconnecting from {#zen_unicast_8_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
[2016-06-07 23:36:08,400][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] disconnecting from {#zen_unicast_4_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
[2016-06-07 23:36:08,400][TRACE][discovery.zen.ping.unicast] [Doctor Spectrum] [1] disconnecting from {#zen_unicast_5_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
[2016-06-07 23:36:08,405][INFO ][cluster.service ] [Doctor Spectrum] new_master {Doctor Spectrum}{T326441uQgiodwOXCsoYMQ}{10.0.151.127}{10.0.151.127:32111}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-06-07 23:36:08,408][TRACE][discovery.zen ] [Doctor Spectrum] stopping join accumulation ([election closed])
[2016-06-07 23:36:08,409][TRACE][discovery.ec2 ] [Doctor Spectrum] cluster joins counter set to [1] (elected as master)
[2016-06-07 23:36:08,427][INFO ][http ] [Doctor Spectrum] publish_address {10.0.151.127:9200}, bound_addresses {[::]:9200}
[2016-06-07 23:36:08,428][INFO ][node ] [Doctor Spectrum] started
[2016-06-07 23:36:08,454][INFO ][gateway ] [Doctor Spectrum] recovered [0] indices into cluster_state
Raw
es-node-gateway.log
[2016-06-07 23:35:54,979][INFO ][node ] [Gateway] version[2.3.3], pid[1], build[218bdf1/2016-05-17T15:40:04Z]
[2016-06-07 23:35:54,982][INFO ][node ] [Gateway] initializing ...
[2016-06-07 23:35:56,307][INFO ][plugins ] [Gateway] modules [reindex, lang-expression, lang-groovy], plugins [cloud-aws], sites []
[2016-06-07 23:35:56,369][INFO ][env ] [Gateway] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/xvda1)]], net usable_space [23.4gb], net total_space [29.4gb], spins? [possibly], types [ext4]
[2016-06-07 23:35:56,370][INFO ][env ] [Gateway] heap size [1007.3mb], compressed ordinary object pointers [true]
[2016-06-07 23:35:56,370][WARN ][env ] [Gateway] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at least [65536]
[2016-06-07 23:35:58,765][DEBUG][discovery.zen.elect ] [Gateway] using minimum_master_nodes [-1]
[2016-06-07 23:35:59,416][DEBUG][discovery.ec2 ] [Gateway] using host_type [PRIVATE_IP], tags [{}], groups [[sg-20d26c44]] with any_group [false], availability_zones [[]]
[2016-06-07 23:35:59,421][DEBUG][discovery.zen.ping.unicast] [Gateway] using initial hosts [127.0.0.1, [::1]], with concurrent_connects [10]
[2016-06-07 23:35:59,437][DEBUG][discovery.ec2 ] [Gateway] using ping.timeout [3s], join.timeout [1m], master_election.filter_client [true], master_election.filter_data [false]
[2016-06-07 23:35:59,440][DEBUG][discovery.zen.fd ] [Gateway] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2016-06-07 23:35:59,443][DEBUG][discovery.zen.fd ] [Gateway] [node ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2016-06-07 23:36:00,194][DEBUG][discovery.ec2 ] [Gateway] using host_type [PRIVATE_IP], tags [{}], groups [[sg-20d26c44]] with any_group [false], availability_zones [[]]
[2016-06-07 23:36:00,195][DEBUG][discovery.ec2 ] [Gateway] using host_type [PRIVATE_IP], tags [{}], groups [[sg-20d26c44]] with any_group [false], availability_zones [[]]
[2016-06-07 23:36:00,202][INFO ][node ] [Gateway] initialized
[2016-06-07 23:36:00,202][INFO ][node ] [Gateway] starting ...
[2016-06-07 23:36:00,309][INFO ][transport ] [Gateway] publish_address {10.0.150.235:32111}, bound_addresses {[::]:9300}
[2016-06-07 23:36:00,316][INFO ][discovery ] [Gateway] hihi/VF45f0A9RMOgAlD0uLbQhg
[2016-06-07 23:36:00,329][TRACE][discovery.zen ] [Gateway] starting to accumulate joins
[2016-06-07 23:36:00,329][TRACE][discovery.ec2 ] [Gateway] starting to ping
[2016-06-07 23:36:00,350][WARN ][com.amazonaws.jmx.SdkMBeanRegistrySupport]
java.security.AccessControlException: access denied ("javax.management.MBeanServerPermission" "findMBeanServer")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at java.security.AccessController.checkPermission(AccessController.java:884)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at javax.management.MBeanServerFactory.checkPermission(MBeanServerFactory.java:413)
at javax.management.MBeanServerFactory.findMBeanServer(MBeanServerFactory.java:361)
at com.amazonaws.jmx.MBeans.getMBeanServer(MBeans.java:111)
at com.amazonaws.jmx.MBeans.registerMBean(MBeans.java:50)
at com.amazonaws.jmx.SdkMBeanRegistrySupport.registerMetricAdminMBean(SdkMBeanRegistrySupport.java:27)
at com.amazonaws.metrics.AwsSdkMetrics.registerMetricAdminMBean(AwsSdkMetrics.java:355)
at com.amazonaws.metrics.AwsSdkMetrics.<clinit>(AwsSdkMetrics.java:316)
at com.amazonaws.AmazonWebServiceClient.requestMetricCollector(AmazonWebServiceClient.java:563)
at com.amazonaws.AmazonWebServiceClient.isRMCEnabledAtClientOrSdkLevel(AmazonWebServiceClient.java:504)
at com.amazonaws.AmazonWebServiceClient.isRequestMetricsEnabled(AmazonWebServiceClient.java:496)
at com.amazonaws.AmazonWebServiceClient.createExecutionContext(AmazonWebServiceClient.java:457)
at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5924)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider.fetchDynamicNodes(AwsEc2UnicastHostsProvider.java:118)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider$DiscoNodesCache.refresh(AwsEc2UnicastHostsProvider.java:230)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider$DiscoNodesCache.refresh(AwsEc2UnicastHostsProvider.java:215)
at org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:55)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider.buildDynamicNodes(AwsEc2UnicastHostsProvider.java:104)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.sendPings(UnicastZenPing.java:335)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.ping(UnicastZenPing.java:240)
at org.elasticsearch.discovery.zen.ping.ZenPingService.ping(ZenPingService.java:106)
at org.elasticsearch.discovery.zen.ping.ZenPingService.pingAndWait(ZenPingService.java:84)
at org.elasticsearch.discovery.zen.ZenDiscovery.findMaster(ZenDiscovery.java:886)
at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:350)
at org.elasticsearch.discovery.zen.ZenDiscovery.access$4800(ZenDiscovery.java:91)
at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1237)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-06-07 23:36:00,356][WARN ][com.amazonaws.metrics.AwsSdkMetrics]
java.security.AccessControlException: access denied ("javax.management.MBeanServerPermission" "findMBeanServer")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at java.security.AccessController.checkPermission(AccessController.java:884)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at javax.management.MBeanServerFactory.checkPermission(MBeanServerFactory.java:413)
at javax.management.MBeanServerFactory.findMBeanServer(MBeanServerFactory.java:361)
at com.amazonaws.jmx.MBeans.getMBeanServer(MBeans.java:111)
at com.amazonaws.jmx.MBeans.isRegistered(MBeans.java:98)
at com.amazonaws.jmx.SdkMBeanRegistrySupport.isMBeanRegistered(SdkMBeanRegistrySupport.java:46)
at com.amazonaws.metrics.AwsSdkMetrics.registerMetricAdminMBean(AwsSdkMetrics.java:361)
at com.amazonaws.metrics.AwsSdkMetrics.<clinit>(AwsSdkMetrics.java:316)
at com.amazonaws.AmazonWebServiceClient.requestMetricCollector(AmazonWebServiceClient.java:563)
at com.amazonaws.AmazonWebServiceClient.isRMCEnabledAtClientOrSdkLevel(AmazonWebServiceClient.java:504)
at com.amazonaws.AmazonWebServiceClient.isRequestMetricsEnabled(AmazonWebServiceClient.java:496)
at com.amazonaws.AmazonWebServiceClient.createExecutionContext(AmazonWebServiceClient.java:457)
at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5924)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider.fetchDynamicNodes(AwsEc2UnicastHostsProvider.java:118)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider$DiscoNodesCache.refresh(AwsEc2UnicastHostsProvider.java:230)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider$DiscoNodesCache.refresh(AwsEc2UnicastHostsProvider.java:215)
at org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:55)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider.buildDynamicNodes(AwsEc2UnicastHostsProvider.java:104)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.sendPings(UnicastZenPing.java:335)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.ping(UnicastZenPing.java:240)
at org.elasticsearch.discovery.zen.ping.ZenPingService.ping(ZenPingService.java:106)
at org.elasticsearch.discovery.zen.ping.ZenPingService.pingAndWait(ZenPingService.java:84)
at org.elasticsearch.discovery.zen.ZenDiscovery.findMaster(ZenDiscovery.java:886)
at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:350)
at org.elasticsearch.discovery.zen.ZenDiscovery.access$4800(ZenDiscovery.java:91)
at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1237)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-06-07 23:36:02,990][TRACE][discovery.ec2 ] [Gateway] building dynamic unicast discovery nodes...
[2016-06-07 23:36:02,992][TRACE][discovery.ec2 ] [Gateway] filtering out instance i-2d049498 based on groups [{GroupName: ecs-production-ECSNodeSG-K9ZR984A5E5Q,GroupId: sg-868a27e2}, {GroupName: RedisClientSG,GroupId: sg-795fbd16}, {GroupName: PostgresClientSG,GroupId: sg-a82af8cd}, {GroupName: GearmanClientSG,GroupId: sg-4f140b23}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:02,992][TRACE][discovery.ec2 ] [Gateway] filtering out instance i-2c049499 based on groups [{GroupName: ecs-production-ECSNodeSG-K9ZR984A5E5Q,GroupId: sg-868a27e2}, {GroupName: RedisClientSG,GroupId: sg-795fbd16}, {GroupName: PostgresClientSG,GroupId: sg-a82af8cd}, {GroupName: GearmanClientSG,GroupId: sg-4f140b23}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:02,995][TRACE][discovery.ec2 ] [Gateway] filtering out instance i-2e04949b based on groups [{GroupName: ecs-production-ECSNodeSG-K9ZR984A5E5Q,GroupId: sg-868a27e2}, {GroupName: RedisClientSG,GroupId: sg-795fbd16}, {GroupName: PostgresClientSG,GroupId: sg-a82af8cd}, {GroupName: GearmanClientSG,GroupId: sg-4f140b23}], does not include all of [sg-20d26c44]
.... snipped filtering out a bunch of machines ....
[2016-06-07 23:36:03,054][TRACE][discovery.ec2 ] [Gateway] filtering out instance i-29feb59b based on groups [{GroupName: DBServerSG,GroupId: sg-f5a5b999}, {GroupName: SaltMinionSG,GroupId: sg-6ad0cf06}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:03,054][TRACE][discovery.ec2 ] [Gateway] filtering out instance i-0ce772b9 based on groups [{GroupName: ecs-production-ECSNodeSG-K9ZR984A5E5Q,GroupId: sg-868a27e2}, {GroupName: RedisClientSG,GroupId: sg-795fbd16}, {GroupName: PostgresClientSG,GroupId: sg-a82af8cd}, {GroupName: GearmanClientSG,GroupId: sg-4f140b23}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:03,054][TRACE][discovery.ec2 ] [Gateway] filtering out instance i-0fe772ba based on groups [{GroupName: ecs-production-ECSNodeSG-K9ZR984A5E5Q,GroupId: sg-868a27e2}, {GroupName: RedisClientSG,GroupId: sg-795fbd16}, {GroupName: PostgresClientSG,GroupId: sg-a82af8cd}, {GroupName: GearmanClientSG,GroupId: sg-4f140b23}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:03,055][TRACE][discovery.ec2 ] [Gateway] adding i-5889eded, address 10.0.150.235, transport_address 10.0.150.235:9300
[2016-06-07 23:36:03,056][TRACE][discovery.ec2 ] [Gateway] filtering out instance i-de52631c based on groups [{GroupName: LoggingTestSG,GroupId: sg-be435ad2}, {GroupName: SaltMinionSG,GroupId: sg-6ad0cf06}, {GroupName: Staging,GroupId: sg-78b5571d}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:03,056][TRACE][discovery.ec2 ] [Gateway] filtering out instance i-6b830231 based on groups [{GroupName: SaltMasterSG,GroupId: sg-6bd0cf07}, {GroupName: CarbonClientSG,GroupId: sg-4ef3ec22}, {GroupName: SaltMinionSG,GroupId: sg-6ad0cf06}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:03,056][TRACE][discovery.ec2 ] [Gateway] filtering out instance i-5b32f3e9 based on groups [{GroupName: SupervisorSG,GroupId: sg-9732fff2}, {GroupName: HaproxyHealthCheck,GroupId: sg-b394fdd6}, {GroupName: SaltMinionSG,GroupId: sg-6ad0cf06}, {GroupName: Staging,GroupId: sg-78b5571d}], does not include all of [sg-20d26c44]
.... snipped filtering out a bunch of machines ....
[2016-06-07 23:36:03,085][TRACE][discovery.ec2 ] [Gateway] filtering out instance i-1010ebd3 based on groups [{GroupName: RedisServerSG,GroupId: sg-775fbd18}, {GroupName: RedisClientSG,GroupId: sg-795fbd16}, {GroupName: SaltMinionSG,GroupId: sg-6ad0cf06}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:03,085][TRACE][discovery.ec2 ] [Gateway] filtering out instance i-c1b66d74 based on groups [{GroupName: LoggingTestSG,GroupId: sg-be435ad2}, {GroupName: SaltMinionSG,GroupId: sg-6ad0cf06}, {GroupName: Staging,GroupId: sg-78b5571d}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:03,085][TRACE][discovery.ec2 ] [Gateway] filtering out instance i-9d90895d based on groups [{GroupName: DBServerSG,GroupId: sg-f5a5b999}, {GroupName: SaltMinionSG,GroupId: sg-6ad0cf06}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:03,086][TRACE][discovery.ec2 ] [Gateway] adding i-e70acb70, address 10.0.151.127, transport_address 10.0.151.127:9300
[2016-06-07 23:36:03,086][TRACE][discovery.ec2 ] [Gateway] filtering out instance i-f9e7276e based on groups [{GroupName: DBServerDevSG,GroupId: sg-4b233d27}, {GroupName: SaltMinionSG,GroupId: sg-6ad0cf06}, {GroupName: Staging,GroupId: sg-78b5571d}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:03,086][TRACE][discovery.ec2 ] [Gateway] filtering out instance i-b6fdb604 based on groups [{GroupName: DBServerSG,GroupId: sg-f5a5b999}, {GroupName: SaltMinionSG,GroupId: sg-6ad0cf06}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:03,087][TRACE][discovery.ec2 ] [Gateway] filtering out instance i-867aff33 based on groups [{GroupName: Docker,GroupId: sg-fb8b8599}, {GroupName: SaltMinionSG,GroupId: sg-6ad0cf06}], does not include all of [sg-20d26c44]
.... snipped filtering out a bunch of machines ....
[2016-06-07 23:36:03,090][TRACE][discovery.ec2 ] [Gateway] filtering out instance i-27a6c492 based on groups [{GroupName: ecs-loadtest-ECSNodeSG-1I91YTXN1CZQC,GroupId: sg-898726ed}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:03,090][TRACE][discovery.ec2 ] [Gateway] filtering out instance i-2ba6c49e based on groups [{GroupName: ecs-loadtest-ECSNodeSG-1I91YTXN1CZQC,GroupId: sg-898726ed}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:03,091][TRACE][discovery.ec2 ] [Gateway] filtering out instance i-2aa6c49f based on groups [{GroupName: ecs-loadtest-ECSNodeSG-1I91YTXN1CZQC,GroupId: sg-898726ed}], does not include all of [sg-20d26c44]
[2016-06-07 23:36:03,091][DEBUG][discovery.ec2 ] [Gateway] using dynamic discovery nodes [{#cloud-i-5889eded-0}{10.0.150.235}{10.0.150.235:9300}, {#cloud-i-e70acb70-0}{10.0.151.127}{10.0.151.127:9300}]
[2016-06-07 23:36:03,094][TRACE][discovery.zen.ping.unicast] [Gateway] replacing {#cloud-i-5889eded-0}{10.0.150.235}{10.0.150.235:9300} with temp node {#zen_unicast_3_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
[2016-06-07 23:36:03,094][TRACE][discovery.zen.ping.unicast] [Gateway] [1] connecting (light) to {#zen_unicast_2#}{::1}{[::1]:9300}
[2016-06-07 23:36:03,094][TRACE][discovery.zen.ping.unicast] [Gateway] [1] connecting (light) to {#zen_unicast_1#}{127.0.0.1}{127.0.0.1:9300}
[2016-06-07 23:36:03,095][TRACE][discovery.zen.ping.unicast] [Gateway] replacing {#cloud-i-e70acb70-0}{10.0.151.127}{10.0.151.127:9300} with temp node {#zen_unicast_4_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
[2016-06-07 23:36:03,096][TRACE][discovery.zen.ping.unicast] [Gateway] [1] connecting (light) to {#zen_unicast_3_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
[2016-06-07 23:36:03,100][TRACE][discovery.zen.ping.unicast] [Gateway] [1] sending to {Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}
[2016-06-07 23:36:03,106][TRACE][discovery.zen.ping.unicast] [Gateway] [1] received response from {Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}: [ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[2], master [null], hasJoinedOnce [false], cluster_name[hihi]}]
[2016-06-07 23:36:03,111][TRACE][discovery.zen.ping.unicast] [Gateway] [1] connecting (light) to {#zen_unicast_4_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
[2016-06-07 23:36:03,135][TRACE][discovery.zen.ping.unicast] [Gateway] [1] failed to connect to {#zen_unicast_4_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
ConnectTransportException[[][10.0.151.127:9300] connect_timeout[30s]]; nested: ConnectException[Connection refused: /10.0.151.127:9300];
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:952)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:916)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:888)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:267)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:395)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: /10.0.151.127:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2016-06-07 23:36:03,141][TRACE][discovery.zen.ping.unicast] [Gateway] [1] failed to connect to {#zen_unicast_3_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
ConnectTransportException[[][10.0.150.235:9300] connect_timeout[30s]]; nested: ConnectException[Connection refused: /10.0.150.235:9300];
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:952)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:916)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:888)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:267)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:395)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: /10.0.150.235:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2016-06-07 23:36:03,146][TRACE][discovery.zen.ping.unicast] [Gateway] [1] connected to {#zen_unicast_1#}{127.0.0.1}{127.0.0.1:9300}
[2016-06-07 23:36:03,146][TRACE][discovery.zen.ping.unicast] [Gateway] [1] sending to {#zen_unicast_1#}{127.0.0.1}{127.0.0.1:9300}
[2016-06-07 23:36:03,146][TRACE][discovery.zen.ping.unicast] [Gateway] [1] connected to {#zen_unicast_2#}{::1}{[::1]:9300}
[2016-06-07 23:36:03,150][TRACE][discovery.zen.ping.unicast] [Gateway] [1] sending to {#zen_unicast_2#}{::1}{[::1]:9300}
[2016-06-07 23:36:03,178][TRACE][discovery.zen.ping.unicast] [Gateway] [1] received response from {#zen_unicast_1#}{127.0.0.1}{127.0.0.1:9300}: [ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[3], master [null], hasJoinedOnce [false], cluster_name[hihi]}]
[2016-06-07 23:36:03,180][TRACE][discovery.zen.ping.unicast] [Gateway] [1] received response from {#zen_unicast_2#}{::1}{[::1]:9300}: [ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[4], master [null], hasJoinedOnce [false], cluster_name[hihi]}]
[2016-06-07 23:36:04,608][TRACE][discovery.zen.ping.unicast] [Gateway] [1] sending to {#zen_unicast_1#}{127.0.0.1}{127.0.0.1:9300}
[2016-06-07 23:36:04,609][TRACE][discovery.zen.ping.unicast] [Gateway] [1] sending to {#zen_unicast_2#}{::1}{[::1]:9300}
[2016-06-07 23:36:04,610][TRACE][discovery.zen.ping.unicast] [Gateway] replacing {#cloud-i-5889eded-0}{10.0.150.235}{10.0.150.235:9300} with temp node {#zen_unicast_5_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
[2016-06-07 23:36:04,610][TRACE][discovery.zen.ping.unicast] [Gateway] [1] connecting (light) to {#zen_unicast_5_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
[2016-06-07 23:36:04,612][TRACE][discovery.zen.ping.unicast] [Gateway] replacing {#cloud-i-e70acb70-0}{10.0.151.127}{10.0.151.127:9300} with temp node {#zen_unicast_6_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
[2016-06-07 23:36:04,613][TRACE][discovery.zen.ping.unicast] [Gateway] [1] connecting (light) to {#zen_unicast_6_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
[2016-06-07 23:36:04,613][TRACE][discovery.zen.ping.unicast] [Gateway] [1] failed to connect to {#zen_unicast_5_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
ConnectTransportException[[][10.0.150.235:9300] connect_timeout[30s]]; nested: ConnectException[Connection refused: /10.0.150.235:9300];
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:952)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:916)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:888)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:267)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:395)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: /10.0.150.235:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2016-06-07 23:36:04,613][TRACE][discovery.zen.ping.unicast] [Gateway] [1] sending to {Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}
[2016-06-07 23:36:04,614][TRACE][discovery.zen.ping.unicast] [Gateway] [1] received response from {#zen_unicast_1#}{127.0.0.1}{127.0.0.1:9300}: [ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[6], master [null], hasJoinedOnce [false], cluster_name[hihi]}]
[2016-06-07 23:36:04,615][TRACE][discovery.zen.ping.unicast] [Gateway] [1] received response from {Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}: [ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[8], master [null], hasJoinedOnce [false], cluster_name[hihi]}]
[2016-06-07 23:36:04,618][TRACE][discovery.zen.ping.unicast] [Gateway] [1] received response from {#zen_unicast_2#}{::1}{[::1]:9300}: [ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[7], master [null], hasJoinedOnce [false], cluster_name[hihi]}]
[2016-06-07 23:36:04,615][TRACE][discovery.zen.ping.unicast] [Gateway] [1] failed to connect to {#zen_unicast_6_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
ConnectTransportException[[][10.0.151.127:9300] connect_timeout[30s]]; nested: ConnectException[Connection refused: /10.0.151.127:9300];
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:952)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:916)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:888)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:267)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:395)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: /10.0.151.127:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2016-06-07 23:36:06,119][TRACE][discovery.zen.ping.unicast] [Gateway] [1] sending to {#zen_unicast_1#}{127.0.0.1}{127.0.0.1:9300}
[2016-06-07 23:36:06,120][TRACE][discovery.zen.ping.unicast] [Gateway] [1] sending to {#zen_unicast_2#}{::1}{[::1]:9300}
[2016-06-07 23:36:06,121][TRACE][discovery.zen.ping.unicast] [Gateway] replacing {#cloud-i-5889eded-0}{10.0.150.235}{10.0.150.235:9300} with temp node {#zen_unicast_7_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
[2016-06-07 23:36:06,121][TRACE][discovery.zen.ping.unicast] [Gateway] [1] connecting (light) to {#zen_unicast_7_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
[2016-06-07 23:36:06,122][TRACE][discovery.zen.ping.unicast] [Gateway] replacing {#cloud-i-e70acb70-0}{10.0.151.127}{10.0.151.127:9300} with temp node {#zen_unicast_8_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
[2016-06-07 23:36:06,122][TRACE][discovery.zen.ping.unicast] [Gateway] [1] sending to {Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}
[2016-06-07 23:36:06,123][TRACE][discovery.zen.ping.unicast] [Gateway] [1] received response from {#zen_unicast_1#}{127.0.0.1}{127.0.0.1:9300}: [ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[9], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[10], master [null], hasJoinedOnce [false], cluster_name[hihi]}]
[2016-06-07 23:36:06,123][TRACE][discovery.zen.ping.unicast] [Gateway] [1] received response from {Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}: [ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[9], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[9], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[11], master [null], hasJoinedOnce [false], cluster_name[hihi]}]
[2016-06-07 23:36:06,123][TRACE][discovery.zen.ping.unicast] [Gateway] [1] connecting (light) to {#zen_unicast_8_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
[2016-06-07 23:36:06,125][TRACE][discovery.zen.ping.unicast] [Gateway] [1] received response from {#zen_unicast_2#}{::1}{[::1]:9300}: [ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[1], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[5], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[9], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[9], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[9], master [null], hasJoinedOnce [false], cluster_name[hihi]}, ping_response{node [{Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}], id[12], master [null], hasJoinedOnce [false], cluster_name[hihi]}]
[2016-06-07 23:36:06,124][TRACE][discovery.zen.ping.unicast] [Gateway] [1] failed to connect to {#zen_unicast_7_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
ConnectTransportException[[][10.0.150.235:9300] connect_timeout[30s]]; nested: ConnectException[Connection refused: /10.0.150.235:9300];
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:952)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:916)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:888)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:267)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:395)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: /10.0.150.235:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2016-06-07 23:36:06,131][TRACE][discovery.zen.ping.unicast] [Gateway] [1] failed to connect to {#zen_unicast_8_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
ConnectTransportException[[][10.0.151.127:9300] connect_timeout[30s]]; nested: ConnectException[Connection refused: /10.0.151.127:9300];
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:952)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:916)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:888)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:267)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:395)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: /10.0.151.127:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2016-06-07 23:36:06,132][TRACE][discovery.ec2 ] [Gateway] full ping responses: {none}
[2016-06-07 23:36:06,132][TRACE][discovery.zen.ping.unicast] [Gateway] [1] disconnecting from {#zen_unicast_3_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
[2016-06-07 23:36:06,132][DEBUG][discovery.ec2 ] [Gateway] filtered ping responses: (filter_client[true], filter_data[false]) {none}
[2016-06-07 23:36:06,133][DEBUG][discovery.ec2 ] [Gateway] elected as master, waiting for incoming joins ([0] needed)
[2016-06-07 23:36:06,142][TRACE][discovery.zen.ping.unicast] [Gateway] [1] disconnecting from {#zen_unicast_2#}{::1}{[::1]:9300}
[2016-06-07 23:36:06,144][INFO ][cluster.service ] [Gateway] new_master {Gateway}{VF45f0A9RMOgAlD0uLbQhg}{10.0.150.235}{10.0.150.235:32111}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-06-07 23:36:06,148][TRACE][discovery.zen.ping.unicast] [Gateway] [1] disconnecting from {#zen_unicast_1#}{127.0.0.1}{127.0.0.1:9300}
[2016-06-07 23:36:06,153][TRACE][discovery.zen ] [Gateway] stopping join accumulation ([election closed])
[2016-06-07 23:36:06,154][TRACE][discovery.zen.ping.unicast] [Gateway] [1] disconnecting from {#zen_unicast_6_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
[2016-06-07 23:36:06,154][TRACE][discovery.zen.ping.unicast] [Gateway] [1] disconnecting from {#zen_unicast_7_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
[2016-06-07 23:36:06,154][TRACE][discovery.zen.ping.unicast] [Gateway] [1] disconnecting from {#zen_unicast_8_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
[2016-06-07 23:36:06,154][TRACE][discovery.zen.ping.unicast] [Gateway] [1] disconnecting from {#zen_unicast_4_#cloud-i-e70acb70-0#}{10.0.151.127}{10.0.151.127:9300}
[2016-06-07 23:36:06,154][TRACE][discovery.zen.ping.unicast] [Gateway] [1] disconnecting from {#zen_unicast_5_#cloud-i-5889eded-0#}{10.0.150.235}{10.0.150.235:9300}
[2016-06-07 23:36:06,156][TRACE][discovery.ec2 ] [Gateway] cluster joins counter set to [1] (elected as master)
[2016-06-07 23:36:06,187][INFO ][http ] [Gateway] publish_address {10.0.150.235:9200}, bound_addresses {[::]:9200}
[2016-06-07 23:36:06,188][INFO ][node ] [Gateway] started
[2016-06-07 23:36:06,206][INFO ][gateway ] [Gateway] recovered [0] indices into cluster_state
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment