Skip to content

Instantly share code, notes, and snippets.

@dav-rob

dav-rob/Beta2StackTrace

Created January 7, 2014 18:29
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save dav-rob/8304130 to your computer and use it in GitHub Desktop.
Save dav-rob/8304130 to your computer and use it in GitHub Desktop.
Download ZIP
Stack Trace from Java Client Failure to Connect.
Raw
Beta2StackTrace
Jan 07, 2014 6:15:15 PM util.ESNodeWrapper init
INFO:
*****************************************
clusteName[CMTDEV1]
multicast[false]
host[166.15.121.56]
hosts[ES_HOSTS=vcll36a-1001.equity.csfb.com:9400,vcll36a-1001.equity.csfb.com:9401,vcll36a-1001.equity.csfb.com:9402,vcll36a-1001.equity.csfb.com:9403,vcll36a-1001.equity.csfb.com:9404,vcll36a-1001.equity.csfb.com:9405,vcll36a-1001.equity.csfb.com:9406,vcll36a-1001.equity.csfb.com:9407,vcll36a-1001.equity.csfb.com:9408,vcll36a-1001.equity.csfb.com:9409,vcll36a-1001.equity.csfb.com:9410]
nodeName[ testNode ]*****************************************
Jan 07, 2014 6:15:15 PM util.ESNodeWrapper init
INFO:
******************* Settings **********************
{index.number_of_shards=1, index.number_of_replicas=0, cluster.name=CMTDEV1, node.client=true, index.mapper=dynamic:false, node.data=false, transport.tcp.port=9400-9450, http.port=9451-9500, compress.lzf.decoder=safe, node.name=testNode, network.host=166.15.121.56, discovery.zen.ping.unicast.enabled=true, http.enabled=true, discovery.zen.ping.unicast.hosts=ES_HOSTS=vcll36a-1001.equity.csfb.com:9400,vcll36a-1001.equity.csfb.com:9401,vcll36a-1001.equity.csfb.com:9402,vcll36a-1001.equity.csfb.com:9403,vcll36a-1001.equity.csfb.com:9404,vcll36a-1001.equity.csfb.com:9405,vcll36a-1001.equity.csfb.com:9406,vcll36a-1001.equity.csfb.com:9407,vcll36a-1001.equity.csfb.com:9408,vcll36a-1001.equity.csfb.com:9409,vcll36a-1001.equity.csfb.com:9410, discovery.zen.ping.multicast.enabled=false}
****************************************************
Jan 07, 2014 6:15:15 PM util.ESNodeWrapper init
INFO: Before Node build for cluster[CMTDEV1]
2014-01-07 18:15:16,283 INFO - [testNode] version[1.0.0.Beta2], pid[21796], build[296cfbe/2013-12-02T15:46:27Z]
2014-01-07 18:15:16,284 INFO - [testNode] initializing ...
2014-01-07 18:15:16,284 DEBUG - [testNode] using home [C:\work\kepler\wksp1\test-Beta2], config [C:\work\kepler\wksp1\test-Beta2\config], data [[C:\work\kepler\wksp1\test-Beta2\data]], logs [C:\work\kepler\wksp1\test-Beta2\logs], work [C:\work\kepler\wksp1\test-Beta2\work], plugins [C:\work\kepler\wksp1\test-Beta2\plugins]
2014-01-07 18:15:16,309 INFO - [testNode] loaded [], sites []
2014-01-07 18:15:16,405 DEBUG - using [UnsafeChunkDecoder] decoder
2014-01-07 18:15:16,405 DEBUG - using [VanillaChunkDecoder] decoder
2014-01-07 18:15:21,370 DEBUG - [testNode] creating thread_pool [generic], type [cached], keep_alive [30s]
2014-01-07 18:15:21,401 DEBUG - [testNode] creating thread_pool [index], type [fixed], size [8], queue_size [200]
2014-01-07 18:15:21,413 DEBUG - [testNode] creating thread_pool [bulk], type [fixed], size [8], queue_size [50]
2014-01-07 18:15:21,413 DEBUG - [testNode] creating thread_pool [get], type [fixed], size [8], queue_size [1k]
2014-01-07 18:15:21,413 DEBUG - [testNode] creating thread_pool [search], type [fixed], size [24], queue_size [1k]
2014-01-07 18:15:21,413 DEBUG - [testNode] creating thread_pool [suggest], type [fixed], size [8], queue_size [1k]
2014-01-07 18:15:21,414 DEBUG - [testNode] creating thread_pool [percolate], type [fixed], size [8], queue_size [1k]
2014-01-07 18:15:21,414 DEBUG - [testNode] creating thread_pool [management], type [scaling], min [1], size [5], keep_alive [5m]
2014-01-07 18:15:21,415 DEBUG - [testNode] creating thread_pool [flush], type [scaling], min [1], size [4], keep_alive [5m]
2014-01-07 18:15:21,415 DEBUG - [testNode] creating thread_pool [merge], type [scaling], min [1], size [4], keep_alive [5m]
2014-01-07 18:15:21,415 DEBUG - [testNode] creating thread_pool [refresh], type [scaling], min [1], size [4], keep_alive [5m]
2014-01-07 18:15:21,415 DEBUG - [testNode] creating thread_pool [warmer], type [scaling], min [1], size [4], keep_alive [5m]
2014-01-07 18:15:21,415 DEBUG - [testNode] creating thread_pool [snapshot], type [scaling], min [1], size [4], keep_alive [5m]
2014-01-07 18:15:21,415 DEBUG - [testNode] creating thread_pool [optimize], type [fixed], size [1], queue_size [null]
2014-01-07 18:15:21,455 DEBUG - [testNode] using worker_count[16], port[9400-9450], bind_host[null], publish_host[null], compress[false], connect_timeout[30s], connections_per_node[2/3/6/1/1], receive_predictor[512kb->512kb]
2014-01-07 18:15:21,475 DEBUG - [testNode] using initial hosts [ES_HOSTS=vcll36a-1001.equity.csfb.com:9400, vcll36a-1001.equity.csfb.com:9401, vcll36a-1001.equity.csfb.com:9402, vcll36a-1001.equity.csfb.com:9403, vcll36a-1001.equity.csfb.com:9404, vcll36a-1001.equity.csfb.com:9405, vcll36a-1001.equity.csfb.com:9406, vcll36a-1001.equity.csfb.com:9407, vcll36a-1001.equity.csfb.com:9408, vcll36a-1001.equity.csfb.com:9409, vcll36a-1001.equity.csfb.com:9410], with concurrent_connects [10]
2014-01-07 18:15:21,483 DEBUG - [testNode] using ping.timeout [3s], master_election.filter_client [true], master_election.filter_data [false]
2014-01-07 18:15:21,484 DEBUG - [testNode] using minimum_master_nodes [-1]
2014-01-07 18:15:21,485 DEBUG - [testNode] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
2014-01-07 18:15:21,527 DEBUG - [testNode] [node ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
2014-01-07 18:15:21,581 DEBUG - [testNode] enabled [true], last_gc_enabled [false], interval [1s], gc_threshold [{default=GcThreshold{name='default', warnThreshold=10000, infoThreshold=5000, debugThreshold=2000}, ParNew=GcThreshold{name='ParNew', warnThreshold=1000, infoThreshold=700, debugThreshold=400}, ConcurrentMarkSweep=GcThreshold{name='ConcurrentMarkSweep', warnThreshold=10000, infoThreshold=5000, debugThreshold=2000}}]
2014-01-07 18:15:21,582 DEBUG - [testNode] Using probe [org.elasticsearch.monitor.os.JmxOsProbe@6aacb24d] with refresh_interval [1s]
2014-01-07 18:15:21,584 DEBUG - [testNode] Using probe [org.elasticsearch.monitor.process.JmxProcessProbe@5bbe870] with refresh_interval [1s]
2014-01-07 18:15:21,597 DEBUG - [testNode] Using refresh_interval [1s]
2014-01-07 18:15:21,597 DEBUG - [testNode] Using probe [org.elasticsearch.monitor.network.JmxNetworkProbe@2f5d6f48] with refresh_interval [5s]
2014-01-07 18:15:21,994 DEBUG - [testNode] net_info
host [GBD03821812]
lo display_name [Software Loopback Interface 1]
address [/127.0.0.1] [/0:0:0:0:0:0:0:1]
mtu [-1] multicast [true] ptp [false] loopback [true] up [true] virtual [false]
net0 display_name [WAN Miniport (SSTP)]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
net1 display_name [WAN Miniport (L2TP)]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
net2 display_name [WAN Miniport (PPTP)]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
ppp0 display_name [WAN Miniport (PPPOE)]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
eth0 display_name [WAN Miniport (IPv6)]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
eth1 display_name [WAN Miniport (Network Monitor)]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
eth2 display_name [WAN Miniport (IP)]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
ppp1 display_name [RAS Async Adapter]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
net3 display_name [WAN Miniport (IKEv2)]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
eth3 display_name [Broadcom NetXtreme Gigabit Ethernet]
address [/166.15.121.56]
mtu [1500] multicast [true] ptp [false] loopback [false] up [true] virtual [false]
net4 display_name [null]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
net5 display_name [null]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
net6 display_name [null]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
net7 display_name [null]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
net8 display_name [null]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
net9 display_name [null]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
net10 display_name [null]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
net11 display_name [null]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
net12 display_name [null]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
net13 display_name [null]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
net14 display_name [null]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
net15 display_name [null]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
eth4 display_name [WAN Miniport (IPv6) - SK-NDIS Miniport]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
eth5 display_name [WAN Miniport (Network Monitor) - SK-NDIS Miniport]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
eth6 display_name [WAN Miniport (IP) - SK-NDIS Miniport]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
eth7 display_name [Broadcom NetXtreme Gigabit Ethernet - SK-NDIS Miniport]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
eth8 display_name [Broadcom NetXtreme Gigabit Ethernet - Juniper Network Agent Miniport]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
net16 display_name [Juniper Networks Virtual Adapter Manager]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
eth9 display_name [WAN Miniport (IP) - Juniper Network Agent Miniport]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
eth10 display_name [Broadcom NetXtreme Gigabit Ethernet - SK-NDIS Miniport-QoS Packet Scheduler-0000]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
eth11 display_name [Broadcom NetXtreme Gigabit Ethernet - SK-NDIS Miniport-WFP LightWeight Filter-0000]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
eth12 display_name [WAN Miniport (IP) - SK-NDIS Miniport-QoS Packet Scheduler-0000]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
eth13 display_name [WAN Miniport (Network Monitor) - SK-NDIS Miniport-QoS Packet Scheduler-0000]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
eth14 display_name [WAN Miniport (IPv6) - SK-NDIS Miniport-QoS Packet Scheduler-0000]
address
mtu [-1] multicast [true] ptp [false] loopback [false] up [false] virtual [false]
2014-01-07 18:15:21,996 DEBUG - [testNode] Using probe [org.elasticsearch.monitor.fs.JmxFsProbe@2caefab3] with refresh_interval [1s]
2014-01-07 18:15:23,115 DEBUG - [testNode] using indices.store.throttle.type [MERGE], with index.store.throttle.max_bytes_per_sec [20mb]
2014-01-07 18:15:23,143 DEBUG - [testNode] using bytebuffer cache with small_buffer_size [1kb], large_buffer_size [1mb], small_cache_size [10mb], large_cache_size [500mb], direct [true]
2014-01-07 18:15:23,162 DEBUG - [testNode] using script cache with max_size [500], expire [null]
2014-01-07 18:15:23,171 DEBUG - [testNode] using node_concurrent_recoveries [2], node_initial_primaries_recoveries [4]
2014-01-07 18:15:23,172 DEBUG - [testNode] using [cluster.routing.allocation.allow_rebalance] with [indices_all_active]
2014-01-07 18:15:23,172 DEBUG - [testNode] using [cluster_concurrent_rebalance] with [2]
2014-01-07 18:15:23,177 DEBUG - [testNode] using initial_shards [quorum], list_timeout [30s]
2014-01-07 18:15:23,401 DEBUG - [testNode] using max_bytes_per_sec[20mb], concurrent_streams [3], file_chunk_size [512kb], translog_size [512kb], translog_ops [1000], and compress [true]
2014-01-07 18:15:23,479 DEBUG - [testNode] using max_chunk_size[8kb], max_header_size[8kb], max_initial_line_length[4kb], max_content_length[100mb], receive_predictor[512kb->512kb]
2014-01-07 18:15:23,489 DEBUG - [testNode] using index_buffer_size [181.6mb], with min_shard_index_buffer_size [4mb], max_shard_index_buffer_size [512mb], shard_inactive_time [30m]
2014-01-07 18:15:23,491 DEBUG - [testNode] using [node] weighted filter cache with size [20%], actual_size [363.3mb], expire [null], clean_interval [1m]
2014-01-07 18:15:23,493 DEBUG - [testNode] using size [-1] [-1b], expire [null]
2014-01-07 18:15:23,519 DEBUG - [testNode] using gateway.local.auto_import_dangled [YES], with gateway.local.dangling_timeout [2h]
2014-01-07 18:15:23,522 DEBUG - [testNode] using enabled [false], host [null], port [9700-9800], bulk_actions [1000], bulk_size [5mb], flush_interval [5s], concurrent_requests [4]
2014-01-07 18:15:23,526 DEBUG - [testNode] using node_concurrent_recoveries [2], node_initial_primaries_recoveries [4]
2014-01-07 18:15:23,526 DEBUG - [testNode] using [cluster.routing.allocation.allow_rebalance] with [indices_all_active]
2014-01-07 18:15:23,526 DEBUG - [testNode] using [cluster_concurrent_rebalance] with [2]
2014-01-07 18:15:23,527 DEBUG - [testNode] using node_concurrent_recoveries [2], node_initial_primaries_recoveries [4]
2014-01-07 18:15:23,527 DEBUG - [testNode] using [cluster.routing.allocation.allow_rebalance] with [indices_all_active]
2014-01-07 18:15:23,527 DEBUG - [testNode] using [cluster_concurrent_rebalance] with [2]
2014-01-07 18:15:23,538 INFO - [testNode] initialized
2014-01-07 18:15:23,538 INFO - [testNode] starting ...
2014-01-07 18:15:23,593 DEBUG - Using select timeout of 500
2014-01-07 18:15:23,593 DEBUG - Epoll-bug workaround enabled = false
2014-01-07 18:15:23,886 DEBUG - [testNode] Bound to address [/166.15.121.56:9400]
2014-01-07 18:15:23,887 INFO - [testNode] bound_address {inet[/166.15.121.56:9400]}, publish_address {inet[/166.15.121.56:9400]}
2014-01-07 18:15:24,032 WARN - [testNode] exception caught on transport layer [[id: 0xdbf73c8b]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:24,110 DEBUG - [testNode] connected to node [[#zen_unicast_2#][inet[vcll36a-1001.equity.csfb.com/169.36.78.96:9401]]]
2014-01-07 18:15:25,511 WARN - [testNode] exception caught on transport layer [[id: 0x164c6998]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:27,016 WARN - [testNode] exception caught on transport layer [[id: 0x168bcd47]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:28,269 DEBUG - [testNode] disconnected from [[#zen_unicast_2#][inet[vcll36a-1001.equity.csfb.com/169.36.78.96:9401]]]
2014-01-07 18:15:28,272 DEBUG - [testNode] filtered ping responses: (filter_client[true], filter_data[false])
--> target [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]], master [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]]
2014-01-07 18:15:28,358 DEBUG - [testNode] connected to node [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]]
2014-01-07 18:15:28,580 WARN - [testNode] Message not fully read (response) for [3] handler future(org.elasticsearch.discovery.zen.membership.MembershipAction$1@38b9f79a), error [true], resetting
2014-01-07 18:15:28,580 INFO - [testNode] failed to send join request to master [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]], reason [org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from stream; org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream; java.io.StreamCorruptedException: unexpected end of block data]
2014-01-07 18:15:28,581 WARN - [testNode] exception caught on transport layer [[id: 0xd283ea6f]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:28,660 DEBUG - [testNode] connected to node [[#zen_unicast_2#][inet[vcll36a-1001.equity.csfb.com/169.36.78.96:9401]]]
2014-01-07 18:15:30,085 WARN - [testNode] exception caught on transport layer [[id: 0x79112363]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:31,586 WARN - [testNode] exception caught on transport layer [[id: 0x1ebfcebb]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:32,827 DEBUG - [testNode] disconnected from [[#zen_unicast_2#][inet[vcll36a-1001.equity.csfb.com/169.36.78.96:9401]]]
2014-01-07 18:15:32,828 DEBUG - [testNode] filtered ping responses: (filter_client[true], filter_data[false])
--> target [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]], master [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]]
2014-01-07 18:15:32,947 WARN - [testNode] Message not fully read (response) for [7] handler future(org.elasticsearch.discovery.zen.membership.MembershipAction$1@714ed0b9), error [true], resetting
2014-01-07 18:15:32,947 INFO - [testNode] failed to send join request to master [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]], reason [org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from stream; org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream; java.io.StreamCorruptedException: unexpected end of block data]
2014-01-07 18:15:32,948 WARN - [testNode] exception caught on transport layer [[id: 0xe96384b5]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:33,027 DEBUG - [testNode] connected to node [[#zen_unicast_2#][inet[vcll36a-1001.equity.csfb.com/169.36.78.96:9401]]]
2014-01-07 18:15:34,450 WARN - [testNode] exception caught on transport layer [[id: 0x9d06f9ad]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:35,952 WARN - [testNode] exception caught on transport layer [[id: 0xd926d2c2]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:37,193 DEBUG - [testNode] disconnected from [[#zen_unicast_2#][inet[vcll36a-1001.equity.csfb.com/169.36.78.96:9401]]]
2014-01-07 18:15:37,194 DEBUG - [testNode] filtered ping responses: (filter_client[true], filter_data[false])
--> target [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]], master [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]]
2014-01-07 18:15:37,310 WARN - [testNode] Message not fully read (response) for [11] handler future(org.elasticsearch.discovery.zen.membership.MembershipAction$1@ef99e17), error [true], resetting
2014-01-07 18:15:37,310 INFO - [testNode] failed to send join request to master [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]], reason [org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from stream; org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream; java.io.StreamCorruptedException: unexpected end of block data]
2014-01-07 18:15:37,311 WARN - [testNode] exception caught on transport layer [[id: 0x1c94b66f]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:37,390 DEBUG - [testNode] connected to node [[#zen_unicast_2#][inet[vcll36a-1001.equity.csfb.com/169.36.78.96:9401]]]
2014-01-07 18:15:38,813 WARN - [testNode] exception caught on transport layer [[id: 0x820bd4a2]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:40,316 WARN - [testNode] exception caught on transport layer [[id: 0xc54971af]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:41,563 DEBUG - [testNode] disconnected from [[#zen_unicast_2#][inet[vcll36a-1001.equity.csfb.com/169.36.78.96:9401]]]
2014-01-07 18:15:41,564 DEBUG - [testNode] filtered ping responses: (filter_client[true], filter_data[false])
--> target [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]], master [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]]
2014-01-07 18:15:41,678 WARN - [testNode] Message not fully read (response) for [15] handler future(org.elasticsearch.discovery.zen.membership.MembershipAction$1@592b3893), error [true], resetting
2014-01-07 18:15:41,679 INFO - [testNode] failed to send join request to master [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]], reason [org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from stream; org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream; java.io.StreamCorruptedException: unexpected end of block data]
2014-01-07 18:15:41,680 WARN - [testNode] exception caught on transport layer [[id: 0xc63a3b16]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:41,758 DEBUG - [testNode] connected to node [[#zen_unicast_2#][inet[vcll36a-1001.equity.csfb.com/169.36.78.96:9401]]]
2014-01-07 18:15:43,182 WARN - [testNode] exception caught on transport layer [[id: 0xfc2aa352]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:44,683 WARN - [testNode] exception caught on transport layer [[id: 0x5e6b77b1]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:45,924 DEBUG - [testNode] disconnected from [[#zen_unicast_2#][inet[vcll36a-1001.equity.csfb.com/169.36.78.96:9401]]]
2014-01-07 18:15:45,925 DEBUG - [testNode] filtered ping responses: (filter_client[true], filter_data[false])
--> target [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]], master [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]]
2014-01-07 18:15:46,042 WARN - [testNode] Message not fully read (response) for [19] handler future(org.elasticsearch.discovery.zen.membership.MembershipAction$1@52a60d93), error [true], resetting
2014-01-07 18:15:46,042 INFO - [testNode] failed to send join request to master [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]], reason [org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from stream; org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream; java.io.StreamCorruptedException: unexpected end of block data]
2014-01-07 18:15:46,044 WARN - [testNode] exception caught on transport layer [[id: 0x6b791d4c]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:46,122 DEBUG - [testNode] connected to node [[#zen_unicast_2#][inet[vcll36a-1001.equity.csfb.com/169.36.78.96:9401]]]
2014-01-07 18:15:47,546 WARN - [testNode] exception caught on transport layer [[id: 0xdf218264]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:49,049 WARN - [testNode] exception caught on transport layer [[id: 0x5d287366]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:50,291 DEBUG - [testNode] disconnected from [[#zen_unicast_2#][inet[vcll36a-1001.equity.csfb.com/169.36.78.96:9401]]]
2014-01-07 18:15:50,291 DEBUG - [testNode] filtered ping responses: (filter_client[true], filter_data[false])
--> target [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]], master [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]]
2014-01-07 18:15:50,411 WARN - [testNode] Message not fully read (response) for [23] handler future(org.elasticsearch.discovery.zen.membership.MembershipAction$1@2d955c97), error [true], resetting
2014-01-07 18:15:50,411 INFO - [testNode] failed to send join request to master [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]], reason [org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from stream; org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream; java.io.StreamCorruptedException: unexpected end of block data]
2014-01-07 18:15:50,413 WARN - [testNode] exception caught on transport layer [[id: 0x753975cb]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:50,492 DEBUG - [testNode] connected to node [[#zen_unicast_2#][inet[vcll36a-1001.equity.csfb.com/169.36.78.96:9401]]]
2014-01-07 18:15:51,915 WARN - [testNode] exception caught on transport layer [[id: 0x1914d169]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:53,417 WARN - [testNode] exception caught on transport layer [[id: 0xd2891b0d]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:54,003 WARN - [testNode] waited for 30s and no initial state was set by the discovery
2014-01-07 18:15:54,003 INFO - [testNode] CMTDEV1/rls-JqdgSkO0YuviXKOxTg
2014-01-07 18:15:54,003 DEBUG - [testNode] can't wait on start for (possibly) reading state from gateway, will do it asynchronously
2014-01-07 18:15:54,032 INFO - [testNode] bound_address {inet[/166.15.121.56:9451]}, publish_address {inet[/166.15.121.56:9451]}
2014-01-07 18:15:54,033 INFO - [testNode] started
Jan 07, 2014 6:15:54 PM util.ESNodeWrapper init
INFO: After Node build for cluster[CMTDEV1]
Jan 07, 2014 6:15:54 PM util.ESNodeWrapper init
INFO: Testing Health and Readiness of Cluster[CMTDEV1]
2014-01-07 18:15:54,674 DEBUG - [testNode] disconnected from [[#zen_unicast_2#][inet[vcll36a-1001.equity.csfb.com/169.36.78.96:9401]]]
2014-01-07 18:15:54,674 DEBUG - [testNode] filtered ping responses: (filter_client[true], filter_data[false])
--> target [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]], master [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]]
2014-01-07 18:15:54,792 WARN - [testNode] Message not fully read (response) for [27] handler future(org.elasticsearch.discovery.zen.membership.MembershipAction$1@4ffe90c2), error [true], resetting
2014-01-07 18:15:54,792 INFO - [testNode] failed to send join request to master [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]], reason [org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from stream; org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream; java.io.StreamCorruptedException: unexpected end of block data]
2014-01-07 18:15:54,793 WARN - [testNode] exception caught on transport layer [[id: 0x5f25ba2b]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:54,873 DEBUG - [testNode] connected to node [[#zen_unicast_2#][inet[vcll36a-1001.equity.csfb.com/169.36.78.96:9401]]]
2014-01-07 18:15:56,296 WARN - [testNode] exception caught on transport layer [[id: 0xd6c8f62f]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:57,797 WARN - [testNode] exception caught on transport layer [[id: 0x7d16bd6a]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:59,043 DEBUG - [testNode] disconnected from [[#zen_unicast_2#][inet[vcll36a-1001.equity.csfb.com/169.36.78.96:9401]]]
2014-01-07 18:15:59,044 DEBUG - [testNode] filtered ping responses: (filter_client[true], filter_data[false])
--> target [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]], master [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]]
2014-01-07 18:15:59,161 WARN - [testNode] Message not fully read (response) for [31] handler future(org.elasticsearch.discovery.zen.membership.MembershipAction$1@563a3ada), error [true], resetting
2014-01-07 18:15:59,161 INFO - [testNode] failed to send join request to master [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]], reason [org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from stream; org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream; java.io.StreamCorruptedException: unexpected end of block data]
2014-01-07 18:15:59,162 WARN - [testNode] exception caught on transport layer [[id: 0x064c1223]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:15:59,241 DEBUG - [testNode] connected to node [[#zen_unicast_2#][inet[vcll36a-1001.equity.csfb.com/169.36.78.96:9401]]]
2014-01-07 18:16:00,664 WARN - [testNode] exception caught on transport layer [[id: 0xf3624602]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:16:02,165 WARN - [testNode] exception caught on transport layer [[id: 0xac3c73c6]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:16:03,398 DEBUG - [testNode] disconnected from [[#zen_unicast_2#][inet[vcll36a-1001.equity.csfb.com/169.36.78.96:9401]]]
2014-01-07 18:16:03,398 DEBUG - [testNode] filtered ping responses: (filter_client[true], filter_data[false])
--> target [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]], master [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]]
2014-01-07 18:16:03,518 WARN - [testNode] Message not fully read (response) for [35] handler future(org.elasticsearch.discovery.zen.membership.MembershipAction$1@3a918a66), error [true], resetting
2014-01-07 18:16:03,518 INFO - [testNode] failed to send join request to master [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]], reason [org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from stream; org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream; java.io.StreamCorruptedException: unexpected end of block data]
2014-01-07 18:16:03,520 WARN - [testNode] exception caught on transport layer [[id: 0xbc4b60b2]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:16:03,599 DEBUG - [testNode] connected to node [[#zen_unicast_2#][inet[vcll36a-1001.equity.csfb.com/169.36.78.96:9401]]]
2014-01-07 18:16:05,023 WARN - [testNode] exception caught on transport layer [[id: 0xa5df30e6]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:16:06,524 WARN - [testNode] exception caught on transport layer [[id: 0x7242c648]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:16:07,771 DEBUG - [testNode] disconnected from [[#zen_unicast_2#][inet[vcll36a-1001.equity.csfb.com/169.36.78.96:9401]]]
2014-01-07 18:16:07,771 DEBUG - [testNode] filtered ping responses: (filter_client[true], filter_data[false])
--> target [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]], master [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]]
2014-01-07 18:16:07,891 WARN - [testNode] Message not fully read (response) for [39] handler future(org.elasticsearch.discovery.zen.membership.MembershipAction$1@74d3dc29), error [true], resetting
2014-01-07 18:16:07,892 INFO - [testNode] failed to send join request to master [[Cheetah][Y-Bo59_1Se-fSsAkz1bqKw][inet[/169.36.78.96:9401]]], reason [org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from stream; org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream; java.io.StreamCorruptedException: unexpected end of block data]
2014-01-07 18:16:07,893 WARN - [testNode] exception caught on transport layer [[id: 0x2849c0ac]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:655)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:618)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:585)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-07 18:16:07,972 DEBUG - [testNode] connected to node [[#zen_unicast_2#][inet[vcll36a-1001.equity.csfb.com/169.36.78.96:9401]]]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment