Skip to content

Instantly share code, notes, and snippets.

@planckiii

planckiii/gist:5947068

Created July 8, 2013 08:13
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save planckiii/5947068 to your computer and use it in GitHub Desktop.
Save planckiii/5947068 to your computer and use it in GitHub Desktop.
Download ZIP
second_node, secondary data center:
Raw
gistfile1.txt
2013-07-05 21:36:16,727][DEBUG][discovery.zen.fd ] [second_node] [master] failed to ping [[first_node][TZ6][inet[/10.10.0.1:9300]]{rack=dcprim_1, max_local_storage_nodes=1, zone=dcprim, master=true}], tried [10] times, each with maximum [30s] timeout
[2013-07-05 21:36:16,729][DEBUG][discovery.zen.fd ] [second_node] [master] stopping fault detection against master [[first_node][TZ6][inet[/10.10.0.1:9300]]{rack=dcprim_1, max_local_storage_nodes=1, zone=dcprim, master=true}], reason [master failure, failed to ping, tried [10] times, each with maximum [30s] timeout]
[2013-07-05 21:36:16,729][INFO ][discovery.zen ] [second_node] master_left [[first_node][TZ6][inet[/10.10.0.1:9300]]{rack=dcprim_1, max_local_storage_nodes=1, zone=dcprim, master=true}], reason [failed to ping, tried [10] times, each with maximum [30s] timeout]
[2013-07-05 21:36:16,731][DEBUG][cluster.service ] [second_node] processing [zen-disco-master_failed ([first_node][TZ6][inet[/10.10.0.1:9300]]{rack=dcprim_1, max_local_storage_nodes=1, zone=dcprim, master=true})]: execute
[2013-07-05 21:36:16,731][DEBUG][cluster.service ] [second_node] cluster state updated, version [6063], source [zen-disco-master_failed ([first_node][TZ6][inet[/10.10.0.1:9300]]{rack=dcprim_1, max_local_storage_nodes=1, zone=dcprim, master=true})]
[2013-07-05 21:36:16,732][INFO ][cluster.service ] [second_node] master {new [second_node][0O8_8hcBSXa6pS5hTagW3w][inet[second_node/10.94.8.48:9300]]{rack=dc4_nova, max_local_storage_nodes=1, zone=dc4, master=true}, previous [first_node][TZ6][inet[/10.10.0.1:9300]]{rack=dcprim_1, max_local_storage_nodes=1, zone=dcprim, master=true}}, removed {[first_node][TZ6GrimmSVSdIaN-7eOntw][inet[/10.190.10.24:9300]]{rack=dc2_nova2, max_local_storage_nodes=1, zone=dc2, master=true},}, reason: zen-disco-master_failed ([first_node][TZ6GrimmSVSdIaN-7eOntw][inet[/10.190.10.24:9300]]{rack=dc2_nova2, max_local_storage_nodes=1, zone=dc2, master=true})
[2013-07-05 21:36:16,736][DEBUG][river.cluster ] [second_node] processing [reroute_rivers_node_changed]: execute
[2013-07-05 21:36:16,737][DEBUG][river.cluster ] [second_node] processing [reroute_rivers_node_changed]: no change in cluster_state
[2013-07-05 21:36:16,738][DEBUG][cluster.service ] [second_node] processing [zen-disco-master_failed ([first_node][TZ6][inet[/10.10.0.1:9300]]{rack=dcprim_1, max_local_storage_nodes=1, zone=dcprim, master=true})]: done applying updated cluster_state
[2013-07-05 21:36:16,741][DEBUG][cluster.service ] [second_node] processing [routing-table-updater]: execute
[2013-07-05 21:36:16,740][DEBUG][transport.netty ] [second_node] disconnected from [[first_node][TZ6][inet[/10.10.0.1:9300]]{rack=dcprim_1, max_local_storage_nodes=1, zone=dcprim, master=true}]
[2013-07-05 21:36:16,743][DEBUG][action.admin.cluster.node.stats] [second_node] failed to execute on node [TZ6]
org.elasticsearch.transport.NodeDisconnectedException: [first_node][inet[/10.10.0.1:9300]][cluster/nodes/stats/n] disconnected
[2013-07-05 21:36:16,756][DEBUG][action.admin.cluster.node.stats] [second_node] failed to execute on node [TZ6]
org.elasticsearch.transport.NodeDisconnectedException: [first_node][inet[/10.10.0.1:9300]][cluster/nodes/stats/n] disconnected
[2013-07-05 21:36:16,758][DEBUG][action.admin.cluster.node.stats] [second_node] failed to execute on node [TZ6]
org.elasticsearch.transport.NodeDisconnectedException: [first_node][inet[/10.10.0.1:9300]][cluster/nodes/stats/n] disconnected
[2013-07-05 21:36:16,751][DEBUG][http.netty ] [second_node] Caught exception while handling client http traffic, closing connection [id: 0xac21213b, /127.0.0.1:38626 :> /127.0.0.1:9200]
java.nio.channels.ClosedChannelException
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:409)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:127)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:99)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:725)
at org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
at org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:704)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:671)
at org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:248)
at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:152)
at org.elasticsearch.rest.action.admin.cluster.node.stats.RestNodesStatsAction$1.onResponse(RestNodesStatsAction.java:156)
at org.elasticsearch.rest.action.admin.cluster.node.stats.RestNodesStatsAction$1.onResponse(RestNodesStatsAction.java:148)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.finishHim(TransportNodesOperationAction.java:220)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.onFailure(TransportNodesOperationAction.java:215)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.access$1000(TransportNodesOperationAction.java:102)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$4.handleException(TransportNodesOperationAction.java:185)
at org.elasticsearch.transport.TransportService$Adapter$2$1.run(TransportService.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
[2013-07-05 21:36:16,762][DEBUG][http.netty ] [second_node] Caught exception while handling client http traffic, closing connection [id: 0xfd8f736a, /127.0.0.1:38646 :> /127.0.0.1:9200]
java.nio.channels.ClosedChannelException
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:409)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:127)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:99)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:725)
at org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
at org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment