-
-
Save brusic/60bdfafd6273c05a3417 to your computer and use it in GitHub Desktop.
Cluster stalls upon node removal
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [2013-01-15 11:09:34,688][DEBUG][discovery.zen.fd ] [search6] [node ] failed to ping [[search8][P7UNCh9oTE623RI8w_zsPw][inet[/<snip>:9300]]], tried | |
| [3] times, each with maximum [2s] timeout | |
| [2013-01-15 11:09:34,689][DEBUG][cluster.service ] [search6] processing [zen-disco-node_failed([search8][P7UNCh9oTE623RI8w_zsPw][inet[/<snip>:9300]] | |
| ), reason failed to ping, tried [3] times, each with maximum [2s] timeout]: execute | |
| [2013-01-15 11:09:34,690][DEBUG][cluster.service ] [search6] cluster state updated, version [58], source [zen-disco-node_failed([search8][P7UNCh9oTE623RI8w_ | |
| zsPw][inet[/<snip>:9300]]), reason failed to ping, tried [3] times, each with maximum [2s] timeout] | |
| [2013-01-15 11:09:34,690][INFO ][cluster.service ] [search6] removed {[search8][P7UNCh9oTE623RI8w_zsPw][inet[/<snip>:9300]],}, reason: zen-disco-nod | |
| e_failed([search8][P7UNCh9oTE623RI8w_zsPw][inet[/<snip>:9300]]), reason failed to ping, tried [3] times, each with maximum [2s] timeout | |
| [2013-01-15 11:09:34,691][DEBUG][river.cluster ] [search6] processing [reroute_rivers_node_changed]: execute | |
| [2013-01-15 11:09:34,691][DEBUG][river.cluster ] [search6] processing [reroute_rivers_node_changed]: no change in cluster_state | |
| [2013-01-15 11:09:34,692][DEBUG][cluster.service ] [search6] processing [zen-disco-node_failed([search8][P7UNCh9oTE623RI8w_zsPw][inet[/<snip>:9300]] | |
| ), reason failed to ping, tried [3] times, each with maximum [2s] timeout]: done applying updated cluster_state | |
| [2013-01-15 11:09:34,692][DEBUG][cluster.service ] [search6] processing [routing-table-updater]: execute | |
| [2013-01-15 11:09:34,693][DEBUG][transport.netty ] [search6] disconnected from [[search8][P7UNCh9oTE623RI8w_zsPw][inet[/<snip>:9300]]] | |
| [2013-01-15 11:09:35,108][DEBUG][discovery.zen.fd ] [search6] [node ] failed to ping [[search11][OMMr4k2DRgSsvMRU8vE-eQ][inet[/<snip>:9300]]], tried | |
| [3] times, each with maximum [2s] timeout | |
| [2013-01-15 11:10:04,693][DEBUG][indices.store ] [search6] failed to execute on node [OMMr4k2DRgSsvMRU8vE-eQ] | |
| org.elasticsearch.transport.ReceiveTimeoutTransportException: [search11][inet[/<snip>:9300]][/cluster/nodes/indices/shard/store/n] request_id [289991] timed out after [30000ms] | |
| at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:342) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) | |
| at java.lang.Thread.run(Thread.java:722) | |
| [2013-01-15 11:10:34,695][DEBUG][indices.store ] [search6] failed to execute on node [OMMr4k2DRgSsvMRU8vE-eQ] | |
| org.elasticsearch.transport.ReceiveTimeoutTransportException: [search11][inet[/<snip>:9300]][/cluster/nodes/indices/shard/store/n] request_id [290160] timed out after [30000ms] | |
| at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:342) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) | |
| at java.lang.Thread.run(Thread.java:722) | |
| [2013-01-15 11:10:52,532][DEBUG][transport.netty ] [search6] disconnected from [[search11][OMMr4k2DRgSsvMRU8vE-eQ][inet[/<snip>:9300]]] | |
| [2013-01-15 11:10:52,533][DEBUG][discovery.zen.publish ] [search6] failed to send cluster state to [[search11][OMMr4k2DRgSsvMRU8vE-eQ][inet[/<snip>:9300]]], should be detected as failed soon... | |
| org.elasticsearch.transport.NodeDisconnectedException: [search11][inet[/<snip>:9300]][discovery/zen/publish] disconnected | |
| [2013-01-15 11:10:52,534][DEBUG][indices.store ] [search6] failed to execute on node [OMMr4k2DRgSsvMRU8vE-eQ] | |
| org.elasticsearch.transport.NodeDisconnectedException: [search11][inet[/<snip>:9300]][/cluster/nodes/indices/shard/store/n] disconnected | |
| [2013-01-15 11:10:52,539][DEBUG][cluster.service ] [search6] cluster state updated, version [59], source [routing-table-updater] | |
| [2013-01-15 11:10:52,541][DEBUG][discovery.zen.publish ] [search6] failed to send cluster state to [[search11][OMMr4k2DRgSsvMRU8vE-eQ][inet[/<snip>:9300]]], should be detected as failed soon... | |
| org.elasticsearch.transport.SendRequestTransportException: [search11][inet[/<snip>:9300]][discovery/zen/publish] | |
| at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:199) | |
| at org.elasticsearch.discovery.zen.publish.PublishClusterStateAction.publish(PublishClusterStateAction.java:97) | |
| at org.elasticsearch.discovery.zen.ZenDiscovery.publish(ZenDiscovery.java:266) | |
| at org.elasticsearch.discovery.DiscoveryService.publish(DiscoveryService.java:115) | |
| at org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:305) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) | |
| at java.lang.Thread.run(Thread.java:722) | |
| Caused by: org.elasticsearch.transport.NodeNotConnectedException: [search11][inet[/<snip>:9300]] Node not connected | |
| at org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:744) | |
| at org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:516) | |
| at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:184) | |
| ... 7 more | |
| [2013-01-15 11:10:52,544][DEBUG][cluster.service ] [search6] processing [routing-table-updater]: done applying updated cluster_state | |
| [2013-01-15 11:10:52,542][DEBUG][river.cluster ] [search6] processing [reroute_rivers_node_changed]: execute | |
| [2013-01-15 11:10:52,545][DEBUG][cluster.service ] [search6] processing [zen-disco-node_failed([search11][OMMr4k2DRgSsvMRU8vE-eQ][inet[/<snip>:9300]]), reason failed to ping, tried [3] times, each with maximum [2s] timeout]: execute | |
| [2013-01-15 11:10:52,545][DEBUG][river.cluster ] [search6] processing [reroute_rivers_node_changed]: no change in cluster_state | |
| [2013-01-15 11:10:52,545][DEBUG][cluster.service ] [search6] cluster state updated, version [60], source [zen-disco-node_failed([search11][OMMr4k2DRgSsvMRU8vE-eQ][inet[/<snip>:9300]]), reason failed to ping, tried [3] times, each with maximum [2s] timeout] | |
| [2013-01-15 11:10:52,546][INFO ][cluster.service ] [search6] removed {[search11][OMMr4k2DRgSsvMRU8vE-eQ][inet[/<snip>:9300]],}, reason: zen-disco-node_failed([search11][OMMr4k2DRgSsvMRU8vE-eQ][inet[/<snip>:9300]]), reason failed to ping, tried [3] times, each with maximum [2s] timeout | |
| [2013-01-15 11:10:52,546][DEBUG][river.cluster ] [search6] processing [reroute_rivers_node_changed]: execute | |
| [2013-01-15 11:10:52,547][DEBUG][river.cluster ] [search6] processing [reroute_rivers_node_changed]: no change in cluster_state | |
| [2013-01-15 11:10:52,547][DEBUG][cluster.service ] [search6] processing [zen-disco-node_failed([search11][OMMr4k2DRgSsvMRU8vE-eQ][inet[/<snip>:9300]]), reason failed to ping, tried [3] times, each with maximum [2s] timeout]: done applying updated cluster_state | |
| [2013-01-15 11:10:52,547][DEBUG][cluster.service ] [search6] processing [routing-table-updater]: execute | |
| [2013-01-15 11:10:52,648][DEBUG][cluster.service ] [search6] cluster state updated, version [61], source [routing-table-updater] | |
| [2013-01-15 11:10:52,650][DEBUG][river.cluster ] [search6] processing [reroute_rivers_node_changed]: execute | |
| [2013-01-15 11:10:52,650][DEBUG][indices.cluster ] [search6] [products-20130113-180241][0] creating shard | |
| [2013-01-15 11:10:52,650][DEBUG][river.cluster ] [search6] processing [reroute_rivers_node_changed]: no change in cluster_state | |
| [2013-01-15 11:10:52,650][DEBUG][index.service ] [search6] [products-20130113-180241] creating shard_id [0] | |
| [2013-01-15 11:10:52,704][DEBUG][index.store ] [search6] [products-20130113-180241][0] using compress.stored [false], compress.tv [false] | |
| [2013-01-15 11:10:52,705][DEBUG][index.deletionpolicy ] [search6] [products-20130113-180241][0] Using [keep_only_last] deletion policy | |
| [2013-01-15 11:10:52,706][DEBUG][index.merge.policy ] [search6] [products-20130113-180241][0] using [tiered] merge policy with expunge_deletes_allowed[10.0], floor_segment[2mb], max_merge_at_once[3], max_merge_at_once_explicit[30], max_merged_segment[5gb], segments_per_tier[3.0], reclaim_deletes_weight[2.0], async_merge[true] | |
| [2013-01-15 11:10:52,706][DEBUG][index.merge.scheduler ] [search6] [products-20130113-180241][0] using [concurrent] merge scheduler with max_thread_count[3] | |
| [2013-01-15 11:10:52,707][DEBUG][index.shard.service ] [search6] [products-20130113-180241][0] state: [CREATED] | |
| [2013-01-15 11:10:52,708][DEBUG][index.translog ] [search6] [products-20130113-180241][0] interval [5s], flush_threshold_ops [5000], flush_threshold_size [200mb], flush_threshold_period [30m] | |
| [2013-01-15 11:10:52,709][DEBUG][indices.memory ] [search6] recalculating shard indexing buffer (reason=created_shard[products-20130113-180241][0]), total is [1.8gb] with [3] active shards, each shard set to [512mb] | |
| [2013-01-15 11:10:52,709][DEBUG][index.engine.robin ] [search6] [products-20130113-180241][4] updating index_buffer_size from [64mb] to [512mb] | |
| [2013-01-15 11:10:52,709][DEBUG][index.shard.service ] [search6] [products-20130113-180241][0] state: [CREATED]->[RECOVERING], reason [from [search5][5TAwP_TZSfaeUA7BRbO_iQ][inet[/<snip>:9300]]] | |
| [2013-01-15 11:10:52,715][DEBUG][cluster.service ] [search6] processing [routing-table-updater]: done applying updated cluster_state | |
| [2013-01-15 11:11:12,727][DEBUG][transport.netty ] [search6] connected to node [[search8][V9IfvFz_T4iPGJhyUPFImA][inet[/<snip>:9300]]] | |
| [2013-01-15 11:11:13,761][DEBUG][cluster.service ] [search6] processing [zen-disco-receive(join from node[[search8][V9IfvFz_T4iPGJhyUPFImA][inet[/<snip>:9300]]])]: execute | |
| [2013-01-15 11:11:13,761][DEBUG][cluster.service ] [search6] cluster state updated, version [62], source [zen-disco-receive(join from node[[search8][V9IfvFz_T4iPGJhyUPFImA][inet[/<snip>:9300]]])] | |
| [2013-01-15 11:11:13,761][INFO ][cluster.service ] [search6] added {[search8][V9IfvFz_T4iPGJhyUPFImA][inet[/<snip>:9300]],}, reason: zen-disco-receive(join from node[[search8][V9IfvFz_T4iPGJhyUPFImA][inet[/<snip>:9300]]]) | |
| [2013-01-15 11:11:13,762][DEBUG][river.cluster ] [search6] processing [reroute_rivers_node_changed]: execute | |
| [2013-01-15 11:11:13,763][DEBUG][river.cluster ] [search6] processing [reroute_rivers_node_changed]: no change in cluster_state | |
| [2013-01-15 11:11:13,763][DEBUG][cluster.service ] [search6] processing [zen-disco-receive(join from node[[search8][V9IfvFz_T4iPGJhyUPFImA][inet[/<snip>:9300]]])]: done applying updated cluster_state | |
| [2013-01-15 11:11:17,908][DEBUG][cluster.service ] [search6] processing [routing-table-updater]: execute | |
| [2013-01-15 11:11:17,910][DEBUG][cluster.service ] [search6] processing [routing-table-updater]: no change in cluster_state | |
| [2013-01-15 11:11:26,183][DEBUG][transport.netty ] [search6] connected to node [[search11][GYA2apevQoehoWZ5ucUwlA][inet[/<snip>:9300]]] | |
| [2013-01-15 11:11:27,247][DEBUG][cluster.service ] [search6] processing [zen-disco-receive(join from node[[search11][GYA2apevQoehoWZ5ucUwlA][inet[/<snip>:9300]]])]: execute | |
| [2013-01-15 11:11:27,247][DEBUG][cluster.service ] [search6] cluster state updated, version [63], source [zen-disco-receive(join from node[[search11][GYA2apevQoehoWZ5ucUwlA][inet[/<snip>:9300]]])] | |
| [2013-01-15 11:11:27,248][INFO ][cluster.service ] [search6] added {[search11][GYA2apevQoehoWZ5ucUwlA][inet[/<snip>:9300]],}, reason: zen-disco-receive(join from node[[search11][GYA2apevQoehoWZ5ucUwlA][inet[/<snip>:9300]]]) | |
| [2013-01-15 11:11:27,249][DEBUG][river.cluster ] [search6] processing [reroute_rivers_node_changed]: execute | |
| [2013-01-15 11:11:27,249][DEBUG][river.cluster ] [search6] processing [reroute_rivers_node_changed]: no change in cluster_state | |
| [2013-01-15 11:11:27,249][DEBUG][river.cluster ] [search6] processing [reroute_rivers_node_changed]: no change in cluster_state | |
| [2013-01-15 11:11:27,249][DEBUG][cluster.service ] [search6] processing [zen-disco-receive(join from node[[search11][GYA2apevQoehoWZ5ucUwlA][inet[/<snip>:9300]]])]: done applying updated cluster_state | |
| [2013-01-15 11:11:27,908][DEBUG][cluster.service ] [search6] processing [routing-table-updater]: execute | |
| [2013-01-15 11:11:27,911][DEBUG][cluster.service ] [search6] processing [routing-table-updater]: no change in cluster_state | |
| [2013-01-15 11:13:52,931][DEBUG][cluster.action.shard ] [search6] received shard started for [products-20130113-180241][6], node[Aja1zGK0TvmOZ4w1Jdky-g], [R], s[INITIALIZING], re: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment